Significant Message Delivery Delays in NATS 3-Node Cluster Despite Low Queue Volume and Idle Consumers

[NoumanNawaz51]

Observed behavior

We encountered an issue with our NATS 3-node cluster where message delivery experienced significant delays, sometimes exceeding timeout (15 seconds). The setup includes a multiple publishing service and multiple consumers subscribing to jet-stream queues.
• Only a few messages were in the queue.
• Not all consumers were busy. (300 consumers we have)
• Random delays noted (1 to 40 seconds)
Despite there being only a few messages in the queue and not all consumers being fully occupied, we consistently observed delays. This behavior is unexpected, given the low message volume and seemingly low load on the consumers.
Did anyone else experience this kind of behavior what could be possibly causing this ?

We are using PushSubscribeSync machanism.
here is config from all nodes.

Expected behavior

All messages pushed over queue should be processed on consumers without delay.

Server and client version

Nats Server Version 2.10.19
Nats Client Version 1.0.8 (Dotnet Application)

Host environment

Nats Server are deployed with Jetstream over Linux.
Applications are dotnet over windows.

Steps to reproduce

Problem is happening randomly.

[Jarema]

Thanks for the report.

Can you share the stream info and consumer info for few consumers, preferably those that might have work but are slow processing it?

[NoumanNawaz51]

[NoumanNawaz51]

Hi, Can I get answer to this query ?

[derekcollison]

What does nats server ls show from the system account?