Symptom: Unexpected nats: timeout
#5859
RitikaLaddha
started this conversation in
General
Replies: 6 comments 4 replies
-
|
Thanks for the report @RitikaLaddha, could you share the nats-server version as well (the one used by the nats container pod)? You could also confirm by checking /varz, for example: https://demo.nats.io:8222/varz |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Also the following configuration will not work ok on k8s, you need to add all the A records from the statefulset to the config or use the nats-io/k8s helm chart: |
Beta Was this translation helpful? Give feedback.
0 replies
-
Hey @wallyqs |
Beta Was this translation helpful? Give feedback.
0 replies
-
can you please specify what are A records we would need to add in config? |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
The routes need the complete dns entries from the stateful set, the nats-io/k8s helm chart does this by default: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hello! We've just upgraded our nats-server container and java client and have started regularly seeing symptoms of this as well. A nontrivial fraction of requests to publish messages into a work queue stream using "Future cancelled, response not registered in time, check connection status." In our case, the network path between our Java application and JetStream has a few hops: Using This started happening around the time we upgraded both our client & server. I haven't tried other combinations to isolate yet.
|
Beta Was this translation helpful? Give feedback.
4 replies
-
|
I enabled The following snippets are from containerized processes all running on the same host in the topology mentioned in my previous comment. In my JetStream leafnode, I observe what appears to be a burst connection closed errors. On the other leaf node where the application is connected, I see the same "Delaying PING" messages as in the title of this discussion nearing 2 minutes. Lastly, on the nats-server, core nats process to which both leaf nodes connect, no obvious signs of trouble. It appears that I am able to reproduce the originally described issue, though in my case, I think there may be trouble with on both the leafnode connection and on the connection from my application to a leafnode. |
Beta Was this translation helpful? Give feedback.
-
|
The Delaying PING message is just part of the periodic ping interval from the server which happens every 2minutes unless there is some recent data being sent (if there was data in the past 2m then it will not send the ping thus logging |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for explaining @wallyqs! Good observation. We have 8 vCPUs, but we're trying to be pretty aggressive on our docker/cgroup limits. We have a somewhat resource constrained edge setup. On the two non-jetstream containers, (nats-server, core) & (leaf, core nats), we have been using And on the jetstream leafnode, additional memory I don't believe we've observed our nats-server processes reaching their CPU limits (eg: in docker stats). The nats-server log message about Also, I see that not all processes selected an identical value for GOMAXPROCS despite having an identical cpu cgroup limit. Should I investigate pinning GOMAXPROCS? |
Beta Was this translation helpful? Give feedback.
-
|
@davidmcote I think for now you could try to set it to GOMAXPROCS=2 |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Observed behavior
Hi
I am facing below intermittent issue because of that Nats is not being stable, as even if its's connected sometimes, it doesn't respond back
Although we get in log of application, that message has been sent. But still it does not returns message.
We have kept 30s timeout, so we do not receive response in 30s of time.
then we get timeout exception
Nats logs
Application logs
Nats configuration
Expected behavior
Application should get response in time
Server and client version
Helm chart used: https://bitnami.com/stack/nats/helm
Kubernetes version: 1.28.12
Application Java version: 21
Nats version: 2.10.18
Host environment
Azure Kubernetes service
Steps to reproduce
No response
Beta Was this translation helpful? Give feedback.
All reactions