Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NATSConnectionException: Authentication Timeout #883

Open
Zetanova opened this issue Mar 28, 2024 · 5 comments
Open

NATSConnectionException: Authentication Timeout #883

Zetanova opened this issue Mar 28, 2024 · 5 comments
Labels
defect Suspected defect such as a bug or regression needs info Additional info is needed

Comments

@Zetanova
Copy link
Contributor

Observed behavior

The nats.client <1.1.4 has sometimes an issue to connect to a server with "Authentication Timeout" exception.
This happens sometimes on vs debug sessions on windows or docker-desktop and in production under k8s
with dotnet 6 and 8. The nats server has an auth timeout of 5s.

With an new aspnet project in production that triggered this issue not only sometimes,
but always at startup in k8s

The nats.client requires real-concurrency (2+ cores) to connect.
A cpu limit env. like docker run --cpus=1 will trigger the lock
after the UserSignatureEventHandler call

Expected behavior

the connection should succeed with cpu limit of 1

Server and client version

nats.client 1.1.4

Host environment

windows and docker
dotnet 6 & 8

Steps to reproduce

set cpu limit to 1 in docker or resources.limits.cpu: 500m in kubernetes

@Zetanova Zetanova added the defect Suspected defect such as a bug or regression label Mar 28, 2024
@scottf
Copy link
Collaborator

scottf commented Jun 25, 2024

@Zetanova Is this still a problem in 1.1.5? Are there any steps to reproduce?

@scottf scottf added the needs info Additional info is needed label Jun 25, 2024
@Zetanova
Copy link
Contributor Author

Yes, I workaround it for now by wrapping the connect inside a LongRunning TaskPool Thread
But this does not solve the issue completely.

Workaround:

var task = Task.Factory.StartNew(state =>
{
    var fc = new ConnectionFactory();
    return fc.CreateConnection((Options)state!);
},
opt,
CancellationToken.None,
TaskCreationOptions.LongRunning, 
TaskScheduler.Default);

return task.GetAwaiter().GetResult();

@scottf
Copy link
Collaborator

scottf commented Jun 26, 2024

Is the problem that the client wants more threads than the environment it's running in can provide? Is this a real production configuration?
This doesn't seem like a defect to me, it just like there are minimum requirements to run the client.

@Zetanova
Copy link
Contributor Author

Yes debug and production on startup.

The issue is the hidden deadlock

task.Wait() and wait-events are used running on the TaskPool
at least one free TaskPool-Thread needs to be idle/free to execute the connect.

Until the TaskPoolManager decides to increase the Thread count on process startup connect
the "Authentication Timeout" exception has already been thrown.

@Zetanova
Copy link
Contributor Author

Kubernetes resources.limits.cpu = 1000m will trigger it instanlty. Its the same as docker --cpus=1 (cgroup cpu=1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect Suspected defect such as a bug or regression needs info Additional info is needed
Projects
None yet
Development

No branches or pull requests

2 participants