I'm currently experiencing major problems with this exact issue. We have two web servers handling about 500 requests a minute during the peak of the day. There are times when that can burst up to several thousand requests for 30 seconds to 2 minutes. We recently switched from IIS Windows Authentication to Entra-based authentication. Prior to that switch, these bursts of web requests were easily handled. Now, bursts of requests cause both web servers to stop responding for several minutes to a half hour or longer.

On the server itself, we can see w3wp.exe start climbing in thread count, approximately one per second, until it spawns thousands of threads while CPU isn't really affected. I've pulled several crash dumps and just about every thread is stuck on Microsoft.Identity.Web.OpenIdConnectCachingSecurityTokenProvider.get_Issuer() or .get_SecurityKeys() where there's a call to TryEnterReadLock().

If there are any recommendations for how to deal with this, I would greatly appreciate it.

There was an issue in Microsoft.Identity.Model 8.0.2 that got fixed in 8.1.1.
Would you have a dependencies on these?

All of the Microsoft.IdentityModel.x packages in my project are 8.1.2. I believe I read about the bug fix you're referring to and made sure to update, but this issue is still popping up. The crash dumps I pored over were taken today with these updates already in place.

There was an issue in Microsoft.Identity.Model 8.0.2 that got fixed in 8.1.1. Would you have a dependencies on these?

I think both @alex-set and myself have clearly identified that the issue lies in OpenIdConnectCachingSecurityTokenProvider.

On my end, we have numerous thread analysis from the Azure portal that point to the excessive locking/synchronization.

As mentioned in my original message, it seems to be a snowball effect, once a few requests hit the situation where they're not processed quickly enough (and don't release the write lock quickly enough), more requests come in, and that aggravates the issue to the point where the app dies.

We've had several production outages yesterday, so we're most likely going to try and implement a workaround that I'll post here once merged, with the hope that:

1. It can be assessed by the team maintaining this repo.
2. It can help other people affected by this issue.

I must admit that I’m very confused about the OpenIdConnectCachingSecurityTokenProvider class. Maybe I’m missing something obvious. As I understand it, it just has two public properties which are necessary to implement IIssuerSecurityKeyProvider with no other externally facing functionality. Every time either of those property getters are called, the following steps are taken:

1. Call RetrieveMetadata
   a. Enter the write lock
   b. Start the _configManager.GetConfigurationAsync method with a thread pool task (via Task.Run)
   c. Wait for the task to complete
   d. Set the value of two local fields
   e. Exit the write lock
2. Enter the read lock
3. Get one of the two local field values that were just set
4. Exit the read lock
5. Return the value

Why does it need a read/write lock when it only uses the read lock immediately after exiting the write lock to access fields that it just set? Couldn’t it just remember the value that was set when it held the lock and return that? In fact, why does it even need the local fields considering that their values are refreshed every single time the getters are called. There is no mechanism to access those fields without first refreshing them, so why not just get the requested value and return it and leave the local fields out of it?

More importantly, why does it use a lock in a property getter to synchronously wait for an async method? Even if _configManager.GetConfigurationAsync returns cached information, it’s still forced to run on a different thread because it’s called using Task.Run. If it takes a second to acquire the thread to invoke the method, it won’t matter if the method returns without doing any async work, it’s still a bottleneck. If thirty other requests pile up behind that lock during that second, each of those are processed one-at-a-time with more requests queuing behind each of those until all the icebergs have melted and the sun envelopes the earth.

Sorry for the over-analysis. Let me know if my assertions are off base.

Agreed.

Adding to what seems to be unnecessary synchronization, there is no timeout when trying to acquire these locks, so threads could be (and are) waiting forever until they can acquire the lock.

Threads can't enter read lock if at least a thread is entering write lock, so as more requests come up, it aggravates the situation.

Thank you @alex-set and @mderriey for sharing your analysis.
This all makes sense to me.
It's a bug that needs to be fixed.