Error when refreshing token

Hello Ben,

Thank you for reaching out. We are exploring the cause of the refresh token request failures. We recognize that the oauth2 token refresh flow is sensitive and take these issues very seriously.

I’ll be in touch as soon as I have any findings that helpful.

For the request you posted originallly, here is the sequence of events that occured. All times in `UTC` and `hashes` refer to non-identifying values which represent an instance of a token. We do not ever have access to the actual token.

1. 2024-04-24T00:31:49.000+00:00 
   1. A user token refreshed many times is successfully refreshed again, generating token hash `3de3198111`.
   2. We return a 200 success with the new token pair
2. 2024-04-24T01:32:22+00:00
   1. A request to refresh token hash `3de319811` is received. Our system refreshes the token and updates the data store with the new pair.
   2. While updating the token history, a data store connection issue occurred leading to an unhandled exception.
   3. We return a 500 internal server error
3. 2024-04-24T01:46:35+00:00
   1. Another request is received to refresh token hash `3de319811`
      1. Because the client didn’t get the refreshed token from step 2, they still see `3de319811` as the current token pair.
      2. But, because our data store was updated before the exception was thrown, we see `3de319811` as a previously used token outside of the 1 minute grace period.
         1. Per the oauth2 spec, when a refresh request is made with a previously used token, the entire chain of user and refresh tokens is revoked
      3. We revoke the original user token, and the current refresh token.
      4. We return a 400 bad request (invalid client,……..)
4. Every request since:
   1. We now consider the token invalid, and return the 400.

This is clearly an issue on our side, and an unfortunate sequence of events.  The takeaways that we are taking and workibng no now:

1. We need to evaluate the connection issue, how it occured, and why it was unhandled.
   1. This investigation is underway.
2. We need to evaluate the oauth2 refresh flow and ensure that anywhere it could fail will appropriate roll back any updates, or at least ensure the new pair is always returned.
   1. This investigation is also underay.
3. We need to update our documentation (which is already going through an overhaul) to state that when a 5xx response is received (or the connection fails), the client should do a retry.
   1. This is one of the goals of the 1 minute grace period.  Although we had updated the refresh pair, there is 1 minute where we still consider the previous pair valid and will return the new pair.
   2. Even with the fixes we will implement for points 1 and 2, there is still the risk of a connection issue causing the client to not receive the new pair. This retry will protect against these scenarios.

I recognize that the recommendation in step 3 is a band-aid. But, if you can introduce a retry mechanism in your auth flow it will protect from this happening again. Unfortunately, the original token (and any others that have had similar problems) is invalid now and can’t be restored. That is why you only see 400s now.

One point of clarification. Have you seen this with multiple user token flows, or just one that is completely broken?

Hmmm, concerning. My interpretation of your original post was that it was more than just the one token. I will expand my research to all tokens by your client.

I have received nebulous reports of dropped connections in some of our microservices that I am investigating now. When I originally saw your post I thought it was going to be the same thing. It turned out not to be with that token, but I am concerned that your other failures might. If it is dropped connections, I believe the retry will protect you as well. But, that is clearly not the solution to the problem.

I’ll be in touch with any additional information I learn. Let me know any status changes on your side, good or bad.

Sounds great. I am adding logic to make it easy to see all failures for a given client and why.  Right now, I have to find and look them up one at a time via logs.  I expect by sometime tomorrow to be able to immediately get a list of client refreshes, and the outcomes. That should make it easy to see macro trends that aren’t visible when looking at individual requests.

Sounds great. Keep me posted.

** Reposting this response as I am not sure it went through:

You mention trying to refresh twice and that made me think of a possibility. We recently had an embed based integration getting refresh token failures. It turned out that in cases where their application had many embeds on the same page, they would all attempt to separately refresh the underlying user token at the same time. (each on their own thread). Because of the multiple parallel requests being so close together, it was hitting a perfect timing window in our logic that lead to a race condition and ultimately a duplication error in our data store.

A fix for that problem is underway now. We are refactoring the refresh token request to be entirely atomic, but its not an insignificant change as we need to ensure it will be able to scale and protect the endpoint from being overwhelmed. I will look at your requests tomorrow to see if it could be the same thing,  but wanted to send this out in case it rings any bells. This is another scenario where the retry should work, but can get a token in a bad state if the requests line up being handled at just the wrong moment in a sub-micro window.

Super glad to hear it. As a matter of professional experience, I remain a bit concerned you will have problems. Hopefully, next week will go smoothly.  Of course, you can resolve this thread whenever you feel the original question has been handled.