Skip to content

fix: refresh the runtime JWT on demand instead of waiting for the next GetProject poll - #10096

Open
nishantmonu51 wants to merge 4 commits into
mainfrom
nishant/fix-runtime-jwt-refresh
Open

nishantmonu51 wants to merge 4 commits into
mainfrom
nishant/fix-runtime-jwt-refresh

Conversation

@nishantmonu51

Copy link
Copy Markdown
Collaborator

Runtime requests in Rill Cloud failed with Timed out waiting for a fresh JWT or the runtime's jwt is expired after a laptop sleep, a network drop or a throttled tab. The 30-minute runtime JWT was only refreshed by the 15-minute GetProject poll, so when a poll was missed, every request waited passively for 60 seconds and then failed together until a reload. Expiry was also counted from when the client first saw the token, so a token handed over from the query cache was credited a full lifetime.

Steps to reproduce

  1. Open a dashboard in Rill Cloud and open DevTools > Network.
  2. Block the request pattern admin.rilldata.com/v1/orgs/*/projects/*, so token refreshes fail while runtime requests still go through.
  3. Wait 30 minutes (the token lifetime), remove the block, then change a filter or the time range.
  4. On main, requests wait for the next scheduled poll, so after 60 seconds every panel fails with Timed out waiting for a fresh JWT (some with rpc error: code = Unauthenticated desc = jwt is expired) until a reload. With this change, the first request fetches a fresh token and the panels load.

What the fix does

  • RuntimeClient takes a JWT refresher. A request that finds its token stale refreshes right away; concurrent requests share one in-flight refresh, which retries with backoff while the admin service is unreachable and fails fast on 401 or 403.
  • Expiry comes from the token's exp claim, capped at its lifetime from receipt, with the default TTL for tokens without claims.
  • A runtime jwt is expired rejection refreshes once and retries the request.
  • RuntimeProvider refreshes a stale token when the tab becomes visible, regains focus or comes back online.
  • Each refresh attempt is abandoned after 20 seconds with its AbortSignal aborted, waiting requests keep their own 60-second deadline, and updateJwt() or dispose() wakes them at once instead of the 50 ms poll.
  • Refreshers are kept per auth context. The project layout refreshes through GetProject (the bearer-token GetProject on public URLs). The edit layout derives the provider's JWT, auth context and refresher from the View As stores, so a user token refresh that lands while View As is active cannot replace the mock user's token.
  • GetProject stops polling after a 401 or 403.

Notes for review

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

…t GetProject poll

A request that found its JWT expired waited passively for up to 60s for the
15-minute GetProject refetch and then failed with "Timed out waiting for a
fresh JWT" whenever a laptop sleep, a network drop or a throttled tab had
skipped that refetch. Expiry was also derived from when the client first saw
the token string, so a token handed over from the query cache was credited
with a full TTL and the runtime rejected it with "jwt is expired".

- RuntimeClient takes a JWT refresher; a request that finds the token stale
  triggers it right away, shares one in-flight refresh, retries with backoff
  while the issuer is unreachable and fails fast on 401/403
- Expiry comes from the token's exp claim (capped at its lifetime from
  receipt), falling back to the default TTL for tokens without claims
- A runtime "jwt is expired" rejection refreshes once and retries the request
- RuntimeProvider refreshes a stale token when the tab becomes visible, gains
  focus or the network comes back
- The project and edit layouts wire a refresher that refetches GetProject,
  the bearer-token GetProject on public URLs or the View As credentials
- GetProject stops polling after a 401/403 instead of repeating it forever
Follow-ups from review of the on-demand JWT refresh:

- A refresh attempt pins the refresher and the auth context it serves before
  the backoff sleep and revalidates both after waking and after the call, so a
  user token refreshed while View As switched the context is never applied
- Each refresher call is abandoned after 20s with its AbortSignal aborted (the
  admin refresher cancels the GetProject query so the retry is not deduped
  into the stalled request), waiting requests are bounded by their own 60s
  deadline, and updateJwt() or dispose() wakes them immediately instead of
  the 50ms poll
- Refreshers are kept per auth context: the provider's stays registered for
  its context while the editor's View As registers one that reissues its mock
  token through GetDeployment, and the user's applies again when View As ends
…lection

The edit layout forwarded every GetProject JWT to RuntimeProvider with the
"user" context, so any user-token update that landed while View As was
active (the 15-minute poll, a focus refetch, or an on-demand refresh that was
still in flight when View As started) replaced the mock token and the preview
showed the user's data under View As.

- The layout derives the provider's jwt, authContext and refreshJwt from the
  View As stores (editSessionRuntimeAuth): the mock user's JWT and a refresher
  that reissues it through GetDeployment while a mock user is selected, the
  user's otherwise
- createUpdateEditSessionDevJWT no longer registers the refresher itself; the
  layout owns it through the provider's props
- A regression plays the layout and provider through an in-flight user
  refresh that completes after View As started and checks the mock
  credentials survive it
@nishantmonu51 nishantmonu51 added Type:Bug Something isn't working Area:Dashboard Size:L Large change: 500-1,999 lines labels Nov 5, 2026
# Conflicts:
#	web-admin/src/routes/[organization]/[project]/+layout.svelte

@AdityaHegde AdityaHegde left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels like we are doing a lot of manual refreshing, something that comes with tanstack query.

Maybe there is a path to replace while of refreshing with it with retry hook triggered by queryClient.refetch in interceptors? One issue might be circular dependancy that needs some workaround.

return { promise, resolve };
}

function sleep(ms: number): Promise<void> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: reuse asyncWait

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area:Dashboard Size:L Large change: 500-1,999 lines Type:Bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants