Skip to content

retry: let the innermost retry interceptor set the attempt timeout - #154

Merged
boks1971 merged 1 commit into
mainfrom
raja/retry-innermost-timeout
Oct 7, 2026
Merged

boks1971 merged 1 commit into
mainfrom
raja/retry-innermost-timeout

Conversation

@boks1971

@boks1971 boks1971 commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

A retry interceptor nested inside another one sets the attempt timeout again, and a timeout the caller sets still wins over both. This fixes livekit/livekit#4955.

Who sets the timeout

A request goes through the client's interceptors in the order they were added. The first one added runs first and wraps the rest:

caller → generic retry → egress retry → send

Each retry interceptor adds its attempt timeout to the request options. At the send, the options apply in order, so the last timeout in the list is used. In the protocol egress client, the generic retry comes from psrpc.timeout (300 ms in the issue), and the egress retry inside it uses 10 s, then 20 s and 40 s.

What #152 fixed

Each interceptor added its timeout at the end of the list. That overrode a timeout the caller set:

caller:          [5s]
generic retry:   [5s, 300ms]
egress retry:    [5s, 300ms, 10s]   → 10s, the caller's 5s is ignored

#152 put the interceptor's timeout at the front instead, so a caller's timeout comes later and wins.

Where #152 broke

An inner interceptor cannot tell the outer interceptor's timeout from a caller's. With no caller timeout:

caller:          []
generic retry:   [300ms]
egress retry:    [10s, 300ms]       → 300ms

The generic timeout now beat the egress one. Every StartEgress attempt timed out before a worker was picked, and the call failed after about 17 s.

The fix

Only the outermost retry interceptor sees the caller's options alone. It checks whether the caller set a timeout and passes the answer down in the context. If the caller set one, no interceptor adds its own. If not, each interceptor adds its timeout at the end, so the innermost one wins:

caller:          []
generic retry:   [300ms]            (notes: no caller timeout)
egress retry:    [300ms, 10s]       → 10s

caller:          [5s]
generic retry:   [5s]               (notes: caller timeout 5s)
egress retry:    [5s]               → 5s
No caller timeout Caller sets a timeout
Before #152 innermost wins ✓ caller ignored ✗
After #152 outermost wins ✗ caller wins ✓
With this fix innermost wins ✓ caller wins ✓

TestNestedInnermostTimeoutWins fails without the fix.

🤖 Generated with Claude Code

#152 put the attempt timeout before the caller's options so a caller's
timeout wins. A retry interceptor nested inside another one then took the
outer one's timeout for a caller's, and the outer timeout won. The
outermost retry interceptor now passes the caller's own timeout down in
the context. Without one, the innermost interceptor's timeout wins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@boks1971
boks1971 merged commit e9701ee into main Oct 7, 2026
8 checks passed
@boks1971
boks1971 deleted the raja/retry-innermost-timeout branch October 7, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Egress fails to start on v1.13.8: psrpc.timeout now overrides the egress client's attempt timeout (psrpc#152)

2 participants