Skip to content

fix(video): enforce one absolute deadline for streaming frame decode - #9393

Merged
lstein merged 3 commits into
invoke-ai:mainfrom
lstein:fix/video-frame-decode-single-deadline
Aug 9, 2026
Merged

fix(video): enforce one absolute deadline for streaming frame decode#9393
lstein merged 3 commits into
invoke-ai:mainfrom
lstein:fix/video-frame-decode-single-deadline

Conversation

@lstein

@lstein lstein commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Follow-on to #9163 (deferred non-merge-blocker).

iter_video_frames charged its timeout twice:

  1. it waited up to timeout to acquire the stream slot and a decoder slot, then
  2. handed _iter_video_frames_unbounded a fresh full timeout for the first frame.

So a caller that waited 29 s for capacity and then hit a hung decoder blocked for ~59 s — twice the bound it believed it was enforcing. Both callers size real budgets on that value: upload probing (which runs on a request worker) and the video nodes.

_run_worker already had the right shape for the one-shot path — take a deadline up front, and pass the remaining time to the worker after acquiring the slot. This applies the same pattern to the streaming path.

Why the first frame only

timeout in the streaming decoder is a decoder-inactivity bound, not a total wall-clock budget: it is restarted after every yielded frame so a long video isn't killed for being long. Shrinking it for all frames (the naive "pass remaining through") would mean a decode that queued 29 s for a slot then got 1 s per frame — spurious failures on perfectly good videos.

So _iter_video_frames_unbounded gains an optional first_frame_timeout that overrides the budget for the first frame only. Post-first-frame behavior is unchanged.

Testing

New regression test test_capacity_wait_and_first_frame_share_one_deadline: holds the single stream slot for 1.6 s of a 2.0 s budget, then releases it against a worker that never produces a frame, and asserts the total stays under one budget plus jitter. Verified it fails on main (~3.6 s, the old hold + timeout) and passes with the fix.

pytest tests/app/util/test_video_thumbnails.py — 54 passed.

🤖 Generated with Claude Code

`iter_video_frames` waited up to `timeout` for a decoder/stream slot and
then handed `_iter_video_frames_unbounded` a fresh full `timeout` for the
first frame, so a call could take nearly 2x the bound its caller thought it
was enforcing (upload probing and the video nodes both size their budget on
that value).

The capacity wait is now charged against the same deadline as the first
frame — the remaining-time-after-acquire pattern `_run_worker` already uses.
Frames after the first still get a full `timeout` each: past the first frame
the budget is a decoder-inactivity bound, not a queueing one, and shrinking
it would kill legitimate long decodes that had queued for a slot.

Deferred non-blocker from PR invoke-ai#9163.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added python PRs that change python files python-tests PRs that change python tests labels Jul 28, 2026
@lstein lstein added the 6.14.1 label Jul 28, 2026
@lstein lstein moved this to 6.14.1: Bug fixes to 6.14.0 in Invoke - Community Roadmap Jul 28, 2026
@JPPhoto JPPhoto changed the title fix(video): bound iter_video_frames by one deadline, not two fix(video): enforce one absolute deadline for streaming frame decode Aug 9, 2026

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Other findings/issues that may be covered in related PRs (I haven't yet gotten to them) follow. If you don't have PRs for these, I suggest making them:

  • invokeai/app/util/video_thumbnails.py:296,369: Relative remainder is converted into a new deadline after worker startup, so slow startup can exceed the promised single timeout. Test: delay worker startup after near-expired capacity wait and assert total time stays within the original deadline.

  • tests/app/util/test_video_thumbnails.py:308-319: Unsynchronized sleep can release the slot before the consumer starts, causing a flaky false failure. Test: synchronize at semaphore acquisition before starting the hold timer.

@lstein
lstein enabled auto-merge (squash) August 9, 2026 17:07
@lstein
lstein merged commit 7b82f29 into invoke-ai:main Aug 9, 2026
17 checks passed
@lstein
lstein deleted the fix/video-frame-decode-single-deadline branch August 9, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.1 python PRs that change python files python-tests PRs that change python tests

Projects

Status: 6.14.1: Bug fixes to 6.14.0

Development

Successfully merging this pull request may close these issues.

2 participants