Repository navigation
Fix the October flaky tests: sync delivery waits, PDF viewer swap invariant, typing hook, menu focus (bd-c72wsugj) - #818
Merged
Conversation
… race; inventory the October harness flakes Plan skeleton plus a deterministic repro. The binary-file entry goes missing because createNewProject yields (await computeSHA256) between its two index changes, automerge-repo's 100 ms trailing throttle splits them across two flushes, hubHasDoc is a presence check satisfied by the partial index, and the creator disconnects with drainMs 0 before the second flush. Delaying crypto.subtle.digest by 150 ms reproduces the CI diff on every run; a drained disconnect passes. Also inventories the Playwright specs flaking in CI (PR #816 and main): the WebKit pdf-viewer swap (two failure shapes), the warm typing hook, and projects-home-dialogs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r swap gate, typing hook, menu focus quarto-sync-client: presence is not delivery. TestHub gains hubHasHeads / hubHasHeadsOf (the hub holds every change up to the client's heads, by change hash); every project-creating test helper waits on that before the creator disconnects, instead of hubHasDoc, which a partially flushed index satisfies. project-creation-delivery.test.ts forces the slow-digest schedule that lost logo.png's index entry under load. The hub-client E2E project factory drains its disconnect for the same reason. hub-client: - pdfViewer promotes a recompile only once pages are loaded, the initial view is set and the current page's view is FINISHED; the first draw at the default zoom, which setInitialView resets, no longer counts. Unit test drives both orders through a fake PDFViewerApplication. - typing() test hook waits for a run start after the last edit and for every run to end, instead of sleeping one debounce. - Menu keeps keyboard focus when the focused item is replaced (the New menu's placeholder giving way to the registry's groups); the dialogs spec waits for the real menu and drives Templates > Default. The five react-hooks/refs lint errors in Menu.tsx are fixed on the way. Rule .claude/rules/hub-client-tests.md gains the sync section and now also covers the sync-client and hub-mcp tests. Audit findings for the rest of the batch are filed on bd-ag1q1eho; the duplicate strands are closed. Plan: claude-notes/plans/2026-10-10-flaky-ci-tests-pass.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ror (bd-c72wsugj) The 5 s promote-anyway fallback is gone. A recompile's viewer is shown only once it has painted the restored view, however long that takes; pdf.js's documenterror, or 30 s without a paint (VIEWER_WATCHDOG_MS), make the swap fail instead, and the previous document stays on top. PdfPreviewPane shows a rejected swap in the same banner a failed compile gets, cleared by the next compile, and its "a PDF is on screen" flag now flips when the viewer has drawn rather than at hand-off. Unit tests cover the document error, the watchdog, and the pane banner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Member
Author
|
@gordonwoodhull This PR changes some of the test assertions you wrote for the PDF viewer to fix some CI flakiness I noticed. I'm just letting you know in case I misunderstood the goal of the test and mangled its implementation. |
Member
|
Sounds good! We should probably have guidelines and best practices to generate less flaky tests, but I’ve just been fixing them on an ad-hoc basis so far. |
Member
Author
Same here, just to clarify, and I agree that we should spend time sorting this out intentionally in the next few weeks or so. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes bd-c72wsugj (and its duplicates bd-fuw5gcni, bd-5sbguner, bd-tg3vnfxg, already closed in braid). Plan:
claude-notes/plans/2026-10-10-flaky-ci-tests-pass.md.What was flaking
doc-inventory.test.ts(localcargo xtask verify, filed four times): a flush race in the test helper. automerge-repo sends local changes through a 100 ms per-doc throttle;createNewProjectawaits a SHA-256 between creating the text and the binary file, so under load the index doc's first flush carries onlymain.qmd. The helper'shubHasDocpresence check is satisfied by that partial index, and the creator then disconnects without draining. Reproduced 5 of 5 by delaying the digest.pandoc-pdf-viewer"the preview pane" (WebKit, the red check on Listings: List.js init script, pagination and category filtering for default/grid listings #816 and twomainruns): the swap gate accepted pdf.js's first draw at the default zoom, which happens beforesetInitialViewapplies the saved zoom and page and resets every page. Under load that draw finishes first, the new frame is promoted, and then empties itself. Both CI snapshots (page 7 selected at 150%, page 1 the only page with content; or all pages empty) are this.pandoc-warmtyping hook (WebKit): the hook slept one debounce after the last edit and checked for runs; the viewer iframe booting pdf.js on the same thread delayed the debounce past it.projects-home-dialogs(Chromium): the New menu's placeholder item is replaced when the WASM registry answers; the focused placeholder unmounts, focus falls to<body>, and Enter goes nowhere. The spec had only ever exercised the placeholder.Changes
quarto-sync-client.
TestHub.hubHasHeads/hubHasHeadsOfcheck that the hub holds every change up to the client's heads (by change hash). Every project-creating test helper waits on that before the creator disconnects.project-creation-delivery.test.tsforces the slow-digest schedule; reverting the helpers tohubHasDocfails it deterministically. The hub-client E2E project factory drains its disconnect and its "flush synchronously" comment is corrected.PDF viewer. The promotion gate is now: pages loaded, initial view set, current page's view FINISHED, re-checked on
documentinitand everypagerendered. The 5 s promote-anyway timeout is gone (decided: the frame on top is never an empty viewer); pdf.js'sdocumenterror, or 30 s without a paint, make the swap fail and keep the previous document.PdfPreviewPaneshows a failed swap in the same banner a failed compile gets, and its "a PDF is on screen" flag flips when the viewer has drawn rather than at hand-off. Unit tests drive a fakePDFViewerApplicationthrough both draw orders, a document error, and the watchdog.Typing hook.
typing()waits for a run start after the last edit and for every run to end, instead of sleeping.Menu.
Menukeeps keyboard focus when the focused item is replaced (tracked by the root's focus events, refocused in a layout effect). The dialogs spec waits for the real menu and drives Templates → Default. The five pre-existingreact-hooks/refslint errors inMenu.tsxare fixed.Docs.
.claude/rules/hub-client-tests.mdgains a "presence is not delivery" section and two Playwright bullets, and now also covers the sync-client and hub-mcp tests. Audit findings for the rest of the October harness batch are filed on bd-ag1q1eho.Verification
cargo xtask verify --skip-hub-build: all steps pass.exit-drain(26 tests) pass.pdfViewer(6),PdfPreviewPane(9),Menu.submenu(8),ProjectsHome.newMenu(4) pass; ESLint clean on touched files.--retries=0 --workers=1: dialogs 8 of 8 under CPU load (a first version of the Menu fix failed 3 of 3 there, which is how the event-based version came about), typing hook 3 of 3 under load, viewer 6 of 6 at normal load and under load.🤖 Generated with Claude Code