Cancel the rest of a concurrent group when one member fails - #203
Open
shellygr wants to merge 1 commit into
Open
Cancel the rest of a concurrent group when one member fails#203shellygr wants to merge 1 commit into
shellygr wants to merge 1 commit into
Conversation
run_pipeline_inner started the pre-formalization work with a bare create_task and did not await it until the whole extraction fan-out had finished, so a fatal error inside it sat unobserved while the extraction agents kept spending on a run that could no longer produce anything. prepare_formalization's own gather had the other half: without return_exceptions it propagates the first child's error immediately and leaves the sibling running. gated_group() wraps a TaskGroup with the single-member unwrap, and the five concurrency points in the driver now share it. Cancelled tasks are recorded in the per-phase breakdown and left out of the failure roll-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
run_pipeline_innerstarted the pre-formalization work with a bareasyncio.create_taskand didnot await it until after the whole property-extraction fan-out had finished. A fatal error inside
it was stored on the task object and sat there unobserved for as long as extraction took. In one
run that was 41 minutes and four bug-analysis agents' worth of spend on a run that could no longer
produce anything, and none of it was kept: extraction batches are only persisted downstream of the
await.
prepare_formalization's ownasyncio.gatherhad the other half of the problem. Withoutreturn_exceptions, gather propagates the first child's exception to its awaiter immediately anddoes not cancel the remaining children. It only cancels them when the gathering future itself is
cancelled. So the sibling detached and ran to completion too. Fixing one level leaves the other,
which is why both are here.
The driver already had the right shape one step earlier. The preflight and the system analysis
share a task group, and the comment there states the policy: neither side outlives the other's
failure. This applies that policy at the concurrency points that were missing it.
How
gated_group()incomposer/io/multi_job.pyyields anasyncio.TaskGroupand unwraps theresulting
BaseExceptionGroup, so the caller sees the failure itself when only one member reallyfailed and keeps the group when two fail at once. It sits beside
maybe_semaphore, and both editsites already import from that module, so there are no new import edges.
Five call sites use it: the preflight and analysis overlap, whose now-duplicated inline unwrap is
deleted rather than left beside the helper; the pre-formalization and extraction overlap; the
per-unit extraction fan-out; the plugin pre-inference fan-out; and the AutoSetup and invariants
pair. Task objects are held rather than completion order, so callers that depend on result order
still get it.
The unwrap re-raises outside the
exceptblock.raise exc from Nonewould clear__cause__andsuppress
__context__, which matters for a wrapper whose message only points at the exception itwraps.
task_loggergains aCancelledErrorbranch, so a cancelled task's tokens are folded into theper-phase breakdown instead of being left in the in-flight map. Cancellations appear in the
per-row status but are excluded from the failure roll-up, so one real failure in an N-unit fan-out
does not print as N failures.
Deliberately unchanged
The per-component
asyncio.gather(..., return_exceptions=True)stays as it is. Per-componentfailure isolation is the opposite policy on purpose.
The four bare fatal gathers in the natspec driver are the same class in a different pipeline, and
are left for a follow-up.
Known gap
A cancelled task does not fire its handler's
on_error, becauserun_taskcatchesExceptionandCancelledErroris not one, so its row stays at RUNNING until the process exits. This predates thechange, since the first overlap already cancelled, but gating the extraction fan-out turns it from
a one-row artefact into a many-row one. Widening the handler touches three UI factories, so it is
out of scope here.
Testing
Four new test modules.
tests/test_gated_group.pycovers the primitive: a failure cancels a 30ssibling and surfaces itself, a double failure keeps the group, cancelling the holder cancels the
members, and a member raised with
fromkeeps its cause through the unwrap.tests/test_extraction_fanout_gate.pydrives the real_extract_allwith only the LLM callstubbed, covering both fan-outs inside it.
tests/test_prover_prepare_formalization_gate.pycoversthe AutoSetup and invariants pair in both directions.
tests/test_cancelled_task_accounting.pycovers the new accounting branch and the roll-up.
tests/test_pipeline_overlap.py's module docstring and its two second-pair tests described the oldbehaviour as intended, so they are rewritten to state the policy, with a non-zero delay on the
surviving side so they assert cancellation rather than winning a race. A third test covers
caller-side cancellation reaching the second pair. Those three tests waited a fixed interval before
cancelling and were flaky under parallel load; they now wait on an event the stub sets when it
starts.
Each new test was checked by putting
asyncio.gatherback in place of the gate and confirming itfails.
Targeted run: 35 passed.
pyright composer/ analyzer sanity_analyzer certora_autosetup: 0 errors.