Conversation
The cursor join tactic played a key's whole cross product into its container builder before checking whether a container had completed, so a key large enough to outrun the driver's fuel buffered its entire output before the iterator returned anything. A single-key join of 10,000 by 10,000 values peaked at 3.2 GB of resident memory for its 100,000,000 matches. The driver's budget never got a chance to act, because `next` did not return until the key was done. Keys now take one of two paths, chosen once the key's edits are loaded. A key whose cross product fits `KEY_WORK_LIMIT`, half the driver's fuel, runs to completion in one call over histories of borrowed values, as before. A larger key is reloaded into histories of value indices, which the iterator owns and which therefore survive a suspension, and is then replayed a container at a time. The indices are what make the state storable: a history of `Cursor::Val<'a>` cannot outlive the storage the iterator owns, so it cannot be held across calls. The same key now peaks at 97 MB. The benchmark added here shows the shapes that still take the uninterrupted path are unaffected: one value per key and a handful of values per key both land within noise of the previous code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The proxy join tactic matched a whole block into its match buffers before handing any of it to the backend. `advance` reports a key entirely within the block that first mentions it, so a block is at least one whole key, and one key's cross product is bounded by nothing; a hot key buffered its entire output the same way the cursor tactic did. Matching now stops once the buffered matches reach `KEY_WORK_LIMIT`, even part-way through a key, and resumes from `at0`, `at1` and `resume` on the next call. Unlike the cursor tactic nothing has to be reloaded: both strategies already address their inputs by offset, and the histories hold identifiers rather than borrowed values, so the state the iterator owns is enough. A backend may now see several `cross` calls for one `advance`, which the trait documents. The corgi backend's block-local `colliding` set is written by `advance` and only read by `cross`, so it stays correct across the extra calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a draft, opened for review of the approach rather than to merge.
Both join tactics buffered a whole key's cross product before yielding anything, so a key larger than the driver's fuel could not be interrupted. A single-key cursor join of 10,000 by 10,000 values peaked at 3.2 GB of resident memory for its 100,000,000 matches; it now peaks at 97 MB.
A key whose cross product fits
KEY_WORK_LIMIT, half the driver's fuel, still runs to completion in one uninterrupted pass. A larger key is replayed a container at a time and can suspend part-way through. The cursor tactic reloads such a key into histories of value indices, because a history ofCursor::Val<'a>cannot outlive the storage the iterator owns; the proxy tactic already addresses its inputs by offset and so needs no reload.One contract change:
ProxyJoinBackend::crossmay now be called several times for oneadvance.differential-dataflow/tests/join_bench.rscovers the shapes that still take the uninterrupted pass, and they land within noise of the previous code.🤖 Generated with Claude Code