Repository navigation
feat(vantage): add pinned RAG-Multi-Corpus benchmark importer and 20-case pilot - #636
Open
josephlongbui-stack wants to merge 1 commit into
Conversation
…case pilot Offline, hash-verified importer for the external RAG-Multi-Corpus benchmark (upstream commit 39071af), a deterministic coverage-balanced 20-case pilot fixture with its source rows, an evaluation input schema, and 45 tests. Adds new files only; no dependency, config or existing-file changes, and no dependency on unmerged Vantage code. Upstream MIT notice preserved in LICENSE.upstream.txt. The upstream README separately describes the dataset as "for research and experimentation"; flagged for maintainer review in the fixture README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@josephlongbui-stack is attempting to deploy a commit to the Timothy Lin's projects Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an offline, hash-pinned importer for the external RAG-Multi-Corpus benchmark and a small, deterministic 20-case pilot fixture for future Vantage evaluation work.
This change is self-contained:
What's included
pipelines/scripts/benchmarks/csv.tspipelines/scripts/benchmarks/rag-multi-corpus.tspipelines/scripts/benchmarks/import-rag-multi-corpus.ts--download(hash-verified, outside the repo only) and offline import /--checkpipelines/scripts/benchmarks/rag-multi-corpus.test.tsfixtures/vantage-benchmarks/rag-multi-corpus/pilot.json(20 cases),pilot-source.csv(their source rows),source-lock.json,LICENSE.upstream.txt,README.mdData integrity
39071af3f4dd25e59f5c59a6f9b6e8e99cd643b3, with a SHA-256 for each of the three downloaded files. The importer rejects any CSV that doesn't match.referenceAnswerisnullandvantageAnnotationsisnullon every case. The dataset provides supporting passages only, and nothing is inferred or invented. The schema rejects either field if it is filled in.--checkreproduces both fixtures byte-for-byte from the full pinned CSV (1,252 rows → 1,075 unique cases, 177 exact duplicates merged with all row locations kept).csv/jsonmodules: same row, case, duplicate and passage counts, and identical IDs for all 20 pilot cases. Two upstream rows are embedded verbatim in the tests as an oracle that this importer did not produce.LICENSE(Copyright (c) 2025 UDAY ALLU). It is included verbatim asLICENSE.upstream.txt, and the test suite checks its SHA-256.@Allu2025RAGMultiCorpusare recorded in the fixture README andsource-lock.json.Testing
Run locally on this branch (based on
main@20b85c78):Selection tests run on synthetic pools larger than the pick. To show they can fail, each scoring rule (coverage, enterprise, category, pair, ID tie-break) was removed in turn, and a dedicated test failed each time.
CI note:
.github/workflows/CI.ymldoes not currently run the@launchstack/pipelinestest suite, so these tests won't run automatically on this PR. Adding pipelines to CI would be a separate change.Not in scope / future work
🤖 Generated with Claude Code