Skip to content

feat(vantage): add pinned RAG-Multi-Corpus benchmark importer and 20-case pilot - #636

Open
josephlongbui-stack wants to merge 1 commit into
Deodat-Lawson:mainfrom
josephlongbui-stack:feat/joseph-vantage-benchmark-import-v1-main
Open

josephlongbui-stack wants to merge 1 commit into
Deodat-Lawson:mainfrom
josephlongbui-stack:feat/joseph-vantage-benchmark-import-v1-main

Conversation

@josephlongbui-stack

Copy link
Copy Markdown

Summary

Adds an offline, hash-pinned importer for the external RAG-Multi-Corpus benchmark and a small, deterministic 20-case pilot fixture for future Vantage evaluation work.

This change is self-contained:

  • It only adds new files: no existing files are modified, and there are no dependency or lockfile changes.
  • It does not depend on any unmerged Vantage code.
  • It adds no routes, database tables, migrations, or model calls.

What's included

Path Purpose
pipelines/scripts/benchmarks/csv.ts Strict CSV reader (quoted commas and newlines, malformed quoting rejected)
pipelines/scripts/benchmarks/rag-multi-corpus.ts Zod evaluation schema, validation, content-hash case IDs, deterministic coverage-balanced selection
pipelines/scripts/benchmarks/import-rag-multi-corpus.ts CLI: explicit --download (hash-verified, outside the repo only) and offline import / --check
pipelines/scripts/benchmarks/rag-multi-corpus.test.ts 45 tests
fixtures/vantage-benchmarks/rag-multi-corpus/ pilot.json (20 cases), pilot-source.csv (their source rows), source-lock.json, LICENSE.upstream.txt, README.md

Data integrity

  • Source pinned to upstream commit 39071af3f4dd25e59f5c59a6f9b6e8e99cd643b3, with a SHA-256 for each of the three downloaded files. The importer rejects any CSV that doesn't match.
  • referenceAnswer is null and vantageAnnotations is null on every case. The dataset provides supporting passages only, and nothing is inferred or invented. The schema rejects either field if it is filled in.
  • --check reproduces both fixtures byte-for-byte from the full pinned CSV (1,252 rows → 1,075 unique cases, 177 exact duplicates merged with all row locations kept).
  • Independently cross-checked with Python's standard csv/json modules: same row, case, duplicate and passage counts, and identical IDs for all 20 pilot cases. Two upstream rows are embedded verbatim in the tests as an oracle that this importer did not produce.
  • Only the 20-case sample (about 31 KB) is committed. The full CSV and the multi-format corpus are not.

⚠️ Licensing: needs maintainer review before merging

  • Upstream ships an MIT LICENSE (Copyright (c) 2025 UDAY ALLU). It is included verbatim as LICENSE.upstream.txt, and the test suite checks its SHA-256.
  • Attribution, source revision, and the citation @Allu2025RAGMultiCorpus are recorded in the fixture README and source-lock.json.
  • The upstream README's "License" section separately says: "This dataset is provided for research and experimentation. All content is synthetic…" It contains no explicit restriction (its citation section mentions "research or product evaluation"), but it isn't a formal license either.
  • No separate permission has been obtained from the upstream authors and no legal review has been done. Please confirm that committing this evaluation sample is acceptable.

Testing

Run locally on this branch (based on main @ 20b85c78):

pnpm --filter @launchstack/pipelines exec vitest run scripts/benchmarks/rag-multi-corpus.test.ts   # 45 passed
pnpm --filter @launchstack/pipelines typecheck                                                     # clean
pnpm exec eslint --max-warnings 0 pipelines/scripts/benchmarks/*.ts                                # clean
pnpm exec prettier --check pipelines/scripts/benchmarks/*.ts fixtures/vantage-benchmarks/rag-multi-corpus/*   # clean

Selection tests run on synthetic pools larger than the pick. To show they can fail, each scoring rule (coverage, enterprise, category, pair, ID tie-break) was removed in turn, and a dedicated test failed each time.

CI note: .github/workflows/CI.yml does not currently run the @launchstack/pipelines test suite, so these tests won't run automatically on this PR. Adding pipelines to CI would be a separate change.

Not in scope / future work

  • Corpus ingestion, retrieval or generation runs, and scoring.
  • Vantage-specific annotations. These will go in a separate reviewed sidecar file, never in this fixture.
  • Any mapping to production Vantage inputs.

🤖 Generated with Claude Code

…case pilot

Offline, hash-verified importer for the external RAG-Multi-Corpus
benchmark (upstream commit 39071af), a deterministic coverage-balanced
20-case pilot fixture with its source rows, an evaluation input schema,
and 45 tests. Adds new files only; no dependency, config or existing-file
changes, and no dependency on unmerged Vantage code.

Upstream MIT notice preserved in LICENSE.upstream.txt. The upstream
README separately describes the dataset as "for research and
experimentation"; flagged for maintainer review in the fixture README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 16, 2026

Copy link
Copy Markdown

@josephlongbui-stack is attempting to deploy a commit to the Timothy Lin's projects Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant