Skip to content

Design generated operator tests (step 5) - #4541

Open
evanchooly wants to merge 1 commit into
audits-js-bindingsfrom
audits-test-generation-design
Open

evanchooly wants to merge 1 commit into
audits-js-bindingsfrom
audits-test-generation-design

Conversation

@evanchooly

Copy link
Copy Markdown
Member

Stacked on #4540. Design only: design/operator-audits-test-generation.md, the plan for step 5 of the operator audit plan.

Problem

Step 5 asks for executable Morphia tests generated from the documented examples. That's typed DSL code, checked separately for serialization, behavior and the published result, without erasing handwritten tests. Nothing for it exists yet.

Measurement. Against a 2025 mongodb/docs checkout there are about 550 documented operator examples. About 521 are already owned by the 556 legacy TemplatedTestBase cases. The ~29 unowned ones are mostly admin stages that need an environment the tests don't have (sessions, search indexes, sharded data distribution), plus the positional update operators, whose fixtures are named differently. Generating tests only for unowned examples would therefore cover almost nothing today.

Design

  • Runner. A core-owned OperatorCaseRunner, built on TestBase, with an OperatorCase descriptor and one scenario.json per case. It checks three things separately:
    • serialization: strict BSON, under declared rules only;
    • behavior: the reference operation against the Morphia one, in two separately seeded databases, comparing every collection touched;
    • the published expectation.
  • Legacy cases join through a compatibility adapter.
    • Bodies: their handwritten bodies stay.
    • References: a case's reference comes from its documented source: the form-migration override for the 138 retired locks, a pinned page for never-locked cases. The 96 still-locked cases run as CURATED_REFERENCE and aren't counted as parity.
  • Fixture ownership. Cutover replaces a case's legacy fixture files with scenario.json in the same change, owned by the generated manifest. A migrated-fixture ledger declares each removed file, so the existing audits checks keep guarding fixture bytes.
  • Generation. It covers unowned and future examples through a reviewed MQL-to-factory binding table, validated against the API inventory. A case it can't express becomes a reported gap, never a placeholder.
  • Candidate, integration and promotion. These include attestation, a check that protected files are unchanged, stale-file deletion and rollback.
  • Seven slices, starting with the runner and one converted case ($round).

Decisions for review

  • Legacy cases go through an adapter instead of being regenerated, and only unowned examples are generated. This is the main call. The alternative, limiting step 5 to the ~29 unowned examples, would leave the legacy cases on the permissive JSON comparison.
  • Slice 1 moves one legacy body ($round) verbatim into the new form, so the runner is proven before the adapter exists.
  • Form-rule and published-constraint comparators are ported to Java with shared golden vectors, instead of a shared test-support module.

Review

Three local review rounds; the progress journal lists what each changed.

A descriptor-driven runner checks serialization, behavior and published
results separately. Legacy cases join it through a compatibility adapter
with their documented sources as references; generation covers examples
no test owns. Fixes the identity, scenario, manifest, migrated-fixture,
candidate and promotion contracts and the slice order.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant