Skip to content

Retire DOCS_ERROR and FIXTURE_CURATION locks through reviewed corrections - #4535

Open
evanchooly wants to merge 1 commit into
audits-numeric-typesfrom
audits-reviewed-corrections
Open

evanchooly wants to merge 1 commit into
audits-numeric-typesfrom
audits-reviewed-corrections

Conversation

@evanchooly

Copy link
Copy Markdown
Member

Stacked on #4534. Step 4, slice 5 of the operator audit plan: the DOCS_ERROR and FIXTURE_CURATION lock families.

Problem

These locks mark fixtures that deliberately differ from the docs. DOCS_ERROR means the docs are wrong: bsonSize prints sizes the server doesn't return, binarySize misspells tea_set in the expected output, and rank documents the wrong rank. FIXTURE_CURATION means Morphia changed something on purpose: $graphLookup reads from the harness collection example_test, and toUUID drops generated ids.

The lock text says why the fixture differs, but not what changed or which source it was checked against. So an upstream change can't be detected, and nobody can see exactly what was edited.

Change

  • FormCorrection(file, pointer, documented?, curated?, index?) is a reviewed correction to one field. It works in three modes:

    • replace a documented value;
    • remove a documented field;
    • add a field at a key position.

    A rename is a remove plus an add. The documented value is pinned in extended JSON, so if the page no longer documents exactly that value, the correction is stale instead of applying to something else.

  • Corrections are discovered field by field, after form rules and number retypes. Discovery keeps going after each one, so every edit in a document needs its own correction. These differences are never corrected and stay gaps:

    • anything reaching a whole file or a top-level row or stage;
    • a key-order difference (that's form);
    • a type change at the same value (that's a retype).

    Form rules are preferred: rule minimization never drops a rule if doing so needs more corrections. Each correction must be necessary.

  • Corrections are allowed only for DOCS_ERROR and FIXTURE_CURATION locks.

  • fetch.py candidates has two fallbacks:

    • when the last commit at the origin date deleted the page during the manual's move, it uses that commit's parent;
    • when the fixture predates every page at that path, it uses the commit that first added the page. That case is recorded as SourceRelation.AFTER_ORIGIN (only $toUUID).
  • The one lockd marker is handled, and each declaration records its marker's name.

Result

52 such locks have no family outside MORPHIA_FORM, NUMERIC_TYPE, DOCS_ERROR and FIXTURE_CURATION.

  • 29 are retired.
    • 11 equal their pinned page exactly. Those pages already had the corrected values before the fixtures were written, so each lock's deviation was against some other, unidentified source.
    • 18 need 34 corrections, and I read each one against its lock reason. They cover wrong documented results, misspelled fields, Decimal128 rounding, a field the server doesn't return, a simpler onNull, dropped ids and the harness collection name.
  • 23 stay locked with a recorded gap: generated include extracts, constructors the strict normalizer rejects, and data or expected files whose rows the page doesn't document.

Totals are now 91 overrides and 51 unresolved. The inventory shows 91 OVERRIDDEN, 143 PRESERVED_CURATED and 145 locked examples. Recapturing the earlier 62 overrides and 28 gaps gave identical results.

Decisions to review

  • degreesToRadians also removes the documented ObjectId _id from its expected row. Its lock only says "the angle_a's were different". I accepted it as a reviewed decision and documented it.
  • toUUID's source page is newer than the fixture (AFTER_ORIGIN). It pins the values the correction changes, but it isn't the fixture's provenance.
  • I declined one review suggestion: comparing a rename's added value with the removed one. The added value is recorded literally in the correction, so nothing is hidden from review.
  • The docs pages are still not committed (CC BY-NC-SA).

Verification

  • Core aggregation and query operator tests: 646 run, 0 failures, 7 skipped, under Critter, then reflection.
  • Audits: 388 tests, 2 optional skips, under both mapper flags.
  • New tests: FormCorrectionsTest (pinning, staleness, remove/add in place, out-of-range position, row floor) and FormMigrationTest#valueCorrectionsNeedADocsErrorOrCurationLock (category gating, field-level discovery, rules kept over corrections, a changed page rejected).
  • Two local review rounds by an independent agent:
    • Round 1 found that corrections taken at a parent document swallowed sibling edits, and on one-row files replaced the whole file. It also found that the exact retirements had been misattributed to later upstream fixes, and that toUUID's provenance wasn't explicit. All of that is fixed.
    • Round 2 confirmed the fixes and all 18 fixtures. Its two lows were an out-of-range add position (now stale) and one wording issue (fixed).

…ions

A FormCorrection replaces, removes or adds one documented field, pinned to
the documented value so a changed page makes it stale. Corrections are
discovered field by field after rules and retypes, never reach a whole file
or top-level row, and are allowed only for DOCS_ERROR and FIXTURE_CURATION
locks. fetch.py falls back to the page before the manual move and to the
first page for fixtures that predate it, recorded as AFTER_ORIGIN.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant