Extract indexing field metadata behind a format-neutral value object - #1351
Open
jwils wants to merge 1 commit into
Open
Extract indexing field metadata behind a format-neutral value object#1351jwils wants to merge 1 commit into
jwils wants to merge 1 commit into
Conversation
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
August 15, 2026 21:58
This was referenced Aug 15, 2026
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
August 21, 2026 13:41
228eda8 to
a6b87d9
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 5, 2026 19:18
a6b87d9 to
2a84c58
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
2 times, most recently
from
September 9, 2026 14:36
672592b to
e3d4ceb
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 10, 2026 17:42
e3d4ceb to
bfdfe9a
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 10, 2026 18:13
bfdfe9a to
b271dba
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 11, 2026 15:12
b271dba to
9dc4994
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 11, 2026 15:25
9dc4994 to
9cad9e4
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 11, 2026 15:39
9cad9e4 to
5b42c62
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 11, 2026 15:55
5b42c62 to
a93dd5e
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 11, 2026 16:34
a93dd5e to
f88251c
Compare
jwils
added a commit
that referenced
this pull request
Sep 11, 2026
## Why `Operation::Factory` performs JSON-schema validation and version-based record-preparer selection inline, which couples the indexer core to the JSON ingestion format. To support other formats such as protobuf, that logic needs to live behind a format-neutral seam. ## What - Add `Indexer::IngestionAdapter`, an interface for validating an event and providing its record preparer - Add `Indexer#ingestion_adapters_by_format`, keyed by the event's `ingestion_format` - Route each event to one adapter by format; untagged events default to `json`, while unknown formats produce an actionable `FailedEventError` - Preserve multiple indexing operations from the selected adapter's event - Extract the existing JSON validation and version selection into `IngestionAdapter::JSONEvents`; the next PR moves it out of `elasticgraph-indexer` - Allow tests to inject the complete adapter registry through `Indexer#initialize` ## Verification No manual testing. ## Stack Current PR is marked with `->`. - -> [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302) - [#1220 Add configurable indexing event decoder](#1220) - [#1351 Extract indexing field metadata behind a format-neutral value object](#1351) - [#1284 Keep ingestion schema versions adapter-owned](#1284) - [#1376 Pass transport metadata to event decoders](#1376)
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
2 times, most recently
from
September 11, 2026 20:26
ecd9040 to
710aae4
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 12, 2026 21:47
710aae4 to
125ea0b
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
2 times, most recently
from
September 12, 2026 22:26
d871497 to
9b4e63e
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
2 times, most recently
from
September 12, 2026 22:36
9b4e63e to
5c75027
Compare
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 12, 2026 22:52
5c75027 to
198b65f
Compare
`Indexer::RecordPreparer` read its per-field metadata straight out of the
JSON schema artifacts: it received raw `{"type" => ..., "nameInIndex" =>
...}` hashes and called `eg_meta.fetch("nameInIndex")`. That made the
JSON schema representation part of the indexer's record-preparation
contract, so an ingestion format that has no JSON schemas could only
supply metadata by synthesizing JSON-schema-shaped hashes.
The metadata is now a `RecordPreparer::FieldMetadata` value object with
`type` and `name_in_index`, and `RecordPreparer::TypeMetadata` exposes
`fields_by_name` instead of `eg_meta_by_field_name`. The JSON schema key
names stay inside `elasticgraph-json_ingestion`, which converts them when
it builds its type metadata. Behavior is unchanged.
jwils
force-pushed
the
joshuaw/indexing-field-metadata
branch
from
September 12, 2026 22:54
198b65f to
46ad2bf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Indexer::RecordPreparerread its per-field indexing metadata straight out of the JSON schema artifacts. It received raw hashes shaped like{"type" => "ID!", "nameInIndex" => "id"}and calledeg_meta.fetch("nameInIndex")directly.That made the JSON schema representation part of the indexer's record-preparation contract.
elasticgraph-indexeris meant to stay ingestion-format-neutral, but an ingestion format with no JSON schemas (such as the plannedelasticgraph-proto_ingestionruntime) could only supply metadata by synthesizing JSON-schema-shaped hashes with camelCase keys.What
RecordPreparer::FieldMetadata, a value object withtypeandname_in_index.RecordPreparer::TypeMetadatanow exposesfields_by_name(aHash<String, FieldMetadata>) instead ofeg_meta_by_field_name.RecordPreparerusesfield.name_in_indexandfield.typeinstead of fetching camelCase JSON schema keys.JSONIngestion::RecordPreparerFactoryconverts the JSON schemaElasticGraphmetadata intoFieldMetadata, so the JSON schema key names stay insideelasticgraph-json_ingestion.RecordPreparerdirectly fromTypeMetadataandFieldMetadata, with no JSON schema involved. This is the path a non-JSON ingestion format will use.Behavior is unchanged. This is a pure refactor of the contract between the indexer and the ingestion format gems.
Follow-up
This removes the shape coupling. A later PR still needs to give non-JSON formats a source for this metadata. Today
name_in_indexand the indexing type are recorded only in the versioned JSON schema artifacts.Runtime metadata is not a sufficient substitute. Its
graphql_fields_by_nameentries recordname_in_index, but they carry no fieldtype, whichRecordPreparerneeds to select a scalar indexing preparer and to recurse into nested objects. The entries are also unversioned and describe the GraphQL field set, not the ingestion field set.Verification
elasticgraph-indexer,elasticgraph-json_ingestion,elasticgraph-indexer_lambda,elasticgraph-warehouse_lambda,elasticgraph-lambda_support,elasticgraph-warehouse,elasticgraph-localandelasticgraph-apollounit suites: 673 examples, 0 failuresscript/lint,script/spellcheck,script/type_check: all greenbundle exec rake schema_artifacts:check: artifacts up to dateStack
Current PR is marked with
->.