Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ events = records_by_type.flat_map do |type_name, records|
{
__typename: type_name,
__version: 1,
__json_schema_version: 1
__schema_version: 1
}.merge(record)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def configure_index(index)
"created_at" => "2019-06-02T12:00:00Z",
"__typename" => "Widget",
"__version" => 1,
"__json_schema_version" => 1
"__schema_version" => 1
}
index_name_for_writes = index_definition.index_name_for_writes(record)
derive_index_from_template(record, datastore_core)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,7 @@ def update_enum_values_in(data, json_schema_defs, type_name)
else
props = json_schema_def.fetch("properties")
data.to_h do |field_name, field_value|
unless [:__version, :__typename, :__json_schema_version].include?(field_name)
unless [:__version, :__typename, :__schema_version].include?(field_name)
field_type = props.fetch(word_to_snake_case(field_name.to_s)).fetch("ElasticGraph").fetch("type")[/\w+/]
field_value = update_enum_values_in(field_value, json_schema_defs, field_type)
end
Expand Down
5 changes: 5 additions & 0 deletions elasticgraph-indexer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,3 +89,8 @@ module MyCompany
end
end
```

A decoded event hash may carry a `schema_version` to request a specific schema artifact version. The
key is optional, because an ingestion format may have no versions at all. Each ingestion adapter
decides what a missing version means for its own format. `elasticgraph-json_ingestion` uses the latest
available JSON schema version.
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ def initialize(config:, schema_artifacts:, logger:)
end

# @param payload [String] a raw payload from the transport
# @return [Array<Hash<String, Object>>] the decoded ElasticGraph indexing events
# @return [Array<Hash<String, Object>>] the decoded ElasticGraph indexing events. An event may
# include a `schema_version`, but does not have to: an ingestion format with no versions
# omits it. Each ingestion adapter decides what a missing version means for its own format.
def decode(payload)
# :nocov: -- must return an array to satisfy Steep type checking but never called
[]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ def handles_event?(event)
# Validates the given event and resolves the record preparer appropriate for the event's
# schema version.
#
# The event's `schema_version` is optional, because an ingestion format may have no versions
# at all. Each adapter decides what a missing version means for its own format.
#
# @param event [Hash<String, Object>] an ElasticGraph indexing event
# @return [ValidationResult] the result of validating the event
def validate_event(event)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,12 +120,18 @@ def calculate_latency_metrics(successful_operations, noop_results)

result = successful_events.include?(event) ? "success" : "noop"

# The schema version is optional, since an ingestion format may have no versions at all.
schema_version = event[SCHEMA_VERSION_KEY]

@logger.info({
"message_type" => "ElasticGraphIndexingLatencies",
"message_id" => event["message_id"],
"event_type" => event.fetch("type"),
"event_id" => EventID.from_event(event).to_s,
JSON_SCHEMA_VERSION_KEY => event.fetch(JSON_SCHEMA_VERSION_KEY),
SCHEMA_VERSION_KEY => schema_version,
# Deprecated alias of `schema_version`, kept so that dashboards and monitors that watch
# the old name keep working.
JSON_SCHEMA_VERSION_KEY => schema_version,
"latencies_in_ms_from" => latencies_in_ms_from,
"slo_results" => slo_results,
"result" => result
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,27 @@ module ElasticGraph
class Indexer
module TestSupport
module Converters
# Attributes that describe the event rather than the record, so they never reach the record.
# `__json_schema_version` is the legacy name of `__schema_version`; projects generated before
# the schema version became ingestion-format-neutral still use it.
EVENT_ONLY_ATTRIBUTES = ["__typename", "__version", "__schema_version", "__json_schema_version"]

# Helper method for testing and generating fake data to convert a factory record into an event
def self.upsert_event_for(record)
{
event = {
"op" => "upsert",
"id" => record.fetch("id"),
"type" => record.fetch("__typename"),
"version" => record.fetch("__version"),
"record" => record.except("__typename", "__version", "__json_schema_version"),
JSON_SCHEMA_VERSION_KEY => record.fetch("__json_schema_version")
"record" => record.except(*EVENT_ONLY_ATTRIBUTES)
}

# The schema version is optional, so include it only when the factory supplies one.
if (schema_version = record["__schema_version"] || record["__json_schema_version"])
event[SCHEMA_VERSION_KEY] = schema_version
end

event
end

# Helper method to create an array of events given an array of records
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ module ElasticGraph
class Indexer
module TestSupport
module Converters
EVENT_ONLY_ATTRIBUTES: ::Array[::String]

def self.upsert_event_for: (::Hash[::String, untyped]) -> ::Hash[::String, untyped]

def self.upsert_events_for_records: (
Expand Down
20 changes: 10 additions & 10 deletions elasticgraph-indexer/spec/acceptance/schema_evolution_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ def build_address_event_without_geolocation
end

def build_widget(json_schema_version:)
event = build_upsert_event(:widget, __json_schema_version: json_schema_version)
event = build_upsert_event(:widget, __schema_version: json_schema_version)
event.merge("record" => (yield event.fetch("record")))
end
end
Expand All @@ -116,7 +116,7 @@ def build_widget(json_schema_version:)
write_address_schema_def(json_schema_version: 2, address_extras: "t.deleted_field 'deprecated'")
dump_artifacts

event = build_upsert_event(:address, id: "abc", deprecated: "foo", __json_schema_version: 1)
event = build_upsert_event(:address, id: "abc", deprecated: "foo", __schema_version: 1)
expect(event.dig("record", "deprecated")).to eq("foo")

boot_indexer.processor.process([event], refresh_indices: true)
Expand Down Expand Up @@ -162,8 +162,8 @@ def get_address_payload(id)
# included at that part of the JSON schema. So here we verify that the factory includes that.
expect(build(:team_season)).to include(__typename: "TeamSeason")

v1_event = build_upsert_event(:team, __json_schema_version: 1)
v2_event = build_upsert_event(:team, __json_schema_version: 2)
v1_event = build_upsert_event(:team, __schema_version: 1)
v2_event = build_upsert_event(:team, __schema_version: 2)
.then { |event| ::JSON.generate(event) }
# Fix the event to align with the v2 schema, since `build_upsert_event` doesn't automatically
# know that the `__typename` should be `SeasonOfATeam` instead of `TeamSeason`.
Expand Down Expand Up @@ -200,8 +200,8 @@ def get_address_payload(id)
end
dump_artifacts

v1_event = build_upsert_event(:team, __json_schema_version: 1)
v2_event = build_upsert_event(:team, __json_schema_version: 2)
v1_event = build_upsert_event(:team, __schema_version: 1)
v2_event = build_upsert_event(:team, __schema_version: 2)

expect {
boot_indexer.processor.process([v1_event, v2_event], refresh_indices: true)
Expand Down Expand Up @@ -244,9 +244,9 @@ def get_address_payload(id)
end
dump_artifacts

v1_event = build_upsert_event(:team, __json_schema_version: 1)
v1_event = build_upsert_event(:team, __schema_version: 1)
v1_event = ::JSON.parse(::JSON.generate(v1_event).gsub('"name":', '"full_name":'))
v2_event = build_upsert_event(:team, __json_schema_version: 2)
v2_event = build_upsert_event(:team, __schema_version: 2)

expect {
boot_indexer.processor.process([v1_event, v2_event], refresh_indices: true)
Expand Down Expand Up @@ -288,7 +288,7 @@ def get_address_payload(id)
end
dump_artifacts

v1_event = build_upsert_event(:team, __json_schema_version: 1)
v1_event = build_upsert_event(:team, __schema_version: 1)

expect {
boot_indexer.processor.process([v1_event], refresh_indices: true)
Expand Down Expand Up @@ -323,7 +323,7 @@ def get_address_payload(id)
write_address_schema_def(json_schema_version: 2, schema_extras: 'schema.deleted_type "Team"')
dump_artifacts

v1_event = build_upsert_event(:team, __json_schema_version: 1)
v1_event = build_upsert_event(:team, __schema_version: 1)
boot_indexer.processor.process([v1_event], refresh_indices: true)

expect(search_for_ids("teams")).to be_empty
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ module Operation
"type" => "Widget",
"version" => 1,
"record" => event["record"],
JSON_SCHEMA_VERSION_KEY => 1
SCHEMA_VERSION_KEY => 1
}

expect(build_expecting_success(event)).to contain_exactly(
Expand Down Expand Up @@ -86,7 +86,7 @@ module Operation
"type" => "Component",
"version" => 1,
"record" => event["record"],
JSON_SCHEMA_VERSION_KEY => 1
SCHEMA_VERSION_KEY => 1
}.merge(latency_timestamps))])
end

Expand All @@ -96,7 +96,7 @@ module Operation
"id" => "1",
"type" => "MyOwnInvalidGraphQlType",
"version" => 1,
JSON_SCHEMA_VERSION_KEY => 1,
SCHEMA_VERSION_KEY => 1,
"record" => {"field1" => "value1", "field2" => "value2", "id" => "1"}
}

Expand All @@ -110,7 +110,7 @@ module Operation
"id" => "1",
"type" => "WidgetOptions",
"version" => 1,
JSON_SCHEMA_VERSION_KEY => 1,
SCHEMA_VERSION_KEY => 1,
"record" => {"field1" => "value1", "field2" => "value2", "id" => "1"}
}

Expand All @@ -127,17 +127,17 @@ module Operation
expect_failed_event_error(event, "missing_keys", "type", expect_no_ops: true)
end

it "notifies an error on missing `#{JSON_SCHEMA_VERSION_KEY}`" do
event = build_upsert_event(:component).except(JSON_SCHEMA_VERSION_KEY)
it "builds operations for an event that carries no `#{SCHEMA_VERSION_KEY}`, since the key is optional" do
event = build_upsert_event(:component).except(SCHEMA_VERSION_KEY)

expect_failed_event_error(event, JSON_SCHEMA_VERSION_KEY)
expect(build_expecting_success(event)).not_to be_empty
end

it "notifies an error on wrong field types" do
event = {
"op" => "upsert",
"id" => 1,
JSON_SCHEMA_VERSION_KEY => 1,
SCHEMA_VERSION_KEY => 1,
"type" => [],
"version" => "1",
"record" => ""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,20 @@ class Indexer

expect(logged_jsons_of_type("ElasticGraphIndexingLatencies").first).to include(
"event_id" => "Component:#{component.fetch("id")}@v#{component.fetch("version")}",
"message_id" => "m1"
"message_id" => "m1",
SCHEMA_VERSION_KEY => component.fetch(SCHEMA_VERSION_KEY),
# Deprecated alias, kept for existing dashboards and monitors.
JSON_SCHEMA_VERSION_KEY => component.fetch(SCHEMA_VERSION_KEY)
)
end

it "logs a nil schema version for an event that carries none, since the key is optional" do
component = upsert_event_with_latency_timestamps(:component, 36, 72).except(SCHEMA_VERSION_KEY)
process([component])

expect(logged_jsons_of_type("ElasticGraphIndexingLatencies").first).to include(
SCHEMA_VERSION_KEY => nil,
JSON_SCHEMA_VERSION_KEY => nil
)
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ module TestSupport
"id" => "1",
"__version" => 1,
"__typename" => "Widget",
"__json_schema_version" => 1,
"__schema_version" => 1,
"field1" => "value1",
"field2" => "value2"
}
Expand All @@ -30,7 +30,43 @@ module TestSupport
"version" => 1,
"type" => "Widget",
"record" => {"id" => "1", "field1" => "value1", "field2" => "value2"},
JSON_SCHEMA_VERSION_KEY => 1
SCHEMA_VERSION_KEY => 1
)
end

it "accepts the legacy `__json_schema_version` attribute, so factories from older projects keep working" do
factory_record = {
"id" => "1",
"__version" => 1,
"__typename" => "Widget",
"__json_schema_version" => 3,
"field1" => "value1"
}

expect(TestSupport::Converters.upsert_event_for(factory_record)).to eq(
"op" => "upsert",
"id" => "1",
"version" => 1,
"type" => "Widget",
"record" => {"id" => "1", "field1" => "value1"},
SCHEMA_VERSION_KEY => 3
)
end

it "omits the schema version when the factory record supplies none, since the key is optional" do
factory_record = {
"id" => "1",
"__version" => 1,
"__typename" => "Widget",
"field1" => "value1"
}

expect(TestSupport::Converters.upsert_event_for(factory_record)).to eq(
"op" => "upsert",
"id" => "1",
"version" => 1,
"type" => "Widget",
"record" => {"id" => "1", "field1" => "value1"}
)
end
end
Expand All @@ -41,7 +77,7 @@ module TestSupport
"id" => "1",
"__typename" => "Widget",
"__version" => 1,
"__json_schema_version" => 1,
"__schema_version" => 1,
"field1" => "value1",
"field2" => "value2"
}
Expand All @@ -50,7 +86,7 @@ module TestSupport
"id" => "2",
"__typename" => "Address",
"__version" => 5,
"__json_schema_version" => 1,
"__schema_version" => 1,
"field3" => "value5"
}

Expand All @@ -63,15 +99,15 @@ module TestSupport
"version" => 1,
"type" => "Widget",
"record" => {"id" => "1", "field1" => "value1", "field2" => "value2"},
JSON_SCHEMA_VERSION_KEY => 1
SCHEMA_VERSION_KEY => 1
},
{
"op" => "upsert",
"id" => "2",
"version" => 5,
"type" => "Address",
"record" => {"id" => "2", "field3" => "value5"},
JSON_SCHEMA_VERSION_KEY => 1
SCHEMA_VERSION_KEY => 1
}
])
end
Expand Down
13 changes: 12 additions & 1 deletion elasticgraph-json_ingestion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,22 @@ end

Beyond schema definition, this gem teaches `elasticgraph-indexer` how to ingest JSON events: it provides an
ingestion adapter that validates each event against the JSON schema identified by the event's
`json_schema_version` and prepares its record for indexing using that version's view of the schema.
`schema_version` and prepares its record for indexing using that version's view of the schema.

No configuration is needed: defining your schema with this gem's `SchemaDefinition::APIExtension` registers
an indexer extension in your schema artifacts' runtime metadata, which the indexer applies when it boots.

### Schema versions

The adapter resolves the version of each event as follows:

- The `schema_version` key selects the JSON schema version. When the exact version is unavailable, the
adapter selects the closest available version and logs `ElasticGraphMissingJSONSchemaVersion`.
- The legacy `json_schema_version` key still works, so a publisher or an in-process caller that predates
the ingestion-format-neutral key needs no change.
- An event that carries neither key gets the latest available JSON schema version. The adapter still
validates the event against that version, so a malformed event still fails.

This gem also provides the `be_a_valid_elastic_graph_event` RSpec matcher (via
`require "elastic_graph/json_ingestion/spec_support/event_matcher"`) for testing that publisher events
conform to your schema.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,13 @@ module JSONIngestion
module IndexerExtension
# Adds the JSON {IngestionAdapter} to the indexer's available ingestion adapters.
#
# The memo uses a name specific to this module. Every extension module in the chain assigns
# while it calls `super`, so a shared name would make the result depend on the order of
# evaluation.
#
# @return [Array<Object>] the available ingestion adapters
def ingestion_adapters
@ingestion_adapters ||= super + [IngestionAdapter.new(schema_artifacts: schema_artifacts, logger: logger)]
@json_ingestion_adapters ||= super + [IngestionAdapter.new(schema_artifacts: schema_artifacts, logger: logger)]
end
end
end
Expand Down
Loading