Add an experimental automatic retrieval path

[jwils]

Why

ElasticGraph should choose how to retrieve fields without asking schema authors to select datastore APIs. Narrow scalar queries may benefit from avoiding source retrieval.

What

Add an opt-in automatic retrieval experiment and benchmarks for both datastore APIs and complete GraphQL execution. Source retrieval remains the default.

How

Set graphql.experimental_field_retrieval: automatic after regenerating schema artifacts and verifying existing index mappings and data. The planner uses doc values only when every requested payload field is an eligible direct scalar. It preserves source fallback for mixed selections, lists, objects, highlighting, and unsupported mappings. Returnable data stays stored in _source.

A consuming project can compare both modes using its settings, queries, and variables. The GraphQL harness checks complete results and records which retrieval path ran. The standalone script compares _source, docvalue_fields, fields, and stored_fields without requiring a candidate library installation.

Risk

The default retrieval behavior is preserved, but query planning and response normalization are shared code. Automatic mode is experimental: current artifacts cannot verify historical physical mappings or data. Small-document or wide-selection queries may regress. The 100-field cap reflects the default datastore limit, not a measured performance cutoff.

Testing

Ran the standalone matrix twice on Elasticsearch 9.0.0. For 64 KiB source padding and 100 hits, source-free doc values reduced median client time by 12–22%. Mixed retrieval ranged from about 1% faster to 5% slower.

Also ran graphql_benchmark.rb twice against the repository's Widget schema, with 2,000 large-source documents, 100 hits, 20 warmups, and 100 samples per mode. Scalar-query median GraphQL execution time fell from 14.322 to 10.935 ms and from 13.511 to 10.378 ms (23–24%). Mixed queries used source in both modes and varied by roughly 1%. Complete results matched on every execution.

Usage and commands · Results, fixture details, and limitations

Bigger picture

Validate supported OpenSearch releases, representative consuming projects, and concurrent load before considering a default automatic policy. The exploratory OpenSearch snapshot results cover the standalone benchmark only. The design explains the roles of all four retrieval APIs.

Stack Info: Originally stacked on #1108 (merged); now targets main.

[myronmarston]

retrieved_from: :doc_values exposes a level of detail that, until now, we haven't exposed to ElasticGraph users. ElasticGraph does many things to query the data as efficiently as possible, without asking users to configure those things.

Can we do something similar here? For example, can we determine in which situations it's more efficient to use doc_values and automatically do so without requiring users to configure it at the field level?

Also, if we're going to change how fields are retrieved, I'd like to take a step back and review a full design that takes into account all the options Elasticsearch/OpenSearch offer us here. Besides _source and docvalue_fields there's also fields and stored_fields. I'd like to consider a unified design rather than just tacking this on as a one-off.