Skip to content

Improve the app-EP weightless contract - #32899

Open
Javier Martinez (javier-intel) wants to merge 10 commits into
microsoft:mainfrom
javier-intel:tighter_weightless_contract
Open

Javier Martinez (javier-intel) wants to merge 10 commits into
microsoft:mainfrom
javier-intel:tighter_weightless_contract

Conversation

@javier-intel

@javier-intel Javier Martinez (javier-intel) commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

Until now, an EP could report only one weightless mode, and apps could only turn weightless mode on or off. Each mode has different runtime requirements: ALL needs the source model at runtime, EXTERNAL_ONLY doesn't. With this change, EPs report every mode they support and apps choose the one that fits, without breaking EPs and apps built for 1.29 and 1.30.

Motivation and Context

Address the following shortcomings from PR 29607:

  • After a system update, a plugin EP could switch to a weightless mode whose requirements the app wasn't prepared to meet.
  • An EP that supports both modes could only advertise one. If it advertised the more demanding one, apps that can't meet those requirements would skip the EP.

Changes

  • Modes: OrtWeightlessSupport stays a sequential enum (ABI-compatible).
    • New value OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY = 3 describes an EP capability only; apps can't request it.
    • Future versions may add values, and apps should ignore values they don't recognize.
  • EP reporting:
    • OrtEp::GetWeightlessSupport() keeps its 1.29 signature. EPs return a single mode or OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY. ORT versions before 1.32 only check that the value isn't NONE, so returning 3 is safe for them too.
    • New EP metadata key weightless_supported_modes, with values "none", "external_only", "all" or "all_or_external_only".
    • The existing weightless_support key keeps its 1.29 values. ORT and apps fall back to it when the new key is absent.
  • App request:
    • New ModelCompilationOptions_SetWeightlessMode(OrtWeightlessSupport) (C++: SetWeightlessMode()) and session option ep.enable_weightless_mode. The option accepts exactly "0", "1" or "2".
    • Deprecated: ModelCompilationOptions_SetWeightlessEnabled(bool) and ep.enable_weightless. They keep their previous behavior when the new option isn't set.
  • Validation:
    • InferenceSession::Initialize() rejects invalid values (INVALID_ARGUMENT) for every API (C, C++, Python), even without plugin EPs or compiled nodes. That includes "3", other spellings like "0x2", and an explicitly empty value.
    • Before creating each plugin EP, ORT checks the requested mode against the EP metadata (EP_FAIL if it isn't supported). The EP then receives a copy of the session options with ep.enable_weightless set to match ("0" or "1"), so EPs built for earlier versions behave correctly. The app's session options aren't modified.
    • During Compile(), ORT checks the mode against GetWeightlessSupport(). It logs a warning if that disagrees with the EP metadata, and includes the mismatch in the error when the mode is rejected.
    • Error codes from plugin EP creation are now preserved instead of becoming ORT_FAIL.
  • Recorded mode: the chosen mode is written to the compiled model's metadata (weightless_mode). ORT warns when a model compiled with ALL is loaded without a source model path or buffer.
  • Fix: the weightless source model buffer moved into onnxruntime::SessionOptions. Previously it was lost when session options were copied or cloned, and the session couldn't see it.
  • Example EP: reports both metadata keys and honors the selected mode, both when compiling and when running directly:
    • NONE: copies all initializers.
    • EXTERNAL_ONLY: copies only internal initializers.
    • ALL: copies none.
  • Docs: updated header docs and a new design doc, docs/design/Weightless_Mode.md. It covers how EPs report modes, how apps choose one, compatibility with older EPs and apps, and the runtime and packaging requirements for each mode.

Compatibility

  • GetWeightlessSupport() and its OrtWeightlessSupport* output are unchanged.
  • weightless_support keeps its values ("none", "external_only", "all").
  • SetWeightlessEnabled(bool) and ep.enable_weightless still work. When ep.enable_weightless_mode is set, it takes precedence, and ORT translates it for EPs that only read the deprecated option.

Testing

  • onnxruntime_autoep_test:
    • Compiling with each mode, including the mode recorded in the compiled model.
    • How the EP handles initializers for internal, external and mixed models, both compiled and run directly.
    • Invalid values and their error codes.
    • Rejection when the EP metadata doesn't support the mode.
    • ep.enable_weightless being set correctly for older EPs.
    • The deprecated APIs.
  • onnxruntime_test_all: PluginExecutionProviderTest.Compile_WeightlessSupportDoesNotMatchEpMetadata checks the mismatch warning.
  • Python (onnxruntime_test_python.py): invalid values are rejected and the ALL source-model warning is logged on the Python path.

@javier-intel Javier Martinez (javier-intel) changed the title Improve the app-EP weightless contract to allow apps to select the be… Improve the app-EP weightless contract Sep 29, 2026
@javier-intel
Javier Martinez (javier-intel) marked this pull request as ready for review October 5, 2026 23:02
Copilot AI balanced review requested due to automatic review settings October 5, 2026 23:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Public callback compatibility and reliable enforcement of selected modes remain unresolved.

Review effort: Balanced
Findings: 1 High severity · 4 Medium severity

Open (5)
What changed in this PR

Extends ONNX Runtime’s weightless compilation contract so applications can discover supported modes and select their runtime requirements.

Changes:

  • Adds explicit mode selection through C/C++ APIs and session options.
  • Validates provider capabilities and records the selected mode in compiled models.
  • Updates contract documentation, example-provider reporting, and tests.
File Description
onnxruntime/​test/​autoep/​test_registration.cc Checks bitmask metadata.
onnxruntime/​test/​autoep/​test_execution.cc Tests selection, validation, and recorded metadata.
onnxruntime/​test/​autoep/​library/​example_plugin_ep/​ep.h Adds configurable capability flags.
onnxruntime/​test/​autoep/​library/​example_plugin_ep/​ep.cc Returns capability flags.
onnxruntime/​test/​autoep/​library/​example_plugin_ep/​ep_factory.cc Advertises modes and supports mismatch testing.
onnxruntime/​core/​session/​utils.cc Warns about missing source models.
onnxruntime/​core/​session/​plugin_ep/​ep_plugin_provider_interfaces.h Stores requested mode settings.
onnxruntime/​core/​session/​plugin_ep/​ep_plugin_provider_interfaces.cc Checks capabilities and metadata consistency.
onnxruntime/​core/​session/​model_compilation_options.h Declares mode selection.
onnxruntime/​core/​session/​model_compilation_options.cc Validates and stores selected modes.
onnxruntime/​core/​session/​compile_api.h Declares the C API implementation.
onnxruntime/​core/​session/​compile_api.cc Implements and registers the new API.
onnxruntime/​core/​framework/​graph_partitioner.cc Records compiled-model mode metadata.
onnxruntime/​core/​framework/​ep_context_options.h Declares the mode parser.
onnxruntime/​core/​framework/​ep_context_options.cc Parses mode values.
include/​onnxruntime/​core/​session/​onnxruntime_session_options_config_keys.h Documents selection and runtime options.
include/​onnxruntime/​core/​session/​onnxruntime_ep_device_ep_metadata_keys.h Defines capability and recorded-mode metadata.
include/​onnxruntime/​core/​session/​onnxruntime_ep_c_api.h Changes capability reporting to a bitmask.
include/​onnxruntime/​core/​session/​onnxruntime_cxx_inline.h Implements the C++ wrapper.
include/​onnxruntime/​core/​session/​onnxruntime_cxx_api.h Exposes C++ mode selection.
include/​onnxruntime/​core/​session/​onnxruntime_c_api.h Defines mode flags and the selection API.
docs/​design/​Weightless_Mode.md Explains negotiation and deployment requirements.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread include/onnxruntime/core/session/onnxruntime_ep_c_api.h Outdated
Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated
Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated
Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated
Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc Outdated
@chilo-ms

Copy link
Copy Markdown
Contributor

Thanks for writing this up. I agree with separating the modes an EP supports from the mode the application selects. For compatibility, I suggest the following approach.

1. Keep the existing callback signature

Preserve:

ORT_API2_STATUS(GetWeightlessSupport,
                _In_ const OrtEp* this_ptr,
                _Out_ OrtWeightlessSupport* support);

Add:

OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY = 3

This means the EP supports both selectable modes and honors the application's explicit selection. It is a capability value, not a compilation request. Applications still select exactly NONE (0), EXTERNAL_ONLY (1), or ALL (2).

Keeping the signature avoids the function-pointer type change. Adding a return value still needs compatibility consideration, although the released v1.30.0 ORT implementation checks only for NONE, so it accepts 3 for a legacy boolean request. When no explicit mode is supplied, upgraded EPs must preserve their established legacy behavior.

2. Normalize legacy options and validate the requested mode

When the new mode option is present, ORT should consistently override the legacy enable flag before CreateEp(). This should cover both the compile API and direct session-option configuration.

For an older EP whose reported scope describes its behavior when weightless is enabled:

New request Older EP's reported scope Action
NONE Any Force ep.enable_weightless = "0"
EXTERNAL_ONLY EXTERNAL_ONLY Set legacy option to "1"
ALL ALL Set legacy option to "1"
Either enabled mode Different or unknown scope Reject the request clearly

For example, an older EP reporting ALL cannot be instructed to choose EXTERNAL_ONLY through its boolean interface alone, unless another documented selector exists. Without a valid mapping, ORT should reject the request rather than claim it was honored.

For an upgraded EP reporting ALL_OR_EXTERNAL_ONLY, either individual mode is accepted, and the EP must honor ep.enable_weightless_mode.

When the new option is absent, preserve the existing legacy setting and behavior. Invalid requests, including 3, should return an invalid-argument error.

3. Preserve the existing metadata key and add a supported-modes key

Keep:

weightless_support = "none" | "external_only" | "all"

Add:

weightless_supported_modes = "none"
                          | "external_only"
                          | "all"
                          | "all_or_external_only"

Preserve the existing weightless_support value for each device, and expose the full supported-mode set through the new weightless_supported_modes key.

For an upgraded EP supporting both modes that previously advertised "all":

weightless_support         = "all"
weightless_supported_modes = "all_or_external_only"

If it previously advertised "external_only", retain that value instead:

weightless_support         = "external_only"
weightless_supported_modes = "all_or_external_only"

This adds capability information for newer applications without changing what older applications read. New applications can prefer the new key and fall back to the old key when it is absent. ORT should compare the full callback capability against the new key when present.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread onnxruntime/core/framework/ep_context_options.cc Outdated
Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc Outdated
Comment thread onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc
Comment thread onnxruntime/core/session/utils.cc Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread onnxruntime/test/autoep/test_execution.cc Outdated
Comment thread docs/design/Weightless_Mode.md
Comment thread include/onnxruntime/core/session/onnxruntime_c_api.h Outdated
Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep.h Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread include/onnxruntime/core/session/onnxruntime_c_api.h Outdated
Comment thread include/onnxruntime/core/session/onnxruntime_c_api.h
Comment thread include/onnxruntime/core/session/onnxruntime_ep_device_ep_metadata_keys.h Outdated
Comment thread include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Explicit NONE translation, C API release bookkeeping, header dependencies, and copy-regression coverage remain unresolved.

2 open findings
4 resolved since last review
Previously missed (3)

In code that hasn't changed since last review

Medium severity Populate explicit NONE option when deprecated key is absent

onnxruntime/​core/​session/​plugin_ep/​ep_plugin_provider_interfaces.cc:178

When the new mode is explicitly NONE and the deprecated key is absent, GetConfigOrDefault(..., "0") makes this branch skip the copy, so the EP observes the old option as absent rather than the promised explicit "0". This contradicts the compatibility contract and the hook can distinguish the two states. Compare the optional entry itself so an absent key is also populated; add the missing { "0", nullptr, 0 } test case.

Medium severity Retain direct string_view header

onnxruntime/​test/​autoep/​library/​example_plugin_ep/​ep_factory.cc:9

This file still declares and accepts std::string_view at lines 36–40, so removing its direct header makes compilation depend on an unrelated transitive include. Retain <string_view> while adding <string>.

Low severity Add regression test for copied source-model buffer

onnxruntime/​core/​framework/​session_options.h:236

The regression this move fixes—retaining the source-model buffer when session options are copied or cloned—is not covered. The existing test only validates setter inputs and never copies the options or reads the buffer afterward. Add a clone/copy regression test that verifies both pointer and size survive, preferably through the EP getter path that previously lost them.

🧠 Review effort: Balanced

Comment thread onnxruntime/core/session/compile_api.cc Outdated
Comment thread onnxruntime/core/session/compile_api.cc Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The example plugin test library currently fails to compile, and minimal builds bypass the documented mode validation.

1 open finding
2 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Validate weightless mode in minimal builds

onnxruntime/​core/​session/​inference_session.cc:2810

Invalid mode validation is compiled out for minimal builds, and the minimal-build option loop below does not reject this key. A minimal ORT-format session can therefore initialize successfully with ep.enable_weightless_mode="3" or an empty value, contrary to the public option contract that only exact values 0–2 are accepted. Keep ValidateWeightlessMode and its call outside the non-minimal guard; only the source-model warning needs to remain guarded if necessary.

Low severity Document weightless_support compatibility fallback

include/​onnxruntime/​core/​session/​onnxruntime_c_api.h:8699

This new API documentation omits the compatibility fallback that the implementation and other public docs require. An application following only this method's contract would miss 1.29–1.31 EPs that expose only weightless_support. State that kOrtEpDevice_EpMetadataKey_WeightlessSupport must be inspected when weightless_supported_modes is absent.

🧠 Review effort: Balanced

Comment thread onnxruntime/test/autoep/library/example_plugin_ep/ep_test_hooks.h

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Legacy metadata currently accepts the new aggregate value, bypassing the intended compatibility and mismatch checks.

0 open findings

1 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity Reject aggregate mode in legacy weightless_support key

onnxruntime/​core/​session/​plugin_ep/​ep_plugin_provider_interfaces.cc:111

The shared parser also accepts "all_or_external_only" for the legacy weightless_support key, although that key is contractually limited to the 1.29 values. An EP that puts the aggregate value only in the legacy key will therefore pass pre-creation validation and avoid the intended metadata-mismatch warning, while older applications cannot understand its metadata. Treat the aggregate as unrecognized unless it came from weightless_supported_modes.

🧠 Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes a public C/EP compatibility contract across ABI-sensitive APIs, runtime validation, serialization, and plugin behavior.

0 open findings

🧠 Review effort: Balanced

@chilo-ms

Copy link
Copy Markdown
Contributor

Found three issues.

  1. [P2] Explicit  NONE  does not disable the oldest weightless flag.
    ep_plugin_provider_interfaces.cc:175–183
    With  ep.context_enable=1 ,  ep.enable_weightless_ep_context_nodes=1 , and  ep.enable_weightless_mode=0 , ORT clears only  ep.enable_weightless . An older EP reading the oldest flag can still produce a weightless artifact despite the explicit disable request. Clear the oldest flag for  NONE  before  CreateEp , and cover this with an EP that ignores the new mode key.
  2. [P3] The source-model warning incorrectly treats  ALL  as always requiring the original ONNX.
    inference_session.cc:2756–2761
    For the documented ORT-provided strategy ( drop_constant_initializers=false ), ORT retains initializers in the compiled graph and supplies them at runtime. Such an  ALL  artifact does not inherently need the original model, but the new warning—and public documentation—asserts that it does. Make the warning conditional or qualify it as EP-dependent; distinguish conservative packaging guidance from actual runtime requirements.
  3. [P3] Valid heterogeneous-device support produces a false metadata-mismatch warning.
    ep_plugin_provider_interfaces.cc:125–137
    Suppose device A supports both modes and device B supports only  EXTERNAL_ONLY . An EP instance spanning both can correctly report their common support,  EXTERNAL_ONLY . Preparation accepts that request, but the exact per-device equality check warns that A disagrees with the callback. Define the instance-level aggregation rule—normally the intersection when all devices participate—and compare against that rather than requiring identical metadata on every device

# and needs the source model at runtime. ORT logs a warning when such a model is loaded without
# "ep.context_source_model_path" or a source model buffer. The warning is written to the native stderr, so the
# session is created in a subprocess.
import onnx # noqa: PLC0415
}
};

for (const Node& node : graph_viewer.Nodes()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dependencies are missed for initializers inside a compiled control-flow node.

The collector still does not traverse subgraphs. If an EP compiles an entire if or Loop, branch/body-local initializers are invisible to this scan. Once the region is replaced by an EPContext node, the final serialized-graph scan cannot recover those hidden dependencies either. The artifact can incorrectly advertise no required source model or external weights.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a partition includes a subgraph and it depends on an initializer the replacement EPContext node that covers the entire partition should cover the initializer as well, either by copying it (removing the dependency) or marking it as an input (maintaining it). Perhaps the actual question is whether a context model can have subgraphs or not, I was under the impression that it cannot.

if (generate_ep_ctx_model_ && weightless_requested) {
// The deprecated option does not select a mode; assume weightless mode for all initializers, as in earlier
// versions.
const OrtWeightlessSupport mode = requested_mode.value_or(OrtWeightlessSupport_ALL);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The legacy option ep.enable_weightless=1  is only a boolean and does not explicitly select internal versus external initializers. However, the existing  GetWeightlessSupport() callback already allowed an EP to report EXTERNAL_ONLY, and ORT accepted that response—it rejected only NONE.

For such an EP, internal weights may be copied into the compiled binary while external weights remain referenced. Therefore, dropping internal initializer inputs does not necessarily create a dependency on the source model.

With requested_mode.value_or(OrtWeightlessSupport_ALL), the new code assumes every legacy enable request means ALL. This can incorrectly record source_model for an external-only EP. We should use the EP’s reported scope where it unambiguously establishes the legacy behavior, rather than infer ALL solely from the boolean option.

// initializers get them from the compiled model. Such EPs may also still locate the source model via the
// "onnx_model_filename" EPContext node attribute, so this is not an error.
static void CheckWeightlessSourceModel(const SessionOptions& session_options, const Model& model,
const logging::Logger& logger) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An ALL artifact explicitly recording no source-model dependency still warns, while a legacy artifact recording source_model without weightless_mode skips the warning. Prefer the dependency metadata when present, retaining mode-based guidance only for older artifacts without it.

// The deprecated option does not select a mode; assume weightless mode for all initializers, as in earlier
// versions.
const OrtWeightlessSupport mode = requested_mode.value_or(OrtWeightlessSupport_ALL);
std::vector<std::string> copied_internal_initializers;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Could we track only the initializer count and one example name instead of accumulating all names? The warning only uses empty(), size(), and front(). We could also return that summary from GetFusedNodeConstantInitializers() to avoid building the per-fused-node vectors.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants