You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Until now, an EP could report only one weightless mode, and apps could only turn weightless mode on or off. Each mode has different runtime requirements: ALL needs the source model at runtime, EXTERNAL_ONLY doesn't. With this change, EPs report every mode they support and apps choose the one that fits, without breaking EPs and apps built for 1.29 and 1.30.
Motivation and Context
Address the following shortcomings from PR 29607:
After a system update, a plugin EP could switch to a weightless mode whose requirements the app wasn't prepared to meet.
An EP that supports both modes could only advertise one. If it advertised the more demanding one, apps that can't meet those requirements would skip the EP.
Changes
Modes: OrtWeightlessSupport stays a sequential enum (ABI-compatible).
New value OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY = 3 describes an EP capability only; apps can't request it.
Future versions may add values, and apps should ignore values they don't recognize.
EP reporting:
OrtEp::GetWeightlessSupport() keeps its 1.29 signature. EPs return a single mode or OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY. ORT versions before 1.32 only check that the value isn't NONE, so returning 3 is safe for them too.
New EP metadata key weightless_supported_modes, with values "none", "external_only", "all" or "all_or_external_only".
The existing weightless_support key keeps its 1.29 values. ORT and apps fall back to it when the new key is absent.
App request:
New ModelCompilationOptions_SetWeightlessMode(OrtWeightlessSupport) (C++: SetWeightlessMode()) and session option ep.enable_weightless_mode. The option accepts exactly "0", "1" or "2".
Deprecated: ModelCompilationOptions_SetWeightlessEnabled(bool) and ep.enable_weightless. They keep their previous behavior when the new option isn't set.
Validation:
InferenceSession::Initialize() rejects invalid values (INVALID_ARGUMENT) for every API (C, C++, Python), even without plugin EPs or compiled nodes. That includes "3", other spellings like "0x2", and an explicitly empty value.
Before creating each plugin EP, ORT checks the requested mode against the EP metadata (EP_FAIL if it isn't supported). The EP then receives a copy of the session options with ep.enable_weightless set to match ("0" or "1"), so EPs built for earlier versions behave correctly. The app's session options aren't modified.
During Compile(), ORT checks the mode against GetWeightlessSupport(). It logs a warning if that disagrees with the EP metadata, and includes the mismatch in the error when the mode is rejected.
Error codes from plugin EP creation are now preserved instead of becoming ORT_FAIL.
Recorded mode: the chosen mode is written to the compiled model's metadata (weightless_mode). ORT warns when a model compiled with ALL is loaded without a source model path or buffer.
Fix: the weightless source model buffer moved into onnxruntime::SessionOptions. Previously it was lost when session options were copied or cloned, and the session couldn't see it.
Example EP: reports both metadata keys and honors the selected mode, both when compiling and when running directly:
NONE: copies all initializers.
EXTERNAL_ONLY: copies only internal initializers.
ALL: copies none.
Docs: updated header docs and a new design doc, docs/design/Weightless_Mode.md. It covers how EPs report modes, how apps choose one, compatibility with older EPs and apps, and the runtime and packaging requirements for each mode.
Compatibility
GetWeightlessSupport() and its OrtWeightlessSupport* output are unchanged.
weightless_support keeps its values ("none", "external_only", "all").
SetWeightlessEnabled(bool) and ep.enable_weightless still work. When ep.enable_weightless_mode is set, it takes precedence, and ORT translates it for EPs that only read the deprecated option.
Testing
onnxruntime_autoep_test:
Compiling with each mode, including the mode recorded in the compiled model.
How the EP handles initializers for internal, external and mixed models, both compiled and run directly.
Invalid values and their error codes.
Rejection when the EP metadata doesn't support the mode.
ep.enable_weightless being set correctly for older EPs.
The deprecated APIs.
onnxruntime_test_all: PluginExecutionProviderTest.Compile_WeightlessSupportDoesNotMatchEpMetadata checks the mismatch warning.
Python (onnxruntime_test_python.py): invalid values are rejected and the ALL source-model warning is logged on the Python path.
Javier Martinez (javier-intel)
changed the title
Improve the app-EP weightless contract to allow apps to select the be…
Improve the app-EP weightless contract
Sep 29, 2026
Thanks for writing this up. I agree with separating the modes an EP supports from the mode the application selects. For compatibility, I suggest the following approach.
This means the EP supports both selectable modes and honors the application's explicit selection. It is a capability value, not a compilation request. Applications still select exactly NONE (0), EXTERNAL_ONLY (1), or ALL (2).
Keeping the signature avoids the function-pointer type change. Adding a return value still needs compatibility consideration, although the released v1.30.0 ORT implementation checks only for NONE, so it accepts 3 for a legacy boolean request. When no explicit mode is supplied, upgraded EPs must preserve their established legacy behavior.
2. Normalize legacy options and validate the requested mode
When the new mode option is present, ORT should consistently override the legacy enable flag before CreateEp(). This should cover both the compile API and direct session-option configuration.
For an older EP whose reported scope describes its behavior when weightless is enabled:
New request
Older EP's reported scope
Action
NONE
Any
Force ep.enable_weightless = "0"
EXTERNAL_ONLY
EXTERNAL_ONLY
Set legacy option to "1"
ALL
ALL
Set legacy option to "1"
Either enabled mode
Different or unknown scope
Reject the request clearly
For example, an older EP reporting ALL cannot be instructed to choose EXTERNAL_ONLY through its boolean interface alone, unless another documented selector exists. Without a valid mapping, ORT should reject the request rather than claim it was honored.
For an upgraded EP reporting ALL_OR_EXTERNAL_ONLY, either individual mode is accepted, and the EP must honor ep.enable_weightless_mode.
When the new option is absent, preserve the existing legacy setting and behavior. Invalid requests, including 3, should return an invalid-argument error.
3. Preserve the existing metadata key and add a supported-modes key
This adds capability information for newer applications without changing what older applications read. New applications can prefer the new key and fall back to the old key when it is absent. ORT should compare the full callback capability against the new key when present.
When the new mode is explicitly NONE and the deprecated key is absent, GetConfigOrDefault(..., "0") makes this branch skip the copy, so the EP observes the old option as absent rather than the promised explicit "0". This contradicts the compatibility contract and the hook can distinguish the two states. Compare the optional entry itself so an absent key is also populated; add the missing { "0", nullptr, 0 } test case.
This file still declares and accepts std::string_view at lines 36–40, so removing its direct header makes compilation depend on an unrelated transitive include. Retain <string_view> while adding <string>.
Add regression test for copied source-model buffer
The regression this move fixes—retaining the source-model buffer when session options are copied or cloned—is not covered. The existing test only validates setter inputs and never copies the options or reads the buffer afterward. Add a clone/copy regression test that verifies both pointer and size survive, preferably through the EP getter path that previously lost them.
Invalid mode validation is compiled out for minimal builds, and the minimal-build option loop below does not reject this key. A minimal ORT-format session can therefore initialize successfully with ep.enable_weightless_mode="3" or an empty value, contrary to the public option contract that only exact values 0–2 are accepted. Keep ValidateWeightlessMode and its call outside the non-minimal guard; only the source-model warning needs to remain guarded if necessary.
This new API documentation omits the compatibility fallback that the implementation and other public docs require. An application following only this method's contract would miss 1.29–1.31 EPs that expose only weightless_support. State that kOrtEpDevice_EpMetadataKey_WeightlessSupport must be inspected when weightless_supported_modes is absent.
The shared parser also accepts "all_or_external_only" for the legacy weightless_support key, although that key is contractually limited to the 1.29 values. An EP that puts the aggregate value only in the legacy key will therefore pass pre-creation validation and avoid the intended metadata-mismatch warning, while older applications cannot understand its metadata. Treat the aggregate as unrecognized unless it came from weightless_supported_modes.
[P2] Explicit NONE does not disable the oldest weightless flag.
ep_plugin_provider_interfaces.cc:175–183
With ep.context_enable=1 , ep.enable_weightless_ep_context_nodes=1 , and ep.enable_weightless_mode=0 , ORT clears only ep.enable_weightless . An older EP reading the oldest flag can still produce a weightless artifact despite the explicit disable request. Clear the oldest flag for NONE before CreateEp , and cover this with an EP that ignores the new mode key.
[P3] The source-model warning incorrectly treats ALL as always requiring the original ONNX.
inference_session.cc:2756–2761
For the documented ORT-provided strategy ( drop_constant_initializers=false ), ORT retains initializers in the compiled graph and supplies them at runtime. Such an ALL artifact does not inherently need the original model, but the new warning—and public documentation—asserts that it does. Make the warning conditional or qualify it as EP-dependent; distinguish conservative packaging guidance from actual runtime requirements.
[P3] Valid heterogeneous-device support produces a false metadata-mismatch warning.
ep_plugin_provider_interfaces.cc:125–137
Suppose device A supports both modes and device B supports only EXTERNAL_ONLY . An EP instance spanning both can correctly report their common support, EXTERNAL_ONLY . Preparation accepts that request, but the exact per-device equality check warns that A disagrees with the callback. Define the instance-level aggregation rule—normally the intersection when all devices participate—and compare against that rather than requiring identical metadata on every device
The reason will be displayed to describe this comment to others. Learn more.
Dependencies are missed for initializers inside a compiled control-flow node.
The collector still does not traverse subgraphs. If an EP compiles an entire if or Loop, branch/body-local initializers are invisible to this scan. Once the region is replaced by an EPContext node, the final serialized-graph scan cannot recover those hidden dependencies either. The artifact can incorrectly advertise no required source model or external weights.
The reason will be displayed to describe this comment to others. Learn more.
If a partition includes a subgraph and it depends on an initializer the replacement EPContext node that covers the entire partition should cover the initializer as well, either by copying it (removing the dependency) or marking it as an input (maintaining it). Perhaps the actual question is whether a context model can have subgraphs or not, I was under the impression that it cannot.
The reason will be displayed to describe this comment to others. Learn more.
The legacy option ep.enable_weightless=1 is only a boolean and does not explicitly select internal versus external initializers. However, the existing GetWeightlessSupport() callback already allowed an EP to report EXTERNAL_ONLY, and ORT accepted that response—it rejected only NONE.
For such an EP, internal weights may be copied into the compiled binary while external weights remain referenced. Therefore, dropping internal initializer inputs does not necessarily create a dependency on the source model.
With requested_mode.value_or(OrtWeightlessSupport_ALL), the new code assumes every legacy enable request means ALL. This can incorrectly record source_model for an external-only EP. We should use the EP’s reported scope where it unambiguously establishes the legacy behavior, rather than infer ALL solely from the boolean option.
The reason will be displayed to describe this comment to others. Learn more.
An ALL artifact explicitly recording no source-model dependency still warns, while a legacy artifact recording source_model without weightless_mode skips the warning. Prefer the dependency metadata when present, retaining mode-based guidance only for older artifacts without it.
The reason will be displayed to describe this comment to others. Learn more.
nit: Could we track only the initializer count and one example name instead of accumulating all names? The warning only uses empty(), size(), and front(). We could also return that summary from GetFusedNodeConstantInitializers() to avoid building the per-fused-node vectors.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Until now, an EP could report only one weightless mode, and apps could only turn weightless mode on or off. Each mode has different runtime requirements: ALL needs the source model at runtime, EXTERNAL_ONLY doesn't. With this change, EPs report every mode they support and apps choose the one that fits, without breaking EPs and apps built for 1.29 and 1.30.
Motivation and Context
Address the following shortcomings from PR 29607:
Changes
Compatibility
Testing