Add RFC-0011: Agent Registry - #39
Conversation
First draft covering the introductory material and user journeys: record-level registry for AI agents with Agent/AgentVersion entities, composition (BOM) with definitional anchors, A2A card as canonical-when-present payload, protocol-typed access bindings, and agent-centric traces and evaluations via a new trace destination. Detailed design and later sections are TBD. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plugins are referenced as composed units and expand through their RFC-0008-recorded members for cross-registry queries, so blast-radius lookups find agents that consume a skill via a plugin. Previously held as an open question; resolved by team direction and confirmed by the registry UI prototype's "Link plugin" flow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Source provenance now covers Git repos, OCI images, and archives via the typed source pointers RFC-0008 defines, matching how agents are actually distributed, instead of Git-only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agent versions are registry-minted serial numbers, per the policy from the earlier registries: adopt the artifact's inherent version when its format defines one, mint serial numbers when it does not. No standard agent artifact defines one; an A2A card's provider-defined version string is preserved as metadata, not adopted as identity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A version's source provenance is now a list of typed pointers, so an
agent built from a Git repo and shipped as an OCI image records both,
and source entries join the queryable axes ("which agents ship image
X?"). Deliberate divergence from RFC-0008's one-source-per-version
rule, with the rationale (agents have no content digest to reconcile
per-source duplicates) stated in Open questions.
Prompted by Ann Marie's review question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An Agent chooses its version scheme at creation: monotonic (registry-assigned serial numbers, the default when no version is supplied), semver, or freeform, with possible autodetection from the first registration. Keeps the simple path as simple as the Skill Registry while letting provider-versioned agents keep their own versioning. Proposed by Matthew Prahl in review discussion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agent's endpoint is the card's system of record: registration imports descriptive metadata (card name seeds display_name) and creates the a2a binding, the UI renders the card read-only by fetching through the binding, and no card payload is persisted. This deliberately departs from the RFC-0004/0008 canonical-payload pattern. Endpoint-only registrations become interface-only records with no definitional anchor, marked as such. The canonicity open question is replaced by whether registration and lifecycle events should capture card copies as audit evidence. Direction from design-doc review comments, confirmed by Bill. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both evaluate sketches now pass agent_version alongside agent_id; version-to-version eval comparison (J3) needs the version captured at evaluation time. From design-doc review feedback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The case for capturing card copies as decision evidence was weak: no customer signal, promotion decisions rest on eval evidence, and registration already imports the card's claim surface into registry fields. The open question is now solely about how thin an interface-only record may be. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two sentences in the versioning journey: the choice is the same judgment developers make for any software, the registry enforces no rule, and the consequence is that version comparison exists only within one agent. From design-doc review discussion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| destination today, and evaluation and trace-search APIs gain agent | ||
| identity alongside experiment identity. Traces and eval results | ||
| appear on the agent's registry page, filterable by version. The | ||
| change is additive: under the hood an agent owns a default |
There was a problem hiding this comment.
I believe you want a default experiment per agent-version rather than a default experiment per agent.
There was a problem hiding this comment.
This is now an explicit open question in the draft ("Is the default experiment per agent or per agent-version?") with both positions stated; reviewer input is welcome. The draft's current position is per agent: the version is recorded on every trace and evaluation run as metadata, so per-version filtering and comparison work within one experiment, and monitoring an agent's behavior across upgrades stays in one place. Whichever default is chosen, users who want the other behavior can override it per deployment, so the open question is which default serves most users best.
(From Bill Murdock with assistance with Claude Code.)
| the cause, cross-referencing the BOM diff from the previous | ||
| journey. | ||
|
|
||
| Backward compatibility is preserved by construction: an agent owns a |
There was a problem hiding this comment.
Here again it would be a default experiment ID per agent-version rather than per agent. The version would be a required argument for this to work.
There was a problem hiding this comment.
Same resolution as the thread above: this is now an open question in the draft with both positions stated. One note on "the version would be a required argument": that requirement is specific to the per-version model, and it is part of the cost weighed in the open question. Under the draft's current per-agent position, the destination resolves from the agent alone, and the version is a label recorded on traces and evaluation runs rather than an input to destination resolution; the latest draft spells out that separation in the trace journey.
(From Bill Murdock with assistance from Claude Code.)
| to that experiment's ID. Where MLflow already accepts a typed | ||
| destination or location (trace destinations, `search_traces` | ||
| locations), agent identity becomes a new accepted value; where it | ||
| does not (`evaluate`), `agent_id` is new, optional surface. Existing |
There was a problem hiding this comment.
The agent_id and agent_version constitutes two values where you used to pass one value, so you can't just pass an agent_id and silently map it to the experiment_id. I think this will probably require more API changes than this paragraph suggests.
There was a problem hiding this comment.
The latest draft addresses this in two ways. First, destination resolution takes one value: the agent maps to its default experiment, and the version is not part of that mapping (it is recorded on traces and evaluation runs as metadata). Needing both values to resolve a destination is a cost specific to the per-version model, which the new open question weighs. Second, fair point that the API surface was understated: the trace journey now spells out the pieces, including a public get_default_experiment_id lookup that takes only the agent, and the backward-compatibility paragraph is explicit about where agent identity becomes a new accepted value for existing parameters versus where it is new optional surface (evaluate).
(From Bill Murdock with assistance from Claude Code.)
|
|
||
| - **Should agent-centric traces and evaluations be a separate RFC?** | ||
| The experiments bridge (`agent_id` resolving to a default | ||
| experiment) touches tracing APIs, evaluation APIs, and UI surface |
There was a problem hiding this comment.
agent_id + agent_version resolving to a default experiment...
There was a problem hiding this comment.
The granularity of that mapping is now its own open question ("Is the default experiment per agent or per agent-version?") with both positions stated. The sentence here describes the draft's current per-agent position and would change if the per-version position wins out.
(From Bill Murdock with assistance from Claude Code.)
|
|
||
| 1. Log traces against the agent instead of an experiment: | ||
| ```python | ||
| mlflow.genai.set_active_agent("acme/billing-agent", version=3) |
There was a problem hiding this comment.
I think you should keep this as two API calls rather than one convenience API, for maintainability. Pseudocode:
replace
mlflow.genai.set_active_agent("acme/billing-agent", version=3)
with
mlflow.genai.set_active_experiment(mlflow.genai.get_default_experiment_id("acme/billing-agent", version=3))
where mlflow.genai.get_default_experiment_id("acme/billing-agent", version=3) returns an experiment ID.
Then if, for example, you have a single agent that has many different users using it in different namespaces and they need different experiment IDs, you can just get the experiment ID using a different lookup rather than using the default for the agent-version, and only one line of code needs to change.
If you don't keep the API calls separate, I think people will be confused as to whether they are using the experiment ID for their specific agent deployment vs. the default experiment ID for all agents with the same version.
As Bill pointed out offline, there are a couple of different uses cases here that we need to keep separate. Keeping the API calls separate lets us support all of them:
- Scale-out copies (one deployment, many replicas serving one user base): all replicas share the same deployment config, so they share a destination and their traces aggregate, which is what I would want there.
- Copies that differ in configuration: under the RFC those are different registered versions, so they already get separate default experiments.
- Identical copies with different owners or user bases: the deployments should then override the default experiment.
There was a problem hiding this comment.
The latest draft adopts something along these lines: the trace journey now defines the lookup as a public API (get_default_experiment_id) and defines set_active_agent as convenience over exactly the two pieces you sketch, setting the destination via that lookup plus recording the agent and version as trace metadata. A deployment that needs a different experiment swaps only the lookup line, as you describe. Two notes: the convenience wrapper is kept because the metadata half still needs declaring even when the destination is overridden, and the lookup takes only the agent under the draft's current per-agent position; whether it should take a version as well is exactly the new granularity open question.
(From Bill Murdock with assistance from Claude Code.)
| scorers=[correctness_scorer], | ||
| agent_id="acme/billing-agent", | ||
| agent_version=3) | ||
| ``` |
There was a problem hiding this comment.
Here as well, I recommend keeping the experiment ID lookup separate rather than bundling it into one convenience API.
There was a problem hiding this comment.
Same resolution as the sibling thread: the lookup is now a public API, and the convenience wrapper is documented as exactly that lookup plus the metadata recording, so the two remain separable.
(From Bill Murdock with assistance from Claude Code.)
The journey no longer claims the diff reveals what broke and then
walks it back. The investigator is framed from the start as someone
other than the change author, the diff is a shared record of
declared changes ("what differs in the declarations for two
versions"), and it gives an investigation its starting facts rather
than answers. From design-doc review discussion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New J3 paragraph: the default experiment is a default, not a router. Deployments log wherever their own configuration points; scale-out replicas aggregate by sharing config; deployments needing separation override the destination (destinations, not tags, because permissions are experiment-scoped). Proposes, as not yet settled, that such deployments notify the registry so the experiment ID is recorded on their access binding. Sync bullet gains trace-location link upkeep. From the deployment/experiment-mapping review discussions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One sentence in the J3 deployment paragraph: a version is an analysis dimension recorded on every trace, not an access boundary, and per-version experiments would break the longitudinal view across upgrades. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The RFC keeps its per-agent position but now acknowledges the fork as an open question with both sides stated, since a reviewer holds the per-version position and the author is not settled. Whichever default is chosen, the other camp overrides per deployment, so the question is which behavior makes the better default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The API sketches implied the version participates in choosing the default experiment. Now stated explicitly in the Summary, basic example, and trace journey: a destination identifies the agent only and resolves to its one default experiment; the version is recorded on every trace and evaluation run as metadata. set_active_agent is defined as convenience over two separable pieces, with get_default_experiment_id exposed as a public agent-only lookup (partially adopting Ann Marie's two-call review suggestion), and deployment overrides swap only the destination piece. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cut redundancy that accreted across incremental revisions: rules stated in multiple places (server-never-fetches, card-not-persisted, default-experiment mechanics) now appear once where they matter most, the blast-radius rationale is consolidated, and digressions are trimmed. No design content changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Summary is now a summary: opening, lineage, a one-paragraph tour of the positions, and RFC relationships. The full position statements move intact to a Design positions subsection under Detailed design, where they seed the eventual full design. No content changed beyond the pointer sentences. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
First draft of RFC-0011: Agent Registry, a follow-on to RFC-0004 (MCP Server Registry) and RFC-0008 (Skill Registry).
This draft deliberately covers only the introductory material and user journeys, in the style of the RFC-0009 and RFC-0010 drafts (#37, #27). Detailed design, drawbacks, alternatives, and adoption strategy are TBD.
What the draft contains:
AgentandAgentVersionentities. Each version snapshots a composition (BOM: skill, MCP server, model, and proposed harness references) plus at least one definitional anchor (source provenance, configuration snapshot, or A2A Agent Card).MCPAccessBinding) carrying a URL plus protocol (a2a,mcp,other), never fields on immutable versions.MlflowAgentLocationtrace destination), lifecycle management, and cross-registry blast-radius queries.Authored by Bill Murdock with assistance from Claude Code.
🤖 Generated with Claude Code