Add RFC-0009: Skill Tracing

[jwm4]

Summary

RFC-0009 proposes Skill Tracing: trace-level links from MLflow traces to the registered skill versions that were active when the trace was produced, with the activation span annotated where one is identifiable, so that adoption, deprecation impact, security exposure, and regressions can be evaluated per skill. Links follow the same lineage mechanism MLflow uses to link prompts to traces, and each link records how it was produced.

Tracing was originally drafted as part of RFC-0008 (MVP Skill Registry, PR #26) and deferred out of that RFC. It now gets its own RFC rather than a follow-on PR.

The RFC covers three instrumentation paths: explicit (span.link_skill(), with a plain OpenTelemetry attribute contract for code that does not use the MLflow SDK), automatic in agent frameworks (LangGraph, the OpenAI Agents SDK, and others) via the in-process mapping recorded when a skill is pulled, and automatic in harnesses (Claude Code and others) via an install-time record, with a best-effort content-marker fallback for harnesses that export OpenTelemetry natively. Harness installation commands are in scope to support that third path. User journeys span instrumentation, adoption measurement, deprecation and security impact, benchmark evaluation, and production comparison.

Numbering note

Extended Skill Bundles, previously RFC-0009 in PR #27, has been renumbered to RFC-0010: Extended Agent Plugins. Skill tracing is the more urgent piece of work, so it takes the 0009 slot. The cross-references in the merged RFC-0008 were updated accordingly in PR #43.

Posted by Bill Murdock with assistance from Claude Code.

[jwm4]

Status as of 2026-08-23: per reviewer guidance on RFC-0008, this draft currently contains the summary, problem statement, and user journeys so the workflows can be agreed on before technical detail. Detailed design and the remaining sections are marked TBD, and they will not be worked on until all the key stakeholders have agreed on the summary, problem statement, and user journeys.

Remaining work:

• Fill in summary, motivation, and user journeys
• Stakeholder agreement on the above
• Fill in detailed design and remaining sections
• Resolve open questions (framework activation hooks first)
• Mark ready for review

Posted by Bill Murdock with assistance from Claude Code.

[jwm4]

Status update: five of the seven open questions are now resolved in the draft, based on research into current MLflow internals and framework documentation.

• Query API: skill trace queries extend the existing search_traces filter syntax rather than adding a skill-specific function. Exact matching on span attribute values becomes a store-level requirement of this RFC.
• Automatic context capture: resolving a skill through MLflow records its identity, and autologgers pick it up when spans are created. This follows the pattern MLflow already uses for other registry entities.
• Harnesses without in-process MLflow tracing: harnesses whose tracing integration is provided by MLflow get full SKILL spans. Harnesses that trace themselves through native OpenTelemetry export get the spans they already emit annotated with skill coordinates when the skill invocation is identifiable; MLflow does not create spans on their behalf. ADK is handled the same way, since MLflow traces it by receiving its OpenTelemetry output.
• Span exposure: the skill context manager manages its span internally and does not expose it to the caller, so recorded coordinates cannot be altered after creation.
• Install record location: project-scoped installs record in the project, user-scoped installs record in the user's MLflow configuration, and project entries take precedence.

Two questions remain open for discussion: OTel alignment (whether the mlflow.skill.* span attribute names are part of the public contract for non-MLflow instrumentation) and digest-based linking (best-effort digest recording and whether digest grouping should work across skill names).

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Could we skip the HTTP call and just put alias in the mlflow.genai.skill_context() call? I don't think you actually need to resolve the skill for the trace to link it.

[jwm4]

Done. link_skill() now accepts alias directly, so the caller writes no resolution call. I kept one property from the original design: the alias is resolved when the link is recorded and the trace stores the concrete version, which follows RFC-0008's convention that aliases are accepted as input everywhere but never stored in place of versions (the same rule its agent plugin member rows and alias URIs follow). A trace that stored @production would change meaning whenever the alias is repointed, which would undermine the deprecation and comparison journeys. So the lookup still happens under the hood for alias callers, but only for them, and only at link time.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

A "Skills" tab on the trace like we have for "Links" would be nice.

[jwm4]

Added. The first journey's UI path now shows a "Skills" tab on the trace view alongside the existing linked-entity tabs, listing the linked skill versions with each linking to its registry detail page. It fits naturally now that the design records skills as trace-level links, the same shape the Links tab already renders.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

Let's align on this outside the scope of skill. We simplified the trace drawer in 3.16 and generally prefer not to have too many tabs in the drawer, so I'm inlined to a consolidated lineage tab across other assets.

[jwm4]

That works for me, and I changed the text accordingly. The RFC no longer asks for a Skills tab in the trace view (or on the experiment page); it now states the functional need only, that linked skill versions appear among a trace's linked entities and link to their registry pages, and says explicitly that how linked entities are presented, such as one consolidated lineage view across prompts, skills, and other assets, is aligned across assets outside this RFC. The other UI items (a "Related traces" link from skill pages, and a linked-skill diff in run comparison) are unchanged, but the same principle applies to their presentation.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I’m not convinced SKILL should be a new span type. The RFC describes a skill as ambient context, while a span conventionally represents an operation with meaningful start/end, status, input, and output. I imagine that in many harnesses we can observe loading of a skill but cannot reliably determine when the skill stops influencing later calls, so parenting LLM/tool spans beneath it may assert stronger causality than the instrumentation can honestly record.

Could this follow the existing prompt pattern instead where it links the concrete skill version to the trace for governance and related-trace queries? We could also annotate the currently active or activation span when that is detected. For example, if a tool call loads the skill, we can add a linked attribute on the tool call span.

This better aligns with OTel's guidance on what a span should be:
https://opentelemetry.io/docs/specs/semconv/how-to-write-conventions/#defining-spans

[jwm4]

I'll admit I resisted this one. My mental model of skills is that they're closer to subroutines than to ambient context: a defined invocation with a beginning and an end. Some harnesses partly support that model: Claude Code delimits skill-driven turns and its forked skills are genuinely bounded operations, and Cursor scopes a manually invoked skill to a single message. But I looked at how activation is described across the rest of the ecosystem, and it leans your way: the Agent Skills spec defines activation as a one-way load with no counterpart, and for most harnesses I found an observable activation but nothing that marks the end of a skill's influence. I wouldn't claim that's conclusive (much of it is reading documentation, and silence isn't proof), but it's enough that the RFC shouldn't build its core model on a boundary most harnesses may not have.

So the latest revision adopts your approach: a trace-level link to the skill version following the prompt-linking pattern, plus an annotation on the activation span where one is identifiable. No new span type. The rejected span design is documented under Alternatives.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I don't think Claude MLflow hook will have that level of detail. Is the plan to parse tool call outputs for any indication that a tool read a skill file (e.g. cat <skill path>) from MLflow?

[jwm4]

It does, essentially in the way you suggest. The MLflow Claude Code plugin already works by parsing the session transcript from its Stop hook, and skill activations are visible there today: a model-invoked skill appears as a Skill tool call whose input carries the skill name, and the tool result carries the resolved command name (the parser already keys on that field to filter skill content injection out of prompt detection). A user-invoked /skill-name takes a different shape (a command expansion with no Skill tool call), so the parser needs handling for that path too, which the detailed design will cover. And a skill run with context: fork already surfaces as a tool span with the subagent's spans nested under it. The revision states the detection approach in the harness journey: harness-native signals where they exist, and otherwise matching tool calls that read a skill's SKILL.md from its installed location, which is your cat <skill path> case.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

It'd be nice if we can detect tool calls that use a script bundled in a skill and annotate the tool call span as having used the skill. This should be possible by matching the paths in the tool call to where MLflow installed skills.

[jwm4]

Adopted. The harness journey now specifies exactly this: tool calls that use a skill's bundled files, such as a script under the skill's directory, are annotated as skill usage by matching paths against the locations recorded at install time. The same location matching also serves as the activation-detection fallback (a tool call reading a skill's SKILL.md), so both ride one mechanism, and it has the property that it never fabricates usage: a path match is positive evidence the skill's files were touched.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

The experiment UI shows "prompts" associated with the experiment. Should we consider adding a skills tab too?

[jwm4]

Added. The benchmark evaluation journey's UI path now includes a "Skills" tab on the experiment page, listing the skill versions linked from the experiment's traces, following the existing prompts tab.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Will the UI show the diff of skills used when comparing to evaluation runs?

[jwm4]

Yes, added. The run-comparison view shows a diff of the linked skill versions alongside the metric deltas, so a change in outcomes is read against the change in skill configuration. It appears in both comparison journeys: the benchmark one, and a new journey comparing skill versions on production traffic, where one evaluation run is produced per skill version from the linked production traces and the diff makes the comparison legible.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I'd suggest consolidating this user journey with the one above.

[jwm4]

Done. The lineage and regression journeys are merged into a single "Evaluate and compare skill versions on a benchmark" journey, which carries the lineage steps (evaluation result to linked skill versions and back) inside the comparison flow.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Could you please make the document more concise? Have Claude determine the lowest value sections to cut. The user journeys section may be too exhaustive.

[jwm4]

The revision trims throughout: the token-cost journey is dropped, lineage and regression are merged, the span-model justification in the summary is reduced to a statement of the design, and the nesting material is gone. One journey was added (comparing skill versions on production traffic, which I think earns its place as the production counterpart of the benchmark journey), and the document is still net shorter than before. If it still reads as too long, pointers to whichever sections feel lowest-value would help me cut further.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Should could probably use a bit more details. It's a bit overlooked in the current RFC.

[jwm4]

Expanded. The harness journey's run step now spells out what the autologger does: it identifies skill activations in the recorded conversation using the harness's own signals where they exist (a dedicated skill tool call, a slash-command invocation), falls back to matching tool calls that read a skill's SKILL.md from its installed location, annotates tool calls that use a skill's bundled files by the same location matching, and links using the install record with no registry call during the run.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Maybe not actionable in this RFC but it'd be nice to be able to search for traces with the same content digest as well.

[jwm4]

The draft partially covers this, but only partially. The impact journey describes a two-step route: the registry's digest index (from RFC-0008) resolves a digest to every version sharing that content, and trace queries then cover those versions. What it does not give you is a single "search traces by digest" query: the skill filter as drafted is exact-match on one name/version, so covering N re-minted versions means unioning N queries by hand. A first-class digest filter form (say skill.digest = '<sha256>', resolved server-side through the registry index, following the same lowering pattern as model_id) would close that gap cleanly. You flagged this as maybe out of scope, so I have left the draft as is, but I would be happy to bring the full digest filter into scope if you think it belongs in this RFC.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Could the OTLP-only path support a fallback based on installer-injected skill metadata? For example, MLflow could append a machine-readable marker containing the skill coordinates and optionally the content digest. If the harness includes that marker in captured LLM input, the MLflow server could recognize it during OTLP ingestion and create the link without needing access to the harness-local install record.

This would not be as authoritative as structured span attributes. Perhaps links could record whether their source was explicit instrumentation, install-record matching, or prompt-content inference.

[jwm4]

I liked this enough to test it before answering. I had Claude run the experiment: a marker-bearing skill in OpenHands and Goose, exporting OTel to a local MLflow server, plus a source read of Gemini CLI. Results: where the harness captures LLM content, the mechanism works well. The HTML comment survives byte-for-byte into gen_ai.input.messages, and it appears only on activation (it rides the skill-load tool result, never the system-prompt catalog), so it is an accurate activation signal, recurring in later turns' inputs. The catch is that content capture is the dominant variable and it trends off-by-default: Goose requires the OTel GenAI capture opt-in, Gemini CLI requires enabling traces and also hard-truncates the input attribute at 10k characters (which can cut a trailing marker), and OpenHands captures by default only because its instrumentation lacks the switch.

So the latest revision adopts it as a best-effort fallback for the OTel-receiver path, alongside your provenance suggestion, which is adopted in full: links record whether they came from explicit instrumentation, in-process resolution, install-record matching, or content-marker inference. The marker route fails closed (a link is created only when the marker's coordinates resolve in the registry and its digest matches the version's, which also blunts spoofed markers), and digest validation excludes the marker line from hashing so it composes with the pre-link validation from your other comment.

On possibly skipping path matching for the MVP in favor of this: I would keep path matching. It runs inside MLflow's own integrations where MLflow controls the data, while the marker depends on each harness's content-capture policy, so the dependable mechanism seems worth keeping over the contingent one.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I think the autologger can validate installed content against the recorded digest before linking. If it mismatches, emit an error log and don't link. Then cache the result locally and accept that the skill could change while the agent is running but unlikely.

[jwm4]

Adopted as stated: the autologger validates installed content against the recorded digest before linking, logs an error and records no link on mismatch, and caches the validation result locally. The open question is resolved accordingly in the latest revision. This also composes with the marker idea in your other comment: if that mechanism lands, digest validation needs only a rule that the marker line is excluded from hashing.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Out of scope of this RFC but we could consider addressing the "digest is client-asserted rather than server-verified" by having a periodic job that downloads new/updated skill versions and validates the content digest and MLflow can add a "verified digest" bool that can be displayed in the UI.

[jwm4]

Agreed on both the idea and keeping this out of the scope of this RFC. Worth noting it is really registry-side follow-up (RFC-0008's domain) rather than tracing work, with two wrinkles: RFC-0008 deliberately keeps the server from fetching user-supplied source URLs, so a verification job would need a trusted carve-out for that; and it could only cover sources that need no credentials, since the server does not store source credentials, so private repositories would stay client-asserted. A "verified digest" flag scoped to publicly fetchable sources still seems worthwhile. If it lands, this RFC benefits automatically: digest-based grouping and link provenance both get stronger when the digest is server-verified, with no change to the tracing design.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I think out of scope of this RFC but I think this grouping would be powerful. One option is to add telemetry to MLflow and see how many agent plugins are "packaged" versus "assembled" to know if this is worth the effort.

[jwm4]

Agreed, and the open question stays open with your endorsement noted. Your telemetry proxy captures what is likely the biggest source well: assembled plugins reference shared skill entries, so identical content stays under one name, while packaged imports auto-create member skills per plugin, which systematically mints identical content under different names. One source it would miss is independent registration of the same upstream source under different names with no plugin involved, which seems plausible in larger organizations. Once digests accumulate, counting digests that appear under more than one skill name measures all of it directly. Either check is cheap and belongs with the implementation rather than this document.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

This looks good to me!

I think for technical details, we probably only need to flush out the "install"/"lockfile" spec for MLflow autologger to know the skill coordinates.

Alternatively, we can further explore https://github.com/mlflow/rfcs/pull/37/changes#r3917274106 and not do path matching for this initial MVP.

[B-Step62]

Thanks for the write up! Left a few questions I'd like to clarify before proceeding.

[B-Step62]

Can we make this API of a span object? It's not clear what does this link skill to, for example, it can be eval run. I think span API is more consistent with Otel syntax below.

[jwm4]

Yes, good idea, and I made this change. link_skill is now a method on the span object, span.link_skill(name=..., version=...), so the target is explicit: it annotates that span as the activation point and links that span's trace to the skill version, with nothing resolved from ambient context. That removes the ambiguity you point out (trace versus span versus an active evaluation run), and it makes the explicit path use the same primitive the automatic paths do, where the autologger annotates the activation span and derives the trace link from it. It also lines up with the plain-OpenTelemetry example, which can only ever annotate a span. I deliberately did not keep a module-level form alongside it, to avoid offering two ways to do one thing; code without a span in hand uses mlflow.get_current_active_span().

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

Resolving alias can have an overhead. Do we have any caching strategy here?

[jwm4]

Good point, and I added a caching strategy to the text. It follows the prompt registry's existing alias cache: resolutions are cached per process with a short expiry (aliases get a short TTL because they can move; pinned versions need no resolution at all), and a caller can shorten or disable the cache per call, mirroring load_prompt's cache_ttl_seconds. That keeps repeated links in a hot path from querying the registry on every call, with the same bounded-staleness trade-off after an alias repoint that prompt aliases already accept. Worth noting the overhead only arises when application code passes an alias to span.link_skill at run time: the harness path resolves aliases at install time and the framework path at pull time, so neither queries the registry during a run.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

Do we have this API under genai module directly? How does MLflow know this call is for a skill?

[jwm4]

Sorry, my response above was answering a different question than the one Yuki asked. I am deleting it now.

[jwm4]

Here is the answer the question Yuki actually asked: The mlflow.genai.pull method is defined here. It defaults to entity_type​ skill​, but the user can override it to agent_plugin​ instead (and could presumably be extended to more entity types in the future too). If you would like, I could add entity_type="skill" (even though that's already the default) to make it less confusing to readers.

[B-Step62]

If we want to keep this at genai level, can we make it to take URL skills://... instead of name? I think it's confusing to implicitly scope the API to skill by default while the API shape tells nothing about it.

[jwm4]

Agreed on the merits: a call whose shape names the entity is clearer than one that defaults to skills silently, and it matches how load_prompt already takes prompts:/ URIs and how RFC-0008's CLI already takes skills:/ URIs positionally. I made the change in this RFC: mlflow.genai.pull accepts a skill URI as its first argument (skills:/code-review@production or skills:/code-review/3), and the journey and detailed design now use that form.

One deliberate choice: I made the URI form additive rather than replacing RFC-0008's keyword signature. RFC-0008 is being implemented right now, with stories already written against the keyword form, and I would rather not disrupt that work with another amendment. So the keyword form stays as specified, the URI form is added as part of this RFC's implementation (which touches pull anyway to record the in-process mapping), and the URI form is the one the documentation leads with. If you would prefer the keyword form deprecated later once both exist, that seems reasonable to me too. Does the additive form address your concern?

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Maybe we can just rename it to pull_skill and keep the keyword approach?

[jwm4]

I would rather not rename, to avoid disrupting the RFC-0008 work already in progress. I added the URI form alongside the keyword form, which gives the call a shape that names the entity, Yuki's concern, without changing RFC-0008. If the keyword form should be deprecated once the URI form exists, that can be its own small change later.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

Can we elaborate how we do this? Where do we store the install record and what does the record look like?

[jwm4]

Fair ask, and I clarified what the record contains in the journey text: it maps each installed skill's harness-local name and install location to its registry coordinates (workspace, organization, name, version) and content digest. Where it lives is in the same step: project-scoped installs record it in the project, user-scoped installs in the user's MLflow configuration, and a project entry wins when both define the same harness-local name. The record's format will be specified in the detailed design; Matt asked for the same thing in his approval, so it is clearly the piece of technical detail this RFC most needs. One thing that shapes that spec is the installation-scope open question: whether MLflow performs the install itself or only writes this record for a skill the user installed by other means.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

Can we discuss on the backend aspect as well? Prompt linking currently logs a lineage record in a table and the search query uses it. Do we clone this mechanism for skills or do we do an attribute based search?

Ref: https://github.com/mlflow/mlflow/blob/0cb8a50f9ba620e8163bd349059d2a6545b10768/mlflow/store/tracking/sqlalchemy_store.py#L5069-L5075

[jwm4]

Clone the lineage mechanism, and I made the text say so explicitly. A skill link is a lineage record associating the trace with the skill version, exactly like prompt links, and the skill filter is an exact-match query over those records the same way the prompt filter works. Attribute-based search is deliberately not the query path: span attributes mark where the activation happened, but the store's span-attribute filtering is substring matching over serialized span content, which cannot give the exact-match semantics an adoption count needs, and building structured attribute search would be new store work for a worse result. One more point I added: links created at ingestion (from mlflow.skill.* attributes on received spans, or from a content marker) produce the same lineage record, so every provenance is reached through the one query path.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

nit: Are we still talking about trace-level link? The APIs and spec in this RFC record skill lineage on spans.

[jwm4]

Good catch, and I changed it. The link is now described as recorded through the span on which the activation was observed, not as trace-level, and the lineage record itself carries that span's id alongside the trace id, so the record says both "this trace used this skill version" and "activated at this span." The association is still stored and queried through the same lineage mechanism as prompt links; the span id is an added field, as provenance is.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

As mentioned in the other thread, can we discuss this offline including designer? I still believe we shouldn't make detailed UI decision like this on text-only RFC without prototype.

[jwm4]

Agreed, and I revised the RFC to match. The journeys no longer specify UI presentation (the Skills tab is gone), and the UI section now says explicitly that both the content and the presentation of these surfaces will be decided with a designer and a prototype outside this RFC, alongside the same decisions for prompts and other linked assets. What remains is labeled as an initial draft of the content the journeys need (linked skill versions visible from a trace, a path from a skill's registry page to its traces, and a way to compare linked skill versions across runs), not a UI specification.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

Similarly to the comment above, let's discuss UI outside the RFC. This is also in the scope of IA change. For context we are trying to reduce the number of tabs in experiment in general.

[jwm4]

Understood, and I revised the RFC accordingly. The experiment-page tab is gone; the text now states only the content need (skill versions linked from the experiment's traces, listed alongside the experiment's other linked entities), and the UI section says that both the content and the presentation of these surfaces will be decided with a designer and a prototype outside this RFC. That discussion is the right place to fit this into the tab reduction.

Posted by Bill Murdock with assistance from Claude Code.

[B-Step62]

LGTM except that I defer UI decision until we have prototype and engage designer.

[jwm4]

Status as of 2026-09-11: the RFC now has a full draft. The technical sections (Basic example, Detailed design, Out of scope, Drawbacks, Alternatives, Adoption strategy) are written at a deliberately high level: they fix the shapes other work depends on (the link model and lineage record, the span.link_skill method and mlflow.skill.* attribute contract, the query forms, the install record and mlflow skills track, autologger behavior by harness tier) and leave internals to the implementation.

Scope decisions reflected in the draft: this RFC specifies the install record and the command that writes it for a skill the user installed by any means; an MLflow installer is possible follow-on work, not committed. mlflow.genai.pull gains a skill-URI form additively, keeping RFC-0008's keyword form to avoid disrupting the implementation in progress. UI content and presentation are deferred to a designer-and-prototype discussion outside this RFC.

Both reviewers have approved. Two open questions remain, both awaiting input: whether the mlflow.skill.* attribute names should live in a vendor-neutral namespace, and whether digest grouping should extend across skill names.

Remaining work:

• Summary, motivation, and user journeys
• Stakeholder agreement on the above
• Detailed design and remaining sections (high level)
• Reviewer confirmation on the additive pull URI form
• Merge

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

Does it make sense to declare this in the agent code at runtime instead of storing this as a file populated by a CLI command?

I think the local file makes sense only when MLflow owns the install.

[jwm4]

You are right that a local file only makes sense when MLflow owns the install, and this RFC does not, so I removed it. The new design needs no declaration at all, in code or in a file: the harness autologger identifies an installed skill by its content. It hashes the skill directory with RFC-0008's digest rule, and then, taking up your suggestion from the other thread, first checks for the content marker in the installed SKILL.md: if the marker's digest equals the computed one, its coordinates identify the version with no registry call, so content pulled through MLflow resolves offline and exactly. Otherwise it resolves the digest through the registry (cached locally), with a name-match-then-latest rule when the same content is registered under several versions or names. Skills installed by any means are linked once tracing is enabled. mlflow skills track and the install record are gone; the trade-offs (a registry lookup for content not pulled through MLflow, and the digest rule implemented identically in each harness integration) are in Drawbacks, the rejected record is documented under Alternatives, and the one registry addition needed is a lookup by digest across skill names.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

It'd be helpful to define what that marker is in the RFC since it seems you did a POC on it already.

[jwm4]

Good idea, and I added a Content marker subsection to the detailed design. The marker is one line, an HTML comment appended by mlflow skills pull as the last line of the pulled SKILL.md: <!-- mlflow-skill: {...} --> with JSON carrying the version's workspace, organization, name, version, and content digest. The digest rule excludes that line so pulled content still verifies against the registered digest. The server matches it only in captured LLM input, never output, and creates a link only when the coordinates resolve in the registry and the digest matches the version's. The subsection also notes what the experiment showed: the marker survived byte-for-byte wherever the harness captured LLM content, and whether content is captured at all is the limiting factor.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

The path could vary based on your local machine or when the agent is deployed, so I think at least we should use a relative path, but as mentioned in 72da3d9#r3992701795, I don't know if this extra file is worth it.

If the content marker idea works, the autologger could parse the marker before recording the trace in MLflow.

[jwm4]

The file is gone, so the path question is moot, and I took up the marker suggestion: the autologger now checks the installed SKILL.md for the content marker first, and if the marker's digest equals the digest it computes from the installed content, the marker's coordinates identify the version with no registry call. Content that carries no marker (installed by means other than mlflow skills pull) falls back to resolving the digest through the registry. Both paths are in the Skill identification section.

Posted by Bill Murdock with assistance from Claude Code.

[jwm4]

Status update, 2026-09-12: following review, the install record and mlflow skills track are removed. Installed skills are now identified by content: the autologger hashes an installed skill with RFC-0008's digest rule and resolves it, from the content marker when the content was pulled through MLflow (offline, exact) and otherwise through the registry, preferring a name match and then the latest version when the same content is registered more than once. This RFC therefore specifies no installer, install record, or package manager integration. The content marker is now defined in the detailed design. Open questions are unchanged in number: the mlflow.skill.* namespace, and whether the cross-name digest lookup this RFC adds should also power digest-based trace grouping.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I think most of the time when tracing is enabled, the LLM content will be traced.

[jwm4]

Agreed, and I revised the bullet. It now states the actual dependency, that content capture is a separate switch from tracing and off by default in the OpenTelemetry GenAI conventions, and then says what you said: users who send traces to MLflow to understand agent behavior generally turn content capture on, so in practice this limits little. The old "commonly off by default" wording overstated it.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I'm okay with this.

[B-Step62]

Same here, I didn't find existing convention for skills in otel semconv. Even if there is in the future, mapping is maintainable.

[jwm4]

Thanks, both. With that agreement the question is closed: the RFC now states that mlflow.skill.* is the public contract, that no OpenTelemetry semantic convention for skills exists today, and that a future convention can be mapped to these attributes without changing the contract. I removed it from Open questions.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I think yes, but it can be deferred.

[jwm4]

Agreed, and I resolved it that way. The RFC now states that digest-based trace grouping across skill names is follow-on work that reuses the cross-name lookup delivered here, and the client-asserted digest was already handled under Out of scope, where server-side verification is listed as RFC-0008 follow-up. With both questions resolved, the Open questions section is removed.

Posted by Bill Murdock with assistance from Claude Code.

[jwm4]

Status update, 2026-09-15: both open questions are resolved with reviewer agreement. The mlflow.skill.* attribute names are the public contract; no OpenTelemetry semantic convention for skills exists today, and a future one can be mapped without changing the contract. Digest-based trace grouping across skill names is follow-on work that reuses the cross-name lookup this RFC adds. The Open questions section is removed. The content-marker drawback is restated: the fallback depends on the harness capturing LLM message content, a separate setting from tracing that users of MLflow tracing generally enable.

With Yuki's confirmation on the additive pull URI form, both reviewers have approved and nothing remains open on my side.

• Summary, motivation, and user journeys
• Stakeholder agreement on the above
• Detailed design and remaining sections (high level)
• Reviewer confirmation on the additive pull URI form
• Merge

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

I was talking with @B-Step62 about this offline. Could we make it so that we only support autologging from skills with the content marker and don't verify it with a call to the registry? For skills that don't have the marker, we can still compute the digest and set it as a span attribute so that if the skill gets registered in the future, it can be detected by digest.

[jwm4]

Yes, good idea, and I made the change. Harness autologgers now link only content that carries the marker written by mlflow skills pull: they hash the installed skill, compare the result with the marker's digest, and link from the marker's coordinates with no registry call. Content without a marker is not linked; the autologger records the computed digest as mlflow.skill.digest on the activation span so that traces can be matched to a version registered later, which the RFC lists as follow-on work. Content whose digest no longer matches its marker is treated as unmarked. The registry lookup by digest, the cross-name lookup, and the name-match-then-latest rule are removed from the design and recorded under Alternatives as a rejected route. The digest provenance value stays, now meaning that an autologger verified the installed content against its marker.

Posted by Bill Murdock with assistance from Claude Code.

[mprahl]

One small comment and then ready for merge.

[jwm4]

Status update, 2026-09-16: following review, harness autologgers link only content that carries the marker written by mlflow skills pull, verified locally against the content digest, with no registry call during a run. Content installed from other sources is not linked; its digest is recorded on the span, and matching it to versions registered later is follow-on work, as is digest-based grouping across skill names. The registry lookup by digest is removed from the design and documented under Alternatives.

Posted by Bill Murdock with assistance from Claude Code.