You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Proposal: optional per-entry robustnessEvidence field with a shared benchmarking methodology #61
Every entry in this list carries a prose robustness claim ("robust to copy-paste", "robust to reordering", "robust to synonym substitution"), but there is no measured evidence behind any of them and no shared way to produce it. An implementer choosing an algorithm, or a validator deciding how much to trust a soft-binding match, has nothing comparable to go on. #60 is a case in point: one of my own entries' claims did not hold once measured, and there was no methodology in the registry to catch that.
Proposal. An optional robustnessEvidence object in an entry's entryMetadata, populated by the algorithm's owner from a shared, reproducible methodology — self-reported, auditable, and explicitly not a cross-owner ranking:
Each owner runs their own algorithm, on media appropriate to it, through the shared methodology and submits numbers. The registry gains comparable evidence over time without any party grading another.
Why now. The conformance program has no robustness testing for soft bindings today. I have a reference methodology, a reproducible harness, and measured results for the text entries, published at https://github.com/writerslogic/c2pa-text-binding/blob/main/ROBUSTNESS.md (format / edit / excerpt / paraphrase, scored as balanced accuracy with analytic false-match bounds), and I'm happy to bring it to the Conformance TF and seed the methodology document.
Scope note. Robustness batteries are media-specific (text vs image vs audio vs video), so the field references a methodology + corpus rather than mandating a single battery. Starting point for discussion — feedback on the field shape and whether this belongs here vs. the conformance program is welcome.
@dcondrey thanks a lot for this proposal, we discussed it during the last WM TF meeting. There was a significant amount of support for a idea along these lines and we will discuss it today in greater details. Feel free to join if you can.
Activity
domguinard commentedon Sep 16, 2026
@dcondrey thanks a lot for this proposal, we discussed it during the last WM TF meeting. There was a significant amount of support for a idea along these lines and we will discuss it today in greater details. Feel free to join if you can.
dcondrey commentedon Sep 22, 2026
@domguinard checking in — any outcome from the WM TF discussion?
domguinard commentedon Sep 30, 2026
Hi @dcondrey yes, here is the corresponding PR that will be discussed today, incorporating some of your suggestions: #79