Add benchmarking information

[domguinard]

Adding a benchmarking information object to the entry schema.

[cdr2265]

Two small notes on the schema.

1. It may be worth scoping each record to the media types it covers. 10 of the 28 entries on the current list declare more than one media type, and one declares five. As drafted, a multi-media entry can attach a single record with nothing indicating which media type was measured, so the entry reads as covered across all of them. A per-record mediaTypes field would keep the link honest without asking the list to judge anything.

2. Supporting the externalValidation and sha256 suggestion in the file review. One addition: date is the date the benchmark was performed, which is not the date the linked content was hashed. Without an as-of date for the hash, a mismatch cannot be told apart from the target simply having been updated.

[dcondrey]

This is the robustnessEvidence shape from #61 — should close that issue once it lands, but not as currently shaped.

#61 proposed results as a structured object with the actual scored numbers (balancedAccuracy per scenario — excerpt, paraphrase, etc.), specifically so entries are comparable in the registry itself without any party grading another. Here results is a URI to wherever the owner wants, in whatever format. Two entries' result pages won't be comparable to each other or to anything — which was the whole point of the issue. Needs the structured, scenario-keyed results back; see the suggestion on the results property below.

Also overlaps with #82 (externalEvidence), same location in entryMetadata. Worth reconciling with that before merge rather than carrying both shapes.

[dcondrey]

Concretely, structured and scenario-keyed per #61:

Suggested change

	"results": {
	"type": "string",
	"format": "uri",
	"description": "A URL containing the benchmark results"
	},
	"results": {
	"type": "object",
	"description": "Scored results keyed by test scenario, produced by the methodology at `methodology`",
	"additionalProperties": {
	"type": "object",
	"properties": {
	"balancedAccuracy": {
	"type": "number",
	"minimum": 0,
	"maximum": 1,
	"description": "Balanced accuracy achieved for this scenario"
	}
	},
	"required": [
	"balancedAccuracy"
	]
	}
	},

[domguinard]

Calling it externalValidation could be confusing (because of the term "validation" which has a defined meaning in C2PA).
I also think there is virtue in calling it a benchmark since it is what it actually is. Either we provide benchmarking information or we do not I would argue.

[domguinard]

Two small notes on the schema.

1. It may be worth scoping each record to the media types it covers. 10 of the 28 entries on the current list declare more than one media type, and one declares five. As drafted, a multi-media entry can attach a single record with nothing indicating which media type was measured, so the entry reads as covered across all of them. A per-record mediaTypes field would keep the link honest without asking the list to judge anything.
2. Supporting the externalValidation and sha256 suggestion in the file review. One addition: date is the date the benchmark was performed, which is not the date the linked content was hashed. Without an as-of date for the hash, a mismatch cannot be told apart from the target simply having been updated.

Thanks @cdr2265, great suggestions.

I added a required mediaType array for each entry and added the sha256. The date of the benchmark was already present in the schema.

[domguinard]

This is the robustnessEvidence shape from #61 — should close that issue once it lands, but not as currently shaped.

#61 proposed results as a structured object with the actual scored numbers (balancedAccuracy per scenario — excerpt, paraphrase, etc.), specifically so entries are comparable in the registry itself without any party grading another. Here results is a URI to wherever the owner wants, in whatever format. Two entries' result pages won't be comparable to each other or to anything — which was the whole point of the issue. Needs the structured, scenario-keyed results back; see the suggestion on the results property below.

Also overlaps with #82 (externalEvidence), same location in entryMetadata. Worth reconciling with that before merge rather than carrying both shapes.

@dcondrey the majority of the WM TF group felt that this proposal was too strict and detailed hence the alternative proposal. If you can join the call on Wednesday we will aim at finalizing this proposal and your input is very welcomed.

[jcollomosse]

Given the community ask I support the inclusion of a non-ontological field e.g. pointer to free-formatted benchmarking information, but I think there are some things that need improvement in this PR.

1. We should name it 'benchmarkingInformation' rather than 'externalValidation'. Validation has a very specific meaning within C2PA in respect of C2PA manifests, which is not related to watermark benchmarking. Just a URL is fine.
2. We should not require a SHA hash i.e. hashed URL. Some benchmark information is in actively maintained leaderboards where content is dynamic. It is useful to have perhaps but only as optional field.
3. We should remove the human readable description of the algorithm since this already exists in the main SBAL fields.

I think less is more here - we want to be able to flexibly and optionally point to resources containing benchmarking data for algorithm, but not over-prescribe the nature or format of that data.

[domguinard]

Discussed during the WM TF meeting, pending the change on the format for the hash the TF suggested this is ready to be taken to the TWG.