From f32b90df9c133c9835f0d29a5e778a8b839c71e7 Mon Sep 17 00:00:00 2001 From: AIwork4me Date: Mon, 10 Aug 2026 20:22:14 +0800 Subject: [PATCH] docs: add community benchmark evidence workflow --- CONTRIBUTING.md | 8 +- README.md | 23 +++--- README.zh-CN.md | 20 +++-- SUPPORT.md | 2 +- docs/HANDOFF-windows-hip.md | 2 +- docs/HANDOFF-windows-vlm-llamacpp.md | 4 +- docs/architecture.md | 15 +++- docs/benchmark-methodology.md | 7 +- docs/community-benchmarks.md | 76 +++++++++++++++++++ docs/hardware-matrix.md | 37 ++++++--- docs/how-it-works.md | 2 +- docs/known-gaps.md | 16 ++-- docs/reproducibility.md | 5 +- model_card.pipeline.json | 2 +- .../omnidocbench/v16/windows-hip/README.md | 5 +- 15 files changed, 172 insertions(+), 52 deletions(-) create mode 100644 docs/community-benchmarks.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 93fae0c..4155853 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,12 +24,16 @@ The `smoke` backend (no GPU) should always pass — keep it working. - `make smoke-test` is green. - `make publish` reports `CONFORMANT`. -- `model_card.json` reflects reality (hardware, badge, artifacts) — don't claim `verified` without committed, reproducible results. +- Model cards reflect reality (hardware, legacy badge, artifacts); result submissions must not assign central platform review or promote legacy badges. - Update `docs/known-gaps.md` — close resolved bullets, add new ones. ## Reporting results -Commit the engine-produced `run_summary.json` + `provenance.json` under `results/omnidocbench/v1.6//` and update `model_card.json.artifacts` to point at them. A maintainer will review for the `verified` badge per the policy in `omnidocbench-rocm/contracts/badge-policy.md`. +For a result produced in this repository, commit the engine-produced evidence +bundle under `results/omnidocbench/v16//` and point the applicable +model card at it. For an independent result, follow the short +[community benchmark checklist](docs/community-benchmarks.md) and provide an +immutable evidence permalink. Raw predictions are useful but optional. ## Code of conduct diff --git a/README.md b/README.md index 12e249e..14c8e25 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,14 @@ [![OmniDocBench v1.6](https://img.shields.io/badge/OmniDocBench-v1.6-blue)](https://github.com/opendatalab/OmniDocBench) [![VLM full](https://img.shields.io/badge/MinerU2.5--Pro%20VLM%20(full)-95.56-green)](#evaluation) [![pipeline full](https://img.shields.io/badge/MinerU%203.4%20pipeline%20(full)-86.48-yellowgreen)](#evaluation) -[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](reproducibility.lock.yaml) +[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](REPRO.yaml) [![license: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0%20(+MinerU%20terms)-blue)](NOTICE) ## At a glance - **What it is.** Tooling to run opendatalab MinerU (3.4 pipeline + 2.5-Pro VLM) on AMD ROCm and score it on OmniDocBench v1.6. -- **Where verified.** AMD **gfx1100 (RDNA3), Radeon PRO W7900 (48 GB), ROCm 7.2**, bf16. Host has 4x W7900; **each benchmark used 1 GPU (no tensor parallel).** +- **Where benchmarked.** Linux ROCm: AMD **gfx1100 (RDNA3), Radeon PRO W7900 (48 GB), ROCm 7.2**, bf16, one GPU per run. Windows-HIP: **gfx1151, Ryzen AI MAX+ 395 / Radeon 8060S, Windows ROCm 7.2.1**. +- **Community coverage.** Reproductions on additional AMD configurations are tracked with the central ROCmDoc [producer-evidence / platform-review model](docs/hardware-matrix.md); see the [submission checklist](docs/community-benchmarks.md). - **Most reliable results.** **MinerU2.5-Pro VLM (vLLM-on-ROCm) full 1651 = 95.56 Overall** (platform CDM-scored; prior standalone score 95.46 from the same prediction set — Δ +0.10 pp, entirely the Formula-CDM submetric 96.46→96.73); **MinerU 3.4 pipeline full 1651 = 86.48 Overall**. - **Most important limitation.** **Not precision-aligned.** No same-engine CUDA control exists; the upstream headline may be measured with a different engine. The official anchor (vlm-engine 95.30) is aligned to the upstream README "Local Deployment" table and is **community-verified, not official support**. - **Upstream.** This is a port OF [opendatalab/MinerU](https://github.com/opendatalab/MinerU); the [omnidocbench-rocm](https://github.com/AIwork4me/OmniDocBench-ROCm) engine is one *optional* consumer (install the `[platform]` extra), not the definition of this repo. @@ -116,13 +117,15 @@ developer debugging — see `docs/reproducibility.md` for the full recipe. The `vlm-vllm` row is reproduced on linux-rocm (self-attested, `badge: community`): 1651/1651 pages attempted, 1649 non-empty predictions (2 empty), no process -crashes; ~7 h on a single GPU (gfx1100); read-order EditDist 0.1240. -Overall 95.56 is consistent with the published upstream reference range -1651/1651 pages attempted, 1649 non-empty predictions (2 empty), no process crashes; ~7 h on a single GPU (gfx1100); read-order EditDist 0.1240. The - lm-llamacpp row is reproduced on windows-hip (Strix Halo): 1649/1651 ok, -2 empty (cover/decoration pages), Overall 95.46 identical to the Linux vLLM -standalone score. See docs/windows-vlm-phase2.md. +`vlm-llamacpp` row records a windows-hip run on Strix Halo: +1649/1651 non-empty predictions, 2 empty cover/decoration pages, and Overall +95.46. See [Windows VLM Phase 2](docs/windows-vlm-phase2.md). + +### Results — MinerU 3.4 pipeline + +| Model / Backend | Overall | Text Edit | Formula CDM | Table TEDS | +|---|---:|---:|---:|---:| | **ours MinerU 3.4 pipeline (ROCm gfx1100, linux-rocm)** | **86.48** | 0.0566 | 83.07 | 82.04 | | **ours MinerU 3.4 pipeline (Strix Halo, windows-hip)** | **86.59** | 0.0565 | 83.39 | 82.04 | @@ -135,7 +138,7 @@ supplementary card in the same repo — see `model_card.pipeline.json` and ## Reproducibility -[`reproducibility.lock.yaml`](reproducibility.lock.yaml) is the single source of +[`REPRO.yaml`](REPRO.yaml) is the single source of truth — pinned commits, byte-exact weight/GT SHA256 cross-checked against the upstream HF repos, environment versions, and the metric formula. Verified values were populated from the full 1651-page reruns completed on 2026-07-19. @@ -169,7 +172,7 @@ Not affiliated with the MinerU Team / OpenDataLab. ## Known Gaps - The `smoke` backend emits placeholder text, not real OCR. CI/conformance can force `smoke` via `--backend smoke` to validate the adapter contract without a GPU. -- **Windows-HIP pipeline Phase 1** is `verified` with a conformant 1651-page CDM bundle; **Phase 2 VLM (llama.cpp HIP)** is `verified` with Overall 95.46, identical to the Linux vLLM baseline (independently reproduced). +- **Windows-HIP pipeline Phase 1** has a conformant 1651-page CDM bundle and Overall 86.59; **Phase 2 VLM (llama.cpp HIP)** has Overall 95.46. See the Hardware Matrix for producer evidence and platform review; legacy badges are not the trust source. - **Provisioning scripts** (`adapter/setup/`) are stubs that document the steps; they do not automate full environment setup. - **Platform-standard artifacts** for `results/omnidocbench/v16/linux-rocm/` were generated on 2026-07-21 (self-contained CDM bundles for `mineru2.5` and `mineru-pipeline`: `run_summary` + `provenance` + `metric_result` + `run_stats` + a SHA256 `prediction_manifest` + `dataset_identity`). The legacy results under `results/omnidocbench/v1.6/` are retained for historical comparison and prediction-source provenance. Validate any bundle with `omnidocbench-rocm validate-bundle results/omnidocbench/v16/linux-rocm`. - **VLM empty outputs:** 2 of 1651 VLM pages produced empty predictions (recorded as failures + listed in the prediction manifest). diff --git a/README.zh-CN.md b/README.zh-CN.md index 17dc783..8015752 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -8,11 +8,14 @@ [![OmniDocBench v1.6](https://img.shields.io/badge/OmniDocBench-v1.6-blue)](https://github.com/opendatalab/OmniDocBench) [![VLM full](https://img.shields.io/badge/MinerU2.5--Pro%20VLM%20(full)-95.56-green)](#evaluation评测) [![pipeline full](https://img.shields.io/badge/MinerU%203.4%20pipeline%20(full)-86.48-yellowgreen)](#evaluation评测) -[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](reproducibility.lock.yaml) +[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](REPRO.yaml) [![license: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0%20(+MinerU%20terms)-blue)](NOTICE) > 国内用户优先使用镜像与 ModelScope 拉取模型/数据集,速度更稳定。 +其他 AMD 配置的社区复现按中央 ROCmDoc 的 [Producer evidence / Platform review](docs/hardware-matrix.md) 模型记录; +提交新结果请使用[社区 benchmark 清单](docs/community-benchmarks.md)。 + ## Install(安装) 核心包不依赖 GPU,也无平台依赖。 @@ -100,6 +103,7 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ... | Model / Backend | Overall | Text Edit | Formula CDM | Table TEDS | |---|---:|---:|---:|---:| | _official_ MinerU2.5-Pro _(上游 README vlm-engine 行;社区验证,非官方支持)_ | 95.30 | — | — | — | +| **ours MinerU2.5-Pro(vlm-llamacpp,Windows-HIP)** | **95.46** | 0.0373 | 97.01 | 93.11 | | **ours MinerU2.5-Pro(vlm-vllm,ROCm)** | **95.56** | 0.0359 | 96.73 | 93.54 | | ours MinerU2.5-Pro(vlm-transformers,ROCm) | _仅采样_ | | | | @@ -108,8 +112,9 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ... 7 小时;阅读顺序 EditDist 0.1240。Overall 95.56 与公开发布的上游参考区间**一致** (vlm-engine 95.30;delta +0.26 pp —— **非**受控 CUDA-vs-ROCm 对照)。上游锚点 取自上游 README "Local Deployment" 表,属**社区验证、非官方支持** —— 见 -`reproducibility.lock.yaml`(`benchmark.official_reference`)。`windows-hip` -仍为 `community-wanted`(暂无结果)。 +`REPRO.yaml`(`benchmark.official_reference`)。Windows-HIP 的 llama.cpp HIP +结果为 95.46,并提供完整证据 bundle;其 trust 状态见 Hardware Matrix。 +该分数对应不同于 Linux 平台 CDM 95.56 的评分配置,不应据此进行硬件排名。 > **历史分数说明:** 此前独立 `mineru-rocm score` 路径在同一份 1651 页预测上得分为 > **95.46**(Formula CDM 96.46);当前平台 CDM 得分 **95.56**(Formula CDM 96.73)。 @@ -122,15 +127,16 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ... |---|---:|---:|---:|---:| | _official_ MinerU 3.4 pipeline | 86.47 | — | — | — | | **ours MinerU 3.4 pipeline(ROCm gfx1100,linux-rocm)** | **86.48** | 0.0566 | 83.07 | 82.04 | -| windows-hip | _community-wanted_ | | | | +| **ours MinerU 3.4 pipeline(Strix Halo,windows-hip)** | **86.59** | 0.0565 | 83.39 | 82.04 | -Pipeline 结果位于 `results/omnidocbench/v1.6/pipeline/`。已知:1 个空输出页面。 +Linux pipeline 结果位于 `results/omnidocbench/v1.6/pipeline/`;Windows-HIP +bundle 位于 `results/omnidocbench/v16/windows-hip/`。 主 registry card 为 `mineru2.5`(VLM);pipeline 为同仓补充 card —— 见 `model_card.pipeline.json`。 ## Reproducibility(可复现性) -[`reproducibility.lock.yaml`](reproducibility.lock.yaml) 是唯一事实来源 —— +[`REPRO.yaml`](REPRO.yaml) 是唯一事实来源 —— 锁定的 commit、与上游 HF 仓交叉校验的逐字节权重/GT SHA256、环境版本,以及 指标公式。已验证值来自 2026-07-19 完成的全量 1651 页重跑。 @@ -156,7 +162,7 @@ license —— 视为授权不明,请勿再分发。完整分解见 [NOTICE](N ## Known Gaps(已知限制) - `smoke` 后端输出的是占位文本,并非真实 OCR。CI/conformance 可通过 `--backend smoke` 验证适配器契约而无需 GPU。 -- **Windows-HIP** 为 `community-wanted` —— 尚无正式结果。两个 model card 的 `windows-hip` badge 均为 `community-wanted`。 +- **Windows-HIP pipeline 与 VLM** 分别发布了 Overall 86.59 与 95.46 的证据 bundle;Producer evidence 与 Platform review 状态见 Hardware Matrix。这不是 AMD、MinerU 或 OpenDataLab 的官方支持声明。 - **环境准备脚本**(`adapter/setup/`)为桩代码,仅记录手动步骤,未完全自动化环境安装。 - **平台标准 artifacts** 已于 2026-07-21 在 `results/omnidocbench/v16/linux-rocm/` 生成(`mineru2.5` 与 `mineru-pipeline` 的自包含 CDM bundle:`run_summary` + `provenance` + `metric_result` + `run_stats` + SHA256 `prediction_manifest` + `dataset_identity`)。`results/omnidocbench/v1.6/` 下的遗留结果保留用于历史对比与预测来源 provenance。可用 `omnidocbench-rocm validate-bundle results/omnidocbench/v16/linux-rocm` 校验任意 bundle。 - **VLM 空输出:** 1651 个 VLM 页面中 2 个产生空预测(已记录为失败)。 diff --git a/SUPPORT.md b/SUPPORT.md index 0e3881a..075a2a2 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -2,7 +2,7 @@ | Question type | Where | |---|---| -| Bug or unexpected result in THIS repo | [GitHub Issues](https://github.com/AIwork4me/MinerU-ROCm/issues) — include the OmniDocBench page id, the backend, and the `reproducibility.lock.yaml` environment block. | +| Bug or unexpected result in THIS repo | [GitHub Issues](https://github.com/AIwork4me/MinerU-ROCm/issues) — include the OmniDocBench page id, the backend, and the `REPRO.yaml` environment block. | | ROCm / gfx1100 compatibility | Open an issue with `rocm-smi --showproductname` output; mark it `rocm-compat`. | | Upstream MinerU behavior (model quality, pipeline options) | [opendatalab/MinerU](https://github.com/opendatalab/MinerU) — this repo wraps upstream, it does not change the models. | | OmniDocBench scoring / metrics | [opendatalab/OmniDocBench](https://github.com/opendatalab/OmniDocBench). | diff --git a/docs/HANDOFF-windows-hip.md b/docs/HANDOFF-windows-hip.md index 6c2c9d4..faf2a74 100644 --- a/docs/HANDOFF-windows-hip.md +++ b/docs/HANDOFF-windows-hip.md @@ -119,7 +119,7 @@ powershell -ExecutionPolicy Bypass -File eval-infra\01-omnidocbench\setup.ps1 powershell -ExecutionPolicy Bypass -File eval-infra\01-omnidocbench\verify.ps1 ``` The dataset revision **must** be `2b161d0` (the pinned dataset + scorer revision -this repo's `reproducibility.lock.yaml` records). Note the images directory path +this repo's `REPRO.yaml` records). Note the images directory path (`eval-infra\01-omnidocbench\data\images`); you'll pass it to the adapter. ```powershell diff --git a/docs/HANDOFF-windows-vlm-llamacpp.md b/docs/HANDOFF-windows-vlm-llamacpp.md index a4277b4..bbe602f 100644 --- a/docs/HANDOFF-windows-vlm-llamacpp.md +++ b/docs/HANDOFF-windows-vlm-llamacpp.md @@ -727,7 +727,7 @@ $tool = "C:\Users\rocm\miniconda3\envs\mineru-win-rocm\Scripts\omnidocbench-rocm - `README.md` Windows VLM 行; - `docs/windows-vlm-phase2.md`; - `docs/HANDOFF-windows-hip.md`; -- `reproducibility.lock.yaml` 的 +- `REPRO.yaml` 的 `benchmark.windows_hip.full_1651.vlm`; - root `model_card.json` 的 Windows badge/artifact 指针(保留 Linux 成绩); - 新 `model_card.windows-hip.json`; @@ -750,7 +750,7 @@ git status --short ```powershell git add README.md docs eval scripts src tests results ` - model_card.json model_card.windows-hip.json reproducibility.lock.yaml + model_card.json model_card.windows-hip.json REPRO.yaml git commit -m "results(windows-hip): publish MinerU2.5 llama.cpp validation" git push -u origin codex/windows-vlm-phase2 ``` diff --git a/docs/architecture.md b/docs/architecture.md index 4b8471f..6c4639e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -23,7 +23,7 @@ It does not fork model weights or replace the upstream model APIs. | Linux pipeline | In-process MinerU 3.4.4 | PyTorch ROCm; pipeline ONNX components may use CPU | | Linux VLM | OpenAI-compatible vLLM server | vLLM ROCm on one gfx1100 GPU | | Windows pipeline | In-process MinerU 3.4.4 | Windows ROCm PyTorch plus ONNX Runtime DirectML | -| Windows VLM | Not released | Phase 2 serving runtime remains exploratory | +| Windows VLM | OpenAI-compatible llama.cpp server | llama.cpp HIP on one gfx1151 GPU | On Windows, all compatible ONNX sessions are DirectML-first. The known DirectML-incompatible `slanet-plus.onnx` control-flow model is routed through an @@ -38,6 +38,13 @@ and a prediction manifest. Bulk Markdown predictions remain outside Git, but every formal prediction is represented by filename and SHA-256 in its manifest. The primary VLM card is `model_card.json`. The pipeline is a supplementary -result with `model_card.pipeline.json` and platform-specific cards. A -`community` badge is self-attested; promotion to `verified` requires an -independent maintainer reproduction. +result with `model_card.pipeline.json` and platform-specific cards. The +legacy model-card badges remain only for compatibility and are deprecated by +the central ROCmDoc standard. The current trust model keeps producer assurance +separate from central `platform_review`; see +[`ROCMDOC_STANDARD.md`](https://github.com/AIwork4me/OmniDocBench-ROCm/blob/main/contracts/ROCMDOC_STANDARD.md) +and [`COMPATIBILITY.md`](https://github.com/AIwork4me/OmniDocBench-ROCm/blob/main/contracts/COMPATIBILITY.md). +This repository does not assign central review outcomes. The +[Hardware matrix](hardware-matrix.md) displays the two dimensions without +creating another normative vocabulary; neither denotes vendor or upstream +support. diff --git a/docs/benchmark-methodology.md b/docs/benchmark-methodology.md index 4348ee4..bc7506f 100644 --- a/docs/benchmark-methodology.md +++ b/docs/benchmark-methodology.md @@ -4,7 +4,7 @@ > the upstream opendatalab/MinerU models on AMD ROCm and scores them on > OmniDocBench v1.6 — the same dataset + scorer the upstream uses. It does NOT > claim bit-identical results vs a CUDA baseline (no same-engine CUDA control -> exists). The numbers below are reproducible from `reproducibility.lock.yaml`. +> exists). The numbers below are reproducible from `REPRO.yaml`. ## What this repo measures @@ -13,6 +13,7 @@ | MinerU 3.4 pipeline | in-process (MinerUPipelineRunner on ROCm) | **86.48** | 0.0566 | 83.07 | 82.04 | | MinerU 3.4 pipeline | Windows ROCm PyTorch + DirectML ONNX | **86.59** | 0.0565 | 83.39 | 82.04 | | MinerU2.5-Pro VLM | vLLM-on-ROCm `0.16.1.dev0+g89a77b108.d20260317` (http-client) | **95.56** | 0.0359 | 96.73 | 93.54 | +| MinerU2.5-Pro VLM | llama.cpp HIP Q8_0 (windows-hip) | **95.46** | 0.0373 | 97.01 | 93.11 | All rows use OmniDocBench v1.6, 1651 pages, `quick_match`, and CDM. Linux used AMD gfx1100 (Radeon PRO W7900, 48 GB; one GPU per benchmark), ROCm 7.2. Windows @@ -49,7 +50,7 @@ a controlled CUDA-vs-ROCm hardware-level comparison. - **The official anchor is community-verified.** The upstream README "Local Deployment" table records vlm-engine 95.30 / pipeline 86.47; we cite these as the comparison anchors (aligned to that table, not as official support). - See `reproducibility.lock.yaml` (`benchmark.official_reference: source: + See `REPRO.yaml` (`benchmark.official_reference: source: verified`). The prior withdrawn unofficial anchor is no longer cited. ## The Overall formula @@ -85,6 +86,6 @@ mineru-rocm score \ mineru-rocm manifest verify --pred-dir ``` -Every input is pinned in `reproducibility.lock.yaml` (code commit, model weight +Every input is pinned in `REPRO.yaml` (code commit, model weight SHAs, dataset SHA, scorer commit, environment versions). See that file for the exact provenance. diff --git a/docs/community-benchmarks.md b/docs/community-benchmarks.md new file mode 100644 index 0000000..f9f4c6a --- /dev/null +++ b/docs/community-benchmarks.md @@ -0,0 +1,76 @@ +# Community benchmarks + +Use this checklist to propose an AMD GPU / ROCm result for the +[Hardware matrix](hardware-matrix.md). Open a GitHub issue with the information +below and link an immutable result summary. Contributors only need to submit +evidence; they do not need to declare a result `verified` or assign platform +review status. + +Trust is governed by the central ROCmDoc +[`ROCMDOC_STANDARD.md`](https://github.com/AIwork4me/OmniDocBench-ROCm/blob/main/contracts/ROCMDOC_STANDARD.md) +and +[`COMPATIBILITY.md`](https://github.com/AIwork4me/OmniDocBench-ROCm/blob/main/contracts/COMPATIBILITY.md). +Under that model, producer assurance records submission completeness +(`submitted` or `evidence-complete`) while `platform_review` is a separate, +downstream action performed by maintainers or the central platform. When a +result is imported into the central platform, its platform review starts as +`not-reviewed`. Contributors do not assign this state or self-assign an +accepted review or reproduction assurance. + +If you use this repository's evaluation path, reuse its model cards and +engine-produced bundle files (`run_summary`, `provenance`, `metric_result`, +`run_stats`, prediction manifest, and dataset identity). Do not create a second +result format just for the submission. + +## Required information + +### Platform and hardware + +- GPU or APU name and GPU architecture, such as `gfx1100` or `gfx1151` +- OS and kernel version +- ROCm and HIP versions + +### Workload + +- MinerU version +- model ID and exact model revision +- backend and backend version or commit +- dtype and any runtime parameters that can affect output + +### Evaluation + +- benchmark name/version, dataset revision, and scorer revision +- pages attempted, succeeded, and failed +- Overall and relevant submetrics +- inference wall time + +### Reproducibility + +- MinerU-ROCm repository/adapter commit, if used +- prediction manifest SHA-256 or an equivalent integrity record +- whether MinerU was patched +- whether ROCm or the runtime was patched +- whether `HSA_OVERRIDE_GFX_VERSION` or a similar override was used +- immutable evidence URL: a release/tag or commit-pinned permalink to the result + summary and provenance + +Raw predictions, downloadable result artifacts, performance logs, and +memory/resource observations are welcome but optional. Raw predictions are not +a requirement for submitting evidence. + +## Shortest submission flow + +1. Complete the benchmark and preserve the fields above. +2. Publish a concise result summary and provenance at an immutable tag or + commit permalink. +3. Open an issue and link that evidence. Maintainers review the revisions, + counts, integrity record, score semantics, and any patches or overrides. +4. Maintainers may add a Hardware Matrix row with the exact platform/backend + path and producer evidence state. Any later platform review remains a + separate central-platform action. + +[Issue #21](https://github.com/AIwork4me/MinerU-ROCm/issues/21) is an external +community result submission example. Issue or draft-PR discussion may document +the review process, while Hardware Matrix inclusion must link finalized, +immutable evidence. Inclusion records one tested configuration and is not an +official hardware-support claim or a cross-platform ranking. diff --git a/docs/hardware-matrix.md b/docs/hardware-matrix.md index 6ec19bc..b6cbbd8 100644 --- a/docs/hardware-matrix.md +++ b/docs/hardware-matrix.md @@ -1,14 +1,32 @@ # Hardware matrix -These entries describe completed, evidence-backed runs. They are not general -support claims for untested AMD architectures. +These entries describe completed benchmark runs on specific hardware and +software paths. They are evidence records, not general support claims for an +AMD architecture, ROCm release, or vendor product. -| Platform | Hardware | Backend | Software path | OmniDocBench v1.6 | Status | -|---|---|---|---|---:|---| -| `linux-rocm` | Radeon PRO W7900, gfx1100, 48 GB | MinerU2.5-Pro VLM | vLLM ROCm, bf16, one GPU | 95.56 | `community` | -| `linux-rocm` | Radeon PRO W7900, gfx1100, 48 GB | MinerU 3.4.4 pipeline | PyTorch ROCm | 86.48 | `community` | -| `windows-hip` | Ryzen AI MAX+ 395 / Radeon 8060S, shared memory | MinerU 3.4.4 pipeline | Windows ROCm PyTorch + DirectML | 86.59 | `community` | -| `windows-hip` | Not yet established | MinerU2.5-Pro VLM | Serving runtime undecided | — | `community-wanted` | +| Platform | Hardware | Architecture | Workload / software path | OmniDocBench v1.6 Overall | Producer evidence | Central platform review | +|---|---|---|---|---:|---|---| +| `linux-rocm` | Radeon PRO W7900, 48 GB | `gfx1100` | MinerU2.5-Pro VLM; vLLM ROCm, bf16, one GPU | 95.56 | `evidence-complete` ([card](../model_card.json), [bundle](../results/omnidocbench/v16/linux-rocm/README.md)) | — | +| `linux-rocm` | Radeon PRO W7900, 48 GB | `gfx1100` | MinerU 3.4.4 pipeline; PyTorch ROCm | 86.48 | `evidence-complete` ([card](../model_card.pipeline.json), [bundle](../results/omnidocbench/v16/linux-rocm/README.md)) | — | +| `windows-hip` | Ryzen AI MAX+ 395 / Radeon 8060S, shared memory | `gfx1151` | MinerU 3.4.4 pipeline; Windows ROCm PyTorch + DirectML | 86.59 | `evidence-complete` ([card](../model_card.pipeline.windows-hip.json), [bundle](../results/omnidocbench/v16/windows-hip/README.md)) | — | +| `windows-hip` | Ryzen AI MAX+ 395 / Radeon 8060S, shared memory | `gfx1151` | MinerU2.5-Pro VLM; llama.cpp HIP Q8_0 | 95.46 | `evidence-complete` ([card](../model_card.windows-hip.json), [bundle](../results/omnidocbench/v16/windows-hip/README.md)) | — | + +## Trust model + +This table uses the central ROCmDoc split between producer evidence and +platform review. The authoritative terminology and criteria live in +[`ROCMDOC_STANDARD.md`](https://github.com/AIwork4me/OmniDocBench-ROCm/blob/main/contracts/ROCMDOC_STANDARD.md), +especially section 8. Producer evidence describes the submitted result bundle; +platform review is a separate record owned by the central platform. Here, +`evidence-complete` is an informational compatibility projection from the +repository's existing provenance-complete legacy results and bundles, not a +canonical v2 producer-assurance record or a central platform-review outcome. +`—` means this repository does not claim or link a central `platform_review` +record. + +Legacy model-card badges remain for compatibility only and are not used as the +trust source for this table. These records do not imply official support from +AMD, MinerU, or OpenDataLab. ## Windows execution detail @@ -19,4 +37,5 @@ support claims for untested AMD architectures. because its ONNX control-flow graph is incompatible with DirectML 1.24.4. See the platform model cards and the bundle README files under -`results/omnidocbench/v16/` for exact submetrics and artifact paths. +`results/omnidocbench/v16/` for exact submetrics and artifact paths. To propose +another AMD GPU result, follow [Community benchmarks](community-benchmarks.md). diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 4499be0..6c518e3 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -53,4 +53,4 @@ This repo ships **two** model cards for one upstream model family (MinerU 2.5 / The platform's `hub/registry.yaml` carries **one row per `model_id`**, and that row is the VLM (`mineru2.5`). The pipeline (`mineru-pipeline`) is a secondary card inside this same repo, surfaced via `model_card.pipeline.json` + the README table — **so no new registry row is needed for the pipeline**. The VLM result now fills the primary `model_card.json` (Overall **95.56** platform CDM, badge linux-rocm `community`); the pipeline card stays as the secondary entry. (The prior standalone-path VLM Overall was **95.46** — Formula CDM 96.46 — on the same 1651 predictions; superseded by the platform CDM 95.56, see `docs/reproducibility.md`.) -> **Registry update note:** the `hub/registry.yaml` lives in the **separate** platform repo [`OmniDocBench-ROCm`](https://github.com/AIwork4me/OmniDocBench-ROCm) and carries the primary `mineru2.5` VLM row. The VLM's `windows-hip` entry remains `community-wanted`; the supplementary pipeline Windows result is recorded here in `model_card.pipeline.windows-hip.json` and its conformant bundle. (`verified` still requires maintainer reproduction.) +> **Registry update note:** the `hub/registry.yaml` lives in the **separate** platform repo [`OmniDocBench-ROCm`](https://github.com/AIwork4me/OmniDocBench-ROCm) and carries the primary `mineru2.5` VLM row. This repository records Windows-HIP VLM and pipeline result evidence; the central trust model separately tracks producer assurance and `platform_review`. Legacy badges remain compatibility metadata and do not establish a central review outcome. diff --git a/docs/known-gaps.md b/docs/known-gaps.md index 4eafef2..53887d4 100644 --- a/docs/known-gaps.md +++ b/docs/known-gaps.md @@ -1,8 +1,9 @@ # Known gaps -Track the open items for `MinerU-ROCm` here. A `verified` badge requires these -to be resolved or explicitly scoped. This list covers the pipeline path -(MinerU 3.4); VLM-path gaps are tracked separately. +Track the open items for `MinerU-ROCm` here. This list covers the pipeline path +(MinerU 3.4); VLM-path gaps are tracked separately. Result trust is represented +by producer evidence and central platform review as linked from the +[Hardware matrix](hardware-matrix.md). - **ONNX tables run on CPU on Linux ROCm.** The RapidOCR / RapidTable ONNX models in the pipeline use the CPU execution provider on the tested Linux @@ -11,9 +12,11 @@ to be resolved or explicitly scoped. This list covers the pipeline path - **`MINERU_FORMULA_CH_SUPPORT` must stay off.** Setting it to `true` pulls native PaddlePaddle-GPU, which has no ROCm wheel for this stack. Keep it `false` and use the fallback formula renderer. -- **Windows-HIP VLM is now verified.** Both Phase 1 pipeline (Overall 86.59) - and Phase 2 VLM llama.cpp HIP (Overall 95.46, identical to the Linux vLLM - CDM-scored 95.56 baseline) are complete on Strix Halo with conformant CDM bundles. +- **Windows-HIP results are published.** Phase 1 pipeline (Overall 86.59) and + Phase 2 VLM llama.cpp HIP (Overall 95.46) are complete on Strix Halo with + conformant CDM bundles. No central platform-review outcome is claimed here. + The VLM score matches the prior Linux standalone 95.46 score; the canonical + Linux platform-CDM score is 95.56 under a different scoring configuration. ## Deferred upstream-readiness backlog @@ -30,4 +33,3 @@ These items do not invalidate the committed full-set result bundles: W7900 GPUs; `gpu_count_per_benchmark: 1` is the load-bearing fact. - **`gpu-smoke.yml`** is not present because a safe self-hosted gfx1100 runner topology has not been established. -- **Windows-HIP VLM** is `verified` (Overall 95.46, Phase 2 complete, independently reproduced); the pipeline result is `verified` (Overall 86.59, Phase 1 complete, independently reproduced). diff --git a/docs/reproducibility.md b/docs/reproducibility.md index d81b30d..f6fe520 100644 --- a/docs/reproducibility.md +++ b/docs/reproducibility.md @@ -1,6 +1,6 @@ # Reproducibility -A score is only meaningful if someone else can reproduce it from the committed repo. The canonical published results are the **OmniDocBench-ROCm platform CDM bundles** under `results/omnidocbench/v16/{linux-rocm,windows-hip}/`; the standalone `mineru-rocm` CLI remains available for developer debugging. `reproducibility.lock.yaml` is the single source of truth (pinned commits, byte-exact weight/GT SHAs, scorer commit, environments, the metric formula, the official anchors, and the ROCm recipes). +A score is only meaningful if someone else can reproduce it from the committed repo. The canonical published results are the **OmniDocBench-ROCm platform CDM bundles** under `results/omnidocbench/v16/{linux-rocm,windows-hip}/`; the standalone `mineru-rocm` CLI remains available for developer debugging. `REPRO.yaml` is the single source of truth (pinned commits, byte-exact weight/GT SHAs, scorer commit, environments, the metric formula, the official anchors, and the ROCm recipes). ## Results (OmniDocBench v1.6, full 1651 pages) @@ -9,6 +9,7 @@ A score is only meaningful if someone else can reproduce it from the committed r | MinerU 3.4 pipeline | **86.48** | 0.0566 | 83.07 | 82.04 | 0.1534 | | MinerU 3.4 pipeline (windows-hip, Strix Halo) | **86.59** | 0.0565 | 83.39 | 82.04 | 0.1531 | | MinerU2.5-Pro VLM (vLLM-on-ROCm, platform CDM) | **95.56** | 0.0359 | 96.73 | 93.54 | 0.1240 | +| MinerU2.5-Pro VLM (windows-hip, llama.cpp HIP) | **95.46** | 0.0373 | 97.01 | 93.11 | 0.1225 | Prior standalone `mineru-rocm score` path: VLM Overall **95.46** (Formula CDM 96.46, Text 0.0360, read-order 0.1236) on the **same 1651 predictions** — Δ −0.10 pp vs the platform CDM result, entirely the Formula-CDM submetric (CDM scoring configuration), not new inference. @@ -89,4 +90,4 @@ The tested server launch script is tracked in `examples/serve_vlm_vllm.sh`; its ## Provenance in the lock -`reproducibility.lock.yaml` records: the `mineru_rocm.release` tag + release commit + annotated-tag object SHA, the **per-run benchmark commits** (`mineru_rocm.benchmark_run_commits`, also recorded in each `run_manifest.json`), the **upstream `mineru`/`mineru_vl_utils` git commits** (resolved via `git ls-remote` against the release tags), byte-exact weight + GT SHAs, the scorer commit, both venvs' full environment, the official anchors, and the metric formula. Deferred fields (`canary_*`, `table_sha256`) are annotated `→ docs/known-gaps.md`. +`REPRO.yaml` records: the `mineru_rocm.release` tag + release commit + annotated-tag object SHA, the **per-run benchmark commits** (`mineru_rocm.benchmark_run_commits`, also recorded in each `run_manifest.json`), the **upstream `mineru`/`mineru_vl_utils` git commits** (resolved via `git ls-remote` against the release tags), byte-exact weight + GT SHAs, the scorer commit, both venvs' full environment, the official anchors, and the metric formula. Deferred fields (`canary_*`, `table_sha256`) are annotated `→ docs/known-gaps.md`. diff --git a/model_card.pipeline.json b/model_card.pipeline.json index 5fbb74b..c3075b1 100644 --- a/model_card.pipeline.json +++ b/model_card.pipeline.json @@ -19,7 +19,7 @@ "vram": "48 GB per GPU", "rocm_driver": "ROCm 7.2.1" }, - "note": "Supplementary model card for the MinerU 3.4 pipeline. The top-level score and hardware remain the canonical linux-rocm result recorded in reproducibility.lock.yaml; the platform-specific Windows-HIP result is in model_card.pipeline.windows-hip.json. This is not the canonical OmniDocBench-ROCm registry card — see model_card.json (mineru2.5) for the primary registry card.", + "note": "Supplementary model card for the MinerU 3.4 pipeline. The top-level score and hardware remain the canonical linux-rocm result recorded in REPRO.yaml; the platform-specific Windows-HIP result is in model_card.pipeline.windows-hip.json. This is not the canonical OmniDocBench-ROCm registry card — see model_card.json (mineru2.5) for the primary registry card.", "artifacts": { "provenance": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_provenance.json", "run_summary": "results/omnidocbench/v16/linux-rocm/mineru-pipeline_v16_quick_match_cdm_run_summary.json", diff --git a/results/omnidocbench/v16/windows-hip/README.md b/results/omnidocbench/v16/windows-hip/README.md index c76929d..92faa03 100644 --- a/results/omnidocbench/v16/windows-hip/README.md +++ b/results/omnidocbench/v16/windows-hip/README.md @@ -16,7 +16,8 @@ VLM via llama.cpp HIP). | Reading-order EditDist | 0.1225 | Overall = `((1 - text_EditDist) * 100 + formula_CDM * 100 + table_TEDS * 100) / 3`. -The VLM result is **identical** to the Linux vLLM baseline (95.46). +The VLM result matches the prior Linux standalone score (95.46). The canonical +Linux platform-CDM score is 95.56 under a different scoring configuration. - Backend: llama.cpp HIP Q8_0 (build `b9892`, commit `ee445f93`), context 8192, greedy sampling, flash attention enabled @@ -36,7 +37,7 @@ The VLM result is **identical** to the Linux vLLM baseline (95.46). | Table TEDS | 82.04 | | Reading-order EditDist | 0.1531 | -Overall = `((1 - text_EditDist) * 100 + formula_CDM + table_TEDS) / 3`. +Overall = `((1 - text_EditDist) * 100 + formula_CDM * 100 + table_TEDS * 100) / 3`. The score is +0.11 pp from the linux-rocm pipeline result (86.48) and passes the Phase 1 tolerance of +/-1.0 pp around the 86.47 upstream reference.