Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,16 @@ The `smoke` backend (no GPU) should always pass — keep it working.

- `make smoke-test` is green.
- `make publish` reports `CONFORMANT`.
- `model_card.json` reflects reality (hardware, badge, artifacts) — don't claim `verified` without committed, reproducible results.
- Model cards reflect reality (hardware, legacy badge, artifacts); result submissions must not assign central platform review or promote legacy badges.
- Update `docs/known-gaps.md` — close resolved bullets, add new ones.

## Reporting results

Commit the engine-produced `run_summary.json` + `provenance.json` under `results/omnidocbench/v1.6/<platform>/` and update `model_card.json.artifacts` to point at them. A maintainer will review for the `verified` badge per the policy in `omnidocbench-rocm/contracts/badge-policy.md`.
For a result produced in this repository, commit the engine-produced evidence
bundle under `results/omnidocbench/v16/<platform>/` and point the applicable
model card at it. For an independent result, follow the short
[community benchmark checklist](docs/community-benchmarks.md) and provide an
immutable evidence permalink. Raw predictions are useful but optional.

## Code of conduct

Expand Down
23 changes: 13 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,14 @@
[![OmniDocBench v1.6](https://img.shields.io/badge/OmniDocBench-v1.6-blue)](https://github.com/opendatalab/OmniDocBench)
[![VLM full](https://img.shields.io/badge/MinerU2.5--Pro%20VLM%20(full)-95.56-green)](#evaluation)
[![pipeline full](https://img.shields.io/badge/MinerU%203.4%20pipeline%20(full)-86.48-yellowgreen)](#evaluation)
[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](reproducibility.lock.yaml)
[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](REPRO.yaml)
[![license: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0%20(+MinerU%20terms)-blue)](NOTICE)

## At a glance

- **What it is.** Tooling to run opendatalab MinerU (3.4 pipeline + 2.5-Pro VLM) on AMD ROCm and score it on OmniDocBench v1.6.
- **Where verified.** AMD **gfx1100 (RDNA3), Radeon PRO W7900 (48 GB), ROCm 7.2**, bf16. Host has 4x W7900; **each benchmark used 1 GPU (no tensor parallel).**
- **Where benchmarked.** Linux ROCm: AMD **gfx1100 (RDNA3), Radeon PRO W7900 (48 GB), ROCm 7.2**, bf16, one GPU per run. Windows-HIP: **gfx1151, Ryzen AI MAX+ 395 / Radeon 8060S, Windows ROCm 7.2.1**.
- **Community coverage.** Reproductions on additional AMD configurations are tracked with the central ROCmDoc [producer-evidence / platform-review model](docs/hardware-matrix.md); see the [submission checklist](docs/community-benchmarks.md).
- **Most reliable results.** **MinerU2.5-Pro VLM (vLLM-on-ROCm) full 1651 = 95.56 Overall** (platform CDM-scored; prior standalone score 95.46 from the same prediction set — Δ +0.10 pp, entirely the Formula-CDM submetric 96.46→96.73); **MinerU 3.4 pipeline full 1651 = 86.48 Overall**.
- **Most important limitation.** **Not precision-aligned.** No same-engine CUDA control exists; the upstream headline may be measured with a different engine. The official anchor (vlm-engine 95.30) is aligned to the upstream README "Local Deployment" table and is **community-verified, not official support**.
- **Upstream.** This is a port OF [opendatalab/MinerU](https://github.com/opendatalab/MinerU); the [omnidocbench-rocm](https://github.com/AIwork4me/OmniDocBench-ROCm) engine is one *optional* consumer (install the `[platform]` extra), not the definition of this repo.
Expand Down Expand Up @@ -116,13 +117,15 @@ developer debugging — see `docs/reproducibility.md` for the full recipe.

The `vlm-vllm` row is reproduced on linux-rocm (self-attested, `badge: community`):
1651/1651 pages attempted, 1649 non-empty predictions (2 empty), no process
crashes; ~7 h on a single GPU (gfx1100); read-order EditDist 0.1240.
Overall 95.56 is consistent with the published upstream reference range
1651/1651 pages attempted, 1649 non-empty predictions (2 empty), no process
crashes; ~7 h on a single GPU (gfx1100); read-order EditDist 0.1240. The
lm-llamacpp row is reproduced on windows-hip (Strix Halo): 1649/1651 ok,
2 empty (cover/decoration pages), Overall 95.46 identical to the Linux vLLM
standalone score. See docs/windows-vlm-phase2.md.
`vlm-llamacpp` row records a windows-hip run on Strix Halo:
1649/1651 non-empty predictions, 2 empty cover/decoration pages, and Overall
95.46. See [Windows VLM Phase 2](docs/windows-vlm-phase2.md).

### Results — MinerU 3.4 pipeline

| Model / Backend | Overall | Text Edit | Formula CDM | Table TEDS |
|---|---:|---:|---:|---:|
| **ours MinerU 3.4 pipeline (ROCm gfx1100, linux-rocm)** | **86.48** | 0.0566 | 83.07 | 82.04 |
| **ours MinerU 3.4 pipeline (Strix Halo, windows-hip)** | **86.59** | 0.0565 | 83.39 | 82.04 |

Expand All @@ -135,7 +138,7 @@ supplementary card in the same repo — see `model_card.pipeline.json` and

## Reproducibility

[`reproducibility.lock.yaml`](reproducibility.lock.yaml) is the single source of
[`REPRO.yaml`](REPRO.yaml) is the single source of
truth — pinned commits, byte-exact weight/GT SHA256 cross-checked against the
upstream HF repos, environment versions, and the metric formula. Verified values
were populated from the full 1651-page reruns completed on 2026-07-19.
Expand Down Expand Up @@ -169,7 +172,7 @@ Not affiliated with the MinerU Team / OpenDataLab.
## Known Gaps

- The `smoke` backend emits placeholder text, not real OCR. CI/conformance can force `smoke` via `--backend smoke` to validate the adapter contract without a GPU.
- **Windows-HIP pipeline Phase 1** is `verified` with a conformant 1651-page CDM bundle; **Phase 2 VLM (llama.cpp HIP)** is `verified` with Overall 95.46, identical to the Linux vLLM baseline (independently reproduced).
- **Windows-HIP pipeline Phase 1** has a conformant 1651-page CDM bundle and Overall 86.59; **Phase 2 VLM (llama.cpp HIP)** has Overall 95.46. See the Hardware Matrix for producer evidence and platform review; legacy badges are not the trust source.
- **Provisioning scripts** (`adapter/setup/`) are stubs that document the steps; they do not automate full environment setup.
- **Platform-standard artifacts** for `results/omnidocbench/v16/linux-rocm/` were generated on 2026-07-21 (self-contained CDM bundles for `mineru2.5` and `mineru-pipeline`: `run_summary` + `provenance` + `metric_result` + `run_stats` + a SHA256 `prediction_manifest` + `dataset_identity`). The legacy results under `results/omnidocbench/v1.6/` are retained for historical comparison and prediction-source provenance. Validate any bundle with `omnidocbench-rocm validate-bundle results/omnidocbench/v16/linux-rocm`.
- **VLM empty outputs:** 2 of 1651 VLM pages produced empty predictions (recorded as failures + listed in the prediction manifest).
Expand Down
20 changes: 13 additions & 7 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,14 @@
[![OmniDocBench v1.6](https://img.shields.io/badge/OmniDocBench-v1.6-blue)](https://github.com/opendatalab/OmniDocBench)
[![VLM full](https://img.shields.io/badge/MinerU2.5--Pro%20VLM%20(full)-95.56-green)](#evaluation评测)
[![pipeline full](https://img.shields.io/badge/MinerU%203.4%20pipeline%20(full)-86.48-yellowgreen)](#evaluation评测)
[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](reproducibility.lock.yaml)
[![status: evaluation-backed](https://img.shields.io/badge/status-evaluation--backed-blue)](REPRO.yaml)
[![license: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0%20(+MinerU%20terms)-blue)](NOTICE)

> 国内用户优先使用镜像与 ModelScope 拉取模型/数据集,速度更稳定。

其他 AMD 配置的社区复现按中央 ROCmDoc 的 [Producer evidence / Platform review](docs/hardware-matrix.md) 模型记录;
提交新结果请使用[社区 benchmark 清单](docs/community-benchmarks.md)。

## Install(安装)

核心包不依赖 GPU,也无平台依赖。
Expand Down Expand Up @@ -100,6 +103,7 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ...
| Model / Backend | Overall | Text Edit | Formula CDM | Table TEDS |
|---|---:|---:|---:|---:|
| _official_ MinerU2.5-Pro _(上游 README vlm-engine 行;社区验证,非官方支持)_ | 95.30 | — | — | — |
| **ours MinerU2.5-Pro(vlm-llamacpp,Windows-HIP)** | **95.46** | 0.0373 | 97.01 | 93.11 |
| **ours MinerU2.5-Pro(vlm-vllm,ROCm)** | **95.56** | 0.0359 | 96.73 | 93.54 |
| ours MinerU2.5-Pro(vlm-transformers,ROCm) | _仅采样_ | | | |

Expand All @@ -108,8 +112,9 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ...
7 小时;阅读顺序 EditDist 0.1240。Overall 95.56 与公开发布的上游参考区间**一致**
(vlm-engine 95.30;delta +0.26 pp —— **非**受控 CUDA-vs-ROCm 对照)。上游锚点
取自上游 README "Local Deployment" 表,属**社区验证、非官方支持** —— 见
`reproducibility.lock.yaml`(`benchmark.official_reference`)。`windows-hip`
仍为 `community-wanted`(暂无结果)。
`REPRO.yaml`(`benchmark.official_reference`)。Windows-HIP 的 llama.cpp HIP
结果为 95.46,并提供完整证据 bundle;其 trust 状态见 Hardware Matrix。
该分数对应不同于 Linux 平台 CDM 95.56 的评分配置,不应据此进行硬件排名。

> **历史分数说明:** 此前独立 `mineru-rocm score` 路径在同一份 1651 页预测上得分为
> **95.46**(Formula CDM 96.46);当前平台 CDM 得分 **95.56**(Formula CDM 96.73)。
Expand All @@ -122,15 +127,16 @@ omnidocbench-rocm publish --predictions-dir <真实预测目录> ...
|---|---:|---:|---:|---:|
| _official_ MinerU 3.4 pipeline | 86.47 | — | — | — |
| **ours MinerU 3.4 pipeline(ROCm gfx1100,linux-rocm)** | **86.48** | 0.0566 | 83.07 | 82.04 |
| windows-hip | _community-wanted_ | | | |
| **ours MinerU 3.4 pipeline(Strix Halo,windows-hip)** | **86.59** | 0.0565 | 83.39 | 82.04 |

Pipeline 结果位于 `results/omnidocbench/v1.6/pipeline/`。已知:1 个空输出页面。
Linux pipeline 结果位于 `results/omnidocbench/v1.6/pipeline/`;Windows-HIP
bundle 位于 `results/omnidocbench/v16/windows-hip/`。
主 registry card 为 `mineru2.5`(VLM);pipeline 为同仓补充 card —— 见
`model_card.pipeline.json`。

## Reproducibility(可复现性)

[`reproducibility.lock.yaml`](reproducibility.lock.yaml) 是唯一事实来源 ——
[`REPRO.yaml`](REPRO.yaml) 是唯一事实来源 ——
锁定的 commit、与上游 HF 仓交叉校验的逐字节权重/GT SHA256、环境版本,以及
指标公式。已验证值来自 2026-07-19 完成的全量 1651 页重跑。

Expand All @@ -156,7 +162,7 @@ license —— 视为授权不明,请勿再分发。完整分解见 [NOTICE](N
## Known Gaps(已知限制)

- `smoke` 后端输出的是占位文本,并非真实 OCR。CI/conformance 可通过 `--backend smoke` 验证适配器契约而无需 GPU。
- **Windows-HIP** 为 `community-wanted` —— 尚无正式结果。两个 model card 的 `windows-hip` badge 均为 `community-wanted`。
- **Windows-HIP pipeline 与 VLM** 分别发布了 Overall 86.59 与 95.46 的证据 bundle;Producer evidence 与 Platform review 状态见 Hardware Matrix。这不是 AMD、MinerU 或 OpenDataLab 的官方支持声明。
- **环境准备脚本**(`adapter/setup/`)为桩代码,仅记录手动步骤,未完全自动化环境安装。
- **平台标准 artifacts** 已于 2026-07-21 在 `results/omnidocbench/v16/linux-rocm/` 生成(`mineru2.5` 与 `mineru-pipeline` 的自包含 CDM bundle:`run_summary` + `provenance` + `metric_result` + `run_stats` + SHA256 `prediction_manifest` + `dataset_identity`)。`results/omnidocbench/v1.6/` 下的遗留结果保留用于历史对比与预测来源 provenance。可用 `omnidocbench-rocm validate-bundle results/omnidocbench/v16/linux-rocm` 校验任意 bundle。
- **VLM 空输出:** 1651 个 VLM 页面中 2 个产生空预测(已记录为失败)。
Expand Down
2 changes: 1 addition & 1 deletion SUPPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

| Question type | Where |
|---|---|
| Bug or unexpected result in THIS repo | [GitHub Issues](https://github.com/AIwork4me/MinerU-ROCm/issues) — include the OmniDocBench page id, the backend, and the `reproducibility.lock.yaml` environment block. |
| Bug or unexpected result in THIS repo | [GitHub Issues](https://github.com/AIwork4me/MinerU-ROCm/issues) — include the OmniDocBench page id, the backend, and the `REPRO.yaml` environment block. |
| ROCm / gfx1100 compatibility | Open an issue with `rocm-smi --showproductname` output; mark it `rocm-compat`. |
| Upstream MinerU behavior (model quality, pipeline options) | [opendatalab/MinerU](https://github.com/opendatalab/MinerU) — this repo wraps upstream, it does not change the models. |
| OmniDocBench scoring / metrics | [opendatalab/OmniDocBench](https://github.com/opendatalab/OmniDocBench). |
Expand Down
2 changes: 1 addition & 1 deletion docs/HANDOFF-windows-hip.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ powershell -ExecutionPolicy Bypass -File eval-infra\01-omnidocbench\setup.ps1
powershell -ExecutionPolicy Bypass -File eval-infra\01-omnidocbench\verify.ps1
```
The dataset revision **must** be `2b161d0` (the pinned dataset + scorer revision
this repo's `reproducibility.lock.yaml` records). Note the images directory path
this repo's `REPRO.yaml` records). Note the images directory path
(`eval-infra\01-omnidocbench\data\images`); you'll pass it to the adapter.

```powershell
Expand Down
4 changes: 2 additions & 2 deletions docs/HANDOFF-windows-vlm-llamacpp.md
Original file line number Diff line number Diff line change
Expand Up @@ -727,7 +727,7 @@ $tool = "C:\Users\rocm\miniconda3\envs\mineru-win-rocm\Scripts\omnidocbench-rocm
- `README.md` Windows VLM 行;
- `docs/windows-vlm-phase2.md`;
- `docs/HANDOFF-windows-hip.md`;
- `reproducibility.lock.yaml` 的
- `REPRO.yaml` 的
`benchmark.windows_hip.full_1651.vlm`;
- root `model_card.json` 的 Windows badge/artifact 指针(保留 Linux 成绩);
- 新 `model_card.windows-hip.json`;
Expand All @@ -750,7 +750,7 @@ git status --short

```powershell
git add README.md docs eval scripts src tests results `
model_card.json model_card.windows-hip.json reproducibility.lock.yaml
model_card.json model_card.windows-hip.json REPRO.yaml
git commit -m "results(windows-hip): publish MinerU2.5 llama.cpp validation"
git push -u origin codex/windows-vlm-phase2
```
Expand Down
15 changes: 11 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ It does not fork model weights or replace the upstream model APIs.
| Linux pipeline | In-process MinerU 3.4.4 | PyTorch ROCm; pipeline ONNX components may use CPU |
| Linux VLM | OpenAI-compatible vLLM server | vLLM ROCm on one gfx1100 GPU |
| Windows pipeline | In-process MinerU 3.4.4 | Windows ROCm PyTorch plus ONNX Runtime DirectML |
| Windows VLM | Not released | Phase 2 serving runtime remains exploratory |
| Windows VLM | OpenAI-compatible llama.cpp server | llama.cpp HIP on one gfx1151 GPU |

On Windows, all compatible ONNX sessions are DirectML-first. The known
DirectML-incompatible `slanet-plus.onnx` control-flow model is routed through an
Expand All @@ -38,6 +38,13 @@ and a prediction manifest. Bulk Markdown predictions remain outside Git, but
every formal prediction is represented by filename and SHA-256 in its manifest.

The primary VLM card is `model_card.json`. The pipeline is a supplementary
result with `model_card.pipeline.json` and platform-specific cards. A
`community` badge is self-attested; promotion to `verified` requires an
independent maintainer reproduction.
result with `model_card.pipeline.json` and platform-specific cards. The
legacy model-card badges remain only for compatibility and are deprecated by
the central ROCmDoc standard. The current trust model keeps producer assurance
separate from central `platform_review`; see
[`ROCMDOC_STANDARD.md`](https://github.com/AIwork4me/OmniDocBench-ROCm/blob/main/contracts/ROCMDOC_STANDARD.md)
and [`COMPATIBILITY.md`](https://github.com/AIwork4me/OmniDocBench-ROCm/blob/main/contracts/COMPATIBILITY.md).
This repository does not assign central review outcomes. The
[Hardware matrix](hardware-matrix.md) displays the two dimensions without
creating another normative vocabulary; neither denotes vendor or upstream
support.
7 changes: 4 additions & 3 deletions docs/benchmark-methodology.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
> the upstream opendatalab/MinerU models on AMD ROCm and scores them on
> OmniDocBench v1.6 — the same dataset + scorer the upstream uses. It does NOT
> claim bit-identical results vs a CUDA baseline (no same-engine CUDA control
> exists). The numbers below are reproducible from `reproducibility.lock.yaml`.
> exists). The numbers below are reproducible from `REPRO.yaml`.

## What this repo measures

Expand All @@ -13,6 +13,7 @@
| MinerU 3.4 pipeline | in-process (MinerUPipelineRunner on ROCm) | **86.48** | 0.0566 | 83.07 | 82.04 |
| MinerU 3.4 pipeline | Windows ROCm PyTorch + DirectML ONNX | **86.59** | 0.0565 | 83.39 | 82.04 |
| MinerU2.5-Pro VLM | vLLM-on-ROCm `0.16.1.dev0+g89a77b108.d20260317` (http-client) | **95.56** | 0.0359 | 96.73 | 93.54 |
| MinerU2.5-Pro VLM | llama.cpp HIP Q8_0 (windows-hip) | **95.46** | 0.0373 | 97.01 | 93.11 |

All rows use OmniDocBench v1.6, 1651 pages, `quick_match`, and CDM. Linux used
AMD gfx1100 (Radeon PRO W7900, 48 GB; one GPU per benchmark), ROCm 7.2. Windows
Expand Down Expand Up @@ -49,7 +50,7 @@ a controlled CUDA-vs-ROCm hardware-level comparison.
- **The official anchor is community-verified.** The upstream README "Local
Deployment" table records vlm-engine 95.30 / pipeline 86.47; we cite these
as the comparison anchors (aligned to that table, not as official support).
See `reproducibility.lock.yaml` (`benchmark.official_reference: source:
See `REPRO.yaml` (`benchmark.official_reference: source:
verified`). The prior withdrawn unofficial anchor is no longer cited.

## The Overall formula
Expand Down Expand Up @@ -85,6 +86,6 @@ mineru-rocm score \
mineru-rocm manifest verify --pred-dir <output/>
```

Every input is pinned in `reproducibility.lock.yaml` (code commit, model weight
Every input is pinned in `REPRO.yaml` (code commit, model weight
SHAs, dataset SHA, scorer commit, environment versions). See that file for the
exact provenance.
Loading
Loading