Add AlchemBERT results and fix classification scoring compatibility

benchmarks/matbench_v0.1_AlchemBERT/INPUT_AUDIT.md

@@ -0,0 +1,68 @@
+# Independent composition-input identity audit
+
+`verify_inputs.py` compares all 40 v1 input archives for `expt_gap`,
+`expt_is_metal`, `glass`, and `steels` with independent official datasets and
+the official ordered train/test fold IDs. Every archive and reference dataset
+is checked by SHA256 before parsing. Full machine-readable results, source
+identities and individual mismatches are in [input_audit.json](input_audit.json).
+
+This extends the earlier internal partition-consistency check: the archived
+formula or alloy description is now compared with the identified reference
+material, and each separate `output` field is compared with its reference label.
+No model is trained and no benchmark result is modified.
+
+| Task | Unique materials | Row appearances across five folds | Input comparison | Target / instruction mismatches |
+| --- | ---: | ---: | --- | --- |
+| `expt_gap` | 4,604 | 23,020 | One formula typo difference, present once in each fold | 0 / 0 |
+| `expt_is_metal` | 4,921 | 24,605 | The same formula typo difference, once in each fold | 0 / 0 |
+| `glass` | 5,680 | 28,400 | Exact formula-prefix agreement | 0 / 0 |
+| `steels` | 312 | 1,560 | All element sets agree; percentages agree within displayed rounding precision | 0 / 0 |
+
+All 77,585 row appearances had the expected three fields, reference-aligned
+targets and task-fixed instructions. No input matched the limited property-word
+scan recorded in the JSON report.
+
+## Explicit treatment of the experimental-dataset typo
+
+The public official Figshare originals for
+[expt_gap](https://api.figshare.com/v2/articles/9765779) and
+[expt_is_metal](https://api.figshare.com/v2/articles/9765803) have their historical
+Matminer checksums. Their bytes differ from the current corrected dataset
+checksums. The script accepts these originals only with the explicit
+`--allow-historical-expt` option and labels them accordingly.
+
+At `mb-expt-gap-1394` and `mb-expt-is-metal-1501`, the historical reference
+formula is `G1128Ga1As0.1P0.9`; the archived input says `Ga1As0.1P0.9`.
+The reference labels are respectively 1.22 and false, and all corresponding
+archive rows agree. These are the only discrepancies found; they are reported,
+not silently normalized.
+
+Matminer's [2020 correction commit](https://github.com/hackingmaterials/matminer/commit/edfcff311fea550c1d361300d40d5c5bdefba2e1)
+specifically attributes the two changed checksums to the `1128G` typo. For
+`expt_gap`, the separately published official `expt_gap_kingsbury` table supplies
+additional evidence: after verifying its checksum, all 4,604 indices and labels
+match the original, and the sole formula change is exactly the correction above.
+Its added MP-ID column is not used. This does not assert that the current
+compressed `matbench_expt_gap` bytes were recovered, or establish the current
+`expt_is_metal` target-table identity.
+
+## Repeating the check
+
+With the pinned Gitee checkout and verified official dataset downloads:
+
+```bash
+python benchmarks/matbench_v0.1_AlchemBERT/verify_inputs.py \
+  --source alchembert-source --dataset-dir official-matbench-data \
+  --allow-historical-expt --output new_input_audit.json
+```
+
+The output must be a new file. Supplying the checksum-verified
+`expt_gap_kingsbury.json.gz` in the same dataset directory enables the additional
+derivative-table comparison. Without `--allow-historical-expt`, historical
+experimental-dataset bytes are rejected.
+
+This audit checks formula prefixes and rounded alloy quantities, not every
+natural-language clause or any structure-task neighborhood description.
+Agreement narrows the sample-identity uncertainty for these archived files;
+it does not establish that the files were used in the original experiments,
+exclude semantic proxies, or prove absence of pretraining-corpus overlap.

benchmarks/matbench_v0.1_AlchemBERT/PAPER_EVIDENCE.md

@@ -0,0 +1,42 @@
+# Publication evidence and remaining provenance gaps
+
+The original citation in `info.json` is retained. Published descriptions supplement
+the recovered code; they do not identify the exact run that produced
+`results.json.gz`.
+
+## Original preprint
+
+The [original ChemRxiv PDF](https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/67540a28085116a133a62b85/original/AlchemBERT__Exploring_Lightweight_Language_Models_for_Materials_Informatics-6.pdf)
+(DOI: `10.26434/chemrxiv-2024-r4dnl`, 13 pages) reports:
+
+| Evidence | Location | Paper-reported detail |
+| --- | --- | --- |
+| Hardware | Methods, page 8 | Nvidia RTX 4090 |
+| Model | Methods, page 8 | BERT-base-cased; first-token representation and a 768-to-1 linear head |
+| Training | Methods, page 8 | 9:1 train/validation split; learning rate 1e-5; batch size 32; early-stopping patience 200 epochs |
+| Text examples | Pages 9–11 | Alloy fractions for steels; formula-based descriptions for other composition tasks; lattice, symmetry and neighbors for structure tasks; fixed task/unit instruction |
+| Pretraining control | Figure 1, page 5 | Predictions without fine-tuning compared with a dummy model |
+
+**Interpretation:** the control does not establish that the pretraining corpus has
+no overlap with benchmark data. Hardware is paper-reported, not verified from
+historical run logs. Runtime, checkpoint hashes, exact input identity and the
+hyperparameter-selection history still require original evidence. The paper's
+template examples cannot recover the missing v1 generator or establish that
+particular input archives were used.
+
+## Later publication and public model assets
+
+The [2025 journal article](https://doi.org/10.1016/j.xcrp.2025.102724) links the
+Gitee repository for Matbench property tasks, and separate resources for
+Matbench-Discovery. The following public metadata were checked:
+
+| Resource | Verified scope |
+| --- | --- |
+| [Figshare 28690583](https://api.figshare.com/v2/articles/28690583), version 1 when checked | One checkpoint explicitly described as Matbench-Discovery |
+| [Figshare 28387640](https://api.figshare.com/v2/articles/28387640), version 1 when checked | WBM IS2RE predictions for Matbench-Discovery |
+| [Zenodo 15985169](https://zenodo.org/records/15985169) | Archive of `WangYuHang-WYH/AlchemBERT` v1.0.0; description specifies Discovery IS2RE |
+
+These assets are evidence for a different benchmark. They do not supply the
+65 task/fold checkpoints required to establish this submission's historical
+training provenance. No Discovery checkpoint or prediction file has been
+substituted for a Matbench property artifact.