Export tool-neutral ARC results and parser evidence, and fix three Molpro defects

[calvinp0]

What changed

Re-homes the ARC-owned result-production pieces from the historical TCKDB integration work onto clean ARC main, without bringing the TCKDB adapter back into ARC, and fixes Molpro and Gaussian defects in files this PR already touches.

• expands tool-neutral calculation, scan, thermo, correction, spin, and TS-path provenance in output.yml
• adds Cartesian Hessian, rich IRC, GSM energy, constraint, and spin parsers
• emits an atomic parser_evidence.json sidecar, schema 1.0, bound to output schema 1.1 by document ID
• preserves and transfers queued xTB-GSM node evidence safely
• delegates opt-in uploads to the standalone tckdb-arc package, with disabled/missing-package no-op behavior
• records the standard-state pressure the entropy, free energy and NASA fit are referred to
• keeps all TCKDB enum, indexing, nesting, and payload translation out of ARC
• reads the Molpro version from modern logs, which reported none since 2021
• writes the Molpro command that exists for each open-shell method
• fixes parse_geometry for Molpro, which raised on every log and reported Bohr as Angstrom
• writes each Gaussian ModRedundant constraint on its own line

13 commits, 50 files, no file touched by more than one commit.

Why

The adapter is moving to its own repository, but ARC must continue to own parsing and publication of the scientific facts generated by ARC jobs. This creates that durable producer/consumer boundary without merging the old tckdb-imp branch wholesale.

The result contract

freq_hessian_method

output.yml gains a per-species/TS key stating how the frequency job's Hessian was built: analytic, finite_difference_gradient, finite_difference_energy, or null.

The archive consuming this record resolves its imaginary-mode noise floor from that fact, but records it only from an explicit keyword (Gaussian Freq=Numer/Numerical/EnOnly, ORCA NumFreq/AnFreq). Gaussian never states its analytic default, so the field was absent on every Gaussian record. ARC, as the producer, knows what it requested and which default applies, and can state it.

The value is derived from ARC's own request — the frequency input deck in preference to the log's echoed route — combined with the documented behaviour of the ESS for that method class. Where the request does not settle the answer, the field is null. It is never inferred from anything weaker:

• Gaussian, bare freq: analytic for HF (RHF/UHF), all DFT functionals including the documented double hybrids, MP2, CIS and CASSCF; gradient differencing for ROHF, MP3, MP4(SDQ), CCD, CCSD, CID, CISD, QCISD and BD; energy differencing for MP4 (which means MP4(SDTQ)), MP5, CCSD(T), QCISD(T), QCISD(TQ), BD(T) and BD(TQ). Composites, semi-empirical and force-field methods yield null, as does any method not on those lists.
• ORCA: Freq is documented as an alias of AnFreq, so it is analytic for HF and non-double-hybrid DFT without RI-JK, and null otherwise; NumFreq is gradient differencing, or energy differencing when combined with NumGrad.
• PySCF: keyed on multiplicity, matching what the script actually branches on.
• Every other ESS, and a job with neither deck nor log, yield null.

Provenance for the applied corrections

output.yml records the atom-energy and bond-additivity corrections it applied, the level they were keyed to and the bac_type, but identified the software only as arkane_git_commit — a git HEAD read from RMG_PATH. A pip or conda installed RMG-Py has no git repository there, so that key came back empty and the record then named no source at all for the corrections.

A new top-level arkane_version sits beside it, mirroring the existing arc_version / arc_git_commit pair. Both fields resolve from one source so they cannot describe different installs: Arkane prints its version and the RMG-Py HEAD in its own log header, which ARC already produces under calcs/statmech/, so that is read first, with rmgpy/version.py under RMG_PATH as the fallback for both together. Logs older than the run's start are ignored, since that directory survives between runs. Either key is null rather than absent when nothing resolves.

importlib.metadata is not a usable source here: RMG-Py ships no dist-info, so it raises PackageNotFoundError even inside a working rmg_env.

The git-commit helper no longer catches every exception. A RMG_PATH that is set but is not a directory now logs a warning, so a misconfiguration is distinguishable from an ordinary non-git install.

These name the Arkane that ARC invoked. They are not a claim about what it produced: a run that started and then failed reports the same identity as one that completed, and the correction tables can be empty while both fields are populated.

The evidence sidecar's producer block carries the same two values beside ARC's own, under the same names the output document uses, so a reader holding only the sidecar can still name the install that produced the corrections it describes. Both are always present and null when unknown, following the block's existing convention for git_commit.

The level's year is carried through, and is part of this provenance rather than incidental to it: Arkane selects its atom-energy correction entry by year, so two levels identical in method and basis resolve to different correction tables when their years differ. A record that publishes the corrections without the key that chose them is incomplete.

Which parameter block each correction came from

A consumer reading a corrected energy could see that a correction was applied but not which library block supplied it. Each per-species correction record reports matched_arkane_key, the Arkane key ARC matched in the section that correction's parameters live in.

The atom-energy and bond-additivity keys are matched independently and can differ, and the difference identifies the refit vintage. Against the current RMG-database — 68 atom-energy keys, 26 Petersson, 17 Melius — three pairs diverge exactly that way, for instance atom energies under wb97mv against bond corrections under wb97mv2023. The bond-additivity record previously published the atom-energy key, so in those cases it named the wrong block. It now reports its own, and null when nothing matched in that section rather than borrowing the other.

One qualification the field's definition carries: the atom-energy key is the model chemistry ARC hands Arkane, so both totals are computed under it. matched_arkane_key on the bond-additivity record names the section ARC matched and the key the run-level table was read under, not a second key the numbers were computed from. Both facts are in the document and neither impersonates the other.

The frequency scale factor now reports freq_scale_factor_key, the entry it resolved to in ARC's own freq_scale_factors.yml. The resolved citation stays alongside it in freq_scale_factor_source: every entry in that file carries an explicit integer index into a top-level sources mapping, written by the entry's author, so the attribution is a 1:1 pointer rather than an inference from neighbouring prose.

The parser-evidence sidecar

The sidecar is named for what it holds — facts ARC's own parsers extracted from ESS logs — rather than for one consumer: parser_evidence.json, schema name arc-parser-evidence, described in output.yml under parser_evidence, produced by arc/parser_evidence.py. It contains no TCKDB enums, payload shapes or upload requests.

Standard-state pressure

Each species thermo block carries standard_state_pressure_pa, the pressure s298_j_mol_k, every thermo_points entropy and free energy, and the NASA fit are referred to. It is not a literal: arc/scripts/save_arkane_thermo.py, which runs under rmg_env, recovers it by inverting RMG's own translational partition function, so it reports the pressure RMG actually divided by and follows any change there. Today that is 101325 Pa, i.e. 1 atm rather than 1 bar.

It is per record rather than per file. Every thermodynamic quantity in output.yml currently comes from Arkane at one standard state, but the value enters ARC in the per-species record, and the paths that could add a second standard state are per species.

What the parsed values guarantee

• Bond-additivity corrections are reported only where Arkane applied them. bac_type is the correction requested; a species' bond_additivity record is emitted only when thermo was computed for it. The kinetics run applies none, and transition states never receive one.
• The IRC reaction coordinate is signed about the TS, negated on the reverse branch, so the two branches form one axis from reactants to products. direction is retained.
• IRC point energies are the total for correlated methods. For MP2 or a double hybrid, electronic_energy_hartree is the post-SCF energy, taken by the same code parse_e_elect uses, not the SCF Done component.
• Partial geometries are refused. An IRC point whose geometry table is truncated is dropped with a warning, and a malformed Gaussian Hessian geometry yields no geometry rather than the atoms read so far.
• An all-zero GSM stringfile energy column is treated as absent, not as a flat 0 kcal/mol profile.
• Log paths outside the project directory are null, so an export does not carry the local directory layout.
• Gaussian constraints round-trip. A target stated on a =value B line is carried onto the matching F line.

Versions

output.yml's schema_version goes 1.0 to 1.1. That is an earned increment: main emits 1.0 today.

Everything else introduced here starts at 1. The evidence sidecar is schema 1.0, and the parser-version strings are arc-hessian-1, arc-irc-path-1 and arc-gsm-stringfile-1. No consumer has received an earlier version of any of these.

Molpro

Three independent defects.

Reading the version

output.yml records the ESS version per job type, so a version that fails to parse is a silently incomplete record. arc/parser/adapters/molpro.py::parse_ess_version matched only a NAME : 2015.1.37 header entry. Molpro 2015 emits that block; 2021 and 2022 do not, so ARC recorded no Molpro version at all for any modern run. Across the 11 Molpro fixtures in the repository, 6 returned nothing, including every 2021.x and 2022.x file.

The banner line Version 2022.3 linked Wed Nov 30 06:40:34 2022 appears in all 11, so it is now the fallback. The NAME entry is still preferred where present, because it carries the patch level the banner truncates — 2015.1.37 against 2015.1. In the 4 fixtures carrying both, the two agree and NAME always comes first.

Molpro release	before	after
2015.1.11, 2015.1.37	parsed	unchanged
2021.2, 2021.3	none	parsed
2022.3	none	parsed

The scan is bounded to the first 200 lines, following the Gaussian adapter's precedent for a header field, but keeps the streaming read rather than Gaussian's readlines() so a large log is never loaded whole. The deepest header among the fixtures is line 69.

ORCA needed no change: it parses correctly across 4.1.2, 5.0.4 and 6.0.0.

The open-shell method name

arc/job/adapters/molpro.py prefixed u to the method for any unrestricted species, unconditionally:

if not is_restricted(self):
    input_dict['restricted'] = 'u'

u<method> is a Molpro command for some methods and not for others, so ARC could write an input Molpro has no command for.

A per-method table decides the open-shell command, with a suffix rule for the methods Molpro does not implement open-shell at all.

requested method	ARC wrote	ARC now writes	basis
ccsd-f12, ccsd(t)-f12, dcsd	uccsd(t)-f12	unchanged	"Open-shell unrestricted UCCSD-F12 approximations ... Restricted open-shell Hartree-Fock (RHF) orbitals are used"
mp2, df-mp2, mp2-f12, df-mp2-f12	ump2-f12	rmp2-f12	no UMP2-F12 command exists; "all methods except ump2 use spin-restricted Hartree-Fock (RHF) reference functions"
anything ending -f12c	uccsd(t)-f12c	NotImplementedError	"Currently CCSD-F12c is not available for open-shell cases"
casscf	ucasscf	casscf	"The program is invoked by the command MULTI, MCSCF, CASSCF, or CASCI"; no UCASSCF exists
anything else	u<method>	u<method>	see below

A rewrite is logged at warning level naming both forms, since it changes the command from the one the user wrote. The substitution is the same theory: RMP2-F12 is open-shell MP2-F12, and the mapping is density-fitting consistent on both rows.

Unrecognised methods keep the u prefix deliberately rather than by omission. It is correct for the whole coupled-cluster and configuration-interaction family, and defaulting to no prefix would silently select Molpro's closed-shell program — a worse failure than an invalid command.

The refusal keys on the -f12c suffix rather than two exact names, so df-ccsd(t)-f12c and mp2-f12c are covered too. It raises NotImplementedError from job construction, matching the existing precedent for Molpro plus IRC in check_argument_consistency.

The multireference path is unaffected: mrci/rs2 already clears the prefix and builds its own command. A test fails if that guard is removed.

Not a change: the u prefix on coupled cluster is correct

Worth stating, because it looks like the same class of problem and is not. Molpro's HF/RHF is the spin-restricted program, and ARC's template hardcodes {hf; ... wf,spin=N,charge=C;}, so ARC never requests UHF orbitals. The u selects the open-shell correlation ansatz on an RHF reference, giving RHF-UCCSD(T)-F12, which the manual requires for open-shell F12. ARC's own checked-in output confirms it: arc/testing/sp/TS_x118_sp_CCSD(T).out pairs wf,spin=2 with uccsd(t)-f12; and reports !RHF-UCCSD(T)-F12a energy. ARC's default single point, ccsd(t)-f12/cc-pvtz-f12, was correct before this change and is unchanged by it.

Geometry parsing

parse_geometry iterated a generator that had already been consumed, so it raised on every Molpro log, and where it did return coordinates they were Molpro's Bohr values labelled as Angstrom. It now iterates a list and converts to Angstrom via bohr_to_angstrom.

Gaussian and uploading

Gaussian constraint lines

The ModRedundant writer emitted each freeze directive without a trailing newline, so a deck with two constraints read B 1 2 FA 1 2 3 =104.50 B, and a deck combining constraints with a troubleshooting freeze block read B 1 2 FB 3 4 F. Each directive now ends its line, and the section is closed by the blank line ModRedundant requires.

Uploading

tckdb_arc is optional and imported only when the input's tckdb block sets enabled: true. It is not on PyPI; it lives at https://github.com/calvinp0/tckdb-adapters. The upload destination is logged as host and port only, so credentials in the URL never reach arc.log. The tckdb block is not written to restart.yml, and a restarted run says so rather than skipping the upload silently.

Validation

• output, output-schema, parser, Gaussian adapter and statmech suites: 640 passed, 65 subtests, serial and under -n 4 --dist worksteal
• parser, output, evidence, CLI and scripts suites: 632 passed, 79 subtests
• arc/scripts/save_arkane_thermo_test.py under rmg_env: 10 tests, OK — it skips in arc_env, which is the only environment CI runs it in

The freq_hessian_method rule table was validated by a quantum-chemistry review against vendor documentation, then by an adversarial code review which mutation-tested it: forcing the rule to always-absolute, always-multiplicative, or with the donor and acceptor partners swapped each fails the tests, so they are not tautological.

A later adversarial review of the Arkane provenance was answered the same way. Removing the schema test's resolver patch, or reverting the append-log guard, each fails a specific test — so both are load-bearing rather than decorative.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JYTE9rn3gHi6jDc7ehuHkn

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.06%. Comparing base (6520c82) to head (cefbbad).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #917      +/-   ##
==========================================
+ Coverage   66.29%   67.06%   +0.77%     
==========================================
  Files         122      123       +1     
  Lines       41825    43597    +1772     
  Branches    10749    11143     +394     
==========================================
+ Hits        27726    29240    +1514     
- Misses      11056    11212     +156     
- Partials     3043     3145     +102     

Flag	Coverage Δ
functionaltests	67.06% <ø> (+0.77%)	⬆️
unittests	67.06% <ø> (+0.77%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 3 comments.

[alongd]

Thanks for this well-though of contribution! I left some minor comments

[alongd]

what's tckdb_arc? do we install it in our devscripts? or is it in the env?

[calvinp0]

It's for https://github.com/calvinp0/tckdb-adapters/
I added in the devtools installation now but it's still a WIP. Splitting the adapter from ARC has somewhat been a challenge
https://github.com/ReactionMechanismGenerator/ARC/blob/56800ae93a52f93d20d8a1017b20413a5ca6629d/devtools/install_tckdb_arc.sh

[alongd]

Thanks! added some comments, specifically about parser tests and makefile.
How does CI work with this new installation?

[alongd]

this imports from ARC? does this imply that we have both Arkane (RMG) and ARC in the same env?

[calvinp0]

https://github.com/ReactionMechanismGenerator/ARC/pull/917/changes#diff-49cd9f82811313427e88378beec4688ae3e4e4b2e00015ea38f4776683137f1dR1445

it's a script so should be subprocess called with rmg env

[alongd]

Another review pass at ddc3081 (testing/mutation, schema contract, security, plus a cross-model adversarial pass). The test suite held up well: 9 targeted mutations (Molpro fixes, _get_torsions scan-key guard, Hessian source flag, parent-dir fsync, PBAC gate, ORCA Hessian values) were all caught, and the changed modules pass (655 tests). Previous-round points (duplicate doc rows, AEC/BAC unit sourcing, parent-dir fsync) are resolved.

Points to fix before merge. Each of these was reproduced by running the code:

1. Broken test for the cp_data → thermo_points rename. arc/scripts/save_arkane_thermo_test.py:156 still reads content['P1']['cp_data']. The file isn't touched by this PR. Running it under rmg_env gives KeyError: 'cp_data'. CI doesn't catch it because this test skips in arc_env and no workflow runs it in rmg_env.

2. Gaussian constraint parser loses the target of the constraints ARC itself writes. arc/job/adapters/gaussian.py:393 emits two lines per constraint: B 1 2 =1.45 B, then B 1 2 F. _parse_gaussian_constraint_line returns None for the first line (B action). It returns target_value: None for the second:

   'B 1 2 =1.45 B' -> None
   'B 1 2 F'       -> {'coordinate_type': 'distance', 'atom_indices': [1, 2], 'target_value': None, ...}
   

   Carrying the value from a preceding =x B line on the same atoms into the matching F would fix it. Please add a test that feeds the parser exactly what the adapter writes.

3. IRC reaction coordinate sign. docs/output_yml_schema.md (~L250) says reaction_coordinate_sqrt_amu_bohr is "signed about the TS". parse_irc_path copies Gaussian's cumulative value unchanged, though, so the reverse branch is positive too: rxn_1_irc_2.out gives ('reverse', 1, 0.07236), ('reverse', 2, 0.14469), .... Either negate for direction == 'reverse' or change the doc to say it's unsigned and to use direction.

Smaller points:

4. TCKDB URL logged verbatim (ARC.py ~L64). If the URL carries userinfo (https://user:token@host), it lands in arc.log. Logging only the hostname would avoid that.
5. Where does tckdb_arc come from? Commit d5353ca's message mentions devtools/install_tckdb_arc.sh and a make install-tckdb-arc target, but neither is in the commit. The warning text points users to a "tckdb-arc package", which invites a guessed pip install tckdb-arc. Please name a pinned, canonical install source, or say explicitly that it's not on PyPI. Also, the adapter call shape is tested only against mocks.
6. _make_rel_path (arc/output.py ~L514) returns things like ../../../home/<user>/... for logs outside the project directory. That leaks the local directory layout into the exports meant for sharing. Consider emitting null, or the basename plus an "outside project" flag.
7. parse_1d_scan_full_result docstring says it never raises, but it raises TypeError on an empty log and FileNotFoundError on a missing path. The only caller wraps it in try, so either fix the docstring or add the guard.
8. Silent skip on restart. The tckdb block is popped before ARC() is built, so it never reaches restart.yml, and a restarted project silently uploads nothing. A one-line INFO log or a doc note would help.

Worth checking. These came from the adversarial pass and were not reproduced:

• arc/output.py ~L205 computes per-species corrections from the global bac_type. On a kinetics-only run (the kinetics adapter uses bac_type=None), output.yml might report a BAC that Arkane never applied.
• GSM: if xTB-GSM writes all-zero stringfile energies to mean "no energy", parser_evidence.py ~L544 exports them as a flat 0 kcal/mol profile.
• _read_irc_geometry_table doesn't require the closing separator. A log truncated mid-table would export a point with only some of its atoms.
• Gaussian IRC point energies come only from SCF Done. For an MP2 or double-hybrid IRC, electronic_energy_hartree would then be the SCF component.

[calvinp0]

@alongd

Thanks, all addressed at cefbbad.

1. cp_data test — now reads thermo_points. Verified under rmg_env (10 tests, OK). You're right that CI never runs it there, since it skips in arc_env.

2. Gaussian constraint targets — the parser now carries the value from a =x B line onto the matching F line for the same atoms. The round-trip test builds the deck with GaussianAdapter.write_input_file() rather than hand-typed lines, so it can't drift from the writer. Doing that exposed a writer bug too: the F line had no trailing newline, so two constraints came out as B 1 2 FA 1 2 3 =104.50 B. That's fixed in its own commit, with a two-constraint test.

3. IRC sign — reverse-branch values are now negated, so the coordinate is signed about the TS and the two branches form one axis. direction is still emitted, and the magnitudes are unchanged.

4. URL — the log shows host and port only (urllib.parse.urlsplit), so userinfo, path and query never reach arc.log.

5. tckdb_arc source — the commit message no longer mentions the dropped install script. The warning now names https://github.com/calvinp0/tckdb-adapters and says it isn't on PyPI. The adapter call itself is still only tested against mocks in ARC, because the package isn't installed in ARC's CI.

6. _make_rel_path — a log outside the project directory now gives null, and the schema and docs say so.

7. parse_1d_scan_full_result — I changed the code to match the docstring rather than the reverse. It guards a missing path and degrades each sub-parser to None, so the caller's try is gone.

8. Restart — ARC.init has no tckdb parameter, so the block can't reach restart.yml, and keeping a possibly credential-bearing block out of the project directory is desirable anyway. It now logs at INFO when the block is dropped, and the docstring notes it.

Worth checking - all four reproduced, all four fixed:

• BAC on kinetics-only runs — confirmed: the kinetics run passes bac_type=None. A species' bond_additivity record is now emitted only when thermo was computed for it. bac_type is documented as the request, and TSs never get one.

• GSM all-zero energies — confirmed on the repo's own fixture. An all-zero column across several frames is now treated as absent.

• Truncated IRC table — confirmed: it produced a 4-atom point among 8-atom ones. The table reader is now all-or-nothing, and the point is dropped with a warning. The Hessian geometry reader was changed to fail closed the same way.

• MP2 / double-hybrid IRC energies — confirmed, off by about 231 kcal/mol, and the error varied along the path. parse_irc_path now takes the post-SCF energy using the same helper as parse_e_elect, extracted so both use one implementation.

Two other changes since ddc3081 that you'll see in the diff: the sidecar is renamed to parser_evidence.json, since it holds ARC's parsed facts and nothing TCKDB-specific. And the thermo block now records standard_state_pressure_pa (101325, i.e. 1 atm). It's derived from RMG's partition function rather than written in as a literal.

[alongd]

Thanks!!