Repository navigation
Refactor fixture tests and fix declaration spacing idempotency - #240
Sha1rholder wants to merge 6 commits into
Conversation
- Discover fixtures exactly two directory levels deep in both runners - Apply per-case configs and compare single-pass output byte for byte - Support optional inputs and save or clear failure artifacts - Remove manual registration, idempotency exceptions and duplicate tests - Populate basic and complex expectations and add missing final newlines - Add runner regression tests and update testing documentation
Base declaration spacing on formatted layout instead of source newlines. Cover table expansion, collection collapse, comments, and margin settings. Update the complex fixture to match the stable output.
|
This is too hard to read and needs to be broken up into more than 1 PR.
|
I've tried my best to reduce the size of the PR, but this is already the smallest PR that can make all CI pass. Alternatively could you create a new temporary branch from main branch, and I'll submit multiple PRs in logical order to that temporary branch for your reference? That way at least I won't break the CI on main branch. Only the last PR would pass all CI. |
|
@fdncred Would it be okay if I split this PR into several smaller PRs but merge them into a branch in my own fork, and re-submit a combined PR to nufmt:main? It won't break any CI then. Otherwise I can just create some stacked PRs. |
There was a problem hiding this comment.
Notes from my clanker.
I ran this locally. cargo test passes, and the Nushell runner passes all 272 checks. I also formatted about 4,000 .nu files under my ~/src with main and with this branch. 23 files come out different on the first pass, and in every one the change is blank lines around a let/const that expands to several lines. Files that change on a second format went from 50 to 27, and nothing that was stable on main became unstable. So the spacing fix does what it says.
Before merge
config.nounlooks like a typo forconfig.nuon. Is it on purpose? It's in the README in 3 places, both runners, the runner tests and the 6 fixtures that have one. Anyone who adds aconfig.nuontoday gets it silently ignored, and the case runs with default settings.- Fixtures without their own config pick up any
nufmt.nuonthat nufmt finds walking up from the repo root, becauseformat_via_stdinonly passes--configwhen the case has one. main's harness had the same problem, so this isn't new. It matters more now because a failure writesunexpected.nuornot_idempotent.nuinto the tree, and someone with a~/nufmt.nuonwould get a pile of bogus failures and files. Always passing--configwith a committed default file, or running nufmt from a temp dir, would fix it.run_ground_truth_tests.nuneeds the same change since it doescd $PROJECT_DIR. - With every fixture inside one
#[test],cargo test let_statementcan't pick out a single case anymore. The old macro allowed that. An env var filter checked inrun_fixtureswould be enough, e.g.NUFMT_FIXTURE=core_language_constructs/let_statement cargo test fixtures.libtest-mimicwould bring back real per-case tests, but that's more work.
A question
Should the new declaration spacing have a nufmt.nuon option? I lean no. The old rule looked at the source layout, which is why it wasn't idempotent, so there's no stable behavior worth keeping. It does change first-pass output for existing files, which is where the 23 files above come from.
Nits
- The doc comment on
separator_newlines_between_top_level_pipelines(blocks.rs:88) still says it decides how many newlines to emit. With the new caller it's a minimum, because newlines that comment handling already wrote are never removed. - When a case has no
input.nu, an oldunexpected.nunever gets cleaned up. The runner test asserts this, so I assume it's intended, but I'd rather remove it. It's oneremove_filecall that ignoresNotFound. - The
spliceinformat_blocklooks safe to me, since it only shifts bytes afterseparator_start. A short comment saying why would help the next person who touches it. It also copies the just-formatted pipeline for every top-level pair when margin is 1 or more. I doubt that's measurable, so no change needed. - The Nushell runner uses whatever is in
target/release/nufmt, even if it's stale. That isn't new, but now a stale binary also writes files into the tree. A release binary left over from main gave me a falseFAIL other/complex.
Dropping the flake.nix patch in favor of CARGO_BIN_EXE_nufmt is a nice cleanup.
This is a major refactor. In both the Rust and Nushell test runners, manually registered fixture tests have been replaced with automatic discovery. Adding a new fixture no longer requires modifying the test code.
tests/fixtures/<category>/<case>/structure. Each case must containexpected.nu, and may optionally containinput.nuandconfig.noun. Redundant inputs have been removed, and the basic, complex, and indentation-related fixtures are now included in the full test suite.format(expected) == expected; if an input file exists, also verify thatformat(input) == expected. Each file is formatted only once, and comparisons are byte-for-byte exact, including whitespace, line-ending type, and the final newline at EOF.not_idempotent.nuorunexpected.nu. Once the corresponding check passes, the generated artifact is automatically cleaned up. Failures from all cases are collected instead of stopping at the first failure. Runner regression tests have also been added to cover fixture discovery, configuration, byte-level comparison, error handling, and artifact files.Example
A deliberately broken test:
After running the tests once, if a test fails, a Git-visible temporary file is created directly inside the corresponding test case. Its filename indicates the type of failure:
unexpected.numeans that formattinginput.nudid not produceexpected.nu, whilenot_idempotent.numeans thatexpected.nuitself is not idempotent. Now you can check the idempotency ofexpected.nuwithout a sameinput.nu.After fixing the issue and running the tests again, these temporary artifacts are automatically removed if the tests pass, so no manual cleanup is required. This makes fixture testing more convenient.
It should also make fixtures more "discoverable", as input, expectations, and error outputs are now stored together.
I did design it but I have to commit that there's a LOT of llm in this PR : (
I'm also not confident with the expression in README because it's translated fron Chinese.