feat: add doc-accuracy reviewer to catch harmful documentation changes

[Iheanacho-ai]

What

Adds make check-doc-accuracy, a reviewer that checks docs for changes that could harm a reader who follows them (commands that lose data, destroy things, or weaken security, not just typos). Also runs it in CI on every docs PR, forks included, as an advisory check.

Why

The worst doc bugs are commands that run fine but still do damage, like a service started without its data mount or a dropped safeguard. They pass every syntax check, so a flag checker won't catch them. You need something that reads the snippet and reasons about what happens if you run it. Until now that only ran when someone remembered to run it locally.

Change

• tools/doc-accuracy: a Go program that drives the claude CLI as a read-only reviewer (it reports, never edits). It reviews the .mdx files you changed and checks claims about Talos/Omni against the upstream source.
• Looks for harm first: data loss, destructive commands, removed safeguards, security downgrades, then plain wrong flags and values.
• Output: a short terminal summary, a JSON file (verdict + a reason and fix per finding), and in CI, inline annotations plus one summary comment that updates in place on each push.
• CI (.github/workflows/doc-accuracy.yaml): uses pull_request_target so it works on fork PRs. Checks out our base branch only, never the fork's code, and reads the PR as a text diff. Nothing from the PR is run. It never blocks merge.

Usage

make check-doc-accuracy                                # files you changed vs HEAD
make check-doc-accuracy DOC=public/omni/....mdx        # one specific file
make check-doc-accuracy DOC_ACCURACY_BASE=origin/main  # PR-style, vs a branch
make check-doc-accuracy-all                            # the whole public/ tree (slow)

Testing

• go test, go vet, and gofmt all pass.
• Ran it against a sample doc with four planted problems (etcd with no volume mount, rm -rf /, --insecure on a bootstrapped node, a stale image tag). It caught all four at the right severity and checked the current Talos version upstream. Verdict FAIL, exit 1.
• Confirmed only a CRITICAL fails the run; warnings and notices don't. No changed docs pass cleanly, and a missing claude CLI gives a clear message.
• Not yet tested on a live PR, since pull_request_target runs the workflow from main, so the annotations and comment only show up once this merges. The ANTHROPIC_API_KEY secret is set.

[sterlingkoch]

From Claude:
--allowedTools is ignored under --permission-mode bypassPermissions (claude-code#12232). The whitelist here isn't actually a whitelist, so Bash is unrestricted and the read-only claim doesn't hold. --disallowedTools Edit,Write should still block those two. --permission-mode dontAsk with this allowedTools list, or --tools as a real whitelist, would do what this is aiming for.

[sterlingkoch]

--insecure as CRITICAL will fail half the Talos bootstrap corpus (talosctl apply-config --insecure). Worth carving out the documented bootstrap case so this doesn't train us to ignore the tool.

[sterlingkoch]

The default base is HEAD, so after a local commit this won't actually review anything (DOC_ACCURACY_BASE=origin/main is in the README but easy to miss). Defaulting to origin/main (with a fallback) would match how this will actually get run.

[sterlingkoch]

If the review step errors (timeout, claude crash, no verdict), this manufactures a PASS. Combined with the "existing comment" branch below, a failed run will overwrite a previous FAIL with "Nothing flagged."

Gate this on steps.review.outcome. If the review step didn't succeed, post "reviewer didn't finish, see the log" and leave the last real comment alone. A missing findings file is not a clean review.

[Iheanacho-ai]

Good catch, fixed. The tool now emits a tri-state verdict (PASS/FAIL/INCOMPLETE); a crash or no-verdict run is INCOMPLETE and posts a separate 'didn't finish' note instead of overwriting the findings comment