Keep tests near their business or runtime boundary.
- Business behavior and user-visible changes need tests. An E2E test runs the whole chain through core, native-bridge, or server APIs, not rendered UI (AGENTS.md).
- A fix adds the smallest regression test that fails without it: Vitest for the core chain or components, Playwright for a window or OS failure. Maintenance with no stable behavior to assert is exempt.
- Existing tests can cover a change. Review behavior and coverage instead of requiring a test-file edit.
- Playwright P0 (
clients/e2e/p0) is only for high-frequency user paths and runtime startup and isolation health. Put other client behavior in the closest package, app, E2E, or visual-regression suite.
Production reads BEAM time and application configuration. Tests control both through runtime functions.
- Time: timers, sleeps, timeouts, and deadlines read
SystemsClock. Suites drive the manual clock withSystemsClock.CaseorSystemsClock.Testand stayasync: false. - Configuration: code reads a replaceable dependency with
SystemsRuntime.get_env/3. A test binds it withSystemsRuntime.Test.put_env/3ordelete_env/2. The test, its tasks, descendants, and allowed processes see the binding until the test exits. A suite whose code runs in application processes callsSystemsRuntime.Test.shared/0in anasync: falsesetup. Do not use globalApplication.put_env/3for these dependencies. It stays correct for keys that code reads withApplication.get_env/3.
mix systems.clock_calls (from systems/, MIX_ENV=test) fails on a direct clock call in compiled modules.
systems/apps/systems_runtime/priv/clock_calls_baseline.exs lists older calls. Use --update-baseline only after removals.
The direct-clock-call, direct-application-env, and direct-randomness rules in lint/rules find the same calls in source.
scripts/ast-grep.sh scan runs them in CI and in the pre-commit hook, with mix format, Prettier, and oxlint.
make, mix deps.get, and pnpm install install the hook from scripts/git-hooks.
All gated server and client tests obey these rules.
- The result of a test depends only on the code and the seed.
- The harness, scheduler, and test clock use no real-time threshold or grace period.
- Unobservable states fail closed and name the stopped process and function.
- Do not loosen assertions to pass. When you change an assertion, explain what it verified and how the replacement still verifies it.
Fix flaky tests at their cause. Do not add retries, longer timeouts, or skips.
Tests connect only to this machine: loopback, *.localhost, unix sockets, and the local Postgres, Redis, MinIO and ClickHouse test services.
Replace an external service with a fixture or fake in the repository. A subprocess that a test starts must not download anything (npm, Go modules, git).
The test harness enforces this. A connection to another host fails the test that made it, and the message names the host and the test.
- BEAM:
SystemsRuntime.NetworkGuard(config :systems_runtime, network_guard: true) formix testandMIX_ENV=testscripts. - Node and Bun:
scripts/test-network-guard.mjs, through VitestsetupFiles,node --importor the Bunpreload. - Playwright: Chromium resolves only this machine's names (
scripts/test-network-browser.mjsin each config'slaunchOptions). A page reaches a host a test routes itself (page.route). - Subprocesses inherit an offline environment: proxies on a closed local port, and npm, pip, uv, cargo, Go and git offline.
COMMA_TEST_NETWORK_ALLOW (comma-separated) adds local test service hosts, such as postgres,redis in a container network.
COMMA_TEST_NETWORK=report lets connections through and lists them (COMMA_TEST_NETWORK_REPORT names a JSONL file).
A test of an external service on purpose has a live tag (live_llm, live_meet, live_semantic_gpu, browser_live). The tag excludes it by default and lets it through the guard. PR CI does not run live tests.
The rules in lint/determinism/rules find real time, timers, deadlines, randomness, fixed ports, and raw Electron changes in tests, and real time in the harness (ex-harness-real-clock).
Only the clock's own production reads of real time are exempt, each with an ast-grep-ignore comment and reason.
Go live tests (*_live_test.go) reach real services and are outside the rules. CI never runs them.
scripts/determinism.sh report counts findings. check runs in the Determinism workflow on pull requests and fails when a change adds a finding to a file.
After dependency setup, run the local chain from the repository root:
pnpm --dir clients --filter @comma/i18n compile
MIX_ENV=test MIX_TEST_PARTITION=57 pnpm --dir clients smoke:auth:abuse:localThe command starts the production Router on loopback and runs the client core through MainSessionHost and its credential store. It checks the enforced and observation flows: Electron, native Apple and Play proof, and native provider failure. CAPTCHA appears only in the enforced flow. PostgreSQL and Redis hold the real challenge and account state. Provider transports and native proof ports are scripted locally; tests make no external requests. The Node process uses a test safeStorage implementation. This command does not verify Electron process isolation or OS encryption. The focused Electron iframe test checks the window boundary separately.
Use local PostgreSQL with user/password postgres/postgres and Redis. Defaults are 127.0.0.1:5432 and redis://127.0.0.1:6379/0. SYSTEMS_TEST_DB_HOST and COMMA_TEST_REDIS_URL change these hosts. Set all five database port variables when the local PostgreSQL port differs. For the development Compose default:
export ALERT_ROUTER_TEST_DB_PORT=15432 SALIX_TEST_DB_PORT=15432 COMMA_TEST_DB_PORT=15432
export BILLING_TEST_DB_PORT=15432 BRIDGE_TEST_DB_PORT=15432The normal Systems BEAM selection runs this same ExUnit chain in the owning comma_web shard. That runner installs the filtered client graph and compiles its messages once. Tests need no Turnstile secret, Apple device, Google decoder credentials, or provider-console access. A missing client driver fails the fixture; cases do not skip.
Real-device App Attest and Play Integrity, provider-console configuration, and live Turnstile remain separate acceptance checks. Run them after merge with the selected app, backend, and proof environment. TestFlight uses the Apple production environment.
Every pull request gets the Required Client CI and Required Client Build aggregate checks.
Configure both as required in the default-branch ruleset. Heavy jobs skip for unrelated changes, and the aggregates still report success.
For behavior that needs a native app or external service:
- State the changed user behavior and failure boundary.
- Record the build revision, environment, fixture account, and exact input.
- Run the real runtime path and observe the user-facing result.
- Record failures apart from untested paths.
- Remove test-owned accounts, files, and workloads.
Keep reports, screenshots, and PDFs outside docs/.
Native browser tests and their browser_live tag live in systems/apps/salix_agent/test/browser_live_test.exs.
Computer Use owns its macOS Accessibility and Screen Recording permissions.
Packaged Comma installs the signed helper under ~/Library/Application Support/Comma Computer Use/<helper bundle ID>/.
Permission checks, authorization, and Connector actions open that app through LaunchServices.
System Settings shows the flavor-specific Computer Use name. Its permission restart does not restart Comma.
Comma replaces the copied helper only when shipped code changes.
Comma permissions do not grant access to this app. Users must authorize Computer Use. The next action restarts it if macOS closed it.
Automated tests cover signature, LaunchServices identity, relaunch, replacement, and failed-copy recovery. They do not grant TCC permissions or show the consent dialog.
To check manually on a fresh macOS test account:
- Open authorization from packaged Comma.
- Grant Accessibility and Screen Recording to the Computer Use app.
- Confirm that System Settings names Computer Use for both grants.
- Select the system restart action. Confirm that Comma's PID does not change.
- Request a screenshot through the Connector. Confirm that it succeeds.
Do not reset a developer's existing permissions for this check.