ci: run pytest for vero and the six baseline agents

[shehabyasser-scale]

Stacked on pr3-harness-bench (#46). One new file: .github/workflows/tests.yml.

Why

No test ever ran in CI on this repository. There is no .github directory on any branch, and the only checks on a pull request are Greptile Review and Socket Security, neither of which invokes pytest.

That is not hypothetical. #61 sat open with a red target suite and nothing surfaced it: the change there reaches the OpenAI client through with_options(...), the test fake was a bare SimpleNamespace, and test_agent_submits_response_and_populates_context had been failing on AttributeError: 'types.SimpleNamespace' object has no attribute 'with_options' since the PR was opened. It only came to light because I ran the suite by hand while rebasing.

What it runs

Two jobs, seven suites, roughly 450 vero tests plus 20 across the agents.

Job	Covers	Measured locally at 3.12
vero	vero/	447 passed, 11 skipped
targets (matrix)	browsecomp-plus	3 passed
	gaia	2 passed
	officeqa	2 passed
	swe-atlas-qna	2 passed
	swe-bench-pro	3 passed
	tau3	8 passed

fail-fast: false on the matrix, so one broken agent does not mask the state of the other five.

Decisions worth reviewing

Two placeholder credentials in the vero job. The task compiler refuses to emit a task whose declared credentials are absent from the environment (compiler.py: declared task credentials are missing). Without OPENAI_API_KEY and OPENAI_BASE_URL present, five tests in test_v05_harbor_build.py fail, which looks like a code failure and is not one. They are placeholders; nothing dials them, and the base URL points at port 1 so anything that ever tried would fail fast rather than hang.

Setting VERO_SKIP_SECRET_CHECK instead would be the wrong lever: I tried it, and it makes test_compiler_checks_secrets_before_writing_and_rejects_source_overlap fail, because that test asserts the check does fire. Placeholders keep the check live.

uv sync --locked, not uv sync. --locked fails rather than relocking, so a pyproject.toml edit that forgets to update uv.lock becomes a visible error instead of a silent re-resolve. Verified safe: uv lock --check is clean in all seven projects today, so this will not fail spuriously on arrival.

UV_PYTHON: "3.12". vero requires >=3.11,<3.14, the six targets require >=3.12; 3.12 is the only version satisfying all of them without relying on the runner image's default.

uv via the official installer rather than a setup action, so there is no action tag to keep current. No dependency cache: a fresh resolve of harbor + openai is cheap, and adding cache keys before there is evidence they are needed is speculative.

No submodules: recursive. The only submodule is the BrowseComp-Plus upstream corpus, which no test reads; the directory is empty locally and all suites pass.

Known flake risk

vero/tests/test_v05_docker_sandbox.py is guarded by @pytest.mark.skipif(not _docker_available()). It skips where there is no daemon, but ubuntu-latest has Docker, so these will actually execute in CI. I saw test_generic_optimization_without_a_shared_filesystem fail once locally and pass on every other run, so it may be order-dependent. Flagging rather than pre-emptively skipping it: if it proves flaky in CI it should be fixed or marked, not hidden.

Not included

Ruff is configured in vero/pyproject.toml but not run here. Adding a lint gate to a codebase that has never had one is a separate change with its own diff, and mixing it in would make this PR's signal ambiguous.

Greptile Summary

This PR introduces the repository's first CI test workflow, running the vero suite and six baseline agent suites on every pull request and push to main. All design decisions are well-considered and documented inline.

• Two parallel jobs: vero (447 tests, 25-min budget) and a targets matrix over six agents with fail-fast: false, so one broken agent doesn't hide the others.
• Concurrency: Push events key by github.sha so every main-branch commit gets its own group and is never cancelled; PR events key by github.ref so rapid re-pushes collapse as intended.
• Reproducibility: uv sync --locked on both jobs and a pinned UV_PYTHON: "3.12" prevent silent re-resolves or interpreter drift.

Confidence Score: 5/5

Safe to merge — adds only a CI workflow file with no changes to application code.

The workflow is a single new file that runs tests; it cannot break production. The concurrency logic is correct, deps are locked, permissions are minimal, and placeholder credentials are scoped to the job that needs them. No issues found.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
.github/workflows/tests.yml	New CI workflow adding two parallel jobs — vero suite (447 tests) and a 6-agent matrix (fail-fast: false) — with correct concurrency keying, locked deps, and minimal permissions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    trigger["on: pull_request / push to main"]
    trigger --> concurrency["Concurrency group\npush → tests-{sha}\nPR → tests-{ref}"]
    concurrency --> parallel["Run in parallel"]

    parallel --> vero["Job: vero\nuv sync --locked\nuv run pytest -q\n(447 tests, ~25 min)"]
    parallel --> matrix["Job: targets (matrix)\nfail-fast: false"]

    matrix --> bc["browsecomp-plus\n3 tests"]
    matrix --> gaia["gaia\n2 tests"]
    matrix --> oqa["officeqa\n2 tests"]
    matrix --> swa["swe-atlas-qna\n2 tests"]
    matrix --> swb["swe-bench-pro\n3 tests"]
    matrix --> tau["tau3\n8 tests"]

    vero --> done["All results reported"]
    bc & gaia & oqa & swa & swb & tau --> done

Loading

_{Reviews (3): Last reviewed commit: "ci: stop main-branch test runs from canc..." | Re-trigger Greptile}

[varunursekar]

thanks!