The ML lifecycle dashboard that replaces five fragmented tools.
Track datasets · version models · orchestrate workflows · audit everything - in one place.
ML teams typically manage 6–10 separate tools: one for data labelling, one for dataset versioning, one for experiment tracking, one for model registry, one for pipeline scheduling, one for training job dispatch. These tools don't talk to each other. Lineage breaks. Bugs hide at integration points. Reproducing a trained model requires an archaeology expedition.
CVOps collapses that stack into a single, versioned, auditable system - from raw video to production model weight, every step is tracked, every artifact is content-addressed, every transition is logged.
raw video / images
│
▼
extract frames ← step: runs inside Docker, outputs frame manifest
│
▼
auto-label ← step: calls CVAT or in-house model, outputs annotation refs
│
▼
human review (CVAT) ← gate: workflow pauses, waits for operator to accept/reject
│
▼
commit dataset ← step: creates immutable Commit on the Dataset object
│
▼
export to YOLO ← step: builds YOLO annotation package, uploads to Garage S3
│
▼
dispatch training ← step: launches Docker training container, tails logs
│
▼
model_version ✓ ← step: registers weights + metadata in the Model Registry
Every node in that graph is a Step - composable, versioned, idempotent. Every edge is tracked. Every artifact is a content-addressed blob.
- Dataset versioning - Git-like commits, refs (branches/tags), and set-diffs on labelled image collections. Roll back, branch, and compare datasets the same way you branch code.
- DAG workflow engine - Define pipelines as directed acyclic graphs. Step outputs are wired to downstream inputs via typed
$steps.<id>.outputs.<name>references. The engine resolves them at runtime. - Human-in-the-loop gates - Any step can be a gate that pauses the run. An operator resolves the gate via API; the engine resumes from exactly where it stopped.
- Idempotent execution - Steps are fingerprinted by
sha256(type + config + resolved inputs). Re-running a workflow reuses outputs from identical prior steps - no redundant compute. - Content-addressed blob storage - Every image, annotation file, and model weight is stored by SHA-256 hash in Garage (S3-compatible). Clients receive presigned S3 URLs. Duplicate uploads are free.
- Append-only audit log - Every status transition emits an
Event. Nothing is deleted. You can replay the full history of any run. - Redis-backed token revocation - JWTs carry a unique
jticlaim. Revoked tokens are blacklisted in Redis for the remainder of their lifetime. Refresh rotation is atomic. - Org-scoped multi-tenancy - All resources are scoped to
org_id. Cross-org access is blocked at the DB query level, not the application layer.
Requires (dev): Docker, Python 3.12+, Node 20+, Tilt. Requires (pre-prod): Docker + Docker Compose.
# 1. Clone and configure
git clone https://github.com/Hozi-AI/CVOps.git
cd CVOps
cp manifests/.env.example manifests/.env # fill in JWT_SECRET and Garage secrets
# 2a. Inner-loop dev: infra in compose, api+frontend as host processes
tilt up
# 2b. Pre-prod / integration test (everything containerised):
cd manifests
docker compose --profile app up # prod-target api + frontend + nginx + mlflow + infraIn dev mode the nginx edge serves the placeholder UI and proxies /api/v1/* to the host API at http://localhost; Vite additionally routes /api/* to http://localhost:8000 for the React app at http://localhost:5173. Behind nginx the API is mounted under the versioned /api/v1 prefix.
In ~30 seconds you have:
| Service | URL |
|---|---|
| REST API | http://localhost:8000 |
| Interactive API docs | http://localhost:8000/docs |
| Garage admin API | http://localhost:3903 |
| MLflow tracking UI | http://localhost:5000 |
Smoke test - register and make your first project:
# Register (creates an Org automatically)
TOKEN=$(curl -s -X POST http://localhost:8000/auth/register \
-H "Content-Type: application/json" \
-d '{"email":"you@example.com","password":"hunter2","org_name":"ACME"}' \
| python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
# Create a project
curl -s -X POST http://localhost:8000/projects \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"helmet-detection","description":"PPE compliance on construction sites"}' \
| python3 -m json.toolExpected output:
{
"id": "018f4b2a-...",
"name": "helmet-detection",
"description": "PPE compliance on construction sites",
"org_id": "018f4b29-...",
"created_at": "2025-06-08T12:00:00Z",
"deleted_at": null
}This section walks through the full lifecycle using curl. All IDs below are placeholders -- substitute your own.
Out of date (2026-10-10). The calls below predate the current API: routes live under
/api/v1/…(e.g.http://localhost:8000/api/v1/projects/…), workflow steps use"type"withstep.*keys (step.extract_frames,step.auto_label,step.human_review— notcvops.*), andstep.auto_labeltakes a trained model (config.model_version_id, required), not a model name. Use the interactive API docs as the reference.
# Request a presigned PUT URL for your zip of images
UPLOAD=$(curl -s -X POST http://localhost:8000/projects/$PROJECT_ID/data-sources \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"site-footage-2025-06","media_type":"image/jpeg"}')
UPLOAD_URL=$(echo $UPLOAD | python3 -c "import sys,json; print(json.load(sys.stdin)['upload_url'])")
SOURCE_ID=$(echo $UPLOAD | python3 -c "import sys,json; print(json.load(sys.stdin)['data_source']['id'])")
# Upload directly to Garage (bypasses the API - no proxying of raw bytes)
curl -X PUT "$UPLOAD_URL" \
-H "Content-Type: image/jpeg" \
--data-binary @./site-footage.zip
# Confirm the upload. The backend registers the blob and, if the project has a
# default_ingest_workflow_id set, auto-dispatches that workflow (params.source_id)
# and returns its run_id — no manual run creation needed. The blob hash is
# verified lazily when extract_frames reads the bytes.
curl -s -X POST http://localhost:8000/data-sources/$SOURCE_ID/confirm-upload \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"blob_hash":"sha256:e3b0c4..."}'
# → {"data_source": {...}, "run_id": "..." | null}# Create a label ontology for the project
ONT_ID=$(curl -s -X POST http://localhost:8000/projects/$PROJECT_ID/ontologies \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"PPE Classes"}' \
| python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
# Add label classes
curl -s -X POST http://localhost:8000/ontologies/$ONT_ID/classes \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"helmet","color":"#22c55e","supercategory":"PPE"}'
curl -s -X POST http://localhost:8000/ontologies/$ONT_ID/classes \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"no-helmet","color":"#ef4444","supercategory":"PPE"}'# Define a two-step pipeline: extract frames → auto-label
# (gate step added before the label commit)
WORKFLOW_ID=$(curl -s -X POST http://localhost:8000/projects/$PROJECT_ID/workflows \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "ingest-and-label",
"definition": {
"steps": [
{
"id": "extract",
"type_key": "cvops.extract_frames",
"config": {"fps": 2, "max_frames": 500}
},
{
"id": "label",
"type_key": "cvops.auto_label",
"config": {"model": "grounding-dino", "ontology_id": "'$ONT_ID'"}
},
{
"id": "review",
"type_key": "cvops.human_review_gate",
"config": {}
}
],
"edges": [
{"from": "extract", "to": "label"},
{"from": "label", "to": "review"}
]
}
}' | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
# Launch a run - engine executes steps in topological order
RUN_ID=$(curl -s -X POST http://localhost:8000/workflows/$WORKFLOW_ID/runs \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"input_refs": {"params": {"source_id": "'$SOURCE_ID'"}}}' \
| python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")# Server-sent events stream - each line is a JSON event
curl -N http://localhost:8000/runs/$RUN_ID/events/stream \
-H "Authorization: Bearer $TOKEN"Output (streamed line by line):
data: {"id":"...","run_id":"...","kind":"status_change","payload":{"status":"running"},"created_at":"..."}
data: {"id":"...","run_id":"...","kind":"step_started","payload":{"step_id":"extract"},"created_at":"..."}
data: {"id":"...","run_id":"...","kind":"step_succeeded","payload":{"step_id":"extract","outputs":{"frame_count":412}},"created_at":"..."}
data: {"id":"...","run_id":"...","kind":"step_started","payload":{"step_id":"label"},"created_at":"..."}
data: {"id":"...","run_id":"...","kind":"gate_reached","payload":{"step_id":"review","gate_data":{...}},"created_at":"..."}
# Operator inspects annotations, then resolves
curl -s -X POST http://localhost:8000/runs/$RUN_ID/gates/review/resolve \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"decision":"accept","notes":"Labels look good, <5% error rate"}'The engine resumes. The run completes. A new Dataset Commit is created.
┌──────────────────────────────────┐
│ nginx (:80) │
└────────────┬─────────────────────┘
│
┌────────────────────────┼─────────────────────────┐
│ │ │
▼ ▼ │
React SPA (:3000) FastAPI (:8000) │
TypeScript · Vite SQLAlchemy 2.0 │
TanStack Query Pydantic v2 │
Zustand python-jose │
┬ │
┌────────────────────────┼────────────────────┬────┘
│ │ │
▼ ▼ ▼
PostgreSQL 16 Garage (S3-compatible) Redis 7
21 ORM models Content-addressed JWT JTI
Alembic migrations blobs (SHA-256) blacklist
asyncpg driver Presigned URLs
Request lifecycle:
Client → nginx → FastAPI → Depends(get_current_user)
│
├─ decode JWT
├─ check Redis blacklist
└─ load User from postgres
┬
▼
Router handler
┬
┌──────────┴──────────┐
▼ ▼
postgres Garage
(reads/writes) (presigned URLs)
Workflow execution runs out of process on Redis Streams (doorbell) with Postgres as the authority — the /runs POST returns immediately and a worker does the work:
POST /workflows/{id}/runs → 201 {"id": "...", "status": "pending"}
│
advance_workflow() (synchronous, in-request)
├─ topological sort (Kahn's)
├─ for each ready step: create pending child run,
│ freeze resolved inputs, compute idem key (sha256)
└─ XADD {job_id, step_type, queue} → Redis Stream
│
worker-preprocessing (separate process)
├─ XREADGROUP, claim child (FOR UPDATE SKIP LOCKED)
├─ process_step(): run step_impl.run(), write results
│ ├─ GateException → status=waiting
│ └─ error → status=failed
└─ on success → advance_workflow() enqueues next ready steps
| Package | Language | Status | Description |
|---|---|---|---|
services/api |
Python 3.12 · FastAPI | ✅ Complete | REST API, workflow engine, DB layer - ~27 models, 100+ endpoints, ~540 tests |
services/frontend |
TypeScript · React 18 | ✅ Implemented | Dashboard UI - Vite · TanStack Query · Zustand · @xyflow/react; auth, all pages, api layer, data-source/frame viewer |
packages/steps |
Python | ✅ Implemented | Steps: extract_frames, import_dataset, auto_label (local YOLO inference, training queue), human_review, commit_dataset, export_yolo, train, plus non-CV chunk_text, parse_sensor, export_jsonl, export_csv |
services/worker-preprocessing |
Python · Redis Streams | ✅ Complete | Consumes the preprocessing stream; runs steps out of the API process |
services/worker-cvat |
Python · Redis Streams | ✅ Implemented | Consumes the cvat stream: human_review push/pull with CVAT, deploy_model to Nuclio (opt-in, tilt up -- --cvat) |
services/worker-training |
Python · Redis Streams · Docker | ✅ Implemented | Consumes the training stream: train, auto_label (opt-in, tilt up -- --training) |
services/model-deployer |
Python · FastAPI | Superseded by worker-cvat (which now serves /models and /deploy on :8001). Not run by compose or Tilt and no longer updated |
All endpoints except /auth/* require Authorization: Bearer <token>.
Auth - 5 endpoints
| Method | Path | Description |
|---|---|---|
POST |
/auth/register |
Register user + org, returns token pair |
POST |
/auth/token |
Login with email + password |
POST |
/auth/refresh |
Rotate token pair (old refresh token blacklisted) |
POST |
/auth/revoke |
Blacklist access + refresh tokens in Redis |
GET |
/auth/me |
Current user info |
Orgs & Members - 6 endpoints
| Method | Path | Description |
|---|---|---|
GET |
/orgs/current |
Get current org |
PATCH |
/orgs/current |
Update org name |
GET |
/orgs/current/members |
List members with roles |
POST |
/orgs/current/members |
Invite a member |
PATCH |
/orgs/current/members/{user_id} |
Update member role |
DELETE |
/orgs/current/members/{user_id} |
Remove member |
Projects - 5 endpoints
| Method | Path | Description |
|---|---|---|
GET |
/projects |
List projects (org-scoped) |
POST |
/projects |
Create project |
GET |
/projects/{id} |
Get project |
PATCH |
/projects/{id} |
Update project |
DELETE |
/projects/{id} |
Soft-delete project |
Data Sources - 9 endpoints
| Method | Path | Description |
|---|---|---|
GET |
/projects/{id}/data-sources |
List data sources |
POST |
/projects/{id}/data-sources |
Create + get presigned PUT URL |
POST |
/projects/{id}/data-sources/check |
Pre-upload duplicate check by blob hash |
POST |
/data-sources/{id}/confirm-upload |
Confirm upload with blob hash |
POST |
/projects/{id}/image-uploads/presign |
Presigned PUT URLs for a batch of images (by SHA-256) |
POST |
/projects/{id}/image-uploads/confirm |
Register uploaded images as samples |
POST |
/projects/{id}/annotated-uploads/confirm |
Register uploaded images + YOLO boxes as samples with annotation revisions (doc 16) |
GET |
/data-sources/{id} |
Get data source |
DELETE |
/data-sources/{id} |
Delete data source |
Imports - 2 endpoints
| Method | Path | Description |
|---|---|---|
POST |
/projects/{id}/imports/upload-url |
Presigned PUT URL for a dataset zip |
POST |
/projects/{id}/imports |
Dispatch import_dataset → [human_review] → commit_dataset from a zip blob or server folder path (doc 16) |
Samples, Ontologies, Datasets, Workflows, Runs, Models, Training Containers
Full endpoint table: 100+ endpoints total - see the interactive API docs for the complete reference with request/response schemas.
Highlights:
GET /projects/{id}/samples?cursor=&source_id=&limit=50- cursor-based paginationGET /samples/{id}/image-url- presigned GET URL (15-min TTL)GET /runs/{id}/events/stream- SSE stream, closes on terminal statusPOST /runs/{id}/gates/{step_id}/resolve- resume a paused workflowGET /datasets/{id}/diff?from=&to=- set-diff between two commitsPOST /datasets/{id}/commits- CAS branch-head advance (concurrent-safe)POST /datasets/{id}/review- dispatch ahuman_reviewrun over the latest commit's samples (one-click "Review in CVAT")
cd services/api
# Set up virtual environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install with dev extras
pip install -e ".[dev]"
# Run the test suite
# Uses testcontainers to spin up a real PostgreSQL - requires Docker
pytest tests/ -q
# Lint and type-check
ruff check src/ tests/
ruff format --check src/ tests/
mypy src/
# Start dev server with hot-reload
uvicorn cvops_api.main:app --reload --port 8000cd services/frontend
npm install
npm run dev # http://localhost:5173tilt up is the recommended dev entry point — infra in containers, api + frontend as host processes with HMR.
Port collisions / non-default boxes (all optional, in manifests/.env):
POSTGRES_HOST_PORT/REDIS_HOST_PORT/EDGE_HOST_PORT(defaults 5432 / 6379 / 80) move the published ports; the Tiltfile points the host API and workers at the same ports.*_BIND_ADDR=127.0.0.1keeps a service off the network.VITE_HMR_CLIENT_PORTsets the port the HMR websocket connects back on (default 80, the nginx edge);autouses the page's own port — needed when:80belongs to something else.- Per-machine compose extras (networks, labels) go in a gitignored
manifests/docker-compose.local.yml, which the Tiltfile merges when present.
To force a containerised dev stack (rare — for reproducing CI failures):
cd manifests
docker compose -f docker-compose.yml -f docker-compose.dev.yml --profile app up# Auth tests only
pytest tests/routers/test_auth.py -v
# DB model tests
pytest tests/db/ -v
# With logging output
pytest tests/ -s --tb=long| Layer | Technology | Notes |
|---|---|---|
| API framework | FastAPI 0.115 | Async, auto-docs, Pydantic v2 |
| Database | PostgreSQL 16 | asyncpg driver |
| ORM | SQLAlchemy 2.0 async | Mapped[T] / mapped_column() |
| Migrations | Alembic | Squashed 0001 baseline + incremental 0002–0007 |
| Blob storage | Garage (S3-compatible) | Content-addressed by SHA-256 |
| Cache / revocation | Redis 7 | JWT JTI blacklist with TTL |
| Auth | python-jose + passlib | JWT HS256, bcrypt cost 12 |
| Validation | Pydantic v2 | Strict types, from_attributes=True |
| Frontend | React 18 + TypeScript | Vite, TanStack Query, Zustand |
| DAG editor | @xyflow/react | Visual workflow builder |
| JSON schema forms | @rjsf/core | Step config editor |
| Reverse proxy | nginx | Routes /api/v1/* to the API and serves / |
| Experiment tracking | MLflow | Postgres backend + Garage proxied artifacts, UI on :5000 |
| Container runtime | Docker Compose | Dev + prod configurations |
| Testing | pytest + testcontainers | Real postgres, moto S3 |
| Linting | Ruff 0.4 | 100-char lines, py312 |
| Type checking | mypy (strict) | Zero ignore budget |
Copy manifests/.env.example to manifests/.env and fill in the values:
cp manifests/.env.example manifests/.env| Variable | Required | Description |
|---|---|---|
JWT_SECRET |
✅ | Min 32-char random string - python -c "import secrets; print(secrets.token_hex(32))" |
POSTGRES_PASSWORD |
✅ | PostgreSQL password |
GARAGE_DEFAULT_ACCESS_KEY |
✅ | Garage S3 access key (must start with GK) |
GARAGE_DEFAULT_SECRET_KEY |
✅ | Garage S3 secret key |
GARAGE_RPC_SECRET |
✅ | Garage cluster RPC secret (32-byte hex) |
GARAGE_ADMIN_TOKEN |
✅ | Garage admin API token |
GARAGE_METRICS_TOKEN |
✅ | Garage metrics token |
WORKER_TOKEN |
✅ | Bearer the API sends to the CVAT model deployer. Not validated on inbound /internal/* calls yet (#71) — those are only /internal/health and the HMAC-verified /internal/cvat/webhook |
DATABASE_URL |
auto | Derived - set in manifests/docker-compose.yml |
REDIS_URL |
auto | Defaults to redis://redis:6379/0 |
| Document | Description |
|---|---|
docs/MASTER_PLAN.md |
Full system reference - start here |
ROADMAP.md |
Current priorities and ordering |
docs/VISION.md |
Product vision |
docs/16-dataset-import.md |
Importing pre-labelled data (YOLO / COCO / raw) |
services/api/CLAUDE.md |
API developer orientation (shared deps, conventions, auth model) |
services/api/docs/db/ |
Per-model database schema documentation |
| Interactive API docs | Swagger UI - live when stack is running |
brand/ |
Logos, color tokens, icons, social assets, brand guide |
Read CONTRIBUTING.md for the full guide. Short version:
git clone https://github.com/Hozi-AI/CVOps.git
cd CVOps
cp manifests/.env.example manifests/.env
sh scripts/git-setup.sh # install git hooks
cd services/api
pip install -e ".[dev]"
pytest tests/ -q # all 146 must pass before a PR- One PR per concern - never mix feature + refactor
- PR title:
<type>: <5–8 word imperative title> - All tests + ruff + mypy must pass (CI enforces this)
- Security issues → SECURITY.md, not a public issue
MIT © 2026 Yehuda Briskman