refactor: make worker startup the only flow compilation path by jumski · Pull Request #672 · pgflow-dev/pgflow

jumski · 2026-09-07T11:17:54Z

refactor: make worker startup the only flow compilation path

Implements #647 ahead of queue identity (#650) and per-step private queues (#651).

Summary

Every flow worker now compiles or verifies its imported flow definition before it tracks its worker function or inserts its worker row. A failed compilation registers nothing and mutates no worker_functions row.
Removed public surfaces: compileFlow() and compareFlowShapes() from @pgflow/dsl, the ControlPlane edge function and its root export, the pgflow compile CLI command, the ControlPlane installer step, all supabase/functions/pgflow/ fixtures, and the complete FlowWorkerConfig.compilation property (including allowDataLoss).
SQL: pgflow.ensure_flow_compiled(flow_slug, shape) keeps two arguments only. The new Atlas migration explicitly drops pgflow.ensure_flow_compiled(text, jsonb, boolean), so production can no longer authorize destructive recompilation.
Local same-slug shape mismatch keeps destructive recompile (dev hot reload); production mismatch returns mismatch without touching data. Structural or unobserved semantic changes use a new concrete slug.
Kept: extractFlowShape(), FlowShape types, SQL _compare_flow_shapes(), _create_flow_from_shape(), create_flow(), add_step(), delete_flow_and_data().
Docs rewritten for startup-only deployment: /concepts/startup-compilation/ is canonical, legacy URLs redirect there, and the pgflow 0.16.0 release notes document the fenced breaking-upgrade procedure.
Changeset bumps the fixed group (@pgflow/core, @pgflow/dsl, @pgflow/client, @pgflow/edge-worker, pgflow) as minor.

Breaking: the migration removes the old three-argument SQL signature — stop old workers before applying it, and upgrade the pgflow package set together. See "Remove manual flow compilation" in the update guide.

Checks

All run locally on this branch:

pnpm nx verify-migrations core, pnpm nx gen-types core, pnpm nx verify-gen-types core --skip-nx-cache — pass
pnpm nx test:pgtap core — 1348/1348 pass (incl. new two-arg signature test)
pnpm nx run-many -t lint build --projects=cli,dsl,edge-worker,client — pass
pnpm nx test cli / test dsl (337) / test client — pass
pnpm nx test:unit edge-worker (271) / test:types edge-worker / test:integration edge-worker (61/61) — pass
pnpm nx e2e cli / e2e edge-worker (12/12) / e2e client (47/47) — pass
pnpm nx verify-exports edge-worker / client, publish dry runs (npm/JSR) — pass
pnpm nx build website — 133 pages, starlight-links-validator pass
CLI help contains no compile command; packed exports contain no ControlPlane/compileFlow/compareFlowShapes

Closes #647

Remove migration-based flow compilation ahead of queue identity (#650) and per-step private queues (#651): every flow worker now compiles or verifies its imported definition before it tracks its function or inserts its worker row, so a failed startup registers nothing. Removed surfaces: public compileFlow() and compareFlowShapes() from @pgflow/dsl, the ControlPlane edge function and its root export, the 'pgflow compile' CLI command, the ControlPlane installer step, all supabase/functions/pgflow fixtures, FlowWorkerConfig.compilation (including allowDataLoss), and the callable production data-loss overload pgflow.ensure_flow_compiled(text, jsonb, boolean) — the new Atlas migration drops it explicitly, which makes this a fenced upgrade: stop old workers before applying it. Kept: extractFlowShape() and FlowShape types, SQL shape comparator, _create_flow_from_shape, create_flow, add_step, delete_flow_and_data. Local same-slug mismatch still recompiles destructively (hot reload); production mismatch returns 'mismatch' without touching data. Docs now describe startup-only deployment everywhere current; legacy URLs redirect to /concepts/startup-compilation/ and /deploy/update-pgflow/ documents the breaking migration procedure. Plan: Plans/2026-09-07-issue-647-startup-only-compilation.md (run: Runs/issue-647-startup-only-compilation). 90 files, +501/-4683.

changeset-bot · 2026-09-07T11:18:04Z

🦋 Changeset detected

Latest commit: 04aeed1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@pgflow/core	Minor
@pgflow/dsl	Minor
@pgflow/client	Minor
@pgflow/edge-worker	Minor
pgflow	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

jumski · 2026-09-07T11:18:07Z

refactor: make worker startup the only flow compilation path #672 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

nx-cloud · 2026-09-07T11:19:05Z

View your CI Pipeline Execution ↗ for commit 58cf585

Command	Status	Duration	Result
`nx run edge-worker:e2e:portable-runtimes`	✅ Succeeded	1m 1s	View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗

☁️ Nx Cloud last updated this comment at 2026-09-07 23:29:49 UTC

jumski

See inline comments.

jumski · 2026-09-07T22:26:39Z

+          value: { status: 'blocked' },
+        },
+      });
+    });


we replaced so many removed lines in this test with a single test?

The removed lines were the describe("compareFlowShapes") block — tests of the TypeScript comparator (compareFlowShapes(), compareSteps(), ShapeComparisonResult), which this PR deletes. Comparison now lives only in PostgreSQL, and that behavior keeps its coverage in the retained pgTAP suites: pkgs/core/supabase/tests/compare_flow_shapes/*.test.sql plus the four ensure_flow_compiled behavior tests. The one added test covers the surviving extraction side: structural modes, if/ifNot patterns, whenUnmet/whenExhausted landing in the shape that feeds the SQL comparator. So tests of deleted code were deleted; live behavior stays covered (SQL side), extraction keeps existing tests plus this one.

jumski · 2026-09-07T22:26:39Z

    });
  });

-  describe('compileFlow includes when_exhausted parameter', () => {


so we have this behaviour tested for extractFlowShape and ensure_flow_compiled?

Yes, both ends are covered:

extractFlowShape: the new test in flow-shape.test.ts ("extracts structural modes and patterns for startup compilation") asserts whenUnmet: "skip-cascade", whenExhausted: "skip", and the requiredInputPattern/forbiddenInputPattern wrappers land in the extracted shape. The no-options default case is asserted in the next test (toEqual with whenUnmet: "skip", whenExhausted: "fail", defined: false wrappers).

ensure_flow_compiled / SQL comparison: pgTAP compare_flow_shapes/condition_mode_drift.test.sql and pattern_differences.test.sql assert drift in these fields is detected, and the ensure_flow_compiled suites cover matching/mismatching/recompile behavior end to end.

What this file keeps is its actual job: DSL acceptance — the options are accepted and stored on the step definition. The deployment chain around them is tested where it lives.

jumski · 2026-09-07T22:26:39Z

+| ✅ Adjusting retry parameters | ❌ Adding/removing steps |
+| ✅ Updating timeout values | ❌ Changing step dependencies |
+| ✅ Tuning delay and worker concurrency options | ❌ Changing step types |
+| ✅ Behavior-preserving refactors of handler code | ❌ Changing step slug names |


why you replaced those?

Two fixes from the final review round, both mechanical:

The breaking column listed "Changing step dependencies" twice (rows 3 and 4 of the old table — a leftover duplicate). The second instance became "Changing step types", which is a genuine shape change (single ↔ map) the comparator detects.

"Tuning delay and concurrency options" was ambiguous — the flow DSL has no concurrency option. It is now "worker concurrency", which is a worker setting (maxConcurrent), not part of the flow shape, so tuning it is safe under the same slug.

jumski · 2026-09-07T22:26:39Z


 The original timestamp allows pgflow to search for and detect which migrations have already been installed, while the new timestamp prefix ensures Supabase can apply the migration without timestamp ordering errors.

+## Remove manual flow compilation


this section should go to the news article about this release

Done in d5b53c0. The full procedure now lives in the new release article /news/pgflow-0-16-0-startup-only-flow-compilation/ (what is removed, fenced-upgrade warning, HTTP and process worker procedures, legacy file/config cleanup). update-pgflow.mdx keeps a short evergreen summary plus a link to the article. Version slug follows the changeset minor bump (0.15.1 → 0.16.0) — rename if the released number differs.

PR review asked for the fenced-upgrade procedure to live in the release news article instead of the evergreen update guide. The new pgflow 0.16.0 article carries the full one-time procedure; update-pgflow keeps a short summary plus a link. Version slug follows the changeset minor bump (0.15.1 -> 0.16.0); rename if the released number differs.

The news section deploys on merge to main, but the 0.16.0 npm release only exists after the Version Packages flow runs. Keep the article as a draft until the release is cut, and make the update-guide pointer plain text so no link targets an unpublished page. Undraft (and re-date) when the release goes out; rewrite if the release ends up batching the queue epic instead.

Add a prominent danger alert to the draft 0.16.0 release article. It lists every removed surface users may still depend on and puts the stop-and-drain requirement before the migration instructions.

Make the startup-only compilation article visible when this branch lands. The danger alert now tells users which removed surfaces to check before the fenced upgrade.

jumski · 2026-09-07T23:16:37Z

Merge activity

Sep 7, 11:16 PM UTC: A user started a stack merge that includes this pull request via Graphite.
Sep 7, 11:24 PM UTC: @jumski merged this pull request with Graphite.

github-actions · 2026-09-07T23:25:50Z

🚀 Production Deployment: Website

✅ Successfully deployed to production!

🔗 Production URL: https://pgflow.dev

📝 Details:

Commit: 489e1bcb8684d02a03678fe8e372f5c7c00d5e4f
View Logs

Deployed at: 2026-09-08T01:24:28+02:00

github-actions · 2026-09-07T23:32:03Z

🔍 Preview Deployment: Website

✅ Deployment successful!

🔗 Preview URL: https://pr-672.pgflow.pages.dev

📝 Details:

Branch: 647-controlplane
Commit: 255ef651120e887309e52e86fe791aba8a1421d2
View Logs

_Last updated: _

jumski deployed to preview September 7, 2026 11:27 — with GitHub Actions Active

jumski mentioned this pull request Sep 7, 2026

test: make Nx own fresh environments and focused slow checks #673

Open

10 tasks

jumski commented Sep 7, 2026

View reviewed changes

jumski deployed to preview September 7, 2026 22:38 — with GitHub Actions Active

jumski deployed to preview September 7, 2026 22:55 — with GitHub Actions Active

Agent added 2 commits September 7, 2026 23:02

docs: flag startup compilation removal as breaking

58cf585

Add a prominent danger alert to the draft 0.16.0 release article. It lists every removed surface users may still depend on and puts the stop-and-drain requirement before the migration instructions.

docs: publish the 0.16.0 release article

04aeed1

Make the startup-only compilation article visible when this branch lands. The danger alert now tells users which removed surfaces to check before the fenced upgrade.

jumski merged commit 489e1bc into main Sep 7, 2026
16 of 17 checks passed

jumski deployed to preview September 7, 2026 23:29 — with GitHub Actions Active


		The original timestamp allows pgflow to search for and detect which migrations have already been installed, while the new timestamp prefix ensures Supabase can apply the migration without timestamp ordering errors.

		## Remove manual flow compilation

Uh oh!

Conversation

jumski commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

refactor: make worker startup the only flow compilation path

Summary

Checks

Uh oh!

changeset-bot Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

jumski commented Sep 7, 2026

Uh oh!

nx-cloud Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jumski left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jumski commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

github-actions Bot commented Sep 7, 2026

🚀 Production Deployment: Website

Uh oh!

github-actions Bot commented Sep 7, 2026

🔍 Preview Deployment: Website

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jumski commented Sep 7, 2026 •

edited

Loading

changeset-bot Bot commented Sep 7, 2026 •

edited

Loading

nx-cloud Bot commented Sep 7, 2026 •

edited

Loading

jumski commented Sep 7, 2026 •

edited

Loading