Skip to content

feat(storage): delete a document's stored bytes durably in the background (3/4, #313) - #611

Merged
kiendle merged 10 commits into
mainfrom
feat/storage-lifecycle-03-deletion-lifecycle
Nov 17, 2026
Merged

kiendle merged 10 commits into
mainfrom
feat/storage-lifecycle-03-deletion-lifecycle

Conversation

@kiendle

@kiendle kiendle commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Deleting a document, a batch of documents, or a version now deletes its stored bytes. One transaction records which objects the document used, queues their deletion through the transactional outbox (ADR-003), and removes the rows. The document disappears immediately from lists, retrieval and the graph, and the delete stays synchronous for the user. The worker deletes the bytes in the background, retries provider outages with the outbox's backoff, and leaves anything it cannot resolve blocked for an operator.

Part 3 of 4 for #313. This is the part that closes the P0. It builds on part 1 (object references and per-provider deletes) and part 2 (the write-time ledger). Part 4 adds operator tooling.

sequenceDiagram
    participant U as User request
    participant DB as Postgres (one transaction)
    participant W as apps/worker (outbox)
    participant P as Provider
    U->>DB: plan items + enqueue storage.deletion.requested + delete rows
    DB-->>U: 200 (document gone everywhere)
    W->>DB: claim event; check references + provenance
    W->>P: delete unreferenced, workspace-owned objects
    W->>DB: record outcome (deleted / retained / blocked …)
Loading
  • Every path that removes document rows goes through the planner: single and batch delete, version delete, connector mailbox removal, audio-transcript rollback, and archive expansion (which used to raw-delete the original ZIP). ADEU apply, DOCX modify, website-preview replacement and revert queue the object they displace.
  • Bytes are deleted only when nothing references them. The check covers every workspace and every stored-URL column, including chat attachments held in JSON. Re-embed and revert share URLs. Bytes are also deleted only when their provenance matches the requesting workspace. A ledgered object owned by another workspace, or a file_uploads row stamped with one, is blocked instead of deleted.
  • Expired and discarded MCP staged files plan their bytes in the transaction that removes the staged row, so they also go through the worker.
  • External links are skipped and ambiguous URLs are quarantined. A dead-lettered event blocks whatever is left; replaying it resumes the request.
  • No feature flag. This is the only delete path.

Related

Fixes #313 together with parts 1, 2 and 4. Supersedes #337 and #357 (part 3 of 4).

Checklist

  • pnpm check passes (lint + typecheck)
  • pnpm --filter @launchstack/web test passes
  • Changesets added (@launchstack/orchestration, @launchstack/conversion, @launchstack/indexing, @launchstack/store)
  • New env vars documented — none added
  • UI changes exercised in a browser — no UI changes; deletes behave as before for users
  • Docs updated (ADR-015, docs/storage-ref-contract.md, docs/runbooks/outbox.md)

Testing

Run on the top of the stack, against a throwaway Postgres (pgvector 16) migrated with both migration sets:

  • pnpm lint: 0 errors; the warning count did not go up, and warnings in the touched files went from 5 to 2 (both remaining ones were already on main). pnpm typecheck: every workspace passes. In this environment pnpm check needs NODE_OPTIONS=--max-old-space-size=12288 to avoid an out-of-memory abort.
  • The CI schema gates pass locally: journal integrity for both migration sets, drizzle-kit generate producing no diff, DDL-only migrations, the push ban, the engine/product boundary, and schemas:check.
  • @launchstack/web jest --runInBand (as CI runs it), after rebasing onto main at ad77da386: 744 suites and 9,215 tests passed, including the DB integration suites. Four suites failed only because of the local environment: vantage-suggestions and workspace-history (machine locale; they pass under LANG=en_US.UTF-8), mindmap-embedding-quality (needs a real OpenAI key, which CI has), and avatar-refresh.route, which is fixed in part 3.
  • vitest: evidence 96, conversion 216, retrieval 29, document-conversion-engine 15, simulation 51, worker 48, and orchestration 56 plus indexing 17 against real Postgres (outbox integration suites included). All pass.
  • Lifecycle integration suite (152 tests, real Postgres): planning plus row removal in one transaction and rollback, shared URLs retained, foreign provenance blocked, version delete freeing bytes, both recording/transcript delete orders, chat-attachment references, /api/files pointers backed by S3, historical URL spellings (dot segments, percent escapes, backslashes, whitespace, control characters, path-relative) retained only for the owning company, references created mid-request re-checked under the item lock, dead-letter then replay, and real file_uploads deletion with a SQL failure retried.
  • End-to-end through the real worker: an outbox tick claims storage.deletion.requested, the worker port deletes the object from SeaweedFS, and the request completes with the event processed.
  • EXPLAIN runs on the SQL the reference functions actually emit confirm the exact, extracted-id and pointer arms use their indexes.
  • End-to-end run against real SeaweedFS S3 and Postgres: upload → two documents share the object → delete A (retained, object still present) → delete B (deleted, HeadObject 404, ledger deleted_at stamped, request completed).

Notes for reviewers

  • Read docs/architecture/ADR-015-storage-deletion-lifecycle.md first. It records the decisions (rows first, outbox dispatch, no flag), the protected reference columns, and the JSON columns that were reviewed and excluded.
  • Then read apps/web/src/server/storage/deletion/plan.ts and worker.ts.
  • Engine indexes on document, document_versions, ocr_jobs and upload_batch_files are built CONCURRENTLY in a -- launchstack:no-transaction migration.
  • Known follow-ups, also listed in the ADR: brand media and avatar deletes do not plan their bytes yet, and a document.url displaced inside createDocumentVersionLifecycle is left to the orphan audit.

Credit

This ports the deletion lifecycle from #337 and #357 by @t1ffnyw (Tiffany Wei), with @karMareal1 and @sfloody1. The coordinator, worker, retry and blocked/quarantined states, tombstone-style audit record and serve-safety goals come from that design. The relational-rows-first ordering and outbox dispatch are deliberate changes, recorded in the ADR.

Co-authored-by: Tiffany Wei tiffany.tong.wei@gmail.com
Co-authored-by: karMareal1 kareemtaher25@gmail.com
Co-authored-by: sfloody1 sfloody1@jhu.edu

@vercel

vercel Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
pdr-ai-v2 Ignored Ignored Nov 16, 2026 11:47pm UTC

@kiendle
kiendle added this pull request to stack #613 November 11, 2026 07:28
@kiendle kiendle changed the title feat/storage lifecycle 03 deletion lifecycle feat(storage): delete a document's stored bytes durably in the background (3/4, #313) Nov 11, 2026
@kiendle
kiendle force-pushed the feat/storage-lifecycle-03-deletion-lifecycle branch 2 times, most recently from 3bdc97a to f5ae529 Compare November 16, 2026 23:16
Base automatically changed from feat/storage-lifecycle-02-object-ledger to main November 16, 2026 23:47
kiendle and others added 10 commits November 16, 2026 18:47
…ound

Deleting a document, a batch of documents, or a version now records which
stored objects it used in the same transaction that removes the rows, and
queues their deletion through the transactional outbox. The document still
disappears immediately everywhere (lists, retrieval, graph); the worker then
deletes the bytes, retrying transient provider failures with the outbox's
backoff and surfacing everything it cannot resolve as blocked for an
operator. Requests and items survive the purge and are the audit record.

Every path that removes document rows goes through the planner:
single/batch delete, connector mailbox removal, audio-transcript rollback,
and archive expansion (which used to raw-delete the original ZIP). DOCX
edits, ADEU apply and website-preview replacement queue the object they
replace.

Bytes are only deleted when nothing still references them, across every
workspace and every stored-URL column (re-embed and revert share URLs), and
only when their provenance matches the requesting workspace: a ledgered
object owned by another workspace, or a database file stamped with one, is
blocked instead of deleted. Unrecognized external links are skipped;
ambiguous URLs are quarantined. A dead-lettered event blocks what is left.

ADR-015 records the design (relational rows first, outbox dispatch, no
feature flag) and the decisions behind it.

Ports the deletion lifecycle from #337/#357.

Related: #313

Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
Review follow-ups for the deletion lifecycle:

- Version delete frees its bytes. Each version's OCR job kept "protecting"
  the version URL after the version was gone; the delete now removes that
  version's jobs, and jobs/batch rows only protect bytes while they are
  still in flight.
- Audio converges in either delete order. A transcript's audioUrl is a
  planner candidate again, new transcripts store the recording's canonical
  URL, and any /api/files/<id> spelling on any origin counts as a live
  reference, so a recording is never deleted while a transcript points at it.
- A failed transcript only plans the transcript object it created, never
  the caller's upload, which a batch re-commit reuses.
- /api/files/<id> rows that point at S3 or Blob plan the provider object
  too, instead of deleting the pointer row and reporting success.
- Chat attachments (session messages, Ask-again payloads, queued sends) are
  protected references.
- Ledgered objects can only be attached by their canonical URL, so the
  reference check matches them exactly through indexes; one SQL arm per
  column, new btree/expression/GIN indexes built concurrently.
- Replaying a dead-lettered deletion event resumes it, and a confirmed
  delete wins over a concurrent retry-budget block.
- Archive cleanup no longer records the ZIP as a user-deleted connector
  source, a replayed extraction of a removed archive converges, version
  permissions are checked before the transaction opens, and revert plans the
  document URL it displaces.

Related: #313

Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
…e-safe

Whole-stack review follow-ups for the deletion lifecycle:

- Every write path that stores a URL the lifecycle protects (persona and
  brand avatars, chat attachments and queued sends, upload batch files)
  now applies the same rule as document ingress: one of our objects can only
  be attached by its canonical ledger URL, owned by the caller's workspace.
  That is what lets ledgered objects be matched exactly.
- Objects without a canonical history (pre-ledger, including rows from the
  ledger backfill) keep the conservative alias check, which now matches
  percent-encoded spellings and runs once per request.
- Database file references are matched on an extracted file id through
  expression indexes, and /api/files pointers through an indexed
  file_uploads.storage_url, instead of whole-table scans per item. The
  URL indexes from this stack are now built concurrently in one migration.
- Provider objects behind an /api/files pointer re-check the pointer's
  workspace on every attempt, so a foreign or unstamped pointer blocks the
  delete even after an item is reset; S3 pointer rows are removed with their
  object.
- Absolute /api/files URLs on any origin are planned as database refs
  (foreign files stay blocked by provenance) instead of being skipped.
- A pending request whose outbox event is dead or missing (a tenant delete
  cascades outbox rows) reports as blocked through a shared status helper.
- The protected-reference definition lives here, shared with the audit.

Related: #313

Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
…ingress does

Round-2 verification follow-ups for the deletion lifecycle:

- Ingress resolves an internal file through URL normalization
  (/api/files/x/../7, /api/./files/7, backslashes, stray whitespace) but
  writers used to persist the client's spelling, which the reference check
  could not see. Every writer now persists the canonical /api/files/{id}
  (or the ledger URL for provider objects), and the historical fallback
  parses normalization-suspicious rows the same way ingress does.
- Queue edits authorize attachments on the transaction that holds the
  session lock instead of a second pool connection, which could deadlock a
  small pool under concurrent edits.
- Each item re-reads its exact and indexed references inside its own locked
  transaction just before deleting; only the expensive legacy scans stay
  request-wide.
- A non-canonical /api/files spelling only protects a file for rows in the
  file's own workspace, so another workspace cannot pin it forever.
- Legacy alias scans send one constant regex per batch (≤64 alternatives)
  instead of joining a per-candidate pattern against every row, and the
  EXPLAIN tests now run the SQL the reference functions actually emit.

Related: #313

Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
Round-3 verification follow-ups for the deletion lifecycle:

- Historical rows that spell an internal file path-relative (api/files/7,
  ./api/files/7) or with control characters are matched by the request-wide
  fallback, so the bytes they reference are retained.
- A non-canonical any-origin /api/files/<id> link is planned as a database
  ref only when that file row exists, and file ids are bigint end to end, so
  a large id in an external link can no longer fail a user's delete.
- The per-item reference refresh no longer scans chat JSON arrays; the
  indexed containment arm covers what current writers can store.
- Chat send and attachment schemas accept the canonical relative
  /api/files/{id} URL that ingress now persists, so queued sends and
  Ask-again round-trip it.

Related: #313

Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
The storage.deletion.requested outbox event is part of the pipeline event
union, so the generated JSON Schema has to describe it too.

Related: #313

Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
- A database candidate is matched against its canonical /api/files/<id>
  spellings as well as the URL its row held, so a canonical chat reference
  created after the request-wide scan still retains the bytes when the
  deleted row used a historical spelling.
- The chat query route no longer parses a canonical relative internal
  attachment URL as a storage URL, so Ask-again and queued sends read it
  when a storage bucket is configured. The chat schemas accept that form
  only for ids that are safe integers, like every other file-id parser.

Related: #313

Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
… the lifecycle

- Expired and discarded MCP staged files now plan their bytes for the
  deletion worker in the same transaction that removes the staged row,
  instead of calling the provider directly.
- The Microsoft 365 deleted-items test passes the deletion context
  deleteDocumentCore now requires.
- main (#614) gates brand routes behind the workspace's Beta apps setting;
  the avatar refresh test now stubs the setting on, as brand-routes does.
@kiendle
kiendle force-pushed the feat/storage-lifecycle-03-deletion-lifecycle branch from f5ae529 to e0877e5 Compare November 16, 2026 23:47
@kiendle
kiendle merged commit 47f1803 into main Nov 17, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: P0 document deletion orphans storage objects

1 participant