Repository navigation
feat(storage): delete a document's stored bytes durably in the background (3/4, #313) - #611
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
kiendle
force-pushed
the
feat/storage-lifecycle-03-deletion-lifecycle
branch
2 times, most recently
from
November 16, 2026 23:16
3bdc97a to
f5ae529
Compare
Base automatically changed from
feat/storage-lifecycle-02-object-ledger
to
main
November 16, 2026 23:47
…ound Deleting a document, a batch of documents, or a version now records which stored objects it used in the same transaction that removes the rows, and queues their deletion through the transactional outbox. The document still disappears immediately everywhere (lists, retrieval, graph); the worker then deletes the bytes, retrying transient provider failures with the outbox's backoff and surfacing everything it cannot resolve as blocked for an operator. Requests and items survive the purge and are the audit record. Every path that removes document rows goes through the planner: single/batch delete, connector mailbox removal, audio-transcript rollback, and archive expansion (which used to raw-delete the original ZIP). DOCX edits, ADEU apply and website-preview replacement queue the object they replace. Bytes are only deleted when nothing still references them, across every workspace and every stored-URL column (re-embed and revert share URLs), and only when their provenance matches the requesting workspace: a ledgered object owned by another workspace, or a database file stamped with one, is blocked instead of deleted. Unrecognized external links are skipped; ambiguous URLs are quarantined. A dead-lettered event blocks what is left. ADR-015 records the design (relational rows first, outbox dispatch, no feature flag) and the decisions behind it. Ports the deletion lifecycle from #337/#357. Related: #313 Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com> Co-authored-by: karMareal1 <kareemtaher25@gmail.com> Co-authored-by: sfloody1 <sfloody1@jhu.edu>
Review follow-ups for the deletion lifecycle: - Version delete frees its bytes. Each version's OCR job kept "protecting" the version URL after the version was gone; the delete now removes that version's jobs, and jobs/batch rows only protect bytes while they are still in flight. - Audio converges in either delete order. A transcript's audioUrl is a planner candidate again, new transcripts store the recording's canonical URL, and any /api/files/<id> spelling on any origin counts as a live reference, so a recording is never deleted while a transcript points at it. - A failed transcript only plans the transcript object it created, never the caller's upload, which a batch re-commit reuses. - /api/files/<id> rows that point at S3 or Blob plan the provider object too, instead of deleting the pointer row and reporting success. - Chat attachments (session messages, Ask-again payloads, queued sends) are protected references. - Ledgered objects can only be attached by their canonical URL, so the reference check matches them exactly through indexes; one SQL arm per column, new btree/expression/GIN indexes built concurrently. - Replaying a dead-lettered deletion event resumes it, and a confirmed delete wins over a concurrent retry-budget block. - Archive cleanup no longer records the ZIP as a user-deleted connector source, a replayed extraction of a removed archive converges, version permissions are checked before the transaction opens, and revert plans the document URL it displaces. Related: #313 Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com> Co-authored-by: karMareal1 <kareemtaher25@gmail.com> Co-authored-by: sfloody1 <sfloody1@jhu.edu>
…e-safe Whole-stack review follow-ups for the deletion lifecycle: - Every write path that stores a URL the lifecycle protects (persona and brand avatars, chat attachments and queued sends, upload batch files) now applies the same rule as document ingress: one of our objects can only be attached by its canonical ledger URL, owned by the caller's workspace. That is what lets ledgered objects be matched exactly. - Objects without a canonical history (pre-ledger, including rows from the ledger backfill) keep the conservative alias check, which now matches percent-encoded spellings and runs once per request. - Database file references are matched on an extracted file id through expression indexes, and /api/files pointers through an indexed file_uploads.storage_url, instead of whole-table scans per item. The URL indexes from this stack are now built concurrently in one migration. - Provider objects behind an /api/files pointer re-check the pointer's workspace on every attempt, so a foreign or unstamped pointer blocks the delete even after an item is reset; S3 pointer rows are removed with their object. - Absolute /api/files URLs on any origin are planned as database refs (foreign files stay blocked by provenance) instead of being skipped. - A pending request whose outbox event is dead or missing (a tenant delete cascades outbox rows) reports as blocked through a shared status helper. - The protected-reference definition lives here, shared with the audit. Related: #313 Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com> Co-authored-by: karMareal1 <kareemtaher25@gmail.com> Co-authored-by: sfloody1 <sfloody1@jhu.edu>
…ingress does
Round-2 verification follow-ups for the deletion lifecycle:
- Ingress resolves an internal file through URL normalization
(/api/files/x/../7, /api/./files/7, backslashes, stray whitespace) but
writers used to persist the client's spelling, which the reference check
could not see. Every writer now persists the canonical /api/files/{id}
(or the ledger URL for provider objects), and the historical fallback
parses normalization-suspicious rows the same way ingress does.
- Queue edits authorize attachments on the transaction that holds the
session lock instead of a second pool connection, which could deadlock a
small pool under concurrent edits.
- Each item re-reads its exact and indexed references inside its own locked
transaction just before deleting; only the expensive legacy scans stay
request-wide.
- A non-canonical /api/files spelling only protects a file for rows in the
file's own workspace, so another workspace cannot pin it forever.
- Legacy alias scans send one constant regex per batch (≤64 alternatives)
instead of joining a per-candidate pattern against every row, and the
EXPLAIN tests now run the SQL the reference functions actually emit.
Related: #313
Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
Round-3 verification follow-ups for the deletion lifecycle:
- Historical rows that spell an internal file path-relative (api/files/7,
./api/files/7) or with control characters are matched by the request-wide
fallback, so the bytes they reference are retained.
- A non-canonical any-origin /api/files/<id> link is planned as a database
ref only when that file row exists, and file ids are bigint end to end, so
a large id in an external link can no longer fail a user's delete.
- The per-item reference refresh no longer scans chat JSON arrays; the
indexed containment arm covers what current writers can store.
- Chat send and attachment schemas accept the canonical relative
/api/files/{id} URL that ingress now persists, so queued sends and
Ask-again round-trip it.
Related: #313
Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com>
Co-authored-by: karMareal1 <kareemtaher25@gmail.com>
Co-authored-by: sfloody1 <sfloody1@jhu.edu>
The storage.deletion.requested outbox event is part of the pipeline event union, so the generated JSON Schema has to describe it too. Related: #313 Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com> Co-authored-by: karMareal1 <kareemtaher25@gmail.com> Co-authored-by: sfloody1 <sfloody1@jhu.edu>
- A database candidate is matched against its canonical /api/files/<id> spellings as well as the URL its row held, so a canonical chat reference created after the request-wide scan still retains the bytes when the deleted row used a historical spelling. - The chat query route no longer parses a canonical relative internal attachment URL as a storage URL, so Ask-again and queued sends read it when a storage bucket is configured. The chat schemas accept that form only for ids that are safe integers, like every other file-id parser. Related: #313 Co-authored-by: Tiffany Wei <tiffany.tong.wei@gmail.com> Co-authored-by: karMareal1 <kareemtaher25@gmail.com> Co-authored-by: sfloody1 <sfloody1@jhu.edu>
… the lifecycle - Expired and discarded MCP staged files now plan their bytes for the deletion worker in the same transaction that removes the staged row, instead of calling the provider directly. - The Microsoft 365 deleted-items test passes the deletion context deleteDocumentCore now requires.
- main (#614) gates brand routes behind the workspace's Beta apps setting; the avatar refresh test now stubs the setting on, as brand-routes does.
kiendle
force-pushed
the
feat/storage-lifecycle-03-deletion-lifecycle
branch
from
November 16, 2026 23:47
f5ae529 to
e0877e5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deleting a document, a batch of documents, or a version now deletes its stored bytes. One transaction records which objects the document used, queues their deletion through the transactional outbox (ADR-003), and removes the rows. The document disappears immediately from lists, retrieval and the graph, and the delete stays synchronous for the user. The worker deletes the bytes in the background, retries provider outages with the outbox's backoff, and leaves anything it cannot resolve
blockedfor an operator.Part 3 of 4 for #313. This is the part that closes the P0. It builds on part 1 (object references and per-provider deletes) and part 2 (the write-time ledger). Part 4 adds operator tooling.
sequenceDiagram participant U as User request participant DB as Postgres (one transaction) participant W as apps/worker (outbox) participant P as Provider U->>DB: plan items + enqueue storage.deletion.requested + delete rows DB-->>U: 200 (document gone everywhere) W->>DB: claim event; check references + provenance W->>P: delete unreferenced, workspace-owned objects W->>DB: record outcome (deleted / retained / blocked …)file_uploadsrow stamped with one, isblockedinstead of deleted.skippedand ambiguous URLs arequarantined. A dead-lettered event blocks whatever is left; replaying it resumes the request.Related
Fixes #313 together with parts 1, 2 and 4. Supersedes #337 and #357 (part 3 of 4).
Checklist
pnpm checkpasses (lint + typecheck)pnpm --filter @launchstack/web testpasses@launchstack/orchestration,@launchstack/conversion,@launchstack/indexing,@launchstack/store)docs/storage-ref-contract.md,docs/runbooks/outbox.md)Testing
Run on the top of the stack, against a throwaway Postgres (pgvector 16) migrated with both migration sets:
pnpm lint: 0 errors; the warning count did not go up, and warnings in the touched files went from 5 to 2 (both remaining ones were already onmain).pnpm typecheck: every workspace passes. In this environmentpnpm checkneedsNODE_OPTIONS=--max-old-space-size=12288to avoid an out-of-memory abort.drizzle-kit generateproducing no diff, DDL-only migrations, the push ban, the engine/product boundary, andschemas:check.@launchstack/webjest--runInBand(as CI runs it), after rebasing ontomainatad77da386: 744 suites and 9,215 tests passed, including the DB integration suites. Four suites failed only because of the local environment:vantage-suggestionsandworkspace-history(machine locale; they pass underLANG=en_US.UTF-8),mindmap-embedding-quality(needs a real OpenAI key, which CI has), andavatar-refresh.route, which is fixed in part 3./api/filespointers backed by S3, historical URL spellings (dot segments, percent escapes, backslashes, whitespace, control characters, path-relative) retained only for the owning company, references created mid-request re-checked under the item lock, dead-letter then replay, and realfile_uploadsdeletion with a SQL failure retried.storage.deletion.requested, the worker port deletes the object from SeaweedFS, and the request completes with the eventprocessed.EXPLAINruns on the SQL the reference functions actually emit confirm the exact, extracted-id and pointer arms use their indexes.retained, object still present) → delete B (deleted,HeadObject404, ledgerdeleted_atstamped, requestcompleted).Notes for reviewers
docs/architecture/ADR-015-storage-deletion-lifecycle.mdfirst. It records the decisions (rows first, outbox dispatch, no flag), the protected reference columns, and the JSON columns that were reviewed and excluded.apps/web/src/server/storage/deletion/plan.tsandworker.ts.document,document_versions,ocr_jobsandupload_batch_filesare builtCONCURRENTLYin a-- launchstack:no-transactionmigration.document.urldisplaced insidecreateDocumentVersionLifecycleis left to the orphan audit.Credit
This ports the deletion lifecycle from #337 and #357 by @t1ffnyw (Tiffany Wei), with @karMareal1 and @sfloody1. The coordinator, worker, retry and blocked/quarantined states, tombstone-style audit record and serve-safety goals come from that design. The relational-rows-first ordering and outbox dispatch are deliberate changes, recorded in the ADR.
Co-authored-by: Tiffany Wei tiffany.tong.wei@gmail.com
Co-authored-by: karMareal1 kareemtaher25@gmail.com
Co-authored-by: sfloody1 sfloody1@jhu.edu