feat(core): replace instruction checkpoints with value-delta sync

[kitlangton]

Summary

Replaces the V2 instruction checkpoint lifecycle with value-delta instruction sync. Mutable privileged context such as AGENTS.md, environment and date, skill/reference/MCP guidance, and API entries can change during a session without persisting rendered prose or rewriting text already sent to the model.

The durable log stores only the irreducible fact: which source values changed, and when. Initial instructions, chronological update messages, and the current comparison state are derived from the durable log, content-addressed values, and current renderer code.

The implemented design is documented in specs/v2/instruction-sync-proposal.md.

Source Model

Instruction producers author a typed Source.Definition<A>:

declare namespace Source {
  interface Definition<A> {
    readonly key: Key
    readonly codec: Schema.Codec<A, Json>
    readonly read: Effect<A | Unavailable | Removed>
    readonly render: {
      readonly initial: (current: A) => string
      readonly changed: (previous: A, current: A) => string
      readonly removed?: (previous: A) => string
    }
  }
}

Instructions.make captures the codec and typed renderers in one JSON-level Source. Instructions is an ordered readonly collection of those sources. Instructions.combine preserves source order and rejects duplicate keys.

This keeps producer code typed while giving the runtime one heterogeneous representation for hashing, persistence, replay, and historical rendering. Unavailable and Removed are tagged singleton values discriminated by identity rather than global symbols.

Architecture

flowchart LR
  D["Source.Definition<A>"] -->|"make: encode + capture renderers"| S["Source: JSON-level"]
  S --> I["Instructions: ordered sources"]
  I -->|"read concurrently once"| V["Canonical JSON values"]
  V -->|"SHA-256"| X{"Initial boundary or changed?"}
  X -->|"yes"| T["One DB transaction"]
  T --> B[(instruction_blob)]
  T --> E["session.instructions.updated.2"]
  E --> F[(instruction_state fold)]
  X -->|"no"| A["Request assembly"]
  B --> A
  F --> A
  S --> A
  A --> M["Initial instructions + chronological updates"]

Loading

Durable Data

session.instructions.updated.2 { delta } is the only durable instruction fact. Changed source keys map to SHA-256 content hashes; an observed removal maps to the literal "removed". The sentinel cannot collide with a 64-character hexadecimal hash and avoids nullable record values that do not survive every client generator.

instruction_blob is a machine-local content-addressed store. Each distinct canonical JSON value is stored once by hash. Object keys are canonicalized by code-unit order before hashing. Blob insertion and event publication share the event transaction.

instruction_state is a rebuildable fold cache containing session_id, epoch_start, through_seq, initial_values, and current_values. A missing or stale row is reconstructed from durable events without publishing another instruction event.

No blob garbage collection ships in this change. Mark-and-sweep can be added later without changing the schema or event format.

Safe Boundary

Before pending user input is promoted, each physical runner attempt performs this sequence:

1. Load and combine built-ins, discovered instructions, skill guidance, reference guidance, MCP guidance, and API entries in fixed order.
2. Read every source concurrently exactly once.
3. Encode each typed value as JSON, canonicalize it for hashing, and compare its hash with instruction_state.current_values.
4. At the initial V2 boundary, require every source to be available and publish one complete delta, including {} for a genuinely empty source set.
5. At later boundaries, retain the stored value silently for unavailable sources and publish a delta only when a hash or explicit removal changed.
6. Insert new blobs and publish the delta atomically.
7. Promote pending input only after instruction admission succeeds.
8. Assemble the model request from the admitted instruction state and projected history.

If initial instruction discovery is temporarily unavailable, the attempt fails with Instructions.InitializationBlocked while pending input remains retryable. Later temporary failures do not erase context the model already knows.

Boundary Sequence

sequenceDiagram
  participant R as Runner
  participant S as Instruction sources
  participant D as Database
  participant P as Pending input
  participant M as Model
  Note over R,D: Source reads and fold validation start concurrently
  R->>S: Read every source once
  R->>D: Ensure instruction_state through latest event
  S-->>R: Value / Removed / Unavailable
  D-->>R: Current source hashes
  alt Initial source unavailable
    R-->>R: Fail InitializationBlocked
    Note over P: Input remains pending
  else Complete initial or later boundary
    R->>R: Encode JSON, canonicalize, hash, diff
    alt Initial boundary or changed value
      R->>D: Atomically insert blobs and publish delta
      D-->>R: Fold cache advanced
    else No instruction change
      Note over R: Publish no instruction event
    end
    R->>P: Promote pending input
    R->>D: Read epoch values, deltas, blobs, and history
    D-->>R: One consistent request snapshot
    R->>R: Render initial text and chronological updates
    R->>M: Send model request
  end

Loading

Request Assembly

Assembly runs inside one database read transaction. It loads the epoch's initial hashes, chronological post-epoch deltas, and referenced blobs, then uses the current source renderers to produce:

• Initial instruction text for values known at the epoch boundary.
• Derived System messages at the durable sequence of each later change.
• Removal narration only when the source explicitly observed Removed and supplied a removal renderer.

Rendered model-visible prose is never stored in events or projected message rows. Instruction update messages are excluded from compaction summaries. Client-facing event notices show changed keys, for example Instructions updated: core/date, without exposing privileged values.

Hydrated updates use Option<Json>: a missing record key means unchanged, Some(value) means set or change, and None means removal. This keeps valid JSON null distinct as Some(null) instead of treating it as deletion.

Epochs, Compaction, And Rebuilds

An instruction epoch starts at the initial complete V2 delta or the latest session.compaction.ended sequence. Completed compaction advances the epoch in one SQL update by making current hashes initial hashes. It does not read sources or publish an instruction event.

The fold consumes exact event type/version pairs. Instruction deltas update current values, compaction advances the epoch, and committed revert or session movement clears the fold so the next safe boundary must establish a complete value set.

Forks

session.forked.2 carries authoritative parentSeq. Forking before message N records message.seq - 1, so the child inherits parent instruction events through the exact fork boundary rather than copying the parent's latest state.

The child stores this cutoff in session.fork_seq. Its instruction fold consists of parent ancestry through the frozen cutoff followed by child events. Replay supports the intentional inherited sequence prefix, so rebuilding a fork preserves the same instruction state. Assembly skips ancestor ranges that cannot contain post-epoch updates.

Moves And Execution Serialization

Moving a session interrupts its active execution and awaits idle before capturing changes or publishing session.moved. This prevents the active old-location drain from admitting input, assembling stale instructions, or racing Git change capture while it shuts down. The ordering matches session removal: interrupt, awaitIdle, then mutate durable state.

API Entries

Each stored entry becomes one source keyed api/<key>. DELETE writes a hidden tombstone instead of physically deleting the row, preserving the renderer needed to admit and narrate removal. List responses hide tombstones, and a later PUT revives the same source.

The SQLite value column is nullable because JSON null is a valid value. The separate removed flag distinguishes that value from a tombstone; removed rows store value = NULL rather than a fake empty string. Null-safe conflict predicates preserve no-op PUT behavior across null and non-null values.

PUT measures encoded JSON as UTF-8 and rejects values larger than 8KB with typed InstructionEntryValueTooLargeError, exposed as HTTP 413. Values are never truncated.

Migration And Compatibility

The consolidated migration:

• Creates nullable-value instruction_blob and rebuildable instruction_state tables.
• Rebuilds only instruction_entry to add the tombstone flag and allow JSON null.
• Deletes pre-beta session.instructions.updated.1 events and only their event-derived System rows.
• Backfills session.fork_seq before deleting old instruction events, preserving authoritative fork cutoffs.
• Rewrites session.forked.1 to .2 with parentSeq.
• Drops instruction_checkpoint.

Unrelated events, System messages, sessions, and canonical V1 tables are left intact. Existing sessions establish one complete V2 delta at their next safe boundary.

Both generated client surfaces were regenerated. The delta wire type remains { [key: string]: string | "removed" }; hash bodies do not cross the wire. The obsolete legacy instruction event no longer appears in schemas or clients.

Verification

• Focused instruction, discovery, guidance, runner, event, move, migration, and assembly suites: 241 passing tests.
• Schema manifest suite: 6 passing tests.
• Workspace typecheck: 31 tasks successful across the configured Turbo workspace.
• SDK typecheck and tests pass; Promise/Effect client generation is current.
• Formatting and oxlint pass for the final source changes.
• Migration generation check reports no schema changes and the generated schema is current.

The full core suite still has the two pre-existing LocationWatcher .git/HEAD watcher timeouts documented during development; focused suites covering this change are green.