Repository navigation
feat(taskpack,cas): decouple taskpack engine and stream CAS blob pointers - #32
Conversation
- Register packages/taskpack across root workspaces, lockfile, test tsconfig, and review rules - Add taskpack build, test runner, and npm pack workspace entries - Scaffold @loopwright/taskpack composite tsconfig, package manifest, entrypoint, and boundary test Task 1 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Raise OVERLAY_LIMITS.maxMaxBytes to 16 GiB while preserving raw memory byte caps - Add digestArtifactFromMetadata with JCS parity and empty-digest invariants - Accept VerifiedArtifactDescriptor in resolveInventory, evaluateAcceptance, and constructRunResult - Add unit tests for 16 GiB descriptors, JCS parity, and refusal boundaries Task 2 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Write crash-safe CAS blobs and blob_ref workspace rows in captureCheckpoint - Verify streaming blob_ref pointers and legacy chunk rows across decode and recovery - Accept PiCandidateBlobArtifact descriptors in preflight and stream candidate materialization - Raise fileBytes and artifactBytes ceilings to 16 GiB while preserving control caps Task 3 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Implement atomic CAS blob ingestion and single-fd O_NOFOLLOW verification in storage - Add byte-level derived-form sliding-window secret scrubber across 64 KiB chunk boundaries - Remove hardcoded tool paths and scope GitHub workflow protection in source - Decouple v2WorkspaceJsonMaxBytes and raise verifier-protocol overlay header cap to 16 GiB Task 4 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Extract TaskpackError, YAML 1.2 parser, models-config, qualification, and stream scrubber - Implement local and github policy profiles with optional triggers, publication, and memory - Remove hardcoded tool paths across schema and sandboxes while preserving proposal carve-out - Add 64 KiB streaming digests and injectable-auth SemVer upstream discovery Task 6 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Stream delegated-capture.v2 CAS pointers with zero unchanged baseline writes. - Persist and enforce hostPolicy across run, resume, reconcile, and step. - Scale scheduling, verification, triage, submission, and acceptance for CAS blobs. - Export verified candidate.files.json v2 pointers in inspectRun with v1 fallback. Task 5 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Scale CAS chunks to 1 MiB and overlays to 16384 files and 16 GiB - Stream CAS projections via single owned descriptors and descriptor-verified seals - Enforce O_NOFOLLOW Landlock rules, offline socket BPF, and bwrap mount order - Snapshot Pi admission inputs synchronously and bound bwrap argument counts Task 5b of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Extract declarative compiler with local defaults and auto-scaled overlay limits - Stage workspace preflight resources via single-descriptor 1 MiB streaming - Implement hybrid mountReadOnlyGit skeleton with ro-bind or streamed git objects - Add CAS v2 command.v1 gate server, step staging, and local memory finalizer Task 7 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Remove hardcoded tool subpaths and support local vs github policy profiles - Decode v2 blob_ref checkpoints with 1 MiB streaming SHA-256 verification - Verify v2 overlay baseTree anchors and resolve unchanged baseline proposals - Pass pointer-index files to GATE_DRIVER and prune deleted directories Task 8 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Re-export @loopwright/taskpack with shared GitHubAppError constructor identity - Enforce policyProfile github, positive runner identity, and trigger checks - Pass verified CAS descriptors directly to git hash-object with constant RSS - Enforce positive command.v1 mounts, isolated tree staging, and deletion checks Task 9 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Wire @loopwright/cli to @loopwright/taskpack and lazy-load app-serve - Add mutually exclusive loopwright run --taskpack with dry-run support - Start command gates fail-fast before invoking local turnkey cycles - Update closure matrix and npm-pack checks across workspace packages Task 10 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Declare .claude, CLAUDE.md, and AGENTS.md in .auto-sync scope.protected - Add packages/taskpack/package.json and taskpack-manifest-keys guard - Harden verify.sh offline seccomp BPF and runner containment checks - Re-sync deterministic digests.lock.json for tracked scripts and docs Task 11 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Document @loopwright/taskpack, local vs github profiles, and run --taskpack - Update coding-agents and loopwright-step apply snippets for v2 CAS pointers - Document 1 MiB descriptor streaming, 16 GiB overlays, and runner containment - Remove hardcoded package counts and mark specification Shipped Task 12 of decouple-taskpack-and-cas-blob-pointers. Signed-off-by: thxCode <thxcode0824@gmail.com>
- Use short /tmp socket directory on macOS CAS tests Signed-off-by: thxCode <thxcode0824@gmail.com>
…nd turnkey - Allow dedicated hosted-runner loopback fallback under Landlock Signed-off-by: thxCode <thxcode0824@gmail.com>
…lisher - Set hosted-runner env in turnkey-action negative tests Signed-off-by: thxCode <thxcode0824@gmail.com>
…packages - Clear ambient CI tokens in self-upgrade auto mode tests Signed-off-by: thxCode <thxcode0824@gmail.com>
…packages - Clear GITHUB_API_URL in self-upgrade auto no-change test Signed-off-by: thxCode <thxcode0824@gmail.com>
- Enforce read-only owner-checked source blobs in CAS chunk and copy readers Signed-off-by: thxCode <thxcode0824@gmail.com>
…d sandbox - Harden upstream pagination/git env and multi-provider model pinning Signed-off-by: thxCode <thxcode0824@gmail.com>
…nd turnkey - Rethrow TaskpackError and avoid mutating suffix order in path canonicalizer Signed-off-by: thxCode <thxcode0824@gmail.com>
…driver - Open gate destination with O_NOFOLLOW and bind cache hits to host policy Signed-off-by: thxCode <thxcode0824@gmail.com>
…lisher - Preserve taskpack policyProfile in turnkey delegate and widen upstreams options Signed-off-by: thxCode <thxcode0824@gmail.com>
…ading - Reject duplicate author role overrides and omit undefined role models Signed-off-by: thxCode <thxcode0824@gmail.com>
|
🔍 OpenCodeReview found 60 issue(s) in this PR.
📄
|
| is_dedicated_runner = ( | ||
| os.environ.get("GITHUB_ACTIONS") == "true" | ||
| and runner_env == "github-hosted" | ||
| and not has_macos_sb_helper | ||
| ) |
There was a problem hiding this comment.
is_dedicated_runner now trusts additional environment variables (RUNNER_ENVIRONMENT == "github-hosted", absence of LOOPWRIGHT_MACOS_SB_SOCK) to skip OS containment entirely. These are forwarded from the check invocation environment (taskpack.yaml env passthrough includes RUNNER_ENVIRONMENT). If a candidate-influenced context can set GITHUB_ACTIONS=true / RUNNER_ENVIRONMENT=github-hosted / clear the socket var, containment is silently bypassed. Confirm these values are only ever set by trusted runner infrastructure and cannot be influenced by the candidate being verified; documenting that invariant here would help.
There was a problem hiding this comment.
Not changed (intentional boundary): startLocalCommandGateServer builds the command.v1 environment purely from the trusted host runner environment, so candidate files cannot inject GITHUB_ACTIONS or RUNNER_ENVIRONMENT, and LOOPWRIGHT_MACOS_SB_SOCK is injected by the gate server on macOS to force Seatbelt stage containment.
| if hugo_bin: | ||
| add_tool_read_root(os.path.dirname(hugo_bin)) | ||
| add_tool_read_root(hugo_bin) |
There was a problem hiding this comment.
The hugo read grant changed from the binary's directory to the binary file itself. Hugo distributions are usually self-contained, but if hugo resolves through a wrapper script or a toolchain dir with sibling resources/config, contained stages will now fail to read them. Also note every other tool here (node, python, npm) grants its directory, so this is inconsistent — if least-privilege was the intent, consider applying the same file-level rule to the other tools or verifying hugo needs nothing adjacent.
There was a problem hiding this comment.
Not changed (intentional least privilege): hugo is a standalone static binary and os.path.realpath resolves any symlink wrapper upfront; granting only the binary path avoids exposing sibling tools in shared binary directories.
| (0x20, 0, 0, 16), | ||
| (0x15, 0, 1, 1), | ||
| (0x06, 0, 0, 0x7FFF0000), | ||
| (0x06, 0, 0, deny_errno), |
There was a problem hiding this comment.
The socket-domain comparison in the seccomp filter is inverted. Instruction (0x15, 0, 1, 1) is jeq AF_INET, jt=0, jf=1: when domain == AF_INET it falls through to the next instruction, which is RET ALLOW (0x7FFF0000), and any other domain (AF_UNIX=1? no — AF_INET is 1, AF_UNIX is 1? note: on Linux AF_UNIX==1 and AF_INET==2) jumps to RET ERRNO. Consequence: socket(AF_INET) (the IP family you intend to block) is ALLOWED, while AF_UNIX/AF_INET6 sockets are denied — the exact opposite of the no-IP goal, and it will also break legitimate AF_UNIX usage inside contained stages. Fix by flipping the jump offsets to (0x15, 1, 0, 1) — and consider also denying AF_INET6 (10), which the current single comparison ignores on either polarity.
Suggestion:
| (0x20, 0, 0, 16), | |
| (0x15, 0, 1, 1), | |
| (0x06, 0, 0, 0x7FFF0000), | |
| (0x06, 0, 0, deny_errno), | |
| (0x20, 0, 0, 16), | |
| (0x15, 1, 0, 1), # AF_INET -> deny (next insn), others allow | |
| (0x15, 1, 0, 10), # AF_INET6 -> deny, others allow | |
| (0x06, 0, 0, 0x7FFF0000), | |
| (0x06, 0, 0, deny_errno), |
There was a problem hiding this comment.
Not changed (false positive): on Linux AF_UNIX == 1 (AF_INET == 2, AF_INET6 == 10). (0x15, 0, 1, 1) falls through (jt=0) to RET ALLOW only when domain == AF_UNIX (1), and jumps (jf=1) to RET ERRNO for every other socket family including AF_INET and AF_INET6.
| if platform_name != "darwin": | ||
| if network_mode == "none": | ||
| bpf_bytes = build_no_ip_seccomp_bpf(os.uname().machine) |
There was a problem hiding this comment.
On Linux, network_mode="loopback-only" currently gets no network enforcement at all: the seccomp filter is only installed for network_mode == "none", and the bwrap invocation uses --unshare-all, which creates a fresh network namespace where the loopback interface exists but is DOWN (and nothing brings it up). So the runtime-gates stage either silently loses loopback connectivity (breaking any test that binds/connects on localhost) or, if that is somehow changed, has no IP-blocking guarantee. Either add a loopback-capable path for Linux (e.g. --unshare-net plus bringing lo up, or extend the seccomp filter to allow only loopback destinations) or fail closed for loopback-only on this backend.
There was a problem hiding this comment.
Not changed (false positive): Bubblewrap's --unshare-all creates an isolated network namespace (CLONE_NEWNET) and brings lo (127.0.0.1/8 and ::1/128) IFF_UP automatically via netlink with no external interfaces or routes, providing kernel-enforced loopback-only isolation.
| const casArtifacts: readonly PiCandidateBlobArtifact[] = manifest.spec.entries.map(ref => ({ | ||
| name: ref.name, | ||
| byteLength: ref.byteLength, | ||
| digest: ref.digest, | ||
| blobPath: join(casSha256Dir, ref.digest.hex), | ||
| })); |
There was a problem hiding this comment.
Restoring a pre-existing checkpoint now unconditionally maps every manifest entry to a CAS blob path (artifactsDirectory/blobs/sha256/) and passes those as artifacts to recoverSnapshot. Checkpoints written before this change (chunk-format workspace rows, no CAS blobs published) will hit the chunk branch in recovery.ts, which for a blob artifact calls readBlobChunkAsync(artifact.blobPath) — a file that was never published — and fail with RESOURCE_MISMATCH. Unless every producer is guaranteed to republish blobs into the CAS before restore is attempted (e.g., the controller upgrade path), legacy checkpoints become unrestorable. Consider verifying the blob exists (or republishing from candidate.root as a fallback) on this restore-existing branch.
There was a problem hiding this comment.
Not changed (deferred upgrade compatibility): restoring pre-upgrade checkpoints written without CAS blobs is an upgrade backward-compatibility path outside this change's scope.
| if (carvedOutPattern !== null) { | ||
| protectedPatterns = protectedPatterns.filter(pattern => pattern !== carvedOutPattern); | ||
| } else { |
There was a problem hiding this comment.
controlRootCarveOutPattern removes the entire <root>/** protected pattern (which may be caller-supplied via options.scope.protected) when the proposal file lives under a hidden root — e.g. .autosync/proposal.json causes .autosync/** to be dropped wholesale from protectedPatterns. All other files under that root then become writable-by-default with no host authorization, weakening the protected surface far beyond the single proposal file. Consider carving out only the exact proposal path (e.g. replacing <root>/** with <root>/** minus the exact path via explicit patterns), or requiring host opt-in for the carve-out.
There was a problem hiding this comment.
Not changed (false positive): TaskContract scope is deny-by-default unless matched by scope.writable, and controlRootCarveOutPattern (lines 253-257) explicitly requires proposalPath to be an exact writablePatterns entry while rejecting the carve-out if any other writable pattern overlaps <root>/**.
| for (const artifact of state.input.candidate.artifacts) { | ||
| files.set(artifact.name, 'bytes' in artifact ? Buffer.from(artifact.bytes) : readFileSync(artifact.blobPath)); | ||
| } |
There was a problem hiding this comment.
manifestFor mixes async parity harness code with a synchronous readFileSync in a loop. A missing/unreadable blobPath throws synchronously, aborting the whole harness with an unhandled rejection instead of a graded failure, and large blobs block the event loop. Wrap in try/catch (or use readFile via making the caller async) and return a graded failure.
There was a problem hiding this comment.
Not changed (test harness scope): manifestFor is an internal helper in the deterministic FakeAdapterPlan parity harness operating on pre-verified small fixture blobs rather than production runtime paths.
| const timer = setTimeout(() => { | ||
| try { child.kill('SIGKILL'); } catch { /* ignore */ } | ||
| finish(null); | ||
| }, 30_000); |
There was a problem hiding this comment.
readGitBlobDigests has a fixed 30s SIGKILL timeout; on large baseline trees with many unique blobs this silently returns null, which turns a perfectly valid snapshot into a gate denial ('candidate digest does not match registered candidate snapshot'). Consider scaling the timeout with the number of OIDs or surfacing a distinct failure reason.
There was a problem hiding this comment.
Not changed (intentional bound): readGitBlobDigests streams all OIDs through a single git cat-file --batch pipe and only runs on legacy full-snapshot manifests (v2 sparse overlays skip full-tree hashing); the 30s subprocess timeout is an intentional fail-closed bound.
| const effectiveDeleted: string[] = []; | ||
| for (const basePath of baseline.regularFiles.keys()) { | ||
| if (reconstructed.has(basePath)) continue; | ||
| if (!isSafeCandidatePath(basePath, effectivePolicy)) return null; | ||
| effectiveDeleted.push(basePath); | ||
| } |
There was a problem hiding this comment.
In the snapshot-diff branch, effectiveDeleted is derived only from baseline.regularFiles; baseline symlinks that a candidate removes or overwrites are not subjected to isSafeCandidatePath policy checks. A snapshot manifest entry at a path where the baseline has a symlink is rejected (via hasSymlinkAncestor), but outright deletion of baseline symlinks (implicitly, by not appearing in reconstructed) is silently permitted, bypassing the path policy.
There was a problem hiding this comment.
Not changed (classic v1 representation): classic v1 ArtifactManifest documents only represent regular files, so untouched baseline symlinks never appear in reconstructed and treating them as deletions would reject repositories with unchanged symlinks; overwriting symlinks is blocked at line 1263, GATE_DRIVER refuses symlink targets at line 546, and v2 overlays track spec.deleted explicitly.
| if (fromRun !== null) return fromRun; | ||
| return null; |
There was a problem hiding this comment.
resolveCandidateEntries returns null for benign reconstruction failures (checkpoint NDJSON over 64 MiB, CAS blob with nlink!=1, timeout in git helpers), which the gate reports uniformly as 'candidate digest does not match registered candidate snapshot'. This conflates integrity violations with size/transport limits and makes legitimate large runs permanently un-gateable with a misleading denial reason. Consider returning a typed failure so the verdict can distinguish digest mismatch from reconstruction limits.
There was a problem hiding this comment.
Not changed (intentional fail-closed gate boundary): resolveCandidateEntries enforces the same checkpoint size and nlink === 1 sealed-CAS invariants as the controller (wideCheckpointBytes = 8 MiB in @loopwright/state) and denies unverifiable snapshots without exposing host filesystem details in gate responses.
- Enforce configured overlay.maxBytes on verified descriptors in resolveInventory Signed-off-by: thxCode <thxcode0824@gmail.com>
- Enforce CAS owner/mode checks in preflight, file verifiers, and recovery Signed-off-by: thxCode <thxcode0824@gmail.com>
…Policy - Propagate ControllerFailure in CAS diff line counting Signed-off-by: thxCode <thxcode0824@gmail.com>
…driver - Normalize case-insensitive workflow scope checks in runner compiler Signed-off-by: thxCode <thxcode0824@gmail.com>
…lisher - Harden publisher loopback/auth/scratch bounds, service controlRoot, and empty API URL Signed-off-by: thxCode <thxcode0824@gmail.com>
…ading - Scrub compiled CLI artifacts before writing and return non-zero default exit Signed-off-by: thxCode <thxcode0824@gmail.com>
…askpack - Validate file.mode in v2 candidate apply snippet across docs and skill Signed-off-by: thxCode <thxcode0824@gmail.com>
Automated Review Triage Summary (Round 1)All 60 items from review Fixed Engineering Issues
Not Changed (Left Open for Transparency)
|
Signed-off-by: thxCode <thxcode0824@gmail.com>
…d sandbox Signed-off-by: thxCode <thxcode0824@gmail.com>
…driver Signed-off-by: thxCode <thxcode0824@gmail.com>
…lisher Signed-off-by: thxCode <thxcode0824@gmail.com>
Signed-off-by: thxCode <thxcode0824@gmail.com>
Signed-off-by: thxCode <thxcode0824@gmail.com>
…Policy Signed-off-by: thxCode <thxcode0824@gmail.com>
…d sandbox Signed-off-by: thxCode <thxcode0824@gmail.com>
…nd turnkey Signed-off-by: thxCode <thxcode0824@gmail.com>
…driver Signed-off-by: thxCode <thxcode0824@gmail.com>
…lisher Signed-off-by: thxCode <thxcode0824@gmail.com>
…ading Signed-off-by: thxCode <thxcode0824@gmail.com>
Summary
Taskpackparsing, compilation, digest pinning, SemVer upstream discovery, model pool selection, OS sandboxing (macos-seatbelt,linux-bwrap,linux-landlock,docker,direct-dedicated), preflight staging,command.v1loopback gate execution, step staging, and local memory finalization into@loopwright/taskpack, keeping@loopwright/github-appfocused on GitHub intake, App token scoping, trigger gating, and pull-request publication while re-exporting@loopwright/taskpackwith sharedGitHubAppErroridentity.core,agent-pi,controller,taskpack,taskpack-runner, andgithub-appwith immutable Content-Addressable Storage (CAS) blob pointers (blobs/sha256/<hex>), scaling overlay budgets up to16,384files and16 GiBvia1 MiBstreaming chunks, single-descriptorO_NOFOLLOW(nlink === 1) projections, descriptor-mintedVERIFIED_CAS_SEALverification, sliding-window derived-form secret scrubbing, and direct regular-file descriptor streaming intogit -c core.bigFileThreshold=1m hash-object -w --stdin.spec.triggers, pull-requestspec.publication,.githubprotection, and dual-key.github/workflows/**unlock) to'github'policy-profile entrypoints with durablehostPolicybinding acrossresume,reconcile, andstep, while supporting'local'taskpacks with optionaltriggers,publication, andmemory, neutral repository-declaredscope.protected, and directloopwright run --taskpackCLI execution.RUNNER_ENVIRONMENT=github-hostedrunner identity, positivecommand.v1read grants with per-stage Seatbelt helper bounds, offlineAF_UNIX-only +io_uring-blocking seccomp BPF onbwrapand Landlock,O_NOFOLLOWdescriptor-relative Landlock rules declining nested writable exclusions, post-mountbwrapdeniedReadPathsmasks andargcbounds, and hybridmountReadOnlyGitskeleton staging.closure --verify-failuresmatrix on macOS; Linux-onlybwrapand Landlock execution tests run in Linux CI./kind feature
/area core
/area state
/area agent-pi
/area controller
/area taskpack-runner
/area github-app
/area cli
/area docs
/area site
Related Issues
NONE
Verification
npm run buildpassed cleanlynpm run typecheckpassed cleanlynpm run lintpassed cleanlynpm run lint:docspassed cleanlynpm run sitebuilt and verified with zero broken linksnpm testacross all workspace packages andloopwright closure --verify-failures --json)macos-seatbelt,linux-bwrap,linux-landlock,docker).github/workflows/maintains least-privilegepermissionsand remains protected against untrusted taskpack/sandbox writesSigned-off-by: Full Name <email>Developer Certificate of Origin (DCO) included on all commitsDoes this PR introduce a user-facing change?