[DO NOT MERGE] distroless: prove image + UID gate together (stacked #36099 + #36101) - #37466
Closed
jasonhernandez wants to merge 16 commits into
Closed
[DO NOT MERGE] distroless: prove image + UID gate together (stacked #36099 + #36101)#37466jasonhernandez wants to merge 16 commits into
jasonhernandez wants to merge 16 commits into
Conversation
2 tasks
Replace Debian-based prod images with distroless. Remove shell entrypoint scripts (no shell in distroless). Add libeatmydata to distroless-prod-base for CI compatibility. Update clusterd mzcompose service to drop shell-dependent options. Distroless ships no init, so add a tini-static mzbuild image (static tini built from source, mirroring openssh-static) and copy it into distroless-prod-base. tini stays PID 1 to forward signals and reap zombies, notably the ssh subprocesses spawned by the SSH tunnel feature. Leaf images wrap their binary in ENTRYPOINT ["/usr/bin/tini", "--", ...]. The deleted environmentd entrypoint slept forever on graceful exit so a fenced-out generation would not be restarted. Distroless has no shell to do this, and a Kubernetes StatefulSet (restartPolicy: Always) would crash-loop a process that exits. Replace it with an in-process idle: when fenced out and supervised directly by Kubernetes, environmentd idles until the orchestrator deletes the StatefulSet (SIGTERM, then SIGKILL) instead of calling exit!(0). The all-in-one image keeps its entrypoint and exit-0 behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nightly cloudtest and orchestratord runs crash-looped every distroless environmentd/clusterd pod with: error while loading shared libraries: libfdb_c.so: cannot open shared object file: No such file or directory ubuntu-base has copied libfdb_c.so from the foundationdb image since FoundationDB consensus/timestamp-oracle support landed (#33819), which postdates the original distroless base. Copy it the same way, and carry over ubuntu-base's RUST_BACKTRACE/RUST_LIB_BACKTRACE settings so crashes in distroless images print backtraces too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_crash_storage and the secrets restart test killed clusterd with kubectl exec ... bash -c 'kill -9 `pidof clusterd`'. Distroless clusterd images ship no shell, so the exec fails with exit 1 and the tests fail. Force-delete the pod instead (SIGKILL, no grace period), matching the crash-simulation idiom in test_replica_restart. The StatefulSet recreates the pod under the same name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The distroless base ships glibc's example nsswitch.conf, which routes passwd/group lookups through the `compat` NSS backend. `compat` is not built into glibc, so a lookup dlopens the image's libnss_compat.so.2. The statically-linked `ssh` embeds a different glibc than that module was built for, and the mismatch jumps through a garbage pointer: audit: ANOM_ABEND comm="ssh" exe="/usr/bin/ssh" sig=11 which the SSH tunnel manager surfaced as "failed to connect to the remote host: No such file or directory" in the Nightly ssh tunnel cloudtest. Ship a files/dns nsswitch.conf instead. Those backends are built into glibc (2.33+), so lookups never dlopen, for the static ssh and the dynamically-linked Rust binaries alike. Verified by rebuilding distroless-prod-base and running the CI-built static ssh inside it as uid 65532 and 65534 (`ssh -G` exits 0, was SIGSEGV). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same no-shell-in-distroless problem as the clusterd kills: replace the exec'd shell kill of environmentd with a force pod delete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Distroless environmentd/clusterd images ship no shell, so kubectl exec ... bash -c 'kill ...' cannot work. Add signal_process_in_pod, which execs into the kind node hosting the pod (a Docker container with a full userland) and signals the process found via crictl. This keeps the tests' original semantics: the process dies or suspends in place and the container restarts within the same pod, unlike a pod delete. It also covers the shared-fate SIGSTOP/SIGCONT cases, which simulate a hung peer and cannot be expressed as a pod delete at all. The helper finds the target among the container's init and its direct children, covering both distroless images (tini is init, target is its child) and Ubuntu-based images (the entrypoint execs the target). Convert test_crash, test_secrets, and both shared-fate suites to the helper, including reverting the earlier pod-delete workaround. Mechanics verified against a local kind cluster (crictl discovery, comm matching, kill -s from the node). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The new tini-static image was pinned to ubuntu:noble-20260210.1, the stale base copied from openssh-static before #37354 bumped it. Match the repo's current base (noble-20260610, per ubuntu-base #37449). Build-stage only; the shipped artifact is the static tini binary from a FROM scratch stage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Match ubuntu-base and tini-static at noble-20260610 (openssh-static was one dependabot cycle behind at noble-20260509.1). Aligning all static builder bases to one tag means the distroless build pulls a single ubuntu image rather than several. Build-stage only; the artifact is the static ssh binary from a FROM scratch stage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tini-static and distroless-prod-base are new images introduced by this migration. Register both with the docker dependabot ecosystem so their base-image pins (ubuntu/debian/gcr.io distroless) stay current on the same weekly cadence as ubuntu-base and openssh-static, rather than drifting until someone notices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…prod-base The libfdb_c.so tag must stay in lockstep with the identical COPY in ubuntu-base and the foundationdb crate's fdb-7_3 API feature. Spell that out so a dependabot bump to one of the three gets recognized as a drift bug in review rather than merged as routine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lint suite requires a copyright header on config files; the nsswitch.conf added for the static-ssh fix was missing one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Distroless images run as nonroot (UID 65534) instead of root. Add version-gating so orchestratord sets the correct runAsUser/runAsGroup based on the Materialize version, avoiding UID mismatches during rolling upgrades from Debian-based to distroless images. Gate versions (verified against release history, 2026-06): - balancerd: V26_18_0. Its ci/Dockerfile switched to distroless-prod-base in v26.18.0 (prod-base in v26.17.x). The original V26_19_0 was off by one and would have forced UID 999 onto v26.18.x balancerd pods that actually run as 65534. - environmentd/clusterd: V26_28_0, matching the release that ships their distroless migration (#36099). The original V26_20_0 predated the actual landing by ~8 releases (main is now 26.28-dev) and would have applied UID 65534 to v26.20-v26.27 images that still run as UID 999. NOTE: the env/clusterd gate assumes #36099 lands in the 26.28 cycle. If it slips, bump V26_28_0 to the actual release. The three distroless PRs (#36099 image, #36100 SIGTERM, #36101 this) must ship in the same release. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
main has moved to 26.32.0-dev and 26.28.0 already shipped as a Debian image. Leaving the gate at 26.28 would apply the nonroot 65534 securityContext to the still-Debian 26.28-26.31 env/clusterd images, which expect uid/gid 999. Bump the gate so only the genuinely distroless images get the nonroot context. The gate must equal the release the distroless env/clusterd images (#36099) first ship in. Set to 26.32 on the assumption this lands in the 26.32 cycle. Re-confirm against the actual release cut before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Trim the pod uid/gid comments to the non-obvious facts and drop the release-history narration. Also correct the env/clusterd comment that still said v26.28+ after the gate moved to v26.32. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 26.32 release was cut without the distroless images (#36099 has not merged), so gating at v26.32 would apply nonroot uid/gid 65534 to Ubuntu-based 26.32.x images that run as 999. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Address QA review findings on the pod security context gate. Use 65532 (the distroless `nonroot` user) instead of 65534 (`nobody`) for the environmentd/clusterd and balancerd gates. The base image runs `USER nonroot`, and Kubernetes `runAsUser` overrides the image user, so 65534 would have forced the pods to run as `nobody`, defeating the goal of matching the image UID/GID and breaking any nonroot-owned paths. Strip the `@sha256:...` digest in `parse_image_ref` before splitting on the tag separator. A digest-pinned ref like `env:v26.32.0@sha256:...` previously parsed the digest hex as the tag, failed, and fell into the parse-failure (assume-distroless/assume-newest) path, wrongly moving a pinned old image onto the new UID/GID. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jasonhernandez
force-pushed
the
jason/distroless-stacked-proof
branch
from
July 8, 2026 02:25
8f92c58 to
0785252
Compare
jasonhernandez
marked this pull request as ready for review
July 8, 2026 02:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Throwaway CI vehicle, never merges. Stacks #36099 (distroless environmentd/clusterd images) on #36101 (orchestratord UID/GID version gate) so the Nightly Orchestratord jobs exercise the real target combination: distroless images running as nonroot 65534 under the version-gated securityContext, in a kind cluster.
Each PR's own Nightly only tests half the story. #36101 alone applies the 65534 gate to Ubuntu-based images (no distroless images exist without #36099), and #36099 alone never runs the orchestratord kind tests with the gate present.
What to look at
The
Orchestratordgroup in the triggered Nightly:idle_when_fenced_out).Also of interest:
ssh-connection-*in PR CI against distroless (staticssh, tini reaping).Disposition
Close once #36099 and #36101 have both merged (or once this run has proven the combination). Findings go on the real PRs.
🤖 Generated with Claude Code