containers: switch environmentd and clusterd to a distroless base image - #37526
Merged
Conversation
jasonhernandez
force-pushed
the
jason/distroless-debug-999
branch
2 times, most recently
from
July 9, 2026 05:55
f64cc90 to
38ea068
Compare
Replace Debian-based prod images with distroless. Remove shell entrypoint scripts (no shell in distroless). Add libeatmydata to distroless-prod-base for CI compatibility. Update clusterd mzcompose service to drop shell-dependent options. Distroless ships no init, so add a tini-static mzbuild image (static tini built from source, mirroring openssh-static) and copy it into distroless-prod-base. tini stays PID 1 to forward signals and reap zombies, notably the ssh subprocesses spawned by the SSH tunnel feature. Leaf images wrap their binary in ENTRYPOINT ["/usr/bin/tini", "--", ...]. The deleted environmentd entrypoint slept forever on graceful exit so a fenced-out generation would not be restarted. Distroless has no shell to do this, and a Kubernetes StatefulSet (restartPolicy: Always) would crash-loop a process that exits. Replace it with an in-process idle: when fenced out and supervised directly by Kubernetes, environmentd idles until the orchestrator deletes the StatefulSet (SIGTERM, then SIGKILL) instead of calling exit!(0). The all-in-one image keeps its entrypoint and exit-0 behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nightly cloudtest and orchestratord runs crash-looped every distroless environmentd/clusterd pod with: error while loading shared libraries: libfdb_c.so: cannot open shared object file: No such file or directory ubuntu-base has copied libfdb_c.so from the foundationdb image since FoundationDB consensus/timestamp-oracle support landed (#33819), which postdates the original distroless base. Copy it the same way, and carry over ubuntu-base's RUST_BACKTRACE/RUST_LIB_BACKTRACE settings so crashes in distroless images print backtraces too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_crash_storage and the secrets restart test killed clusterd with kubectl exec ... bash -c 'kill -9 `pidof clusterd`'. Distroless clusterd images ship no shell, so the exec fails with exit 1 and the tests fail. Force-delete the pod instead (SIGKILL, no grace period), matching the crash-simulation idiom in test_replica_restart. The StatefulSet recreates the pod under the same name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The distroless base ships glibc's example nsswitch.conf, which routes passwd/group lookups through the `compat` NSS backend. `compat` is not built into glibc, so a lookup dlopens the image's libnss_compat.so.2. The statically-linked `ssh` embeds a different glibc than that module was built for, and the mismatch jumps through a garbage pointer: audit: ANOM_ABEND comm="ssh" exe="/usr/bin/ssh" sig=11 which the SSH tunnel manager surfaced as "failed to connect to the remote host: No such file or directory" in the Nightly ssh tunnel cloudtest. Ship a files/dns nsswitch.conf instead. Those backends are built into glibc (2.33+), so lookups never dlopen, for the static ssh and the dynamically-linked Rust binaries alike. Verified by rebuilding distroless-prod-base and running the CI-built static ssh inside it as uid 65532 and 65534 (`ssh -G` exits 0, was SIGSEGV). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same no-shell-in-distroless problem as the clusterd kills: replace the exec'd shell kill of environmentd with a force pod delete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Distroless environmentd/clusterd images ship no shell, so kubectl exec ... bash -c 'kill ...' cannot work. Add signal_process_in_pod, which execs into the kind node hosting the pod (a Docker container with a full userland) and signals the process found via crictl. This keeps the tests' original semantics: the process dies or suspends in place and the container restarts within the same pod, unlike a pod delete. It also covers the shared-fate SIGSTOP/SIGCONT cases, which simulate a hung peer and cannot be expressed as a pod delete at all. The helper finds the target among the container's init and its direct children, covering both distroless images (tini is init, target is its child) and Ubuntu-based images (the entrypoint execs the target). Convert test_crash, test_secrets, and both shared-fate suites to the helper, including reverting the earlier pod-delete workaround. Mechanics verified against a local kind cluster (crictl discovery, comm matching, kill -s from the node). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The new tini-static image was pinned to ubuntu:noble-20260210.1, the stale base copied from openssh-static before #37354 bumped it. Match the repo's current base (noble-20260610, per ubuntu-base #37449). Build-stage only; the shipped artifact is the static tini binary from a FROM scratch stage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Match ubuntu-base and tini-static at noble-20260610 (openssh-static was one dependabot cycle behind at noble-20260509.1). Aligning all static builder bases to one tag means the distroless build pulls a single ubuntu image rather than several. Build-stage only; the artifact is the static ssh binary from a FROM scratch stage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tini-static and distroless-prod-base are new images introduced by this migration. Register both with the docker dependabot ecosystem so their base-image pins (ubuntu/debian/gcr.io distroless) stay current on the same weekly cadence as ubuntu-base and openssh-static, rather than drifting until someone notices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…prod-base The libfdb_c.so tag must stay in lockstep with the identical COPY in ubuntu-base and the foundationdb crate's fdb-7_3 API feature. Spell that out so a dependabot bump to one of the three gets recognized as a drift bug in review rather than merged as routine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lint suite requires a copyright header on config files; the nsswitch.conf added for the static-ssh fix was missing one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Draft variant of the distroless migration that keeps a shell and matches the orchestrator's uid, as an incremental step off the Ubuntu base. - distroless-prod-base: switch to the `debug-nonroot` variant of the same distroless build the nonroot pin tracked, which ships a busybox shell at /busybox/sh. Create a `materialize` user at uid/gid 999 in a Debian builder stage and copy its passwd/group/home into the final image, so the process uid matches the orchestrator's hardcoded runAsUser/fsGroup and no orchestrator UID gate is needed. - Restore src/clusterd/ci/entrypoint.sh and src/environmentd/ci/entrypoint.sh, ported to busybox ash (#!/busybox/sh, `hostname -f`, POSIX `[`). The leaf Dockerfiles run them under tini again. - Revert the cloudtest kill helpers to shelling into the pod, now that the image has a shell, using `sh` (busybox) rather than `bash`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ztDqVCrRgMdtjJxvPWa4B
jasonhernandez
force-pushed
the
jason/distroless-debug-999
branch
from
July 9, 2026 06:29
38ea068 to
b45239c
Compare
jasonhernandez
marked this pull request as ready for review
July 9, 2026 06:33
jasonhernandez
changed the title
Empty commit to regenerate the CI pipeline now that the PR is ready and labeled ci-nightly, so the image-triggered nightly (cloudtest / orchestratord / k8s) runs against the 999 debug image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
environmentd and clusterd dynamically link liblzma.so.5 (xz), which the distroless base does not include (ubuntu-base gets it implicitly as a system package). Without it both binaries abort at startup with "error while loading shared libraries: liblzma.so.5: cannot open shared object file", so every pod crash-loops and all orchestratord/cloudtest/k8s nightly jobs fail. Copy it from the Debian stage, matching the ubuntu-base runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jasonhernandez
changed the title
Contributor
Author
|
FYI I also have a path to shrink the discrepancy between materialized and these repos - moving materialized to debian:13-slim is feasible with a bump to postgres 17. |
Contributor
|
How does this relate to #36099 ? |
2 tasks
2 tasks
Contributor
Author
|
I'm closing #36099 and its related PRs - this will supersede it as a simpler, incremental move to dramatically slim down our footprint. I think removing the shell requires more significant work that I'm leaving out of scope for now. |
2 tasks
def-
approved these changes
Jul 13, 2026
def-
left a comment
def-
left a comment
Contributor
There was a problem hiding this comment.
If CI is happy, I'm happy. I don't know if this causes any downstream effects for the Cloud team though.
Contributor
|
CI was not happy! https://buildkite.com/materialize/nightly/builds/17100 failed in Cloudtest, related to this PR. |
def-
added a commit
to def-/materialize
that referenced
this pull request
Jul 14, 2026
def-
added a commit
to def-/materialize
that referenced
this pull request
Jul 14, 2026
def-
added a commit
to def-/materialize
that referenced
this pull request
Jul 14, 2026
def-
added a commit
that referenced
this pull request
Jul 14, 2026
Based on https://buildkite.com/materialize/nightly/builds/17267 Follow-ups to #37526 & #37588 & #37493 Test run: https://buildkite.com/materialize/nightly/builds/17295
Contributor
|
Terraform and Canary Deploy in Staging Cloud also seem to fail in main related to this. I'm trying to get it fixed: https://buildkite.com/materialize/nightly/builds/17334 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrate the
environmentdandclusterdcontainer images from Ubuntu to a distroless base, shrinking the image and its attack surface.This is a lighter approach compared to #36099 / #36101 / #36876 which attempted to entirely remove the shell. I've closed those.
We keep a shell, but it becomes busybox and we aim for pure POSIX / sh compatibility instead of implicitly depending on bash.
Base image
gcr.io/distroless/cc-debian13:debug-nonroot. Thedebugvariant ships a busybox shell at/busybox/sh. Keeping a shell letsentrypoint.shandkubectl execcontinue to work. This is a deliberate, incremental step off Ubuntu; dropping the shell is a possible future hardening.materializeuser with uid/gid 999, created in a Debian builder stage and copied into the final image (distroless has no shell to runuseradd). uid/gid 999 matches the pod securityContext the orchestrator already applies (runAsUser/runAsGroup/fsGroup = 999), so nothing on the orchestrator side changes and existing uid-999 persistent volumes need no re-chown.Runtime dependencies copied in
The distroless base is minimal, so the image explicitly provides what the binaries need at runtime:
tinias PID 1, to forward signals and reap zombies (e.g. ssh-tunnel subprocesses).ssh, since distroless ships none and SSH tunnels require it. (busybox includes some SSH features but we prefer to use openssh built with aws-lc for eventual FIPS support + stronger cryptographic foundations)libfdb_c.so(FoundationDB) andliblzma.so.5(xz), both dynamically linked by the binaries. Withoutliblzmathey abort at startup witherror while loading shared libraries.nsswitch.confusing thefiles/dnsbackends, which avoids a glibccompat-backend segfault in the statically-linkedssh.This might look ugly, but we know what we're shipping vs. the entire Ubuntu userspace!
Entrypoint
entrypoint.sh(for both binaries) is POSIXsh:#!/bin/sh,set -eu, andhostname -f(works under both busybox and GNU). The Dockerfiles invoke it as/busybox/sh entrypoint.sh, since distroless has no/bin/sh. Under Kubernetes it advertises the pod FQDN for the CTP peer check and derives the process ordinal from the StatefulSet pod name.Scope
Container layer only, no product-code changes: image definitions, the two entrypoints, cloudtest adjustments (pod processes are signalled via
kubectl exec … sh -c 'kill'), and dependabot coverage for the new base images.Test plan
HOME=/home/materialize; the entrypoint executes end-to-end andhostname -fresolves.Cloudtestshard flakes independently of this change; it also fails onmain.)