orchestratord: version-gate pod security context UID/GID for distroless by jasonhernandez · Pull Request #36101 · MaterializeInc/materialize

jasonhernandez · 2026-04-15T05:16:36Z

Summary

Distroless images run as nonroot (uid/gid 65534) instead of root. This version-gates the pod security context in orchestratord so each Materialize version gets the matching runAsUser/runAsGroup, avoiding UID mismatches during rolling upgrades.

Gates: balancerd V26_18_0 (already distroless), environmentd/clusterd V26_33_0. The env/clusterd gate must equal the release #36099 ships in; see the coordination comment below.

Test plan

cargo test -p mz-orchestratord passes
Pods for new versions get nonroot uid/gid, old versions keep root

🤖 Generated with Claude Code

github-actions · 2026-04-15T05:16:44Z

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

The PR title is descriptive and will make sense in the git log.
This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

Distroless images run as nonroot (UID 65534) instead of root. Add version-gating so orchestratord sets the correct runAsUser/runAsGroup based on the Materialize version, avoiding UID mismatches during rolling upgrades from Debian-based to distroless images. Gate versions (verified against release history, 2026-06): - balancerd: V26_18_0. Its ci/Dockerfile switched to distroless-prod-base in v26.18.0 (prod-base in v26.17.x). The original V26_19_0 was off by one and would have forced UID 999 onto v26.18.x balancerd pods that actually run as 65534. - environmentd/clusterd: V26_28_0, matching the release that ships their distroless migration (#36099). The original V26_20_0 predated the actual landing by ~8 releases (main is now 26.28-dev) and would have applied UID 65534 to v26.20-v26.27 images that still run as UID 999. NOTE: the env/clusterd gate assumes #36099 lands in the 26.28 cycle. If it slips, bump V26_28_0 to the actual release. The three distroless PRs (#36099 image, #36100 SIGTERM, #36101 this) must ship in the same release. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jasonhernandez · 2026-07-01T17:36:09Z

🔗 Distroless migration — coordination

Part of a set that ships together: #36099 (distroless image), #36872 (clusterd process ordinal, merged), #36876 (removes the unused CTP FQDN check), and this PR (orchestratord UID/GID gating).

This PR version-gates the pod securityContext so distroless images (uid/gid 65534) and older Ubuntu-based images (999) each get the correct values.

Merge order: this PR lands before or with #36099. The gate is version-conditional, so it is a no-op until distroless images of the gated version exist and is safe to land ahead. If #36099 ships nonroot images while this gate is absent, orchestratord applies the wrong UID/GID.

Gates:

balancerd: V26_18_0 (distroless since v26.18.0).
environmentd/clusterd: V26_33_0. Must equal the release containers: switch environmentd and clusterd to distroless base image #36099 ships in. The 26.32 cycle passed without containers: switch environmentd and clusterd to distroless base image #36099 landing, so the gate was bumped from V26_32_0 (2026-07-06). If containers: switch environmentd and clusterd to distroless base image #36099 lands outside the 26.33 cycle, bump this constant to match.

Separate, pre-existing: balancerd runs distroless (uid 65534) as of v26.18.0 but this gating is not yet merged, so balancerd pods can carry a UID mismatch today unless enable_security_context is off in practice. Tracked independently.

…36872) ## Summary Distroless clusterd has no shell entrypoint to set `CLUSTERD_PROCESS` (the process ordinal), which `entrypoint.sh` derived from the pod hostname before the distroless migration (#36099) removes it. clusterd now derives it itself when running under Kubernetes and the value is unset: the ordinal is the last `-`-delimited segment of the StatefulSet pod name, matching how orchestrator-kubernetes parses the process id back out of pod names. No signal handling is added. tini (added to the distroless image in #36099) runs as PID 1 and forwards signals, so clusterd is a child process and SIGTERM takes its default action rather than being ignored by PID 1. Part of the distroless migration: #36099 (image), #36101 (UID/GID gating), #36876 (removes the unused CTP FQDN check, stacks on this). Supersedes the closed #36100. ## Merge order Land before or with #36099. Without it, distroless clusterd has no process ordinal and won't start. Safe to land first: on the current Ubuntu images `entrypoint.sh` still sets `CLUSTERD_PROCESS`, and the in-process derivation only runs when it is unset. ## Test plan - [x] `cargo check -p mz-clusterd` - [ ] Confirm a distroless clusterd pod in k8s picks up its ordinal and joins the cluster Co-authored-by: Jason Hernandez <7144515+jasonhernandez@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Distroless images run as nonroot (UID 65534) instead of root. Add version-gating so orchestratord sets the correct runAsUser/runAsGroup based on the Materialize version, avoiding UID mismatches during rolling upgrades from Debian-based to distroless images. Gate versions (verified against release history, 2026-06): - balancerd: V26_18_0. Its ci/Dockerfile switched to distroless-prod-base in v26.18.0 (prod-base in v26.17.x). The original V26_19_0 was off by one and would have forced UID 999 onto v26.18.x balancerd pods that actually run as 65534. - environmentd/clusterd: V26_28_0, matching the release that ships their distroless migration (#36099). The original V26_20_0 predated the actual landing by ~8 releases (main is now 26.28-dev) and would have applied UID 65534 to v26.20-v26.27 images that still run as UID 999. NOTE: the env/clusterd gate assumes #36099 lands in the 26.28 cycle. If it slips, bump V26_28_0 to the actual release. The three distroless PRs (#36099 image, #36100 SIGTERM, #36101 this) must ship in the same release. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

main has moved to 26.32.0-dev and 26.28.0 already shipped as a Debian image. Leaving the gate at 26.28 would apply the nonroot 65534 securityContext to the still-Debian 26.28-26.31 env/clusterd images, which expect uid/gid 999. Bump the gate so only the genuinely distroless images get the nonroot context. The gate must equal the release the distroless env/clusterd images (#36099) first ship in. Set to 26.32 on the assumption this lands in the 26.32 cycle. Re-confirm against the actual release cut before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Trim the pod uid/gid comments to the non-obvious facts and drop the release-history narration. Also correct the env/clusterd comment that still said v26.28+ after the gate moved to v26.32. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The 26.32 release was cut without the distroless images (#36099 has not merged), so gating at v26.32 would apply nonroot uid/gid 65534 to Ubuntu-based 26.32.x images that run as 999. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Alphadelta14 · 2026-07-07T12:46:46Z

i was very confused walking into the review when i was like "omg how did i miss distroless being merged" but now I have the context of the comments...

Alphadelta14

good luck 🤞🏻
general approach seems fine (though I am not a huge fan of the version conditionals and how those will need to be updated-- but that's just how this works).

def-

Two concerns from the QA LLM review:

Finding 1 - MEDIUM - Distroless pods are forced to run as `nobody`, not the image's `nonroot` user

The PR hard-codes the distroless UID/GID as 65534 in both the balancerd gate and the environmentd/clusterd gate:

src/orchestratord/src/controller/balancer.rs:164-185
src/orchestratord/src/controller/materialize/generation.rs:901-912

That is not the UID/GID of the distroless nonroot user used by Materialize's distroless base image. The checked-in base image is gcr.io/distroless/cc-debian13:nonroot-28078d2e5e77671d2046dcc9e2c75334e31efa4d with USER nonroot, and inspecting that exact image config reports User = "65532". Its /etc/passwd maps nonroot:x:65532:65532 and nobody:x:65534:65534. The operator Helm chart also already runs the distroless orchestratord pod as 65532 for fsGroup, runAsGroup, and runAsUser.

Because Kubernetes runAsUser overrides the image user, this PR makes distroless balancerd, environmentd, and clusterd run as nobody instead of the image's configured nonroot user. That defeats the stated goal of matching the pod security context to the image UID/GID. It can also break any file or directory in the image that is readable or writable only by nonroot, and it leaves $HOME=/home/nonroot paired with the nobody identity. For clusterd this value is passed through --orchestrator-kubernetes-service-fs-group and consumed as run_as_user, run_as_group, and fs_group in src/orchestrator-kubernetes/src/lib.rs:1121-1126, so the blast radius includes replicas launched by the new environmentd.

Use 65532 for the distroless case, or make the images deliberately run as nobody and update the base image, chart, comments, and tests consistently.

Tracker check: GitHub searches for 65534 nonroot distroless, runAsUser 65534, and distroless uid 65532 did not find a duplicate in MaterializeInc/database-issues or MaterializeInc/materialize. Linear searches found only the broader distroless migration tickets DB-44 and CLO-26, not this UID mismatch.

Finding 2 - MEDIUM - Digest-pinned old image tags are misclassified as distroless

The newly public parse_image_ref still splits the image ref with rsplit_once(':') and immediately treats the right-hand side as the tag:

src/cloud-resources/src/crd/materialize.rs:1621-1632

That does not handle valid OCI references of the form image:v26.17.0@sha256:.... For such refs, the final colon is the digest algorithm separator, so parse_image_ref sees the digest hex as the tag and returns None. This PR's new gates then treat None as distroless:

balancerd returns true from the parse-failure arm and uses the distroless UID/GID at src/orchestratord/src/controller/balancer.rs:175-185.
environmentd/clusterd use Materialize::meets_minimum_version, whose parse-failure behavior is also to return true, before setting service_fs_group at src/orchestratord/src/controller/materialize/generation.rs:909-917.

The result is that a valid digest-pinned old image, for example materialize/balancerd:v26.17.0@sha256:... or materialize/environmentd:v26.32.0@sha256:..., is no longer kept on the old 999 UID/GID path. It gets the distroless path solely because a digest was appended. Digest pinning is a realistic deployment configuration for self-managed operators, and this breaks the PR's rolling-upgrade safety property for those users.

The existing parse_image_tag helper in src/orchestratord/src/lib.rs already handles :tag@sha256:... by stripping the digest before splitting on the tag. parse_image_ref should do the same, then parse the stripped tag as semver. Add tests for v26.17.0@sha256:... staying below the balancerd gate and v26.32.0@sha256:... staying below the environmentd/clusterd gate.

Address QA review findings on the pod security context gate. Use 65532 (the distroless `nonroot` user) instead of 65534 (`nobody`) for the environmentd/clusterd and balancerd gates. The base image runs `USER nonroot`, and Kubernetes `runAsUser` overrides the image user, so 65534 would have forced the pods to run as `nobody`, defeating the goal of matching the image UID/GID and breaking any nonroot-owned paths. Strip the `@sha256:...` digest in `parse_image_ref` before splitting on the tag separator. A digest-pinned ref like `env:v26.32.0@sha256:...` previously parsed the digest hex as the tag, failed, and fell into the parse-failure (assume-distroless/assume-newest) path, wrongly moving a pinned old image onto the new UID/GID. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Distroless images run as nonroot (UID 65534) instead of root. Add version-gating so orchestratord sets the correct runAsUser/runAsGroup based on the Materialize version, avoiding UID mismatches during rolling upgrades from Debian-based to distroless images. Gate versions (verified against release history, 2026-06): - balancerd: V26_18_0. Its ci/Dockerfile switched to distroless-prod-base in v26.18.0 (prod-base in v26.17.x). The original V26_19_0 was off by one and would have forced UID 999 onto v26.18.x balancerd pods that actually run as 65534. - environmentd/clusterd: V26_28_0, matching the release that ships their distroless migration (#36099). The original V26_20_0 predated the actual landing by ~8 releases (main is now 26.28-dev) and would have applied UID 65534 to v26.20-v26.27 images that still run as UID 999. NOTE: the env/clusterd gate assumes #36099 lands in the 26.28 cycle. If it slips, bump V26_28_0 to the actual release. The three distroless PRs (#36099 image, #36100 SIGTERM, #36101 this) must ship in the same release. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

alex-hunt-materialize

LGTM, assuming there is another PR that changes the actual containers that will be merged in the same release.

jasonhernandez · 2026-07-09T19:27:48Z

closing this in favor of the simpler #37526 approach

…ge (#37526) Migrate the `environmentd` and `clusterd` container images from Ubuntu to a distroless base, shrinking the image and its attack surface.<br>This is a lighter approach compared to #36099 / #36101 / #36876 which attempted to entirely remove the shell. I've closed those. <br>We keep a shell, but it becomes busybox and we aim for pure POSIX / sh compatibility instead of implicitly depending on bash. ## Base image * Uses `gcr.io/distroless/cc-debian13:debug-nonroot`. The `debug` variant ships a busybox shell at `/busybox/sh`. Keeping a shell lets `entrypoint.sh` and `kubectl exec` continue to work. This is a deliberate, incremental step off Ubuntu; dropping the shell is a possible future hardening. * Runs as a `materialize` user with uid/gid **999**, created in a Debian builder stage and copied into the final image (distroless has no shell to run `useradd`). uid/gid 999 matches the pod securityContext the orchestrator already applies (`runAsUser`/`runAsGroup`/`fsGroup = 999`), so nothing on the orchestrator side changes and existing uid-999 persistent volumes need no re-chown. ## Runtime dependencies copied in The distroless base is minimal, so the image explicitly provides what the binaries need at runtime: * **static** `tini` as PID 1, to forward signals and reap zombies (e.g. ssh-tunnel subprocesses). * **static** `ssh`, since distroless ships none and SSH tunnels require it. (busybox includes some SSH features but we prefer to use openssh built with aws-lc for eventual FIPS support + stronger cryptographic foundations) * `libfdb_c.so` (FoundationDB) and `liblzma.so.5` (xz), both dynamically linked by the binaries. Without `liblzma` they abort at startup with `error while loading shared libraries`. * a `nsswitch.conf` using the `files`/`dns` backends, which avoids a glibc `compat`-backend segfault in the statically-linked `ssh`. This might look ugly, but we know what we're shipping vs. the entire Ubuntu userspace! ## Entrypoint `entrypoint.sh` (for both binaries) is POSIX `sh`: `#!/bin/sh`, `set -eu`, and `hostname -f` (works under both busybox and GNU). The Dockerfiles invoke it as `/busybox/sh entrypoint.sh`, since distroless has no `/bin/sh`. Under Kubernetes it advertises the pod FQDN for the CTP peer check and derives the process ordinal from the StatefulSet pod name. ## Scope Container layer only, no product-code changes: image definitions, the two entrypoints, cloudtest adjustments (pod processes are signalled via `kubectl exec … sh -c 'kill'`), and dependabot coverage for the new base images. ## Test plan - [X] Standalone image on the pinned base runs as uid 999 with `HOME=/home/materialize`; the entrypoint executes end-to-end and `hostname -f` resolves. - [X] CI nightly cloudtest / orchestratord / k8s-recovery jobs pass: managed replicas come up healthy as uid 999 and the CTP check passes against the in-cluster pod FQDN. (One `Cloudtest` shard flakes independently of this change; it also fails on `main`.) --------- Co-authored-by: Jason Hernandez <7144515+jasonhernandez@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jasonhernandez mentioned this pull request Apr 15, 2026

containers: prepare environmentd and clusterd for distroless migration #35859

Closed

4 tasks

jasonhernandez force-pushed the jason/distroless-orchestratord-uid branch from 89eab31 to cf2002f Compare June 2, 2026 17:03

jasonhernandez force-pushed the jason/distroless-orchestratord-uid branch from cf2002f to cbe3af5 Compare June 30, 2026 21:53

jasonhernandez force-pushed the jason/distroless-orchestratord-uid branch from a033313 to aa2ade1 Compare July 6, 2026 23:51

jasonhernandez added the ci-nightly PR CI control: also trigger Nightly label Jul 7, 2026

jasonhernandez and others added 4 commits July 6, 2026 17:55

jasonhernandez force-pushed the jason/distroless-orchestratord-uid branch from aa2ade1 to 7b7add8 Compare July 7, 2026 00:56

jasonhernandez mentioned this pull request Jul 7, 2026

[DO NOT MERGE] distroless: prove image + UID gate together (stacked #36099 + #36101) #37466

Closed

jasonhernandez marked this pull request as ready for review July 7, 2026 12:28

jasonhernandez requested a review from a team as a code owner July 7, 2026 12:28

jasonhernandez requested review from Alphadelta14, alex-hunt-materialize and doy-materialize and removed request for a team and alex-hunt-materialize July 7, 2026 12:28

Alphadelta14 approved these changes Jul 7, 2026

View reviewed changes

Alphadelta14 self-requested a review July 7, 2026 12:51

alex-hunt-materialize approved these changes Jul 7, 2026

View reviewed changes

def- reviewed Jul 7, 2026

View reviewed changes

alex-hunt-materialize approved these changes Jul 8, 2026

View reviewed changes

jasonhernandez mentioned this pull request Jul 9, 2026

containers: switch environmentd and clusterd to a distroless base image #37526

Merged

2 tasks

jasonhernandez closed this Jul 9, 2026

Navigation Menu

Uh oh!

orchestratord: version-gate pod security context UID/GID for distroless - #36101

orchestratord: version-gate pod security context UID/GID for distroless#36101
jasonhernandez wants to merge 5 commits into
mainfrom
jason/distroless-orchestratord-uid

jasonhernandez commented Apr 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 15, 2026

Uh oh!

jasonhernandez commented Jul 1, 2026 •

edited

Loading

Uh oh!

Alphadelta14 commented Jul 7, 2026

Uh oh!

Alphadelta14 left a comment

Uh oh!

def- left a comment

Uh oh!

alex-hunt-materialize left a comment

Uh oh!

jasonhernandez commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jasonhernandez commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

github-actions Bot commented Apr 15, 2026

PR title guidelines

Pre-merge checklist

Uh oh!

jasonhernandez commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Distroless migration — coordination

Uh oh!

Alphadelta14 commented Jul 7, 2026

Uh oh!

Alphadelta14 left a comment

Choose a reason for hiding this comment

Uh oh!

def- left a comment

Choose a reason for hiding this comment

Finding 1 - MEDIUM - Distroless pods are forced to run as nobody, not the image's nonroot user

Finding 2 - MEDIUM - Digest-pinned old image tags are misclassified as distroless

Uh oh!

alex-hunt-materialize left a comment

Choose a reason for hiding this comment

Uh oh!

jasonhernandez commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jasonhernandez commented Apr 15, 2026 •

edited

Loading

jasonhernandez commented Jul 1, 2026 •

edited

Loading

Finding 1 - MEDIUM - Distroless pods are forced to run as `nobody`, not the image's `nonroot` user