Skip to content

feat(gateway): add hashed service-routing hostnames for multi-gateway clusters - #4842

Open
jhjaggars wants to merge 1 commit into
NVIDIA:mainfrom
jhjaggars:feat/4840-gateway-name-service-hostname/jhjaggars
Open

jhjaggars wants to merge 1 commit into
NVIDIA:mainfrom
jhjaggars:feat/4840-gateway-name-service-hostname/jhjaggars

Conversation

@jhjaggars

Copy link
Copy Markdown
Contributor

🏗️ build-from-issue-agent

Summary

Adds an optional service_routing_gateway_name gateway setting that disambiguates exposed-service URLs when multiple gateways share a Kubernetes cluster, ingress, and wildcard certificate. When configured, service hostnames become a deterministic 16-character lowercase base32 SHA-256 label of (gateway name, workspace, sandbox, service) under the same base domain — exactly one label deep, so a single shared *.domain wildcard certificate keeps working, and each gateway's URLs are globally unique. Unset gateways preserve the existing readable hostname format byte-for-byte.

Related Issue

Closes #4840

Changes

  • crates/openshell-core/src/config.rs: ServiceRoutingConfig.service_routing_gateway_name field, with_service_routing_gateway_name builder, DNS-1123-label validation, and config tests.
  • crates/openshell-server/src/config_file.rs: [openshell.gateway] service_routing_gateway_name TOML parsing/serialization.
  • crates/openshell-server/src/cli.rs: --service-routing-gateway-name / OPENSHELL_SERVICE_ROUTING_GATEWAY_NAME with existing CLI > env > TOML precedence, startup validation, and precedence tests.
  • crates/openshell-server/src/service_routing.rs: SHA-256/NUL-delimited tuple hashing with a hand-rolled fixed-size lowercase base32 encoder (no new dependency), hashed endpoint_host output, ServiceRouteIndex (register/reserve/remove/lookup/rebuild), async state-aware resolve_service_host supporting both hashed and legacy hosts, and routing tests with fixed vectors.
  • crates/openshell-server/src/lib.rs: route index in ServerState, hydrated from persisted ServiceEndpoint records at startup before listeners bind; ambiguous persisted hashes fail startup.
  • crates/openshell-server/src/http.rs: HTTP middleware and loopback plaintext router use the async state-aware service-host classification.
  • crates/openshell-server/src/grpc/service.rs: hash route reserved before endpoint persistence (AlreadyExists on collision, no write), rolled back on persistence failure, removed after successful delete; collision test.
  • crates/openshell-server/src/compute/mod.rs: sandbox cleanup removes route index entries for deleted endpoints.
  • deploy/helm/openshell/: server.serviceRoutingGatewayName value rendered into gateway TOML via _toml.tpl (empty stays unset; explicit gatewayConfig takes precedence), chart tests, regenerated README.
  • Docs: docs/how-it-works/gateways/configuration.mdx (field reference), docs/how-it-works/gateways/overview.mdx (hostname grammar — also fixes the pre-existing omission of the workspace segment), docs/how-it-works/sandboxes/overview.mdx, docs/kubernetes/ingress.mdx (operator-managed exact-host ingress guidance for multi-gateway shared-cert deployments), docs/kubernetes/setup.mdx.

Deviations from Plan

None functionally. The route reservation API internally tracks whether the current call inserted the route so persistence-failure rollback cannot remove a pre-existing idempotent reservation from a concurrent request (an implementation detail within the plan's intent).

Testing

  • mise run pre-commit passes (format, license headers, Helm lint) — with one pre-existing, unrelated failure: rust:lockfiles:check rejects examples/supervisor-middleware-content-guard/Cargo.lock because clap 4.6.7 was yanked from crates.io; that lockfile is untouched by this PR and fails on unmodified main as well.
  • Unit tests added/updated
  • E2E tests: N/A — no changes to sandbox/supervisor protocols, persistence schemas, or service relay transport; no files under e2e/ were modified.

Tests added:

  • Unit:
    • openshell-core config: default/accept/reject validation, with_server_sans preserves the routing name.
    • openshell-server service_routing: fixed hash vectors (enh47uqyj7bpyttz, eiud2z5apv3yeucg), hashed URL construction, index idempotency/removal/collision/rebuild-from-store, configured-hash host resolution for Host header and HTTP/2 authority, legacy-host acceptance when configured, unknown-hash fallthrough.
    • openshell-server grpc::service: expose_service_rejects_route_hash_collision_before_persistence.
    • openshell-server config_file/cli: TOML parse/serialize, file-populated value, CLI/env override precedence, effective-config reach.
    • openshell-cli: service_url_for_gateway preserves the hashed label while normalizing the port.
  • Integration: index rehydration from a shared store (restart path) and cross-state miss-triggered refresh, exercised in openshell-server service_routing tests.
  • E2E: N/A (see above).

Verification runs (local):

  • cargo fmt --check, git diff --check — pass
  • cargo clippy -p openshell-core -p openshell-server -p openshell-cli --all-targets -- -D warnings — pass
  • cargo test -p openshell-core — 558 unit + 1 doctest pass
  • cargo test -p openshell-server — 2205 pass (20 Postgres-dependent ignored as expected)
  • cargo test -p openshell-cli service_url — pass
  • mise run helm:test — 345 chart tests pass

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Documentation updated:

  • docs/how-it-works/gateways/configuration.mdx: new TOML field reference (default, validation, precedence, hash URL example, distinction from gateway_jwt.gateway_id)
  • docs/how-it-works/gateways/overview.mdx: corrected hostname grammar (workspace segment) + legacy vs hashed formats
  • docs/how-it-works/sandboxes/overview.mdx: consume printed URLs, not constructed hostnames
  • docs/kubernetes/ingress.mdx: exact-host ingress guidance for multi-gateway shared-wildcard-cert deployments
  • docs/kubernetes/setup.mdx: chart-values table entry
  • deploy/helm/openshell/README.md: regenerated chart values docs

@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@TaylorMutch

Copy link
Copy Markdown
Collaborator

/ok to test 3f1fd80

@jhjaggars

jhjaggars commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor Author

🏗️ build-from-issue-agent

Cluster Verification Report

Verified end-to-end on a real multi-tenant OpenShift (ROSA) cluster with two gateways sharing one base domain and wildcard certificate — the exact ambiguous scenario from #4840.

Environment

Item Value
Cluster ROSA, mixed-arch (arm64 + amd64), OpenShift Routes as the shared ingress
Gateways openshell-a (name gateway-a) and openshell-b (name gateway-b), separate namespaces, images built from PR head 3f1fd809bda8
Shared base domain operator-owned zone (shown here as apps.example.com) via server_sans = ["*.apps.example.com"] on both
Wildcard certificate Single Let's Encrypt *.apps.example.com (cert-manager DNS-01), presented by both gateways through passthrough routes
Auth OIDC via existing Keycloak realm (device grant)
Sandboxes Identical workspace=default, sandbox=demo, service=web (HTTP servers returning distinct content) exposed on both gateways

Results

# Test Result
T1 Distinct hashed hostnames for identical workspace/sandbox/service across gateways ✅ pyldrduzj554prit vs 2scisywrf3egxnmz; both match independently computed SHA-256/NUL-joined/10-byte/lowercase-base32 values
T2 Correct routing through shared ingress — the reported bug scenario ✅ hash-a → served-by-gateway-a, hash-b → served-by-gateway-b; no cross-talk; full TLS verification (LE wildcard presented, ssl_verify_result=0)
T3 Legacy readable hostnames (default--demo--web.<base-domain>) still accepted on a hash-configured gateway ✅ routed correctly (transition compatibility)
T4 Gateway pod restart → route index rehydration from persisted endpoints ✅ hashed URL routes correctly after restart
T5 URL regeneration ✅ service list regenerates identical hashed URLs on both gateways
T6 Unknown hash-shaped hostname falls through to normal routing ✅ HTTP 404, not proxied to any sandbox

Operator-managed ingress was used per the PR's documented model: exact-host passthrough Routes created per emitted hashed hostname, each targeting its own gateway's Service.

Also exercised during setup: Helm rendering of server.serviceRoutingGatewayName into gateway.toml (service_routing_gateway_name = "gateway-a"), image multi-arch (arm64+amd64 manifest lists), and chart install on OpenShift with cert-manager TLS and podSecurityContext/securityContext UID overrides for restricted-v2 SCC (note: fixed runAsUser: 1000/fsGroup: 1000 defaults required nulling on OpenShift — possibly worth a follow-up for OpenShift users).

… clusters

Closes NVIDIA#4840

Add an optional service_routing_gateway_name gateway setting (TOML, CLI,
env, Helm) that disambiguates exposed service URLs when multiple gateways
share a cluster, ingress, and wildcard certificate. When configured,
exposed-service hostnames become a deterministic 16-character lowercase
base32 SHA-256 label of (gateway name, workspace, sandbox, service) under
the same base domain, keeping every hostname exactly one label deep for
wildcard-certificate compatibility. Unset gateways preserve the existing
readable hostname format byte-for-byte, and configured gateways still
accept legacy hosts during migration.

Hashed hosts resolve through an in-memory route index hydrated from
persisted ServiceEndpoint records at startup, refreshed on miss for
multi-replica deployments, maintained across expose, delete, and sandbox
cleanup, and enforced for collisions before endpoint persistence.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
@jhjaggars
jhjaggars force-pushed the feat/4840-gateway-name-service-hostname/jhjaggars branch from 3f1fd80 to 28551bd Compare October 23, 2026 00:35
@jhjaggars

Copy link
Copy Markdown
Contributor Author

Rebased onto the latest main (9a85667d0) — new head is 28551bd8. The only conflict resolution was in crates/openshell-server/src/compute/mod.rs, where the new supervisor_host field from #4702 landed alongside the service_route_index field added here; both were kept. Full local verification passed: clippy, fmt, openshell-core (581 tests), openshell-server suite (2,443 tests), CLI service-URL tests, and helm:test including the config-parser preflight across all CI overlays.

@TaylorMutch the branch was force-pushed for the rebase, so the earlier /ok to test (pinned to 3f1fd809b) no longer covers the head — could you re-approve for 28551bd8 when you have a moment?

@mrunalp

mrunalp commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 28551bd

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add gateway discriminator to exposed service hostnames for multi-gateway clusters

3 participants