Deployment Guide

Deploy madengine workloads to Kubernetes or SLURM clusters for distributed execution.

Overview

madengine supports these deployment backends:

Kubernetes - Cloud-native container orchestration
SLURM - HPC cluster job scheduling
llm-d - Benchmarking a Kubernetes-native distributed inference stack; see llm-d.md

Deployment is configured via --additional-context and happens automatically during the run phase.

Deployment Workflow

Build once, then deploy the resulting manifest to any target:

flowchart LR
    B["1. Build Phase<br/>(local or CI/CD)<br/>madengine build --tags model"] --> M[(build_manifest.json<br/>+ image in registry)]
    M --> D["2. Deploy Phase<br/>madengine run --manifest-file build_manifest.json<br/>--additional-context '{...}'"]
    D --> T{detect target}
    T -->|llm_d| I[llm-d benchmark Job]
    T -->|k8s / kubernetes| K[K8s Job]
    T -->|slurm| S[SLURM script]
    T -->|neither| L[Local Docker]

Deployment Target Inference

No explicit deploy field is required — the target is inferred from the config structure (Convention over Configuration):

flowchart TD
    A[additional_context] --> Q{which key?}
    Q -->|llm_d| I[llm-d benchmark]
    Q -->|k8s / kubernetes| K[Kubernetes deployment]
    Q -->|slurm| S[SLURM deployment]
    Q -->|neither| L[Local Docker execution]

llm_d is checked first: an llm-d config also carries a k8s block, which configures the benchmark-client Job.

Kubernetes Job naming

Batch Jobs force the pod template label job-name to equal metadata.name, and label values are at most 63 characters. madengine therefore builds Job names with sanitize_k8s_object_name(..., max_total_len=63) so long model card paths (for example llm-d MoE cards) hash/truncate instead of failing admission. PVC and ConfigMap names append a short suffix (-results, -config) and may still use the longer RFC 1123 subdomain limit (253).

Kubernetes Deployment

Prerequisites

Kubernetes cluster with GPU support
GPU device plugin installed (AMD or NVIDIA)
Kubeconfig configured (~/.kube/config or in-cluster)
Docker registry accessible from cluster

Quick Start

Minimal Configuration (Recommended)

{
  "k8s": {
    "gpu_count": 1
  }
}

This automatically applies intelligent defaults for namespace, resources, image pull policy, etc.

Build and Deploy

# 1. Build image
madengine build --tags my_model \
  --registry my-registry.io \
  --additional-context-file k8s-config.json

# 2. Deploy to Kubernetes
madengine run \
  --manifest-file build_manifest.json \
  --timeout 3600

The deployment target is automatically detected from the k8s key in the config.

Configuration Options

k8s-config.json:

{
  "k8s": {
    "gpu_count": 2,
    "namespace": "ml-team",
    "gpu_vendor": "AMD",
    "memory": "32Gi",
    "cpu": "16",
    "image_pull_policy": "Always"
  }
}

Configuration Priority (lowest to highest precedence):

Base defaults
GPU vendor preset (AMD/NVIDIA)
GPU vendor multi-GPU preset (AMD only, for multi-GPU/multi-node)
Profile preset (single-gpu/multi-gpu/multi-node)
User config (--additional-context-file) — highest precedence, applied last

See examples/k8s-configs/ for complete examples.

Secrets and credentials

By default (k8s.secrets.strategy: from_local_credentials), madengine run creates Kubernetes Secrets from a local credential.json when present: Docker Hub pull credentials (when configured) and an opaque Secret for runtime use. Pods also reference the pre-created pull secret dockerhub-rocm (k8s.secrets.image_pull_secret_names). Credentials are not embedded in the ConfigMap in that case. For GitOps or clusters without client-side files, use existing or omit and set k8s.secrets.image_pull_secret_names / k8s.secrets.runtime_secret_name as needed. See Configuration and examples/k8s-configs/README.md.

Validating rendered manifests

With "debug": true in additional context, madengine run writes rendered manifests under ./k8s_manifests (or the path you configure). To lint those YAML files against the Kubernetes OpenAPI schema, install kubeconform and run from the repository root:

./tests/scripts/k8s_validate_manifests.sh ./k8s_manifests

The script exits successfully if kubeconform is missing (skip) or if validation passes.

Storage and runtime launchers

A multi-node job uses a shared ReadWriteMany volume when k8s.nfs_storage_class (default nfs-banff) can serve one. On a cluster whose only StorageClass is local-path, each pod gets its own ReadWriteOnce disk (results_layout / data_layout: auto, or force shared / per_pod). Performance CSVs are then taken from pod logs. See examples/k8s-configs/README.md.

madengine run --additional-context distributed (launcher, nnodes, nproc_per_node) is applied at submit time and wins over the copy saved in build_manifest.json. n_gpus: -1 on a model card means "all GPUs on the node", not a process count. Multi-node vLLM and SGLang pods join the headless Service subdomain so pod-0 DNS resolves.

Multi-Node Training

For distributed training across multiple nodes:

{
  "k8s": {
    "gpu_count": 8
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 2,
    "nproc_per_node": 4
  }
}

This creates:

Kubernetes Indexed Job with 2 completions
Headless service for pod discovery
Automatic rank assignment via JOB_COMPLETION_INDEX
MAD_MULTI_NODE_RUNNER environment variable with torchrun command

Supported Launchers:

torchrun - PyTorch DDP/FSDP
deepspeed - ZeRO optimization
megatron-lm - Megatron-LM training
torchtitan - LLM pre-training
primus - Primus unified pretrain (Megatron / TorchTitan / MaxText YAML)
vllm - LLM inference
sglang - Structured generation
sglang-disagg - Disaggregated SGLang (multi-node)
slurm_multi / slurm-multi - Self-managed multi-container topologies (SLURM only, escape hatch)

See Launchers Guide for details.

Monitoring

# Check job status
kubectl get jobs -n your-namespace

# View pod logs
kubectl logs -f job/madengine-job-xxx -n your-namespace

# Check pod status
kubectl get pods -n your-namespace

Cleanup

Finished Jobs are not removed unless you set k8s.ttl_seconds_after_finished to a positive number of seconds; the Job manifest then includes ttlSecondsAfterFinished so the control plane can garbage-collect the Job after it finishes. The deploy step may still delete Secrets it created when cleaning up a failed or cancelled deploy—see runtime logs for details.

Manual cleanup:

kubectl delete job madengine-job-xxx -n your-namespace

SLURM Deployment

Prerequisites

Access to SLURM login node
SLURM commands available (sbatch, squeue, scontrol)
Shared filesystem for MAD package and results
Module system or container runtime (Singularity/Apptainer)

Quick Start

Configuration

slurm-config.json:

{
  "slurm": {
    "partition": "gpu",
    "gpus_per_node": 4,
    "time": "02:00:00",
    "account": "my_account"
  }
}

Build and Deploy

# 1. Build image (on build node or locally)
madengine build --tags my_model \
  --registry my-registry.io \
  --additional-context-file slurm-config.json

# 2. SSH to SLURM login node
ssh user@hpc-login.example.com

# 3. Deploy to SLURM
cd /shared/workspace
madengine run \
  --manifest-file build_manifest.json \
  --timeout 7200

The deployment target is automatically detected from the slurm key in the config.

Configuration Options

slurm-config.json:

{
  "slurm": {
    "partition": "gpu",
    "account": "research_group",
    "qos": "normal",
    "gpus_per_node": 8,
    "nodes": 1,
    "time": "24:00:00"
  }
}

Common SLURM Options:

partition: SLURM partition name
account: Billing account
qos: Quality of Service
gpus_per_node: Number of GPUs per node
nodes: Number of nodes (for multi-node)
nodelist: Comma-separated node names to run on (e.g. "node01,node02"); when set, job runs only on these nodes and node health preflight is skipped
reservation: SLURM reservation name; forwarded to srun health/cleanup commands
time: Wall time limit (HH:MM:SS)
exclusive: Exclusive node access (default: true)

See examples/slurm-configs/ for complete examples.

Multi-Node Training

For distributed training across SLURM nodes:

{
  "slurm": {
    "partition": "gpu",
    "nodes": 4,
    "gpus_per_node": 8,
    "time": "48:00:00"
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 4,
    "nproc_per_node": 8
  }
}

SLURM automatically provides:

Node list via $SLURM_JOB_NODELIST
Master address detection
Network interface configuration
Rank assignment via $SLURM_PROCID

Monitoring

# Check job queue
squeue -u $USER

# Monitor job progress
squeue -j <job_id>

# View job details
scontrol show job <job_id>

# Check output logs
tail -f slurm-<job_id>.out

Pre-Built Images and Build-on-Compute

For workloads that use externally maintained Docker images (e.g. SGLang, vLLM releases):

# Skip Docker build, use a pre-built image
madengine build --tags model --use-image lmsysorg/sglang:latest

# Auto-detect image from model card's DOCKER_IMAGE_NAME
madengine build --tags model --use-image auto

# Build on a SLURM compute node and push to registry
madengine build --tags model --build-on-compute --registry docker.io/myorg

The manifest generated by --use-image merges the model card's distributed and slurm config into deployment_config, so the run phase auto-detects SLURM deployment without additional --additional-context.

slurm_multi Launcher (Self-Managed)

For workloads that orchestrate their own per-node Docker containers (e.g. SGLang Disaggregated proxy + prefill + decode topologies), use the slurm_multi launcher:

{
  "distributed": {
    "launcher": "slurm_multi"
  },
  "slurm": {
    "partition": "gpu",
    "nodes": 3,
    "gpus_per_node": 8,
    "reservation": "my-reservation"
  }
}

Unlike templated launchers, slurm_multi runs the model's .slurm script directly on baremetal. The script manages its own Docker containers via srun internally. See Launchers Guide — slurm_multi for details.

Running Inside salloc

When madengine run detects an existing SLURM allocation (SLURM_JOB_ID is set, e.g. inside salloc), the slurm_multi launcher runs the generated wrapper script synchronously with bash instead of nesting another sbatch. Templated launchers do the same for multi-node runs: the job script runs in place and starts its tasks with srun on the first nodes hosts of the allocation (or on slurm.nodelist, when set). Each run gets its own paths, so several runs can share one allocation. Single-node runs still submit their own sbatch, because in place their script would run on the submitting host.

Those paths are job-scoped directories, not the SLURM submission directory. A single-node job uses $SLURM_TMPDIR/madengine_job_<job id> when that directory is writable, otherwise /tmp/madengine_job_<job id>. A multi-node task appends _node_<rank>, and an in-place run also appends a per-run suffix so two runs in one allocation do not share a directory. A multi-node task removes its workspace only after the task exits 0 and every artifact has been copied out; a failure, or a copy that did not finish, keeps the directory. A later job from the same user deletes madengine_job_* directories older than 7 days under that scratch root when squeue -u lists this user's jobs and the directory's job id is not among them. An unanswered squeue deletes nothing.

salloc --nodes=3 --gpus-per-node=8 --partition=gpu
madengine run --manifest-file build_manifest.json
# → Detects salloc, runs synchronously

Cancellation

# Cancel job
scancel <job_id>

# Cancel all your jobs
scancel -u $USER

Deployment Comparison

Feature	Kubernetes	SLURM	llm-d
Environment	Cloud, on-premise	HPC clusters	Kubernetes
Orchestration	Automatic	Job scheduler	Inference Gateway + helm
Dependencies	Python library (`kubernetes`)	CLI commands only	`kubernetes` + `helm` (managed mode)
Multi-node Setup	Headless service + DNS	SLURM env vars	`InferencePool` + Endpoint Picker
Resource Management	Declarative (YAML)	Batch script	Helm charts
Best For	Cloud deployments, microservices	Academic HPC, supercomputers	Benchmarking disaggregated serving

Configuration Examples

Single-GPU Development (K8s)

{
  "k8s": {
    "gpu_count": 1,
    "namespace": "dev"
  }
}

Multi-GPU Training (K8s)

{
  "k8s": {
    "gpu_count": 4,
    "memory": "64Gi",
    "cpu": "32"
  },
  "distributed": {
    "launcher": "torchrun",
    "nnodes": 1,
    "nproc_per_node": 4
  }
}

Multi-Node Training (K8s)

{
  "k8s": {
    "gpu_count": 8,
    "namespace": "ml-training"
  },
  "distributed": {
    "launcher": "torchtitan",
    "nnodes": 4,
    "nproc_per_node": 8
  }
}

Single-Node SLURM

{
  "slurm": {
    "partition": "gpu",
    "gpus_per_node": 8,
    "time": "12:00:00"
  }
}

Multi-Node SLURM

{
  "slurm": {
    "partition": "gpu",
    "nodes": 8,
    "gpus_per_node": 8,
    "time": "72:00:00",
    "account": "research_proj"
  },
  "distributed": {
    "launcher": "deepspeed",
    "nnodes": 8,
    "nproc_per_node": 8
  }
}

Troubleshooting

Kubernetes Issues

Image Pull Failures:

# Check image exists
docker pull <registry>/<image>:<tag>

# Verify image pull secrets
kubectl get secrets -n your-namespace

# Check pod events
kubectl describe pod <pod-name> -n your-namespace

Node Reported as FAILED but Pod Succeeded:

In multi-node jobs, madengine may report a node as FAILED even though Kubernetes shows the pod as Succeeded. This occurs when the kubelet on the node becomes unreachable after the job completes, preventing madengine from collecting stdout logs (and therefore parsing performance metrics).

To verify:

# Check actual pod status — if Succeeded, the workload ran fine
kubectl describe pod <pod-name> | grep Status

# Check the node's kubelet health
kubectl get nodes
kubectl describe node <node-name> | grep -A5 Conditions

PVC artifacts are still collected in this scenario. Only the API-based pod log retrieval fails, which means performance metrics for that node will be missing from the results table.

Resource Issues:

# Check node resources
kubectl describe nodes | grep -A5 "Allocated resources"

# Check GPU availability
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.capacity.'amd\.com/gpu'

SLURM Issues

Job Pending:

# Check reason
squeue -j <job_id> -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"

# Check partition status
sinfo -p gpu

Out of Resources:

# Check available resources
sinfo -o "%P %.5a %.10l %.6D %.6t %N"

# Adjust resource requests in config

Best Practices

For Kubernetes

Use minimal configs with intelligent defaults
Specify resource limits to prevent over-allocation
Use appropriate namespaces for isolation
Configure image pull policies based on registry location
Monitor pod resource usage with kubectl top

For SLURM

Start with conservative time limits
Use appropriate QoS for priority
Monitor job efficiency with seff <job_id>
Use shared filesystem for input/output
Test with single node before scaling

Next Steps

Launchers Guide - Distributed training and inference launchers
llm-d Guide - Benchmarking the llm-d distributed inference stack
K8s Examples - Complete Kubernetes configurations
SLURM Examples - Complete SLURM configurations
llm-d Examples - Complete llm-d configurations
Usage Guide - General usage instructions

FilesExpand file tree

deployment.md

Latest commit

History

deployment.md

File metadata and controls

Deployment Guide

Overview

Deployment Workflow

Deployment Target Inference

Kubernetes Job naming

Kubernetes Deployment

Prerequisites

Quick Start

Minimal Configuration (Recommended)

Build and Deploy

Configuration Options

Secrets and credentials

Validating rendered manifests

Storage and runtime launchers

Multi-Node Training

Monitoring

Cleanup

SLURM Deployment

Prerequisites

Quick Start

Configuration

Build and Deploy

Configuration Options

Multi-Node Training

Monitoring

Pre-Built Images and Build-on-Compute

slurm_multi Launcher (Self-Managed)

Running Inside salloc

Cancellation

Deployment Comparison

Configuration Examples

Single-GPU Development (K8s)

Multi-GPU Training (K8s)

Multi-Node Training (K8s)

Single-Node SLURM

Multi-Node SLURM

Troubleshooting

Kubernetes Issues

SLURM Issues

Best Practices

For Kubernetes

For SLURM

Next Steps