Skip to content

[None][feat] agent-flow: modeling-v2-onboard planner/worker workflow - #20792

Draft
WeiHaocheng wants to merge 27 commits into
NVIDIA:mainfrom
WeiHaocheng:feat/modeling-v2-onboard
Draft

WeiHaocheng wants to merge 27 commits into
NVIDIA:mainfrom
WeiHaocheng:feat/modeling-v2-onboard

Conversation

@WeiHaocheng

@WeiHaocheng WeiHaocheng commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

Adds modeling-v2-onboard, an agent-flow workflow that brings a new model up on ModelingV2 with one planner and parallel workers, and the changes that came out of running it end to end on nemotron4-nano-preview (GB300).

Orchestration

  • The planner writes plan.md: tasks tagged new_op / target / merge / module_accuracy / e2e_accuracy / perf, plus an Execution Graph. Workers run tasks in parallel, each in its own git worktree, with a system prompt built from the task's tags.

  • An event loop replaces the plan / build / replan phases. A task starts as soon as its dependencies are done, and the planner takes a turn whenever workers finish while the others keep running. A failure blocks only its dependents. State is checkpointed, so a run resumes after a crash.

  • The planner releases a first batch of tasks before it reads the code, then adds batches turn by turn.

  • Scheduling follows a priority class on each task. Free worker slots and pool GPUs both go to the most urgent ready task, in this order:

    1. accuracy: everything on the way to the first whole-model accuracy result. That means the missing new_op entries, the module checks, assembling the target and the first gate.
    2. feature: further capabilities and their checks.
    3. perf.

    The class follows what a task unblocks, not its tag, so a new_op that the first target needs is accuracy. The planner can change a task's class in any turn. Resumed tasks go first, and the graph order breaks ties within a class.

  • Checks never block joins: a failed check gets a fix task and a new join.

GPU resources

  • An optional Slurm GPU pool (slurm-environment.gpu_pool). Holder jobs each hold a whole node and are submitted at start, so Slurm queueing overlaps planning.
  • gpu_run locks GPUs with flock and runs each command as a step of a holder job inside the container. Waiting commands are served by the same priority classes, re-read on every poll. A few GPUs are reserved and perf work may not take them.
  • Placement is per node, so one-GPU commands cannot starve a whole-node command. The planner can grow the pool.

Worker start-up cost

  • new_op and perf workers start as forks of a primer session that has already read the entry contract and exemplars.
  • Workers reuse an existing build. The environment wrapper is a first-batch task, recorded in a shared test_commands.md.

Human control and scope

  • --feedback / --feedback-file messages wake the planner and override the plan in any phase.
  • Free slots only go to completion criteria, the requested features, and then perf work.
  • Bugs unrelated to the goal are recorded in issues.md instead of being chased.
  • Unmerged perf work becomes side work that does not hold up landing. It is kept on -unmerged/<id> branches and listed in UNMERGED_OPTIMIZATIONS.md.

Robustness

  • git worktree operations are serialized and cannot be interrupted.
  • Claude Code auto-memory is off for every agent.
  • Node commits and merges are signed off.
  • A stopped worker's changes are committed to its branch before the worktree is removed.
  • Cancelling a local detached job stops its whole process group.
  • Cancelling a task abandons it: the worktree, branch and jobs are kept, and the task is listed in abandoned.md. The reset machinery is removed.

Prompt rules

  • Prompts state rules, not command recipes. The exceptions are the build commands and the Slurm job check.
  • Perf work need not match the baseline bit for bit. A new implementation, kernel or fusion may differ in the last bits, as long as the error stays as small as its arithmetic explains.

End-to-end result. nemotron4-nano-preview on GB300:

  • The run went from start to an automatic land in 35.5 h: 149 tasks, 152 planner turns, 2 mid-run human messages and 3 orchestrator restarts.
  • After the human's turn-108 instructions, throughput rose 10-41% across tp1 / dep4 / tp4.

Test Coverage

  • tests/workflows/modeling_v2_onboard/: 111 tests, all passing. They cover plan validation, the event loop and planner turns, cancellation and abandonment, resume, priorities, the GPU pool and gpu_run placement, side tasks, and the inbox.
  • End-to-end: the nemotron4-nano-preview v3 run described above.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

…-flow

A two-role agent-flow workflow for bringing up ModelingV2 models and targets,
built on the concurrent orchestration engine.

- The planner writes plan.md: one section per task, tagged new_op / target /
  merge / module_accuracy / e2e_accuracy, plus an Execution Graph of their
  dependencies. The plan is validated (flat graph, known tags, multi-dependency
  tasks are merge tasks, a single shared final task) and errors are sent back
  to the planner.
- Workers run the tasks in parallel through NodeScheduler, each in its own git
  worktree, with a system prompt composed from the task's tags, and end by
  writing task_result_<id>.md.
- A FAILED task pauses the run; the planner revises the plan (REVISED / DONE /
  ABORT), the failed and changed tasks are reset, and the build resumes. There
  is no round limit.
- Task branches are kept until the run ends so a re-run task still starts from
  its dependencies' code (the scheduler otherwise reclaims a dependency branch
  once its only dependent has failed).
- --backend switches every agent between claude-code (default) and codex.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
… perf tag

The planner is told --max-parallel and asked to keep that many workers
busy: independent work side by side, feature work alongside the accuracy
gate, and perf tasks (alternative implementations, new kernels, fusion)
in the remaining slots. The new perf tag briefs a worker to measure,
adopt only a measured win, and revert otherwise.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
Replace the plan/build/replan phases with an event loop of its own: every
task whose dependencies are DONE starts at once (up to --max-parallel),
and whenever workers finish the planner gets a turn while the others keep
running. A turn may add tasks, rewrite or remove tasks not yet started,
retry/rewrite/remove FAILED tasks, cancel RUNNING ones, FINISH (release
the final task) or ABORT. A FAILED task only holds back its dependents.

DONE and RUNNING tasks are frozen, including their plan.md section. An
invalid turn is discarded after three fixes and reported on the next one;
three stalled turns in a row stop the run. Task states and the running
plan are checkpointed, so a rerun resumes and restarts running tasks on
their branches.

Also fix task-section lookup matching a prefix id (gsm8k vs gsm8k-final).

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
A fresh worktree has no compiled artifacts. When the brief or the shared
command cache gives a way to link a built checkout's artifacts, workers use
it instead of a full build_wheel per worktree.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
Each execution-graph node may carry priority: accuracy | feature | perf,
its claim on shared GPUs. Left out, perf-tagged tasks are perf, the path
to the first whole-model accuracy result is accuracy, and the rest is
feature. The orchestrator publishes the running plan's classes in
gpu_priorities.json for a GPU pool to serve; changing a class never
re-runs a task.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…irst gate on a target

A perf task the plan puts on the way to the first whole-model gate is on
the critical path, so it is accuracy too. The first gates are the
e2e_accuracy tasks that build on a target task with no such gate
upstream.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…, safe git ops

- --feedback / --feedback-file leave a message in <workspace>/inbox/. The
  orchestrator polls the inbox, wakes the planner, shows the messages first
  in its turn prompt, and moves them to inbox/consumed/ only once that turn
  applied. Tasks that finished since the last turn are kept in .state.json,
  so a restart no longer drops them.
- The planner fills free slots only with completion-criteria work, then the
  features task.yaml asks for, then perf work; unrequested verification or
  polishing (other models' suites, upstreaming tests) is ruled out because
  it holds up landing. Human messages override the plan.
- Git worktree operations run one at a time and are never interrupted by a
  cancellation: concurrent `git worktree add` raced on .git metadata, and a
  cancelled add left a locked worktree that could not be removed.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…nstead of chasing them

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…gent ready task

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…t merge

Once task.yaml's goals are met, the planner marks the unmerged perf tasks
side: true so the run can land without another round of merging. Side tasks
keep using free slots until the final task is DONE, get --side-wait-minutes
to finish afterwards, and are kept as <landed branch>-unmerged/<id> branches
listed in a committed UNMERGED_OPTIMIZATIONS.md.

A human can still ask for side work to be merged: the planner drops side and
merges it before the final task, or after it through a chain of shared
landing tasks. A cancelled shared task's changes to the checkout are undone.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
… the plan

The planner writes the tasks it can define before the deep reading (the
execution setup, contract reviews, references, ops with no entry at all) and
ends its first turn with planning: true. Workers start on that batch while
the planner keeps adding batches, one turn after another, until it writes
planning: false.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…ner plans

With slurm-environment.gpu_pool in task.yaml, the orchestrator keeps holder
jobs for whole nodes submitted from the start of the run and cancels them at
its end. Workers run GPU commands through gpu_run, which locks GPUs on a
holder job, serves waiters by their task's priority class and runs the
command as a step in the container.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…er session

A primer session reads what every new_op and perf worker needs (the entry
contract, exemplar entries and tests, how GPU tests run) while the planner
plans, and writes a summary. Those workers start as forks of its session;
when it is not ready within 20 minutes, failed, or a fork cannot start, they
start afresh and read the summary. agent-flow's sessions can now fork an
earlier session (Claude backend) and report their session id.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…d task, not a pool setup hook

Drop gpu_pool.setup. Preparing the container environment (reusing a build,
the Python environment, model paths) is now a first-batch task: a worker
writes the wrapper into the workspace and records it in test_commands.md,
and the other tasks run their GPU commands through it.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…e auto-memory

Claude Code keeps one auto-memory per repository, shared by every worktree
of the checkout, so a run's agents read memories of unrelated work and of
earlier runs, and left their own to later runs. The CLI now disables it for
every agent it starts.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…the first batch

The first batch is written from task.yaml, config.json and the ModelingV2
README alone; the reference code, the safetensors index, the catalog and the
engine source wait for the second batch.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…s are placed per node

gpu_run places each waiter ahead of a command on a node first; a waiter
that fits nowhere claims the node with the most free GPUs, so one-GPU
commands can no longer starve a whole-node command.

Each planner turn shows the pool, and the turn file's gpu_nodes grows it
up to gpu_pool.max_nodes (default: the worker count).

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…grows in any phase

A merge task depends only on the tasks whose code it joins; module checks,
accuracy gates, benchmarks and profiles run beside the next step, and a
failed check gets a fix task and a new join. The planner's example plan
follows the rule, and the default priorities find the first gates by the
code they check rather than by gate-to-gate edges.

The planner may grow the GPU pool whenever commands wait for it, in any
phase and for any class of work.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…tops the run

Cancelling a task read its jobs.json as the job registry and raised when the
worker had written the file itself in another shape, ending the run. The
orchestrator now logs and skips such a file, and the worker prompt says the
file belongs to the run_detached wrapper.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
A merge node's worktree merges its dependencies' branches with git merge. A
TensorRT-LLM checkout with its pre-commit hooks installed runs a DCO check on
every commit message, so the unsigned merge commit was refused and the run
stopped. The merges now carry a sign-off.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…worker's work

The per-node output commit is now signed off, so a node branch passes a
repository's DCO check. When the run stops, each running worktree task's
changes are committed to its branch before the worktree is removed, so a
resumed task starts from them instead of from its last commit.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…mmand recipes

The worker prompt's GPU pool, Slurm and detached-job blocks now say what must hold and leave the exact commands to the worker. The detached-job block no longer assumes Slurm jobs in a pool run.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…urm job check in the worker prompt

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
A worktree directory that git no longer counts as a worktree (its .git file
is gone) is now removed as plain files instead of failing the release. The
tasks a planner turn resets are saved in the run state until each one is
dropped, so a run stopped in the middle of a reset finishes it when it
resumes instead of leaving the task's branch, notes and jobs behind.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…mands it started

A local detached job leads its own session; its cancel now signals the whole process group, so a GPU command it started does not keep running after the task is reset.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
Cancelling a running task now only stops its worker's session. Its
worktree, branch, notes and GPU jobs are left as they are; the task must
leave the plan in the same turn and its id is never used again, so redoing
the work is a task with a new id. A FAILED task removed from the plan is
abandoned the same way, and every abandoned task is listed in
abandoned.md for a human to clean up. The final task, which works in the
checkout itself, cannot be cancelled.

A retried or rewritten FAILED task carries on from its own branch; its
last result is kept as an earlier attempt. The reset machinery (job
cancellation, branch and worktree removal, checkout restore, and finishing
an interrupted reset on resume) is gone.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
…ne bit for bit

A new implementation, kernel or fusion may accumulate in another order and
differ from the baseline in the last bits. The perf worker prompt accepts
that while the error stays as small as the arithmetic explains (the entry
still passes its band against its reference), and the planner no longer
asks perf tasks for bitwise identity.

Signed-off-by: Fred Wei <20514172+WeiHaocheng@users.noreply.github.com>
@WeiHaocheng WeiHaocheng changed the title Feat/modeling v2 onboard [None][feat] agent-flow: modeling-v2-onboard planner/worker workflow Oct 22, 2026
Wanli-Jiang added a commit to Wanli-Jiang/TensorRT-LLM that referenced this pull request Oct 24, 2026
…s and merges, cancel a local job's process group

Agent-flow changes the trtllm_modeling_v2 workflow builds on, the first
three ported from NVIDIA#20792:

- A session can start as a fork of an earlier one
  (SessionConfig.fork_from): the Claude Code backend resumes the earlier
  session with fork_session, and the Codex backend refuses with
  NotImplementedError. A turn's result carries the backend's session id,
  and the session layer keeps the last one.
- Node commits and merges in git worktrees are signed off, for
  repositories that require DCO.
- Cancelling a local job sends SIGTERM to its whole process group, so the
  commands it started stop with it; a pid that leads no group gets the
  signal alone.
- The Codex backend launches $CODEX_BIN, else the codex on PATH when it is
  newer than the pinned runtime (a login offers a new model only to a
  client that knows it), and a session given no model runs the one Codex
  is configured with.

Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
@ruocheng-nv

Copy link
Copy Markdown
Collaborator

Hi, heads-up: #22286 has merged, so ModelingV2 now lives at tensorrt_llm/_torch/modeling_v2 (module tensorrt_llm._torch.modeling_v2), and the old _torch/_experimental/modeling_v2 path no longer exists.

Since this PR touches modeling_v2, please rebase it onto the latest main. You can use these migration commands to move any files you added under the old directory and update any remaining references to the old path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants