feat(agent-profiles): one launch pipeline for every conversation start - #5154
simonrosenberg wants to merge 5 commits into
Conversation
Collapse the several code paths that built a launch agent into prepare_agent_launch(), the single SDK function that resolves an Agent Profile's references and applies the runtime-dependent and per-launch pieces. conversation_service, the Docker runtime's mediation and the materialize preview all call it, so a profile named `default` and a named one build the same agent, and a preview can no longer disagree with a launch. Adds an inline `agent_profile` draft and a per-launch `llm_profile_ref` override, deprecates `agent_settings` (converted into an inline profile so it takes the same pipeline), and returns one structured error for dangling LLM/MCP references. Fixes #5141 Co-authored-by: openhands <openhands@all-hands.dev> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
📁 PR Artifacts Notice This PR contains a |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Co-authored-by: openhands <openhands@all-hands.dev> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage Report •
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
…agent Co-authored-by: openhands <openhands@all-hands.dev> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: openhands <openhands@all-hands.dev> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🤖 OpenHands is reviewing this PR. Head commit: This comment was posted by an AI agent (OpenHands). |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.
Summary
This PR unifies every conversation-start path (stored profile, inline profile, raw agent, deprecated agent_settings, materialize preview, and the Docker runtime) behind a single prepare_agent_launch function. I reviewed the resolver, the agent-server launch glue, the Docker mediation double-resolve, the request model changes, and the test suite.
No material bugs found. The design is clean and achieves what it set out to do: each profile field is now implemented once instead of once-per-launch-path.
What holds up under scrutiny
- Exception hierarchy is correct.
UnresolvedProfileReferences -> AgentLaunchError -> ValueError, and both routers catchAgentLaunchErrorbeforeValueError, so the structured 422 detail (code/dangling_llm_profile_ref/dangling_mcp_server_refs) is preserved rather than collapsed to a plain string.ProfileNotFoundstays a separateException-> 404. - Docker double-resolve is sound.
prepare_startruns withbuild_agent=False(andbrowser_available=False) so a dangling ref fails before any container is provisioned;finish_startre-resolves the sameLaunchSource/catalog with the container's realbrowser_availablefromGET /server_info(usable_toolsfield -- verified correct). Skill discovery happens once in the catalog and is reused, not re-run. Container cleanup (registry.stop) is wired on every failure branch. - Secret scoping is enforced server-side in
apply_launch(plan.allowed_secretsfiltersrequest.secrets), andagent_settings_launch_sourcecorrectly setsallowed_secrets=None(unrestricted) to match the legacy path. The additions-cannot-widen-scope invariant is asserted by a real test. - Backward compatibility is preserved.
LaunchedAgentProfile.inline/llm_profile_refandAgentLaunchAdditions.llm_profile_refare additive fields with defaults;LaunchedAgentProfilehas noextra="forbid", so old persisted conversations load.agent_settingsis deprecated withdeprecated_in=1.50.0->removed_in=1.55.0(5 minor releases, meeting the policy) and is converted to an inline profile through the same pipeline rather than dropped. - Tests exercise real code paths, not mock wiring:
test_launch.pycovers runtime pieces, additions, provenance, dangling refs, and the deprecatedagent_settingsround-trip (verifying non-profile fields likecritic_api_key,user_message_suffix,agent_context.secretssurvive). The parity test diffs every launch path field-by-field.
Eval / benchmark risk -- flagging for a human maintainer
This PR changes agent launch behavior in ways that could plausibly move benchmark numbers, and there is no eval-monitor link or maintainer eval confirmation in the PR description or comments:
- The
agent_settingspath now goes through the unified pipeline, so it gains forcedstream=True,load_project_skills=True, browser injection (when the runtime has it andtoolsis null), and a freshcurrent_datetimeinstead of the saved timestamp. Canvas already sends these explicitly so it's unaffected, but a hand-rolled REST client relying on the old defaults would see a change. - ACP skill sourcing in Docker switched from the host's answer to the container's (
openhands_managed), so an ACP profile in Docker now gets managed skills where it previously got none.
Per the repo's review policy I'm leaving a COMMENT rather than approving. Recommend a maintainer run lightweight evals (or confirm Canvas-only impact) before merging.
Minor note (non-blocking)
warn_deprecated(..., deprecated_in="1.50.0") is called from the agent-server while the current SDK version is 1.49.1. _should_warn compares current >= deprecated_in, so the runtime warning won't actually fire until the SDK ships as 1.50.0 -- which is presumably the release this PR targets, so this is consistent, just worth knowing the warning is effectively inert until then.
Risk assessment: MEDIUM -- no correctness/security issues, but behavior changes on the agent_settings/ACP-in-Docker paths that warrant eval confirmation.
Verdict: Worth merging after a maintainer confirms no eval regression.
Improve this review? If any feedback above seems incorrect or irrelevant to this repository, you can teach the reviewer to do better:
- Add a
.agents/skills/custom-codereview-guide.mdfile to your branch (or edit it if one already exists) with the/codereviewtrigger and the context the reviewer is missing. See the customization docs for the required frontmatter format.- Re-request a review - the reviewer reads guidelines from the PR branch, so your changes take effect immediately.
- When your PR is merged, the guideline file goes through normal code review by repository maintainers.
Resolve with AI? Install the iterate skill in your agent and run
/iterateto automatically drive this PR through CI, review, and QA until it's merge-ready.Was this review helpful? React with thumbs up or thumbs down to give feedback.
|
Thanks — on the eval-risk flag, here is what I can evidence from this branch so a maintainer has the facts to decide. I have not run evals. Who actually sees the Cloud is not on this path. The enterprise app server builds a concrete ACP-in-Docker. Previously the host's Unchanged for stored-profile launches (the path evals exercise): the parity e2e diffs On the |
2 participants
HUMAN:
Filing the SDK half of #5141: the
defaultprofile and named profiles have to build the same agent from one pipeline, so new profile fields stop needing an implementation per launch path. Canvas and cloud adoption follow separately.AGENT:
Why
Launching the
defaultAgent Profile and launching a named one built different agents from the same stored settings, because the launch had several code paths and each set agent fields its own way: a Pydantic validator foragent_settings, theagent_profile_idbranch ofconversation_service, a second copy of that branch in the Docker runtime'sprepare_start, and a separate dry-run formaterialize. Every profile field had to be implemented, and kept correct, once per path — #3967, #4014, #4016 and #4542 were that cost paid one field at a time.This is step 1–3 of #5141 (the SDK steps). Canvas and cloud adoption are the follow-ups listed below.
Summary
openhands.sdk.profiles.prepare_agent_launch(source, *, catalog, runtime, additions, profile_origin, build_agent)resolves a profile'sllm_profile_ref/mcp_server_refs/disabled_skillsand owns the launch-time fields: tool defaults and browser injection, forced streaming, skill catalog and ACP skill sourcing, project-skill loading, suffix + additions,current_datetime,load_memory, and the secret scope. Runtime-dependent answers come in as an explicitAgentLaunchRuntime, so the Docker runtime can pass its container's answer instead of the host's.conversation_service, Docker mediation andmaterializeall call it through the newagent_server/agent_launch.py;_resolve_agent_from_profile,_with_load_memoryand_apply_acp_skill_sourcingare gone, as is the duplicated profile branch inprepare_start.materializeis the same call with side effects off (build_agent=False, which skipscreate_agent()— the only step that can refresh a subscription LLM's credentials over the network). Its verdict andresolved_settingsnow come from the launch itself.StartConversationRequestaccepts an inlineagent_profiledraft (resolved exactly like a stored one, never saved), andagent_settingsis deprecated (deprecated: truein OpenAPI, removal target v1.55.0). During the window the server converts it into an inline profile with the payload's own LLM/MCP/skills as its catalog, so it takes the same pipeline instead of the validator shortcut; fields no profile models (critic_api_key,user_message_suffix,agent_context.secrets,acp_isolate_data_dir, …) are carried through, not dropped.AgentLaunchAdditions.llm_profile_refgives the chat LLM picker a per-launch override, recorded inLaunchedAgentProfile(which also gainedinline). Additions stay additive: they carry no tools, MCP servers, skills or secrets, and a test asserts a scoped profile's tools/MCP/skills/secret scope are byte-identical with and without them.UnresolvedProfileReferencesand the start endpoint returns 422 with{"code": "unresolved_profile_references", "message", "dangling_llm_profile_ref", "dangling_mcp_server_refs"}— no silent fallback to a different path.REST API contract changes
Compared with base OpenAPI
c00f26a0c602for public/api/**paths.Issue Number
Fixes #5141
How to Test
Unit tests:
End-to-end against a real agent-server (this is the interesting one — it reproduces the table in #5141 without canvas):
It boots
python -m openhands.agent_serveron a temp persistence dir, stores two identically-configured profiles (defaultanddefault-copy), launches a conversation throughagent_profile_idfor each, through an inlineagent_profiledraft, and through the deprecatedagent_settings, then diffs the agents the server actually built against thematerializepreview of the same profile. Output in.pr/launch_parity_e2e_output.txt:Compared field by field:
llm(whole dump), tools, MCP keys, skills, suffix,disabled_skills,load_project_skills,load_memory, condenser, critic, concurrency, switch-LLM and whether a timestamp is present. The one deliberate exception is the skill catalog on theagent_settingspath: that payload carries the client's own catalog (canvas assembles one today), which is exactly what the canvas follow-up removes.Type
Notes
Behavior changes reviewers should weigh:
code/dangling_llm_profile_ref(the oldmessage/dangling_mcp_server_refskeys are unchanged). Canvas never reads that status — it pre-checks the profile list and downgrades toagent_settings— and that rule is what the follow-up deletes.agent_settingsis no longer converted in the validator, soStartConversationRequest(agent_settings=...).agentisNoneuntil the server resolves it, and an invalid payload is rejected by the start endpoint (422) rather than at parse time. The field also lostexclude=Trueso it round-trips over the wire. An explicit"agent": nullalongside another source no longer crashes.agent_settingspath now gets the launch-owned fields too, because it goes through the same pipeline: streaming forced on,load_project_skills=True, browser added when the payload'stoolsis null and the runtime has it, and a freshcurrent_datetimeinstead of the saved one. Canvas already sendsstream: true,load_project_skills: trueand an explicit tool list, so its payload is unaffected; a hand-rolled REST client that relied on those staying off would see the change.build_agent=False, so a dangling ref still fails fast without paying for a container) and once after, with the container's own runtime answer (GET /server_infofor browser availability,openhands_managedskills). Previously the host's answers were used for a container agent, which meant an ACP profile in Docker got no managed skills./server_infoadvertisesunified_agent_launch_v1.Not fixed here, found while testing: a condenser's
max_tokensinheritance keys offmodel_fields_set, so a stored profile (loaded from JSON, every field "set") does not inherit the LLM's token limit while an in-memory one does. It is consistent across today's product paths (both read persisted JSON) and predates this PR, so I left it alone rather than widen the diff.Follow-ups, per #5141:
defaultname rule, the LLM-mismatch and dangling-ref fallbacks, theuse-llm-configuredcopy and client-side agent assembly, and send<RUNTIME_SERVICES>viaagent_launch_additions.prepare_agent_launch, stop overwriting fields the profile carries, surface resolution failures.default-profile refresh from globalagent_settingsbelongs with the canvas step: doing it before the settings pages edit the active profile would just let it drift again.profile_launch.pyis superseded byagent_launch.py.🤖 Generated with Claude Code
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:97f40ab-pythonRun
All tags pushed for this build
About Multi-Architecture Support
97f40ab-python) is a multi-arch manifest supporting both amd64 and arm6497f40ab-python-amd64) are also available if needed