chore(release): v6.13.8 (DO NOT MERGE) - #9496
Open
lstein wants to merge 18 commits into
Open
Conversation
…room before decode/encode (#9305) * fix(qwen): estimate VAE working memory so the cache frees room before decode/encode The Qwen Image l2i/i2l invocations called `model_on_device()` without a `working_mem_bytes` estimate, unlike the SD/SDXL path. The model cache therefore only reserved the default `device_working_mem_gb` and never evicted the resident transformer/text encoder before the VAE decode. On a near-full card (e.g. Qwen Image Edit Q8_0 with transformer + text encoder resident) the decode then OOMs trying to allocate its working set into the fragmented remainder. Add `estimate_vae_working_memory_qwen_image()` and pass it into both the decode and encode paths so the cache makes room (evicting other models when needed) before the operation runs. The constant is calibrated against a measured decode on an AMD W7900: at 1248x832 the decode grew CUDA reserved memory by ~10.06 GiB (implied constant ~5082), rounded up to 5500 for headroom. It tracks peak *reserved* (not just allocated) memory so that whenever the cache declines to free room (free >= estimate) the decode is still guaranteed to fit. Encode uses ~half, matching the other estimators (not independently measured). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(qwen): cover VAE working-memory estimate is passed to cache Address review feedback from @Pfannkuchensack on #9305: - Add test_qwen_image_working_memory.py mirroring the z-image pattern, asserting both decode and encode paths call model_on_device with the estimated working_mem_bytes (regression guard for the OOM fix). - Clarify the qwen estimator comment: the encode constant is not independently measured (half of decode, matching siblings' ratio) and should be recalibrated against a measured encode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(qwen): recalibrate VAE working-memory constants from a measured grid Add scripts/calibrate_qwen_vae_working_memory.py, a backend-portable (CUDA/ROCm) harness that measures peak reserved-memory growth for VAE decode/encode across a resolution grid, one fresh subprocess per point. Calibrating on an AMD W7900 (fp16) showed the encode constant was wrong: the previous 2750 ("half of decode") under-estimated by ~2x at every measured resolution, the exact OOM mode Qwen Image Edit (which encodes a real image) would hit. Raise encode 2750 -> 6300. Decode 5500 is confirmed safe across the full 512^2..2048^2 range and left unchanged. The grid also showed memory is super-linear in area above ~1792^2 (an attention term) and non-monotonic (likely an SDPA-backend crossover on ROCm); both documented in the estimator. Constants are the conservative ROCm side and will be max-merged with a pending NVIDIA/CUDA run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(qwen): branch VAE working-memory constants by backend (ROCm vs CUDA) Calibrating the same fp16 grid on an NVIDIA card showed CUDA reserves ~2x (decode) to ~4x (encode) less than ROCm: the Qwen VAE is attention- heavy, and CUDA's Flash/efficient attention is O(area) and flat while the ROCm math-attention fallback is O(area^2). The backends diverge far more than any headroom, so a single constant either under-estimates on ROCm (OOM) or massively over-budgets CUDA (needless eviction). Select constants via torch.version.hip: decode: ROCm 5500 / CUDA 2900 encode: ROCm 6300 / CUDA 1600 Each verified to cover its measured grid (19 points/backend) with ~8% headroom. The CUDA run also confirms the linear model holds with Flash attention (the ROCm super-linear/non-monotonic behavior is a math- attention artifact), and that "encode is half of decode" is CUDA-only. Add parametrized tests asserting the constant selected for each (operation, backend) so a refactor can't silently swap them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(backend): ruff * calibrate: support single-file Qwen Image VAE checkpoints The calibration script only loaded the Qwen VAE from a diffusers directory via from_pretrained, so passing a single .safetensors file failed. Add _load_vae, which loads a directory as before and handles a single-file checkpoint by loading the state dict directly: a strict load for the diffusers layout, falling back to convert_wan_vae_to_diffusers for the original Qwen-Image/Wan release layout (downsamples/residual/ time_conv keys) before retrying. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev>
* docs: add 3d party GPU hosting services * Update docs/src/content/docs/index.mdx Co-authored-by: Josh Corbett <joshwcorbett@icloud.com> * docs: restyle hosted options as LinkCards in a wrapper card Implements joshistoast's suggested design: a bordered "Hosted Options" wrapper containing Starlight CardGrid/LinkCard entries, replacing the text separator and hand-rolled cards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev> Co-authored-by: Josh Corbett <joshwcorbett@icloud.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…s 5.x (#9333) Qwen2_5_VLForConditionalGeneration._checkpoint_conversion_mapping is empty in transformers 5.x, so the `if key_mapping:` guard skipped remapping and ComfyUI single-file checkpoints (`visual.*`, `model.*`) failed to load: their keys were reported as unexpected while `model.visual.*` params stayed as meta tensors. Fall back to the legacy mapping when transformers no longer provides one. The negative lookahead keeps new-layout checkpoints untouched.
A number of model-management and app-info routes carried no auth dependency at all. In multi-user mode this left them reachable by a fully unauthenticated network attacker, contradicting the documented model that model management is administrator-only. The highest-impact route was `GET /api/v2/models/scan_folder`, which takes an attacker-controlled filesystem path and recursively enumerates it, returning absolute paths of model-like files. Its distinct 200/400/500 responses also formed an existence/readability oracle for arbitrary paths. While auditing, `GET /api/v1/app/runtime_config` turned out to be worse: it served the raw `InvokeAIAppConfig` — including `remote_api_tokens` and the `external_*_api_key` values — in plaintext, to anyone. Changes: - `scan_folder` now requires `AdminUserOrDefault`, and every failure mode returns one generic 400 (details go to the server log) so the response cannot be used as a filesystem oracle. Arbitrary paths remain scannable by admins, consistent with `install_model`, which already accepts local paths. - Admin-only (`AdminUserOrDefault`): `missing`, `hugging_face`, `starter_models`, `stats`, `hf_login`, `external_providers/config`, `logging` (GET/POST) and the `invocation_cache` routes. All are either operator-only or rendered exclusively behind the admin-gated install panel. - Authenticated-user (`CurrentUserOrDefault`): model record reads, which ordinary users need for generation, plus `runtime_config`, `app_deps`, `patchmatch_status` and `external_providers/status`. - `runtime_config` now masks API keys and download tokens server-side rather than shipping them to the browser; the UI only ever needed the is-configured signal. Single-user mode is unaffected — `*OrDefault` resolves to a system admin when `multiuser` is off. Not changed: the binary-serving routes (image full/thumbnail, model cover image, workflow thumbnail) are consumed via `<img src>` and cannot carry a Bearer header, so closing those needs a signed-URL or cookie scheme. Closes #9365 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The added auth dependencies make FastAPI emit a `security: [{HTTPBearer: []}]`
requirement on each affected route, so the checked-in schema no longer matched
the generated one.
Regenerated with the openapi-checks command. The diff is exactly the 21 routes
gated in the previous commit gaining a security requirement; no route loses
one, and the rest of the document is byte-identical.
`schema.ts` is unchanged — openapi-typescript does not encode security
requirements in the generated types.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Serving the runtime config to every authenticated user relied on a denylist: `_redact_config_secrets` masks the credential fields we know about, so any future secret added to `InvokeAIAppConfig` would leak by default until someone remembered to update it. Admin-only is an allowlist posture without that failure mode. Nothing outside the admin UI actually needed it. The three non-admin call sites read `config.multiuser` purely to derive "admin or single-user", which is already available from the unauthenticated `/auth/status` route as `multiuser_enabled` — the same source `useIsModelManagerEnabled` uses. The remaining fields (`max_queue_history`, `image_subfolder_strategy`) only feed admin-gated edit controls. - Extract that predicate into `useIsAdmin`, and have `useIsModelManagerEnabled` delegate to it. Semantics are preserved exactly, so this is a pure refactor. - Use it in the three settings components instead of `runtime_config`, and skip the query for non-admins so they generate no failed requests. - AboutModal's debug blob now omits the config section for non-admins. Its client-side redaction is dropped: it only ever masked `remote_api_tokens` and never the `external_*_api_key` fields, so a logged-in non-admin could read provider API keys out of it. The server now masks both. Server-side masking is kept as defense in depth — an admin's browser has no use for the raw values either. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… gating Three issues from JPPhoto's review of #9367: - `PATCH /runtime_config` echoed the updated config back unredacted, so an admin changing an unrelated setting still pulled every external API key and `remote_api_tokens` value into the browser's RTK Query cache, defeating the masking added to the GET route. It now returns `_redact_config_secrets(config)` as GET does. - `GET /models/missing` is `CurrentUserOrDefault` again, not admin-only. The frontend's model hooks (`modelsByType.ts`) subtract this set from the model list so unusable records stay out of the generation dropdowns. Under an admin-only gate a non-admin got 403, the subtraction became a silent no-op, and missing models were offered for selection, failing only at execution. - The invocation-cache panel lives on the universally available Queue tab, but its routes are admin-only. A non-admin saw zeroed statistics and an active Enable button that always 403s. The panel is now gated with `useIsAdmin`, and the enable/disable/clear hooks additionally refuse to offer the mutation to a non-admin, so the guard does not depend on the render site alone. `useIsAdmin`'s predicate is extracted as a pure `getIsAdmin` so it can be unit-tested alongside the new cache-control predicates. Tests: - `test_runtime_config_patch_response_has_no_secrets` patches a writable setting with secrets configured and asserts they are absent from the response. - `test_non_admin_can_filter_out_missing_models` gives a non-admin one present and one missing model record and checks the missing one is not selectable. - `invocationCacheControls.test.ts` asserts a connected non-admin is offered no cache mutation, and that admin semantics are unchanged. openapi.json is unchanged: `/models/missing` keeps the same HTTPBearer security requirement either way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… gating Backend: - scan_for_models: guard scan_path before pathlib.Path() — the query param defaults to None and Path(None) raised a TypeError (500) when omitted. - scan_for_models: mid-scan failures return a detail-free 500 again instead of the generic 400. By that point is_dir() has confirmed existence, so a distinct status leaks nothing, and it restores the caller-error vs server-fault distinction for admins and retrying clients. Details still go to the log only. - _redact_config_secrets docstring no longer overclaims: coverage is by naming convention, and a differently-named future credential must extend the function. - openapi.json/schema.ts regenerated — the list_missing_models docstring was edited after the last regeneration, which would have failed typegen CI. Tests: - New default-deny meta-test walks app.routes and asserts every route carries an auth dependency or sits on an explicit 12-entry public allowlist, with a staleness check in the other direction. A hand-maintained route list cannot catch the next unauthenticated route — the exact failure mode behind #9365. - The oracle test no longer scans '/' (a real, unbounded ModelSearch walk of the host filesystem); it uses tmp_path fixtures, and the mid-scan-failure path is exercised deterministically by stubbing ModelSearch. - New test for the missing-scan_path 400. Frontend: - SettingsModal hides the Max Queue History field for non-admins instead of rendering it permanently blank-and-disabled (the backing runtime_config query is skipped for them), matching SettingsImageSubfolderStrategySelect. - onModelInstallError only fetches HF token status for admins: the event is broadcast to every client, but /hf_login is admin-only, so every non-admin session fired a doomed request and logged a spurious 403. - Consolidated five inline copies of the admin predicate (UseCacheCheckbox, useStarterModelsToast, NoContentForViewer, ModelPicker, UpscaleWarning) onto useIsAdmin, and getIsCustomNodesEnabled now delegates to getIsAdmin. The toast/viewer/picker copies treated a still-loading setup status as admin=true, disagreeing with the canonical predicate. - getIsAdmin unit tests moved to a colocated useIsAdmin.test.ts per the frontend CLAUDE.md colocation rule; the cache-control tests no longer import it. Test results: tests/app/routers 500 passed; frontend vitest 1354 passed; eslint/prettier/tsc clean; production bundle rebuilt via vite build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y its formatting Local ruff passed but CI pins ruff@0.11.2, which flags zip() without strict= (B905) and formats two signatures differently. Replaced the zip with a plain loop over the paths and ran the pinned version's formatter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e admin matrix - create_image_upload_entry (POST /api/v1/images/) now requires CurrentUserOrDefault. It was allowlisted as public only because it is a 501 stub, but a later implementation would have shipped unauthenticated without the route-audit test noticing. The allowlist entry is removed; unauthenticated requests get 401 and an authenticated request still reaches the stub (501, covered by a new test). - POST /api/v1/app/logging and GET /api/v1/app/invocation_cache/status added to the authorization matrices (both PROTECTED_ROUTES and ADMIN_ONLY_ROUTES for the former, ADMIN_ONLY_ROUTES for the latter), so demoting either dependency from AdminUserOrDefault would now fail a test. - openapi.json regenerated (HTTPBearer marker on the upload-entry op); schema.ts unchanged by typegen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…RLs for server-side downloads (#9492) * fix(download_queue): confine downloads to the cache dir and block non-public URLs The download endpoint accepted an arbitrary http(s) source and a relative destination from any authenticated user. Two problems followed. The destination was resolved against the server process's working directory rather than the download cache, so the caller chose where on the filesystem the download landed. An existing file at that path is unlinked and replaced, not skipped. In a source install started from the root directory that reaches `custom_nodes_dir`, whose top-level packages are imported at startup; in the container image the working directory holds the application's own package and the built frontend. The source URL was fetched with no restriction on where it pointed, so the job could be used to reach loopback services, private-network hosts and the cloud metadata endpoint. Redirects were followed unchecked, so a public URL could bounce the request onto a private address. Changes: - Anchor `dest` under `download_cache_path` and check containment after resolving, so the working directory no longer takes part. - Add `invokeai/app/util/ssrf.py`. A session built by `build_guarded_session()` validates the peer address of every socket before the request is written; checking the connected socket is what makes this hold against DNS rebinding and against host spellings that `requests` percent-decodes and we do not. `validate_download_url()` is a cheap up-front check that keeps the API's errors useful without being relied on alone. - Re-check every redirect hop through a response hook. - Require admin on all download_queue routes. The queue is server-wide: its jobs carry remote URLs and local paths, and cancelling one affects whoever started it. The web client does not use these routes. - Reject Content-Disposition and URL-derived filenames that are not a single safe path component. `..\evil`, `C:evil` and a trailing `..` segment all escaped the destination directory. - Add `allow_private_download_urls` for installs that mirror models on their own network. Warn at startup when a proxy is configured, since address policy belongs to the proxy in that case. Single-user mode is unaffected: `AdminUserOrDefault` resolves to the system administrator when multiuser is off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ssrf): reject legacy IPv4 literals * fix(download): close SSRF review gaps * fix(download): close remaining SSRF review gaps * chore: typegen/openapi * fix(model-install): serialize download completion wait * fix(download): serialize job paths as POSIX `_api_job` hands the API a relativised `dest`, which pydantic serialises with `str()`. On Windows that yields `models\sd15.safetensors` for a request that submitted `models/sd15.safetensors`, so the value a client gets back depends on the server OS -- and `test_download_accepts_relative_dest` fails on both windows-cpu CI legs. Serialise `dest` and `download_path` with `as_posix()`, matching what the download events in `events_common.py` already do. The JSON schema is unchanged: both fields are still plain strings, so no artifact regeneration is needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(api): remove the download_queue REST router entirely Smoke-tested removal supersedes the confinement approach for the router itself: the /api/v1/download_queue/* endpoints were a thin remote-control surface over the internal DownloadQueueService with no first-party consumer (the frontend only ever saw them as generated schema types; model install drives the same service in-process), and the enqueue route existed mostly as SSRF/dest-traversal attack surface. Deleting it removes the risk rather than fencing it. Everything protecting downloads the server performs on its own behalf stays: the guarded requests session (socket-peer public-address enforcement), per-connect URL validation and per-redirect-hop re-checks in DownloadQueueService, the server-derived-filename check, the allow_private_download_urls / download_proxy settings with proxy credential redaction, and admin-room scoping of download/model events. DownloadJob remains in the OpenAPI schema via ModelInstallJob.download_parts. schema.ts and openapi.json regenerated with the CI pipelines (typegen + prettier pass). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(download): honor download_proxy when allow_private_download_urls is set Surfaced by adversarial review of the router-removal commit: with both settings configured, the private-opt-in branch built a bare Session() that never received the explicit proxy — downloads went direct (or via ambient *_PROXY, the opposite of the documented policy) with no hint. The proxy is applied per request rather than on the session because request-level proxies are the only kind that take precedence over ambient *_PROXY variables in a plain Session (environment proxies are folded into the request dict with setdefault, and that dict wins the merge against session.proxies). Regression test verified to fail against the previous branch logic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev> Co-authored-by: JPPhoto <jpollack@jpollackphoto.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com>
lstein
requested review from
JPPhoto,
Pfannkuchensack,
blessedcoolant and
dunkeroni
as code owners
August 13, 2026 12:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE
This is the 6.13.8 patch-release branch, opened against
mainonly so that CI runs againstit. Every substantive commit here is already in
main(squash-merged); this branch carries thosefixes on top of the 6.13.x line together with the release version bumps. Merging it would replay
already-merged work and move
main's version backwards.What's new in 6.13.8
The only content change since v6.13.7 is a back-port of #9492:
fix(download): remove the download_queue REST API; block non-public URLs for server-side downloads/api/v1/download_queue/*REST router. It accepted an arbitraryhttp(s)source and a destination path from any authenticated user, had no first-partyconsumer, and existed mostly as SSRF / path-traversal attack surface.
invokeai/app/util/ssrf.py.build_guarded_session()validates the peer address of theconnected socket before the request is written, which is what holds against DNS rebinding and
against host spellings that
requestspercent-decodes differently than we do. Every redirecthop is re-checked through a response hook.
path component —
..\evil,C:eviland a trailing..all escaped the destination directory.allow_private_download_urlsanddownload_proxysettings, with proxy credentialsredacted from the runtime-config endpoint.
server-side paths.
Downloads the server performs on its own behalf — model installs from URLs — keep every one of
these protections.
Back-port notes
The release line is ~105 commits behind
main, so several hunks needed adapting. Each is adeliberate deviation from the upstream commit, not a merge artifact:
sockets.py— install/download events are addressed to admin sockets, not theadminroom. Upstream can use the room because it joins it in
_handle_connect; on this branch theroom is only entered on
subscribe_queue. Using the room here would have dropped these eventsfor the whole connect→subscribe window, and permanently for any client that never subscribes —
a regression against the previous unroomed broadcast. A
_admin_sids()helper (from_socket_users, populated at connect) addresses them directly instead, leaving this branch'squeue-event room semantics untouched.
sockets.py— model-load events stay broadcast. Upstream routesModelLoadStarted/CompleteEventtouser:{user_id}before the admin check, but that came froma separate
main-only commit and those events have nouser_idfield on this branch. Applyingit verbatim would have made the loading-models spinner admin-only. Their
configdoes carrypath/source, but any authenticated user can already read those for every model vialist_model_records, so broadcasting discloses nothing new.test_model_install.py—_wait_for_restore_complete()takes notimeoutand returnsNoneon this branch, so the new regression test calls it accordingly.test_invocation_event_socketio.py— the file ismain-only, so it was recreated with justthe tests that describe this branch's behavior.
main's invocation-event tests assert asingle
room=["user:x", "admin"]emit; this branch still does two separate emits.openapi.json,schema.tsanddocs/src/generated/settings.jsonwere regenerated from thisbranch's backend. The schema delta came out identical to
main's (285 / 325 lines), and the onlyadditions are the two new config fields.
Verification
ruff@0.11.2(CI's pin) check + format: cleanabove before this branch was pushed; it is fixed and covered by a test here.
Known non-blockers (identical upstream, not introduced here)
wait_for_installs()can still return with a job left in the non-terminalPAUSEDstate, via_download_cancelled_callback's resume-required path.download_proxyset, requests route through a separateProxyManager, so the socket-peerguard does not run — the proxy is responsible for address policy, as the new docs state.