Skip to content

chore(release): v6.13.8 (DO NOT MERGE) - #9496

Open
lstein wants to merge 18 commits into
mainfrom
lstein/chore/v6.13.8
Open

chore(release): v6.13.8 (DO NOT MERGE)#9496
lstein wants to merge 18 commits into
mainfrom
lstein/chore/v6.13.8

Conversation

@lstein

@lstein lstein commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

DO NOT MERGE

This is the 6.13.8 patch-release branch, opened against main only so that CI runs against
it. Every substantive commit here is already in main (squash-merged); this branch carries those
fixes on top of the 6.13.x line together with the release version bumps. Merging it would replay
already-merged work and move main's version backwards.

What's new in 6.13.8

The only content change since v6.13.7 is a back-port of #9492:

fix(download): remove the download_queue REST API; block non-public URLs for server-side downloads

  • Removes the general-purpose /api/v1/download_queue/* REST router. It accepted an arbitrary
    http(s) source and a destination path from any authenticated user, had no first-party
    consumer, and existed mostly as SSRF / path-traversal attack surface.
  • Adds invokeai/app/util/ssrf.py. build_guarded_session() validates the peer address of the
    connected socket
    before the request is written, which is what holds against DNS rebinding and
    against host spellings that requests percent-decodes differently than we do. Every redirect
    hop is re-checked through a response hook.
  • Rejects server-derived filenames (Content-Disposition, URL path) that are not a single safe
    path component — ..\evil, C:evil and a trailing .. all escaped the destination directory.
  • Adds allow_private_download_urls and download_proxy settings, with proxy credentials
    redacted from the runtime-config endpoint.
  • Scopes download and model-install socket events to administrators; they carry source URLs and
    server-side paths.

Downloads the server performs on its own behalf — model installs from URLs — keep every one of
these protections.

Back-port notes

The release line is ~105 commits behind main, so several hunks needed adapting. Each is a
deliberate deviation from the upstream commit, not a merge artifact:

  1. sockets.py — install/download events are addressed to admin sockets, not the admin
    room.
    Upstream can use the room because it joins it in _handle_connect; on this branch the
    room is only entered on subscribe_queue. Using the room here would have dropped these events
    for the whole connect→subscribe window, and permanently for any client that never subscribes —
    a regression against the previous unroomed broadcast. A _admin_sids() helper (from
    _socket_users, populated at connect) addresses them directly instead, leaving this branch's
    queue-event room semantics untouched.
  2. sockets.py — model-load events stay broadcast. Upstream routes
    ModelLoadStarted/CompleteEvent to user:{user_id} before the admin check, but that came from
    a separate main-only commit and those events have no user_id field on this branch. Applying
    it verbatim would have made the loading-models spinner admin-only. Their config does carry
    path/source, but any authenticated user can already read those for every model via
    list_model_records, so broadcasting discloses nothing new.
  3. test_model_install.py_wait_for_restore_complete() takes no timeout and returns
    None on this branch, so the new regression test calls it accordingly.
  4. test_invocation_event_socketio.py — the file is main-only, so it was recreated with just
    the tests that describe this branch's behavior. main's invocation-event tests assert a
    single room=["user:x", "admin"] emit; this branch still does two separate emits.

openapi.json, schema.ts and docs/src/generated/settings.json were regenerated from this
branch's backend. The schema delta came out identical to main's (285 / 325 lines), and the only
additions are the two new config fields.

Verification

  • Full Python suite: 1961 passed, 106 skipped, 8 xfailed, 0 failed
  • ruff@0.11.2 (CI's pin) check + format: clean
  • Every new regression test confirmed to fail when its production fix is reverted
  • An adversarial fresh-context review of the back-port caught the admin-room defect in note 1
    above before this branch was pushed; it is fixed and covered by a test here.

Known non-blockers (identical upstream, not introduced here)

  • wait_for_installs() can still return with a job left in the non-terminal PAUSED state, via
    _download_cancelled_callback's resume-required path.
  • The URL-derived-filename check has no direct test (the Content-Disposition path does).
  • With download_proxy set, requests route through a separate ProxyManager, so the socket-peer
    guard does not run — the proxy is responsible for address policy, as the new docs state.

lstein and others added 18 commits June 29, 2026 20:48
…room before decode/encode (#9305)

* fix(qwen): estimate VAE working memory so the cache frees room before decode/encode

The Qwen Image l2i/i2l invocations called `model_on_device()` without a
`working_mem_bytes` estimate, unlike the SD/SDXL path. The model cache
therefore only reserved the default `device_working_mem_gb` and never
evicted the resident transformer/text encoder before the VAE decode. On a
near-full card (e.g. Qwen Image Edit Q8_0 with transformer + text encoder
resident) the decode then OOMs trying to allocate its working set into the
fragmented remainder.

Add `estimate_vae_working_memory_qwen_image()` and pass it into both the
decode and encode paths so the cache makes room (evicting other models when
needed) before the operation runs.

The constant is calibrated against a measured decode on an AMD W7900: at
1248x832 the decode grew CUDA reserved memory by ~10.06 GiB (implied
constant ~5082), rounded up to 5500 for headroom. It tracks peak *reserved*
(not just allocated) memory so that whenever the cache declines to free room
(free >= estimate) the decode is still guaranteed to fit. Encode uses ~half,
matching the other estimators (not independently measured).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(qwen): cover VAE working-memory estimate is passed to cache

Address review feedback from @Pfannkuchensack on #9305:
- Add test_qwen_image_working_memory.py mirroring the z-image pattern,
  asserting both decode and encode paths call model_on_device with the
  estimated working_mem_bytes (regression guard for the OOM fix).
- Clarify the qwen estimator comment: the encode constant is not
  independently measured (half of decode, matching siblings' ratio) and
  should be recalibrated against a measured encode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(qwen): recalibrate VAE working-memory constants from a measured grid

Add scripts/calibrate_qwen_vae_working_memory.py, a backend-portable
(CUDA/ROCm) harness that measures peak reserved-memory growth for VAE
decode/encode across a resolution grid, one fresh subprocess per point.

Calibrating on an AMD W7900 (fp16) showed the encode constant was wrong:
the previous 2750 ("half of decode") under-estimated by ~2x at every
measured resolution, the exact OOM mode Qwen Image Edit (which encodes a
real image) would hit. Raise encode 2750 -> 6300. Decode 5500 is confirmed
safe across the full 512^2..2048^2 range and left unchanged.

The grid also showed memory is super-linear in area above ~1792^2 (an
attention term) and non-monotonic (likely an SDPA-backend crossover on
ROCm); both documented in the estimator. Constants are the conservative
ROCm side and will be max-merged with a pending NVIDIA/CUDA run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(qwen): branch VAE working-memory constants by backend (ROCm vs CUDA)

Calibrating the same fp16 grid on an NVIDIA card showed CUDA reserves
~2x (decode) to ~4x (encode) less than ROCm: the Qwen VAE is attention-
heavy, and CUDA's Flash/efficient attention is O(area) and flat while the
ROCm math-attention fallback is O(area^2). The backends diverge far more
than any headroom, so a single constant either under-estimates on ROCm
(OOM) or massively over-budgets CUDA (needless eviction).

Select constants via torch.version.hip:
  decode: ROCm 5500 / CUDA 2900
  encode: ROCm 6300 / CUDA 1600
Each verified to cover its measured grid (19 points/backend) with ~8%
headroom. The CUDA run also confirms the linear model holds with Flash
attention (the ROCm super-linear/non-monotonic behavior is a math-
attention artifact), and that "encode is half of decode" is CUDA-only.

Add parametrized tests asserting the constant selected for each
(operation, backend) so a refactor can't silently swap them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(backend): ruff

* calibrate: support single-file Qwen Image VAE checkpoints

The calibration script only loaded the Qwen VAE from a diffusers
directory via from_pretrained, so passing a single .safetensors file
failed. Add _load_vae, which loads a directory as before and handles a
single-file checkpoint by loading the state dict directly: a strict load
for the diffusers layout, falling back to convert_wan_vae_to_diffusers
for the original Qwen-Image/Wan release layout (downsamples/residual/
time_conv keys) before retrying.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev>
* docs: add 3d party GPU hosting services

* Update docs/src/content/docs/index.mdx

Co-authored-by: Josh Corbett <joshwcorbett@icloud.com>

* docs: restyle hosted options as LinkCards in a wrapper card

Implements joshistoast's suggested design: a bordered "Hosted Options"
wrapper containing Starlight CardGrid/LinkCard entries, replacing the
text separator and hand-rolled cards.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev>
Co-authored-by: Josh Corbett <joshwcorbett@icloud.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…s 5.x (#9333)

Qwen2_5_VLForConditionalGeneration._checkpoint_conversion_mapping is empty
in transformers 5.x, so the `if key_mapping:` guard skipped remapping and
ComfyUI single-file checkpoints (`visual.*`, `model.*`) failed to load:
their keys were reported as unexpected while `model.visual.*` params stayed
as meta tensors. Fall back to the legacy mapping when transformers no longer
provides one. The negative lookahead keeps new-layout checkpoints untouched.
A number of model-management and app-info routes carried no auth dependency
at all. In multi-user mode this left them reachable by a fully unauthenticated
network attacker, contradicting the documented model that model management is
administrator-only.

The highest-impact route was `GET /api/v2/models/scan_folder`, which takes an
attacker-controlled filesystem path and recursively enumerates it, returning
absolute paths of model-like files. Its distinct 200/400/500 responses also
formed an existence/readability oracle for arbitrary paths.

While auditing, `GET /api/v1/app/runtime_config` turned out to be worse: it
served the raw `InvokeAIAppConfig` — including `remote_api_tokens` and the
`external_*_api_key` values — in plaintext, to anyone.

Changes:

- `scan_folder` now requires `AdminUserOrDefault`, and every failure mode
  returns one generic 400 (details go to the server log) so the response
  cannot be used as a filesystem oracle. Arbitrary paths remain scannable by
  admins, consistent with `install_model`, which already accepts local paths.
- Admin-only (`AdminUserOrDefault`): `missing`, `hugging_face`,
  `starter_models`, `stats`, `hf_login`, `external_providers/config`,
  `logging` (GET/POST) and the `invocation_cache` routes. All are either
  operator-only or rendered exclusively behind the admin-gated install panel.
- Authenticated-user (`CurrentUserOrDefault`): model record reads, which
  ordinary users need for generation, plus `runtime_config`, `app_deps`,
  `patchmatch_status` and `external_providers/status`.
- `runtime_config` now masks API keys and download tokens server-side rather
  than shipping them to the browser; the UI only ever needed the
  is-configured signal.

Single-user mode is unaffected — `*OrDefault` resolves to a system admin when
`multiuser` is off.

Not changed: the binary-serving routes (image full/thumbnail, model cover
image, workflow thumbnail) are consumed via `<img src>` and cannot carry a
Bearer header, so closing those needs a signed-URL or cookie scheme.

Closes #9365

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The added auth dependencies make FastAPI emit a `security: [{HTTPBearer: []}]`
requirement on each affected route, so the checked-in schema no longer matched
the generated one.

Regenerated with the openapi-checks command. The diff is exactly the 21 routes
gated in the previous commit gaining a security requirement; no route loses
one, and the rest of the document is byte-identical.

`schema.ts` is unchanged — openapi-typescript does not encode security
requirements in the generated types.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Serving the runtime config to every authenticated user relied on a denylist:
`_redact_config_secrets` masks the credential fields we know about, so any
future secret added to `InvokeAIAppConfig` would leak by default until someone
remembered to update it. Admin-only is an allowlist posture without that
failure mode.

Nothing outside the admin UI actually needed it. The three non-admin call
sites read `config.multiuser` purely to derive "admin or single-user", which is
already available from the unauthenticated `/auth/status` route as
`multiuser_enabled` — the same source `useIsModelManagerEnabled` uses. The
remaining fields (`max_queue_history`, `image_subfolder_strategy`) only feed
admin-gated edit controls.

- Extract that predicate into `useIsAdmin`, and have `useIsModelManagerEnabled`
  delegate to it. Semantics are preserved exactly, so this is a pure refactor.
- Use it in the three settings components instead of `runtime_config`, and skip
  the query for non-admins so they generate no failed requests.
- AboutModal's debug blob now omits the config section for non-admins. Its
  client-side redaction is dropped: it only ever masked `remote_api_tokens` and
  never the `external_*_api_key` fields, so a logged-in non-admin could read
  provider API keys out of it. The server now masks both.

Server-side masking is kept as defense in depth — an admin's browser has no use
for the raw values either.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… gating

Three issues from JPPhoto's review of #9367:

- `PATCH /runtime_config` echoed the updated config back unredacted, so an admin
  changing an unrelated setting still pulled every external API key and
  `remote_api_tokens` value into the browser's RTK Query cache, defeating the
  masking added to the GET route. It now returns `_redact_config_secrets(config)`
  as GET does.

- `GET /models/missing` is `CurrentUserOrDefault` again, not admin-only. The
  frontend's model hooks (`modelsByType.ts`) subtract this set from the model
  list so unusable records stay out of the generation dropdowns. Under an
  admin-only gate a non-admin got 403, the subtraction became a silent no-op,
  and missing models were offered for selection, failing only at execution.

- The invocation-cache panel lives on the universally available Queue tab, but
  its routes are admin-only. A non-admin saw zeroed statistics and an active
  Enable button that always 403s. The panel is now gated with `useIsAdmin`, and
  the enable/disable/clear hooks additionally refuse to offer the mutation to a
  non-admin, so the guard does not depend on the render site alone.

`useIsAdmin`'s predicate is extracted as a pure `getIsAdmin` so it can be
unit-tested alongside the new cache-control predicates.

Tests:
- `test_runtime_config_patch_response_has_no_secrets` patches a writable setting
  with secrets configured and asserts they are absent from the response.
- `test_non_admin_can_filter_out_missing_models` gives a non-admin one present
  and one missing model record and checks the missing one is not selectable.
- `invocationCacheControls.test.ts` asserts a connected non-admin is offered no
  cache mutation, and that admin semantics are unchanged.

openapi.json is unchanged: `/models/missing` keeps the same HTTPBearer security
requirement either way.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… gating

Backend:
- scan_for_models: guard scan_path before pathlib.Path() — the query param
  defaults to None and Path(None) raised a TypeError (500) when omitted.
- scan_for_models: mid-scan failures return a detail-free 500 again instead of
  the generic 400. By that point is_dir() has confirmed existence, so a distinct
  status leaks nothing, and it restores the caller-error vs server-fault
  distinction for admins and retrying clients. Details still go to the log only.
- _redact_config_secrets docstring no longer overclaims: coverage is by naming
  convention, and a differently-named future credential must extend the function.
- openapi.json/schema.ts regenerated — the list_missing_models docstring was
  edited after the last regeneration, which would have failed typegen CI.

Tests:
- New default-deny meta-test walks app.routes and asserts every route carries an
  auth dependency or sits on an explicit 12-entry public allowlist, with a
  staleness check in the other direction. A hand-maintained route list cannot
  catch the next unauthenticated route — the exact failure mode behind #9365.
- The oracle test no longer scans '/' (a real, unbounded ModelSearch walk of the
  host filesystem); it uses tmp_path fixtures, and the mid-scan-failure path is
  exercised deterministically by stubbing ModelSearch.
- New test for the missing-scan_path 400.

Frontend:
- SettingsModal hides the Max Queue History field for non-admins instead of
  rendering it permanently blank-and-disabled (the backing runtime_config query
  is skipped for them), matching SettingsImageSubfolderStrategySelect.
- onModelInstallError only fetches HF token status for admins: the event is
  broadcast to every client, but /hf_login is admin-only, so every non-admin
  session fired a doomed request and logged a spurious 403.
- Consolidated five inline copies of the admin predicate (UseCacheCheckbox,
  useStarterModelsToast, NoContentForViewer, ModelPicker, UpscaleWarning) onto
  useIsAdmin, and getIsCustomNodesEnabled now delegates to getIsAdmin. The
  toast/viewer/picker copies treated a still-loading setup status as admin=true,
  disagreeing with the canonical predicate.
- getIsAdmin unit tests moved to a colocated useIsAdmin.test.ts per the
  frontend CLAUDE.md colocation rule; the cache-control tests no longer import it.

Test results: tests/app/routers 500 passed; frontend vitest 1354 passed;
eslint/prettier/tsc clean; production bundle rebuilt via vite build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y its formatting

Local ruff passed but CI pins ruff@0.11.2, which flags zip() without strict=
(B905) and formats two signatures differently. Replaced the zip with a plain
loop over the paths and ran the pinned version's formatter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e admin matrix

- create_image_upload_entry (POST /api/v1/images/) now requires
  CurrentUserOrDefault. It was allowlisted as public only because it is a
  501 stub, but a later implementation would have shipped unauthenticated
  without the route-audit test noticing. The allowlist entry is removed;
  unauthenticated requests get 401 and an authenticated request still
  reaches the stub (501, covered by a new test).
- POST /api/v1/app/logging and GET /api/v1/app/invocation_cache/status
  added to the authorization matrices (both PROTECTED_ROUTES and
  ADMIN_ONLY_ROUTES for the former, ADMIN_ONLY_ROUTES for the latter), so
  demoting either dependency from AdminUserOrDefault would now fail a test.
- openapi.json regenerated (HTTPBearer marker on the upload-entry op);
  schema.ts unchanged by typegen.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…RLs for server-side downloads (#9492)

* fix(download_queue): confine downloads to the cache dir and block non-public URLs

The download endpoint accepted an arbitrary http(s) source and a relative
destination from any authenticated user. Two problems followed.

The destination was resolved against the server process's working directory
rather than the download cache, so the caller chose where on the filesystem the
download landed. An existing file at that path is unlinked and replaced, not
skipped. In a source install started from the root directory that reaches
`custom_nodes_dir`, whose top-level packages are imported at startup; in the
container image the working directory holds the application's own package and
the built frontend.

The source URL was fetched with no restriction on where it pointed, so the job
could be used to reach loopback services, private-network hosts and the cloud
metadata endpoint. Redirects were followed unchecked, so a public URL could
bounce the request onto a private address.

Changes:

- Anchor `dest` under `download_cache_path` and check containment after
  resolving, so the working directory no longer takes part.
- Add `invokeai/app/util/ssrf.py`. A session built by `build_guarded_session()`
  validates the peer address of every socket before the request is written;
  checking the connected socket is what makes this hold against DNS rebinding
  and against host spellings that `requests` percent-decodes and we do not.
  `validate_download_url()` is a cheap up-front check that keeps the API's
  errors useful without being relied on alone.
- Re-check every redirect hop through a response hook.
- Require admin on all download_queue routes. The queue is server-wide: its
  jobs carry remote URLs and local paths, and cancelling one affects whoever
  started it. The web client does not use these routes.
- Reject Content-Disposition and URL-derived filenames that are not a single
  safe path component. `..\evil`, `C:evil` and a trailing `..` segment all
  escaped the destination directory.
- Add `allow_private_download_urls` for installs that mirror models on their
  own network. Warn at startup when a proxy is configured, since address policy
  belongs to the proxy in that case.

Single-user mode is unaffected: `AdminUserOrDefault` resolves to the system
administrator when multiuser is off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ssrf): reject legacy IPv4 literals

* fix(download): close SSRF review gaps

* fix(download): close remaining SSRF review gaps

* chore: typegen/openapi

* fix(model-install): serialize download completion wait

* fix(download): serialize job paths as POSIX

`_api_job` hands the API a relativised `dest`, which pydantic serialises with
`str()`. On Windows that yields `models\sd15.safetensors` for a request that
submitted `models/sd15.safetensors`, so the value a client gets back depends on
the server OS -- and `test_download_accepts_relative_dest` fails on both
windows-cpu CI legs.

Serialise `dest` and `download_path` with `as_posix()`, matching what the
download events in `events_common.py` already do. The JSON schema is unchanged:
both fields are still plain strings, so no artifact regeneration is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore(api): remove the download_queue REST router entirely

Smoke-tested removal supersedes the confinement approach for the
router itself: the /api/v1/download_queue/* endpoints were a thin
remote-control surface over the internal DownloadQueueService with no
first-party consumer (the frontend only ever saw them as generated
schema types; model install drives the same service in-process), and
the enqueue route existed mostly as SSRF/dest-traversal attack
surface. Deleting it removes the risk rather than fencing it.

Everything protecting downloads the server performs on its own behalf
stays: the guarded requests session (socket-peer public-address
enforcement), per-connect URL validation and per-redirect-hop
re-checks in DownloadQueueService, the server-derived-filename check,
the allow_private_download_urls / download_proxy settings with proxy
credential redaction, and admin-room scoping of download/model events.

DownloadJob remains in the OpenAPI schema via
ModelInstallJob.download_parts. schema.ts and openapi.json regenerated
with the CI pipelines (typegen + prettier pass).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(download): honor download_proxy when allow_private_download_urls is set

Surfaced by adversarial review of the router-removal commit: with both
settings configured, the private-opt-in branch built a bare Session()
that never received the explicit proxy — downloads went direct (or via
ambient *_PROXY, the opposite of the documented policy) with no hint.

The proxy is applied per request rather than on the session because
request-level proxies are the only kind that take precedence over
ambient *_PROXY variables in a plain Session (environment proxies are
folded into the request dict with setdefault, and that dict wins the
merge against session.proxies).

Regression test verified to fail against the previous branch logic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev>
Co-authored-by: JPPhoto <jpollack@jpollackphoto.com>
Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com>
@github-actions github-actions Bot added api python PRs that change python files invocations PRs that change invocations backend PRs that change backend files services PRs that change app services frontend PRs that change frontend files python-tests PRs that change python tests docs PRs that change docs labels Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api backend PRs that change backend files docs PRs that change docs frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests services PRs that change app services

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants