Skip to content

fix(download): remove the download_queue REST API; block non-public URLs for server-side downloads - #9492

Merged
lstein merged 13 commits into
invoke-ai:mainfrom
lstein:fix/download-queue-ssrf-and-dest-confinement
Aug 13, 2026
Merged

fix(download): remove the download_queue REST API; block non-public URLs for server-side downloads#9492
lstein merged 13 commits into
invoke-ai:mainfrom
lstein:fix/download-queue-ssrf-and-dest-confinement

Conversation

@lstein

@lstein lstein commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes a reported security issue in POST /api/v1/download_queue/i/, reproduced against main at 45ad17b.

In multiuser mode any authenticated non-admin could supply an arbitrary http(s) source and a relative dest. Two things followed:

  1. Unconfined destination. dest was resolved against the server process's working directory instead of the download cache, so the caller chose where the file landed. An existing file at that path is unlinked and replaced, not skipped. The working directory is not arbitrary — in a source install started from the root directory it contains custom_nodes_dir, whose top-level packages are imported at startup; in the container image (WORKDIR ${INVOKEAI_SRC}) it contains the invokeai/ package and the built frontend. So the practical impact is code execution on restart, or replacing frontend assets served to every user.

  2. Unrestricted source. The URL was fetched with no restriction on where it resolved, so the job could be used to reach loopback services, private-network hosts and the cloud metadata endpoint. Redirects were also followed unchecked, which the original report did not cover — a public URL returning a 302 to a private address was followed and its body fetched.

Changes

The /api/v1/download_queue/* router is removed entirely. Earlier revisions of this PR confined it (admin-only, destination anchored under a dedicated directory, URL validation on enqueue); after smoke-testing the removal we are dropping it instead. The router was a thin remote control over the internal DownloadQueueService with no first-party consumer — the web client never called it (its only trace was generated schema types), and model install drives the same service in-process through /api/v2/models/install. Deleting the endpoint removes the destination-traversal surface outright instead of fencing it. DownloadJob legitimately remains in the OpenAPI schema via ModelInstallJob.download_parts; schema.ts and openapi.json are regenerated with the CI pipelines.

The SSRF protections stay, because the server still downloads user-supplied URLs on its own behalf (model installs from URLs being the main path):

  • New invokeai/app/util/ssrf.py. build_guarded_session() returns a requests.Session whose adapter validates the peer address of every socket before any request byte is written. validate_download_url() is a cheap up-front check for good error messages. DownloadQueueService builds its own sessions guarded; a caller-supplied session (tests) is left as given.
  • Every redirect hop is re-validated through a requests response hook, and the URL is re-checked at connect time, not just submit time.
  • Filename hardening. Content-Disposition and URL-derived names must be a single safe path component; ..\evil, C:evil (drive-relative) and a trailing .. URL segment all escaped the destination directory.
  • allow_private_download_urls (default false) for installs that mirror models on their own network.
  • Proxy policy. The guarded session ignores ambient HTTP_PROXY/HTTPS_PROXY/ALL_PROXY (resolved with trust_env=False) so an environment proxy cannot route around the socket guard; an explicit download_proxy setting is provided for mandatory-egress environments, with a startup warning that address policy then belongs to the proxy. Proxy credentials are redacted from the runtime-config endpoint. download_proxy is honored with allow_private_download_urls enabled too (applied per request in that mode, the only level ambient variables cannot override).
  • Download and model-install socket events are scoped to the admin room — they carry signed source URLs and server filesystem paths and feed the admin-only model manager UI. Single-user mode is unaffected (AdminUserOrDefault semantics).

Why the guard is at the socket layer

Two earlier iterations of this fix validated the URL string, and both were bypassable:

  • requests runs URLs through requote_uri(), which percent-decodes unreserved characters in the host before connecting. A check on urlsplit().hostname therefore validates a different string than the one dialled.
  • Any resolve-then-connect check loses to DNS rebinding: the guard and the client each call getaddrinfo independently.

Checking sock.getpeername() is the only check that cannot be desynchronised from what the client actually does. The up-front check is kept because it gives the API a useful 400 and avoids turning the socket layer into a port-existence oracle, but it is explicitly not load-bearing.

Address classification is worth a careful look during review. not ip.is_global alone is not safe: for IPv6, CPython defines is_global as not is_private, and the IPv6 private list excludes reserved/unallocated space — so NAT64 (64:ff9b::/96), IPv4-compatible (::/8) and 4000::/3 all report is_global == True. Conversely is_private/is_reserved alone miss IPv4 100.64.0.0/10. The predicate needs all three of not is_global, is_reserved, is_multicast, applied to the address and to every IPv4 it can wrap (v4-mapped, 6to4, Teredo, ISATAP) — yielding rather than substituting, so a wrapper with a public payload is still judged as a wrapper.

Verification

  • App-side suite (tests/ minus tests/backend): 2828 passed, 0 failures, including router, download-service, model-install, SSRF and socket.io suites. Lint (ruff 0.11.2) and tsc clean.
  • App boots with the router gone: /api/v1/download_queue/* returns 404, remaining routes intact; frontend smoke test of the full app passed on the equivalent standalone removal branch.
  • The original report's claims were each reproduced end-to-end before fixing, and every guard was sensitivity-checked: reverted individually to confirm its test fails against the broken code.
  • Checked for over-blocking by resolving A and AAAA records for huggingface.co, cdn-lfs.hf.co, civitai.com, github.com, objects/raw.githubusercontent.com, pypi.org, files.pythonhosted.org and cdn.jsdelivr.net — nothing blocked. A live https://example.com fetch through the guarded session returns 200, and Range/streaming/redirect behaviour is unchanged.

Notes for review

  • API removal is a breaking change for any external script that used the download-queue REST API. There is no in-product consumer; scripted downloads can use model install, or the operator can fetch files themselves.
  • Proxies. Ignoring ambient proxy variables is itself a behaviour change for installs that relied on them; such installs must set download_proxy explicitly. This is deliberate: an ambient proxy silently defeats the peer-address guard, and no_proxy interactions made the degraded mode hard to reason about.

Known gap, deliberately not fixed here

download_and_cache_model returns cache entries unverified and consumers pass them to torch.load/torch.jit.load, so a write inside the cache — by whatever means — can still poison a cached model that any user's node may load. Removing the REST endpoint eliminates the only remote write path this PR knew about, but hash verification in the cache is still worth doing separately.

Two adjacent pre-existing items, also out of scope: external_generation/providers/alibabacloud.py::_download_image fetches a provider-response URL with no guard, and custom_nodes.py passes source straight to git clone without a scheme check.

🤖 Generated with Claude Code

…-public URLs

The download endpoint accepted an arbitrary http(s) source and a relative
destination from any authenticated user. Two problems followed.

The destination was resolved against the server process's working directory
rather than the download cache, so the caller chose where on the filesystem the
download landed. An existing file at that path is unlinked and replaced, not
skipped. In a source install started from the root directory that reaches
`custom_nodes_dir`, whose top-level packages are imported at startup; in the
container image the working directory holds the application's own package and
the built frontend.

The source URL was fetched with no restriction on where it pointed, so the job
could be used to reach loopback services, private-network hosts and the cloud
metadata endpoint. Redirects were followed unchecked, so a public URL could
bounce the request onto a private address.

Changes:

- Anchor `dest` under `download_cache_path` and check containment after
  resolving, so the working directory no longer takes part.
- Add `invokeai/app/util/ssrf.py`. A session built by `build_guarded_session()`
  validates the peer address of every socket before the request is written;
  checking the connected socket is what makes this hold against DNS rebinding
  and against host spellings that `requests` percent-decodes and we do not.
  `validate_download_url()` is a cheap up-front check that keeps the API's
  errors useful without being relied on alone.
- Re-check every redirect hop through a response hook.
- Require admin on all download_queue routes. The queue is server-wide: its
  jobs carry remote URLs and local paths, and cancelling one affects whoever
  started it. The web client does not use these routes.
- Reject Content-Disposition and URL-derived filenames that are not a single
  safe path component. `..\evil`, `C:evil` and a trailing `..` segment all
  escaped the destination directory.
- Add `allow_private_download_urls` for installs that mirror models on their
  own network. Warn at startup when a proxy is configured, since address policy
  belongs to the proxy in that case.

Single-user mode is unaffected: `AdminUserOrDefault` resolves to the system
administrator when multiuser is off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added api python PRs that change python files services PRs that change app services frontend PRs that change frontend files python-tests PRs that change python tests docs PRs that change docs labels Aug 12, 2026
@lstein lstein moved this to 6.14.x Theme: USER EXPERIENCE in Invoke - Community Roadmap Aug 12, 2026
@lstein lstein added the 6.14.0 label Aug 12, 2026
@Pfannkuchensack

Copy link
Copy Markdown
Member

Findings

High: invokeai/app/api/routers/download_queue.py:24-66 confines dest into the download cache, which is exactly the directory download_and_cache_model treats as trusted and never verifies. Any non-admin user then triggers the payload.

Chain:

  1. invokeai/app/services/model_install/model_install_default.py:770-772 derives the cache directory as download_cache_path / slugify(str(source)) - fully predictable from the public model URL.
  2. invokeai/app/services/model_install/model_install_default.py:784-787 returns contents[0] of that directory whenever it is non-empty. No hash, no size, no picklescan, no download.
  3. invokeai/app/api/routers/download_queue.py:62 accepts dest = "<slug>/big-lama.pt" - a plain relative path with no .. - and resolves it to exactly that directory.
  4. invokeai/backend/image_util/infill_methods/lama.py:52 calls torch.jit.load(model_path) on whatever came back; invokeai/backend/image_util/hed.py:86, invokeai/backend/image_util/lineart.py:107 and invokeai/backend/image_util/lineart_anime.py:152 call torch.load on the same class of cache entries.
  5. invokeai/app/services/shared/invocation_context.py:663,711 is the node-facing entry point, so the load is triggered by any user running a LaMa Infill or a DWOpenpose/lineart node - not by the admin who wrote the file.

The docstring at invokeai/app/api/routers/download_queue.py:34-38 acknowledges this and justifies it with "an administrator can already install arbitrary models". That justification does not hold as stated: the ordinary install path is picklescanned (invokeai/backend/model_manager/model_on_disk.py:121 / invokeai/backend/model_manager/util/model_util.py:66, unsafe_disable_picklescan defaults to False at invokeai/app/services/config/config_default.py:243), whereas the cache path this PR now legitimises has no scan at all. So the change converts "admin can install a scanned model" into "admin can plant an unscanned torch.jit archive that fires under another user's session". In multiuser deployments the admin role is not the machine owner, so this is a privilege boundary, not a no-op.

Minimum viable mitigation inside this PR's scope: reject a dest whose first path component matches an existing download_cache_path entry, or refuse dest values that resolve into download_cache_path at all and give manual downloads their own subdirectory.

To expose this issue, add a test that writes a file into download_cache_path / slugify(<lama url>) via the router and then asserts download_and_cache_model(<lama url>) refuses to return it.


Medium: invokeai/app/api/routers/download_queue.py:98-113 performs synchronous DNS resolution inside an async def handler, stalling the whole server event loop.

download() is a coroutine, so FastAPI runs it on the event loop thread. Line 109 calls validate_download_url(), which reaches invokeai/app/util/ssrf.py:110-112 -> socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP). That is a blocking syscall with no timeout, and nothing in the repo sets socket.setdefaulttimeout. invokeai/app/util/ssrf.py:161-169 iterates over up to two host spellings, so a single request can issue two such lookups.

Trigger: a caller submits a URL whose hostname is delegated to an unresponsive authoritative nameserver. The resolver blocks for its full retry budget (glibc default 5s x 2 attempts per nameserver) while the event loop is held, so every other in-flight HTTP request and every socket.io event for every user is stalled. Repeat the POST to hold it indefinitely.

Note this is reachable without any credentials in the default single-user configuration: AdminUserOrDefault short-circuits to the system admin when multiuser is off (invokeai/app/api/auth_dependencies.py, get_current_user_or_default pattern at lines 100-103). The same validation inside invokeai/app/services/download/download_default.py:438 is fine, because that runs on a worker thread.

The repo already has the idiom for this: invokeai/app/api/routers/model_manager.py:479 and invokeai/app/api/routers/utilities.py:188 use asyncio.to_thread, invokeai/app/api/routers/videos.py:292 uses run_in_threadpool. Wrap the validate_download_url call (and ideally _validate_dest, which also does mkdir + two resolve() calls at lines 57-62) the same way.

To expose this issue, add a test that monkeypatches invokeai.app.util.ssrf._resolve to sleep, issues the POST against an ASGI transport, and asserts a concurrently-issued request to a trivial route completes while the download request is still in flight.


Medium: invokeai/app/util/ssrf.py:161-169 fails open on Windows for legacy IPv4 spellings, and the two test cases asserting otherwise fail on the windows-2022 CI leg.

_resolve() delegates to socket.getaddrinfo, whose acceptance of octal and 32-bit-integer IPv4 literals is a libc behaviour, not a Python one. Verified on Windows 10 with the PR checked out:

0177.0.0.1  ERR gaierror [Errno 11001] getaddrinfo failed
2130706433  ERR gaierror [Errno 11001] getaddrinfo failed
127.0.0.1   ('127.0.0.1', 0)

The OSError is swallowed by except (OSError, UnicodeError, ValueError): continue at line 168, so validate_download_url("http://0177.0.0.1/x") returns cleanly. Running the PR's own suite here:

FAILED tests/app/util/test_ssrf.py::test_rejects_non_public_addresses[http://0177.0.0.1/x]
FAILED tests/app/util/test_ssrf.py::test_rejects_non_public_addresses[http://2130706433/x]
2 failed, 88 passed

.github/workflows/python-tests.yml:38-60 runs the full pytest on windows-2022 for both py3.11 and py3.12, so this is red CI on 2 of 6 matrix legs. The PR body's "Full suite: 4243 passed" was a Linux-only observation.

Security impact is nil (urllib3 shares the resolver, so it cannot dial these spellings either), but the assertion is a portability defect and the guard does silently degrade. The equivalent router cases pass only incidentally: AnyHttpUrl runs the WHATWG IPv4 parser and normalises http://2130706433/ to http://127.0.0.1/ before validate_download_url ever sees it. Fix by parsing these forms explicitly (e.g. ipaddress.ip_address(int(host)) / socket.inet_aton) rather than relying on the platform resolver, and keep the assertion platform-independent.


Medium: the admin-gating of the router does not achieve its stated confidentiality goal, because the same fields are still broadcast to every connected socket.

The PR justifies moving list_downloads / get_download_job / cancel_download_job from CurrentUserOrDefault to AdminUserOrDefault (invokeai/app/api/routers/download_queue.py:73,125,145) on the grounds that "jobs carry remote URLs and local filesystem paths". Those exact fields continue to reach non-admins:

  1. invokeai/app/services/events/events_common.py:448-451 - DownloadEventBase.source is the full remote URL (which for a signed/presigned mirror URL carries the credential in its query string).
  2. invokeai/app/services/events/events_common.py:460 and :474 - DownloadStartedEvent.download_path / DownloadProgressEvent.download_path is the absolute server filesystem path.
  3. invokeai/app/api/sockets.py:610-612 - _handle_model_event emits these with no room=, i.e. broadcast to every connected socket, with an explicit comment that this is intentional.

So a non-admin in multiuser mode cannot GET /api/v1/download_queue/ any more, but still receives download_started / download_progress for every admin-initiated job. Either route the DownloadEventBase subclasses to room="admin" alongside this change, or drop the confidentiality argument from the justification and keep only the "cancelling one affects whoever started it" integrity argument.

To expose this issue, add a test that asserts _handle_model_event emits DownloadStartedEvent with a non-empty room when multiuser is enabled.


Medium: no test binds DownloadQueueService to the guarded session, so the layer the PR calls "the one that actually holds" is unprotected against regression.

invokeai/app/services/download/download_default.py:79-85 is the only production wiring of build_guarded_session() (invokeai/app/api/dependencies.py:180 constructs the service with no requests_session). Every service-level test passes an explicit requests_session=, which takes the line-80 branch and skips the guard entirely; the two new SSRF service tests therefore only exercise _validate_url, the up-front check that invokeai/app/util/ssrf.py:19-22 explicitly documents as "must never be relied on alone". Replacing build_guarded_session() with requests.Session() at line 84 would leave every test green. The socket guard is only tested against build_guarded_session() called directly.

Same gap for the opt-in branch: nothing asserts that allow_private_download_urls=True yields an unguarded session, so a future refactor could silently guard it and break LAN mirrors.

To expose this issue, add a test that constructs DownloadQueueService() with no requests_session and asserts session.get_adapter("https://x") is an SsrfGuardedAdapter, plus its inverse under allow_private_download_urls=True.


Low: invokeai/app/services/download/download_default.py:615-621 raises out of a requests response hook, leaking the streamed connection.

_reject_unsafe_redirect is dispatched from Session.send after adapter.send() returns but before the response is consumed or closed. Because the request is issued with stream=True (line 440), raising here means resp.close() is never called and the connection is not returned to the pool - it is only reclaimed on garbage collection. Each rejected redirect leaks one socket. Wrap the validation so the response is closed before the exception propagates.


Low: invokeai/app/util/ssrf.py:130-137 turns the route into an internal-DNS oracle, which partly negates the reason the up-front check exists.

check_address embeds the resolved address in the exception text, and invokeai/app/api/routers/download_queue.py:110-111 returns it verbatim as the 400 detail. A caller can distinguish "internal name, resolves to 10.x.y.z" (400 with the address) from "does not resolve" (accepted, job created) and read the private address straight out of the error body. The module docstring at invokeai/app/util/ssrf.py:20-22 says the up-front check exists partly to avoid "the socket layer being used as a port-existence oracle"; it substitutes an address-disclosure oracle instead. Return a generic "resolves to a non-public address" detail to the API and log the specific IP server-side.

Open Questions

  • invokeai/app/util/ssrf.py:167 passes parts.port positionally into _resolve; urlsplit(...).port raises ValueError for out-of-range ports, which is caught by line 168 and turns into continue (silent skip of validation for that spelling). requests would also reject such a URL, so I could not build a chain to a real fetch - flagging as an open question rather than a finding.
  • invokeai/app/util/ssrf.py:100-107 blocks every IPv4-mapped IPv6 address (::ffff:a.b.c.d falls in ::/8, so is_reserved). On a host where getaddrinfo returns v4-mapped results (AI_V4MAPPED), _check_socket would reject an entirely public destination. urllib3's allowed_gai_family() does not request AI_V4MAPPED, so I could not construct the failure, but it is the one plausible over-block in the classifier.

Verification

Reviewed PR #9492 (fix/download-queue-ssrf-and-dest-confinement, head c13e341dd9) against merge base 5e5d7fba81: 11 files, +863/-31.

Ran locally on Windows 10 / Python 3.11 / urllib3 2.7.0 / requests 2.34.2:

  • pytest tests/app/util/test_ssrf.py tests/app/routers/test_download_queue_router.py -q -> 2 failed, 88 passed (the two Windows failures above).
  • Confirmed the guard's urllib3 attachment points are real in 2.7.0: HTTPConnection._new_conn exists, HTTPSConnection does not override it (connect() does self.sock = sock = self._new_conn()), HTTPConnectionPool._new_conn instantiates self.ConnectionCls, and PoolManager.pool_classes_by_scheme is an instance attribute aliasing a module-level dict - so the adapter's assignment at invokeai/app/util/ssrf.py:216-219 correctly avoids mutating urllib3's global.
  • Live smoke of build_guarded_session() against huggingface.co/api/models/..., civitai.com and raw.githubusercontent.com: all 200, including the redirect-following case. No over-blocking of real download hosts.

Static checks performed:

  • i18n: nothing to check. The only frontend changes are regenerated invokeai/frontend/web/openapi.json and invokeai/frontend/web/src/services/api/schema.ts; no TSX, no new visible strings, no en.json delta needed. Schema and openapi are consistent with invokeai/app/services/config/config_default.py:242 and the new dest description, and docs/src/generated/settings.json matches.
  • Auth-change blast radius: grepped invokeai/frontend/web/src for any consumer of the download_queue operations - none outside schema.ts. The PR's claim that the web client is unaffected by the admin gating holds.
  • Coverage of the wider SSRF surface: model installs from arbitrary URLs land in invokeai/app/services/model_install/model_install_default.py:849-860, whose fallback path routes through multifile_download -> _do_download, so they now inherit the guard. That is a genuine positive side effect not claimed in the PR body.

Residual Risk

  • The _reject_unsafe_redirect hook and the socket guard overlap; I did not build a case where the hook catches something the socket guard would miss on a non-proxied session. If it exists only for the proxied/injected-session case, that should be stated, since as written it costs an extra getaddrinfo per redirect hop on the happy path (HF -> CDN).
  • Proxy fail-open (trust_env left on, proxy_manager_for unguarded) is documented and warned at invokeai/app/util/ssrf.py:245-262; I did not test behaviour behind a real egress proxy.
  • TOCTOU on _validate_dest: containment is decided by resolve() at line 62, and the write happens later on a worker thread. A symlink planted into the cache between the two would escape. The window requires local filesystem access, so I did not raise it as a finding.
  • Negative-path coverage is good on dest syntax, address classification and the auth matrix. It is thin on cleanup/lifecycle: nothing asserts that a job rejected by _validate_url or by the redirect hook leaves no .downloading residue in dest beyond the two assert not any(tmp_path.iterdir()) checks, and nothing covers a redirect chain that goes public -> public -> private.
  • Behaviour change not called out in the PR body: DownloadJob.dest is now echoed back to API clients as an absolute server path rather than the relative string the caller supplied. Admin-only, but it is a response-contract change for external consumers.

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge blockers:

  • invokeai/app/api/routers/download_queue.py:34-38,62: Admin can place an unverified payload in the exact cache path consumed by download_and_cache_model; non-admin inference then loads it via torch.load/torch.jit.load. Test: download a malicious archive to download_cache_path / slugify(source), then assert cache consumers reject it.

  • invokeai/app/api/routers/download_queue.py:106-111: Synchronous socket.getaddrinfo() runs inside async def, allowing a slow resolver to stall the entire event loop. Test: delay _resolve, issue POST concurrently with a trivial request, assert the trivial request completes before DNS returns.

Other findings/issues:

  • invokeai/app/api/sockets.py:610-612: Download events still broadcast signed source URLs and absolute filesystem paths to non-admin sockets, defeating the route's stated confidentiality goal. Test: emit DownloadStartedEvent in multiuser mode and assert it targets admin/authorized rooms.

  • invokeai/app/util/ssrf.py:265-267,291-305: With HTTP_PROXY/ALL_PROXY, proxy-side DNS can resolve attacker-controlled names to private hosts while the socket guard sees only the proxy. Test: route a hostname unresolved locally through a test proxy to loopback and assert the request is blocked. Expected docs: docs/src/content/docs/features/Multi-User Mode/admin-guide.mdx should state proxy-side enforcement is required.

  • invokeai/app/services/download/download_default.py:615-620: A rejected redirect raises from a stream=True response hook without closing the response, leaking connections on repeated attempts. Test: custom adapter returns a 302 with closable raw; assert raw.close() before UnsafeDownloadURLException propagates.

  • invokeai/app/util/ssrf.py:169-176: The API returns resolved private IPs in 400 responses, exposing internal DNS data. Test: mock _resolve as 10.0.0.7, call the route, and assert the response omits the address.

Suggestions:

  • Consider retaining invokeai/app/util/ssrf.py:130-166 and its regression cases; pure parsing fixes octal, decimal, and hex IPv4 forms independent of platform resolver behavior.

  • Consider adding a production-wiring test for DownloadQueueService() without requests_session; current service tests inject sessions and could miss removal of the guarded adapter.

@JPPhoto

JPPhoto commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

@lstein @Pfannkuchensack Fixed since c13e341:

  • Rejects legacy octal, decimal, and hexadecimal IPv4 literals portably.
  • Separates API downloads from the trusted model cache.
  • Moves path and URL validation off the async event loop.
  • Hides resolved private IPs from API errors while logging them server-side.
  • Rejects invalid ports and tests public IPv4-mapped IPv6 addresses.
  • Ignores ambient HTTP proxy settings for guarded sessions.
  • Routes download and model-install events to admin sockets only.
  • Closes responses when rejecting unsafe redirects.
  • Updates admin documentation and generated API schemas.
  • Adds regression tests for cache separation, proxy behavior, event routing, redirect cleanup, service wiring, and SSRF edge cases.

@JPPhoto
JPPhoto self-requested a review August 12, 2026 19:15
@Pfannkuchensack

Copy link
Copy Markdown
Member

Findings

High: invokeai/app/util/ssrf.py:103 blocks every IPv4-mapped IPv6 literal, and the new test added in this revision asserts the opposite. The suite fails deterministically on every CI leg.

The revision adds test_allows_public_ipv4_mapped_ipv6_literal, asserting validate_download_url("http://[::ffff:8.8.8.8]/x") succeeds. _is_blocked was not changed and rejects it on two independent grounds:

  1. CPython lists ::ffff:0:0/96 in the IPv6 private networks, and for IPv6 is_global is defined as not is_private, so not c.is_global is True for the wrapper.
  2. ::ffff:8.8.8.8 also falls in ::/8, so c.is_reserved is True.

Observed locally, and it is a pure literal-parse decision with no resolver or network involvement, so it reproduces on ubuntu, macOS and windows alike:

FAILED tests/app/util/test_ssrf.py::test_allows_public_ipv4_mapped_ipv6_literal
1 failed, 137 passed
ip = IPv6Address('::ffff:808:808'), host = '::ffff:8.8.8.8'
UnsafeDownloadURLException: Refusing to download from '::ffff:8.8.8.8' ...

This is not just a wrong assertion, it is an unresolved spec conflict inside the module. invokeai/app/util/ssrf.py:60-88 documents the rule as "Both the wrapper and what it wraps must be acceptable, which is why this yields rather than substitutes". Under that rule no v4-mapped literal can ever be accepted, so the new test is unsatisfiable without changing the classifier. And the same file's reject-list still asserts http://[::ffff:127.0.0.1]/x and http://[::ffff:10.0.0.1]/x are blocked - which they are only because all v4-mapped literals are blocked, not because the inner address is private. So the two tests encode contradictory intents about the same code path.

Decide one way: either keep the blanket block and delete the new test, or make _candidates substitute rather than yield for ipv4_mapped specifically - in which case the two reject cases start passing for the right reason instead of by accident.


Medium: invokeai/app/util/ssrf.py:274 sets session.trust_env = False, which also disables REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE and .netrc. Custom-CA deployments will fail every download with a TLS error.

The stated intent is to stop ambient proxies from moving destination resolution out of process. trust_env is not that narrow. Verified in the installed requests 2.34.2 Session.merge_environment_settings:

if self.trust_env:
    # Set environment's proxies.
    ...
    # Look for requests environment configuration
    # and be compatible with cURL.
    if verify is True or verify is None:
        verify = (
            os.environ.get("REQUESTS_CA_BUNDLE")
            or os.environ.get("CURL_CA_BUNDLE")
            or verify
        )

The CA-bundle lookup is inside the same guard. Chain:

  1. invokeai/app/services/download/download_default.py:84 builds the session via build_guarded_session() in the default configuration.
  2. invokeai/app/util/ssrf.py:274 clears trust_env.
  3. Every download then uses certifi's bundle regardless of REQUESTS_CA_BUNDLE.
  4. In any environment with a TLS-inspecting egress appliance, an internal CA, or a container image that exports REQUESTS_CA_BUNDLE (common), model installs fail with SSLError: CERTIFICATE_VERIFY_FAILED.

Session.prepare_request's netrc lookup is gated the same way, so .netrc credentials for private mirrors also stop working. Neither side effect is mentioned in the module docstring, in invokeai/app/services/config/config_default.py, or in the admin guide - the docs only mention proxies.

Fix surgically: keep trust_env = True and drop only the proxy contribution, e.g. a Session subclass whose merge_environment_settings calls super() and then blanks the returned proxies, or override rebuild_proxies. That preserves CA-bundle and netrc behaviour while closing the proxy path.

To expose this issue, add a test that sets REQUESTS_CA_BUNDLE to a temp file and asserts the guarded session's merge_environment_settings("https://x", None, None, None, None)["verify"] equals that path.


Medium: after invokeai/app/util/ssrf.py:274, an install behind a mandatory egress proxy has exactly one working configuration - allow_private_download_urls: true - which also turns off the private-address block entirely.

invokeai/app/services/download/download_default.py:79-85 has only two production branches: guarded session with trust_env off, or a bare requests.Session() with trust_env on. There is no "use my proxy, keep blocking private addresses" state. The admin guide's advice - "Use direct outbound access for guarded downloads, or explicitly enable private download URLs only when the proxy's destination policy is trusted" - is not actionable in networks where direct egress does not exist, and the fallback it offers is strictly weaker than the previous revision's position (trust_env on, proxy documented as a known gap, private addresses still blocked at the socket for every non-proxied host).

Note this also reverses the previous revision's own stated reasoning without a changelog entry: "Setting it to False would ... break installs behind a mandatory egress proxy, whose DNS may only resolve proxy-side." That trade-off was correct then and is unaddressed now. A separate download_proxy / trust_env_proxies setting would decouple the two concerns; as written, one boolean controls both proxy support and SSRF enforcement.


Low: invokeai/app/util/ssrf.py:126-160 does not strip a trailing dot, so http://127.0.0.1./x re-opens exactly the platform-dependent fail-open the new parser was added to close.

_parse_ipv4_literal splits on .; a trailing dot yields a 5th empty part, 1 <= len(parts) <= 4 fails, and the host falls through to _resolve. Verified on Windows with the PR checked out:

'127.0.0.1.'   parsed= None
'0177.0.0.1.'  parsed= None
getaddrinfo('127.0.0.1.') -> ERR [Errno 11001] getaddrinfo failed
validate_download_url('http://127.0.0.1./x')    -> ALLOWED
validate_download_url('http://0177.0.0.1./x')   -> ALLOWED

On glibc the same URLs are blocked, because getaddrinfo strips the trailing dot. So the up-front check's verdict is still resolver-dependent for a spelling that is a standard SSRF filter-bypass form, which is what test_rejects_legacy_ipv4_literal_without_dns set out to prevent ("must not depend on resolver normalization").

Exploitability is nil today - urllib3 calls the same getaddrinfo, so Windows cannot dial it either, and on Linux it is blocked - so this is a correctness and consistency defect rather than a hole. Strip one trailing dot before parsing.

To expose this issue, add "127.0.0.1." and "0177.0.0.1." to the test_rejects_legacy_ipv4_literal_without_dns parametrisation, which monkeypatches _resolve to fail and therefore fails on all platforms today.


Low: the new event-loop regression test is wall-clock-based and covers only half the fix.

test_download_validation_does_not_block_event_loop asserts time.monotonic() - start < 0.4 against a threading.Timer(0.5, ...). On a loaded GitHub runner a 0.4s budget for coroutine scheduling plus a to_thread hop is tight; a timing assertion is the wrong instrument when the property under test is "the loop was not blocked". Prefer asserting that a concurrently scheduled coroutine completes while the validator is still parked, with no absolute time bound.

Two coverage gaps in the same test:

  • It monkeypatches download_queue_router.validate_download_url only, so nothing asserts that _validate_dest at invokeai/app/api/routers/download_queue.py:110 is also off-loop, even though it performs mkdir and two resolve() calls.
  • It invokes download_queue_router.download(...) as a plain coroutine rather than through the ASGI stack, so converting the route back to def - which would also fix the blocking, differently - or dropping either to_thread inside a dependency would not be distinguished.

Open Questions

  • invokeai/app/api/routers/download_queue.py:116-117 collapses every UnsafeDownloadURLException into "Download URL resolves to a non-public address." validate_download_url also raises that type for a bad scheme, a missing host, and an invalid port. I confirmed all three are unreachable through this route today - AnyHttpUrl rejects non-http schemes, hostless URLs, and :65536 before the handler runs - so I could not build a chain to a misleading response. It remains a fragile coupling if the parameter type ever loosens.
  • Non-admin sockets no longer receive download_* or model_install_* events after invokeai/app/api/sockets.py:610-615. invokeai/frontend/web/src/services/events/setEventListeners.tsx:300-470 consumes them solely to maintain listModelInstalls and to toast unidentified models, both admin-only surfaces, so I could not identify a non-admin UI that degrades. Worth a maintainer confirming no non-admin view depends on ModelInstalls cache freshness.

Verification

Reviewed the updated head 601182ec78 (fix(download): close SSRF review gaps, on top of 2a178ce9d9 fix(ssrf): reject legacy IPv4 literals) against merge base 5e5d7fba81: 13 files, +1104/-45.

All six findings from the previous round were addressed, and I re-verified each against the code rather than the commit message:

  • Model-cache poisoning: dest is now anchored in invokeai/app/api/routers/download_queue.py:25-27's sibling directory, confirmed resolving to <root>/models/.download_cache.downloads, disjoint from download_cache_path. download_and_cache_model's slugify directories are no longer reachable from the route. Covered by a new negative test.
  • Event-loop blocking: both validators moved to asyncio.to_thread at invokeai/app/api/routers/download_queue.py:110-115; HTTPException still propagates correctly across the thread boundary.
  • Legacy IPv4 literals: _parse_ipv4_literal at invokeai/app/util/ssrf.py:126 now normalises octal/hex/integer/short forms in-process. Spot-checked 127.1, 0x7f.1, 2130706433, 0x7f000001, 010.0.0.1 -> 8.0.0.1 (octal, matching both glibc inet_aton and the WHATWG parser). The two Windows failures from the previous round are gone.
  • Socket broadcast: invokeai/app/api/sockets.py:610-615 routes DownloadEventBase and ModelEventBase to room="admin". Confirmed single-user sockets join "admin" at invokeai/app/api/sockets.py:219 and multiuser admins at :200, so the admin model-manager UI is unaffected in both modes.
  • Guarded-session wiring: now pinned by two tests asserting the adapter type on a service built with no injected session, and its inverse under the opt-in.
  • Connection leak and IP oracle: response.close() on the raising path, generic 400 detail with the address moved to a server-side warning; both pinned by tests.

Local run on Windows 10 / Python 3.11 / urllib3 2.7.0 / requests 2.34.2, across test_ssrf.py, test_download_queue_router.py, test_download_queue.py, test_invocation_event_socketio.py: 1 failed, 137 passed - the single failure is the High finding above.

Environment note affecting confidence: my first run showed three additional failures in test_multifile_download, test_multifile_download_error and test_multifile_cancel. Those are an artefact of my shell exporting HF_ENDPOINT=http://cm3588:8090, a LAN HuggingFace mirror; metadata.download_urls() builds URLs against that endpoint and the new guard correctly rejects 192.168.178.95. With HF_ENDPOINT unset all 34 download-queue tests pass, so this is not a CI failure. It is, however, an accidental live demonstration of the deployment class this change breaks by default: a LAN HF_ENDPOINT mirror stops working until allow_private_download_urls: true is set. The error message does name the setting, which is good, but the admin guide does not mention HF_ENDPOINT as an affected configuration.

Checked and found benign:

  • The new .downloads directory sits inside models_path, but invokeai/backend/model_manager/search.py:110 filters out any entry whose name starts with ., so _register_orphaned_models cannot descend into it even with scan_models_on_startup enabled. No exclusion-list entry is needed.

Residual Risk

  • Unrelated to the fix but now more visible: invokeai/app/services/model_install/model_install_default.py:1115-1121 compares an absolute resolved_path against the raw config values convert_cache_dir and download_cache_dir, which default to relative paths (models/.download_cache). is_relative_to is therefore always False and those two exclusions are dead code; only the absolute models_path / "core" entry works. Pre-existing, out of scope, but it means "just add the new directory to that list" would not have helped.
  • DownloadJob.dest is still echoed back to API clients as an absolute server path rather than the relative string supplied. Admin-only, but an unannounced response-contract change for external consumers.
  • I did not exercise the guarded session behind a real egress proxy or with a real custom CA bundle; findings 2 and 3 rest on the requests source and the two production branches, not on a live reproduction.
  • Negative-path coverage is now good on redirects (single and multi-hop), address classification, legacy literals, dest confinement and the auth matrix. Still uncovered: a redirect chain that goes public -> public -> private with a pooled connection reused on the first two hops, and any assertion that a job rejected mid-stream leaves no .downloading residue beyond the existing assert not any(tmp_path.iterdir()) checks.

@JPPhoto

JPPhoto commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

@Pfannkuchensack @lstein Fixed since 601182e:

  • Accepts public IPv4-mapped IPv6 addresses while rejecting mapped private addresses.
  • Rejects trailing-dot legacy IPv4 literals.
  • Preserves Requests CA-bundle and .netrc support while ignoring ambient proxies, including across redirects.
  • Adds explicit download_proxy configuration with production wiring and documentation.
  • Keeps download API destinations relative in responses.
  • Strengthens ASGI event-loop regression coverage.
  • Redacts proxy credentials from runtime configuration responses.
  • Adds regression tests for all changes.

JPPhoto and others added 4 commits August 12, 2026 15:48
`_api_job` hands the API a relativised `dest`, which pydantic serialises with
`str()`. On Windows that yields `models\sd15.safetensors` for a request that
submitted `models/sd15.safetensors`, so the value a client gets back depends on
the server OS -- and `test_download_accepts_relative_dest` fails on both
windows-cpu CI legs.

Serialise `dest` and `download_path` with `as_posix()`, matching what the
download events in `events_common.py` already do. The JSON schema is unchanged:
both fields are still plain strings, so no artifact regeneration is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lstein

lstein commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Adversarial review — head 5a281e093f

Reviewed the full diff (merged with current main), traced every new piece of state, attacked the guard empirically, and ran the tests. No blocking defects.

Verified holds

  • The socket-layer SSRF guard is not bypassable on the default (no-proxy) path. Exercised the full bypass catalog against a real build_guarded_session(); each was blocked at the socket and the up-front check: direct loopback, 169.254.169.254, [::1], legacy IPv4 literals (0x7f000001, 2130706433, 017700000001), localhost, v4-mapped ::ffff:127.0.0.1, CGNAT 100.64.1.1, NAT64 64:ff9b::7f00:1; example.com passes. sock.getpeername() fired on a real loopback connection, so DNS-rebinding and percent-decode desync are genuinely neutralized.
  • dest confinement. Anchoring under a separate {cache}.downloads dir (not download_cache_path) with a post-resolve() containment check structurally closes the earlier model-cache-poisoning gap. Absolute paths, .. segments (POSIX + Windows parsing), null bytes, drive-relative C:evil, and ..-suffixed URL segments are all rejected, with tests.
  • model_install lock change is correct. _download_cache enqueue now runs under _lock (model_install_default.py:1351) before submit (:1358), so the completion callback can't pop a missing entry; lock released before the download-queue call (no reentrancy); stop() and the wait-loop read it consistently. No deadlock path.
  • Event scoping is safe. Download/model events emit only to room="admin", and admins join that room on connect (sockets.py:200,218), so the model-manager UI still receives them.

139 targeted tests (ssrf, router, download-queue) pass at this HEAD.

Minor, non-blocking (tracked in a follow-up issue)

  1. _api_job relativizes dest but not download_path, so once a download starts the response leaks the absolute server path. Admin-only, cosmetic.
  2. A configured download_proxy bypasses the socket guard entirely — requests routes proxied connections through proxy_manager_for(), whose pools never get the guarded ConnectionCls (confirmed: loopback proxy returned 200, not blocked). Documented ("the proxy must enforce the public-address policy"), but worth an explicit note that enabling download_proxy disables the in-process guard.
  3. The router catches UnsafeDownloadURLException and always reports "resolves to a non-public address," even for the scheme/host/port variants (unreachable given source: AnyHttpUrl, so cosmetic).

The two items the PR flags as out of scope (alibabacloud._download_image unguarded fetch; custom_nodes.py unchecked git clone source) remain valid follow-ups, correctly excluded here.

@JPPhoto JPPhoto left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no merge blockers!

I found one more thing to address:

  • invokeai/app/services/download/download_default.py:79-80: An injected ordinary requests.Session bypasses the socket SSRF guard when private URLs are disabled; only DNS preflight remains. DNS rebinding can therefore reach loopback. Test: inject requests.Session(), make _resolve() return a public address for localhost, and request a local server.
    • Instead of implicitly trusting injected sessions, require an explicit unsafe-session opt-in or apply the guarded adapter whenever private URLs are disabled.

lstein and others added 4 commits August 12, 2026 22:08
Smoke-tested removal supersedes the confinement approach for the
router itself: the /api/v1/download_queue/* endpoints were a thin
remote-control surface over the internal DownloadQueueService with no
first-party consumer (the frontend only ever saw them as generated
schema types; model install drives the same service in-process), and
the enqueue route existed mostly as SSRF/dest-traversal attack
surface. Deleting it removes the risk rather than fencing it.

Everything protecting downloads the server performs on its own behalf
stays: the guarded requests session (socket-peer public-address
enforcement), per-connect URL validation and per-redirect-hop
re-checks in DownloadQueueService, the server-derived-filename check,
the allow_private_download_urls / download_proxy settings with proxy
credential redaction, and admin-room scoping of download/model events.

DownloadJob remains in the OpenAPI schema via
ModelInstallJob.download_parts. schema.ts and openapi.json regenerated
with the CI pipelines (typegen + prettier pass).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…is set

Surfaced by adversarial review of the router-removal commit: with both
settings configured, the private-opt-in branch built a bare Session()
that never received the explicit proxy — downloads went direct (or via
ambient *_PROXY, the opposite of the documented policy) with no hint.

The proxy is applied per request rather than on the session because
request-level proxies are the only kind that take precedence over
ambient *_PROXY variables in a plain Session (environment proxies are
folded into the request dict with setdefault, and that dict wins the
merge against session.proxies).

Regression test verified to fail against the previous branch logic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lstein lstein changed the title fix(download_queue): confine downloads to the cache dir and block non-public URLs fix(download): remove the download_queue REST API; block non-public URLs for server-side downloads Aug 13, 2026
@lstein

lstein commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Course change per maintainer decision, pushed as 3facee9 + 27e3fe1: the /api/v1/download_queue/* router is now removed entirely instead of confined — it had no first-party consumer (the web client never called it; model install drives the service in-process), so deletion beats fencing. All service-level SSRF protections stay, since model installs still download user-supplied URLs: guarded session with socket-peer checks, per-connect URL validation, per-redirect-hop re-checks, filename hardening, allow_private_download_urls/download_proxy, proxy credential redaction, and admin-room event scoping. The second commit fixes a config-coherence bug an adversarial review pass surfaced (download_proxy was silently ignored when allow_private_download_urls was enabled). PR title and body updated to match; smoke test of the full app with the router gone passed. App-side suite: 2828 passed.

@lstein
lstein merged commit 1f64b0b into invoke-ai:main Aug 13, 2026
17 checks passed
@lstein
lstein deleted the fix/download-queue-ssrf-and-dest-confinement branch August 13, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.0 api docs PRs that change docs frontend PRs that change frontend files python PRs that change python files python-tests PRs that change python tests services PRs that change app services

Projects

Status: 6.14.x Theme: USER EXPERIENCE

Development

Successfully merging this pull request may close these issues.

3 participants