chore: sync v7 with upstream and restore webv2 parity - #26
Merged
Conversation
…oke-ai#9295) The patcher decided whether a patch's layer keys were flattened (legacy underscore-joined) or real dotted module paths by inspecting only the first key. For FLUX.2 Klein diffusers LoRAs whose first converted layer is a dotless top-level module (e.g. `context_embedder`), the whole patch was misclassified as flattened, causing `assert "." not in layer_key` to fail on subsequent dotted keys and crashing LoRA application. Inspect all keys instead: a flattened key never contains a dot, so the patch is flattened only if no key contains one. Add a regression test covering the mixed dotless/dotted key ordering. Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Add new model type integration guide Comprehensive documentation covering all steps required to integrate a new model type into InvokeAI, including: - Backend: Model manager, configs, loaders, invocations, sampling - Frontend: Graph building, state management, parameter recall - Metadata, starter models, and optional features (ControlNet, LoRA, IP-Adapter) Uses FLUX.1, FLUX.2 Klein, SD3, SDXL, and Z-Image as reference implementations. * docs: improve new model integration guide - Move document to docs/contributing/ directory - Fix broken TOC links by replacing '&' with 'and' in headings - Add code example for text encoder config (section 2.4) - Add text encoder loader example (new section 3.3) - Expand text encoder invocation to show full conditioning flow (section 4.2) * docs: move new model integration guide into astro docs Move the New Model Type Integration guide from the legacy docs/contributing path into the astro docs tree and restyle it to match current documentation conventions (frontmatter, <Steps>, code block titles, :::tip checklists, <FileTree>). --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* feat(model-manager): support GGUF-quantized T5 text encoders Add loading support for single-file GGUF T5 encoders (e.g. city96/t5-v1_1-xxl-encoder-gguf, llama.cpp naming), mirroring the existing Qwen3 GGUF encoder path. - Add T5Encoder_GGUF_Config (single-file, detects enc.blk.* keys + GGML tensors) and register it in the AnyModelConfig union - Add T5EncoderGGUFModel loader: remaps llama.cpp T5 keys to transformers naming, infers T5Config from tensor shapes, dequantizes token/relative-attention-bias embeddings, ties embed_tokens to shared - Work around transformers T5DenseGatedActDense casting activations to the uint8 GGML weight dtype (int8 guard doesn't cover uint8), which would corrupt the feed-forward output - Reject T5 encoders in the Qwen3 GGUF/checkpoint configs so the two stay mutually exclusive (both carry token_embd.weight; the factory resolves multi-matches from a set, so this is not order-safe) Reuse the vendored T5-XXL tokenizer instead of downloading it: move it out of Anima into a neutral invokeai/backend/t5 module shared by Anima and the GGUF loader, and update the package-data path accordingly. * Chore Typegen + Openapi * Add T5 Recalling * Add 2 gguf T5 to the Starter Models * Chore Ruff * Chore Typegen * test(t5-gguf): add unit tests for GGUF T5 loader helpers + fail-loud FFN patch guard - Add unit coverage for the pure, high-risk parts of T5EncoderGGUFModel: key remapping (_convert_t5_gguf_to_transformers), config inference (_infer_t5_config_from_state_dict), and the wo-dtype workaround. - Make _make_feed_forward_gguf_safe raise if it patches no feed-forward modules, so a future transformers class rename fails loudly at load time instead of silently corrupting encoder output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(t5-gguf): drop dead _shape_of branch, document T5 v1.1 XXL assumptions - Inline tensor.shape in _infer_t5_config_from_state_dict: GGMLTensor.shape already returns the dequantized (logical) shape, so the _shape_of helper's fallback branch was unreachable. Remove the helper. - Document that config inference targets the T5 v1.1 XXL family and that the hardcoded architectural constants (rel-attention max distance, layer-norm epsilon, gated-gelu) are that family's defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ders (invoke-ai#9338) * feat(qwen3): bundle Qwen3 tokenizer for offline single-file/GGUF encoders" -m "Single-file (safetensors) and GGUF Qwen3 encoder checkpoints used by Anima (0.6B) and Z-Image (4B/8B) ship weights only — no tokenizer files. The loader pulled the tokenizer from Qwen/Qwen3-4B on HuggingFace, which fails offline / airgapped and whenever the HF cache is not persisted (e.g. Docker without a cache volume). Vendor the self-contained Qwen3 fast tokenizer (Apache-2.0, from Qwen/Qwen3-4B) in the package and load it locally, mirroring the bundled T5-XXL tokenizer (invoke-ai#9244). The Qwen3 BPE tokenizer is identical across the 0.6B/4B/8B variants, so a single copy serves every Qwen3 encoder. Removes the HuggingFace download path from both the checkpoint and GGUF loaders. * fix(qwen3): gzip bundled tokenizer to pass LFS check The vendored Qwen3 tokenizer.json is ~11MB, over the repo's 10MB lfs-warning threshold, failing the "lfs checks" CI job. Git LFS is unsuitable here since the file must ship inside the wheel for offline use. Vendor it gzip-compressed (~2MB) instead and decompress into a temp dir at load time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(qwen3): fix stale tokenizer-loader comments and method name The single-file/GGUF Qwen3 loaders now use the vendored tokenizer, but the call-site comments still described the removed HuggingFace download path and the method was still named _load_tokenizer_with_offline_fallback despite having no fallback. Rename to _load_bundled_tokenizer and update the comments to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(qwen3): restore chat_template in bundled tokenizer config The vendored tokenizer_config.json was missing the chat_template that Qwen/Qwen3-4B ships. The Z-Image text encoder formats prompts via tokenizer.apply_chat_template(), which raises ValueError: Cannot use chat template functions because tokenizer.chat_template is not set ... so GGUF/single-file Qwen3 encoders failed at encode time. The old HF-download path pulled the full config (template included), so this was a regression introduced by bundling. Restore the exact upstream Qwen3-4B chat_template and add a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ai#9349) * fix empty collector graph stalls * fix empty collector after grouped materialization * address empty collector review feedback --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
…for correct mask size (invoke-ai#9356) * Mirror width/height calculation for correct mask size * test(anima): cover regional-prompting token grid for odd latent dims Extract the image token grid computation into AnimaDenoiseInvocation._compute_img_token_grid so it can be unit tested, and add TestComputeImgTokenGrid. The key test cross-checks the grid against the transformer's real MiniTrainDIT._pad_to_patch_size so the mask sizing and the transformer's patchified grid cannot silently drift, plus a regression case for the 1080x1920 (8160 vs 8040) mask shape mismatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(anima): correct token-grid docstrings to ceil division The img_token grid uses ceiling division (ceil(latent / patch_size)) to mirror the transformer's padding, but two docstrings still described it as floor division (H // patch_size). Update them to match the actual math. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* fix(vae): decode on the model's intended compute device, not current residency (invoke-ai#9373) VAE decode inferred its device via get_effective_device(vae). Under partial loading with VRAM pressure, all VAE weights can be temporarily offloaded to RAM, so that returned CPU — placing the latents on CPU and, because the autocast layers follow the input device, running the entire decode on CPU. Expose LoadedModel.compute_device (the model's intended device, stable across partial-load residency and still honoring cpu_only) and use it in all 8 decode invocations. Adds a regression test reproducing the offloaded-weights case. * fix(text-encoder): encode on the intended compute device, not current residency Same class of bug as invoke-ai#9373, for text encoders: get_effective_device() returns CPU when partial loading has offloaded all weights, running the whole encode on CPU. Fully autocast-capable encoders (e.g. CLIP) are affected because repair_required_tensors_on_device() pins nothing to the compute device. Use LoadedModel.compute_device in compel, sd3, z_image, flux2_klein, cogview4 and qwen_image encoders, and thread it through HFEncoder for the FLUX encoder. Updates the affected tests to assert the intended-device behavior. * fix(tests): set vae_info.compute_device in qwen/z-image decode mocks Like the anima decode test, the qwen-image and z-image working-memory tests build vae_info as a bare MagicMock and drive invoke(). Since invoke-ai#9373 places latents on vae_info.compute_device, latents.to(device=...) raised TypeError — but both tests wrapped invoke() in `except Exception: pass`, so the failure was silently swallowed and the decode path never actually ran. Set compute_device to torch.device("cpu") so the tests exercise the real decode path instead of masking the error. * fix(text-encoder): anima encodes on the intended compute device, not current residency The anima text encoder inferred its device via text_encoder.device (HF PreTrainedModel residency), which returns CPU when partial loading has temporarily offloaded all Qwen3 weights to RAM — running the whole encode on the CPU. This is the same class of bug as invoke-ai#9373; the other text encoders and VAE decodes in this PR were already fixed, but the anima encoder was missed. Use LoadedModel.compute_device instead. This also corrects the device passed to TorchDevice.choose_anima_inference_dtype(). Adds a regression test covering both the offloaded-accelerator case and the cpu_only case. --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* feat(canvas): pressure-opacity brush support * feat(canvas): smooth pressure-opacity brush strokes * Tweak options labels * fix(canvas): preserve pressure-opacity self-overlaps on commit * fix(canvas): sort CanvasBrushToolModule imports * fix(canvas): keep pressure opacity rendering stable on reload --------- Co-authored-by: dunkeroni <dunkeroni@gmail.com>
* security: enforce multifile download path boundary * fix multifile download relative destinations --------- Co-authored-by: Ersa-tech <186737553+Ersa-tech@users.noreply.github.com> Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: JPPhoto <jpollack@jpollackphoto.com>
* Extend the draggability fix to all browsers. * Small comment fixes. * Fix bad import in useRefImageDnd --------- Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev> Co-authored-by: Josh Corbett <joshwcorbett@icloud.com>
The pytest matrix jobs already run 9-11 minutes per platform on main, and heavier PRs (e.g. invoke-ai#9163) push windows-cpu past the 15-minute cap, cancelling the run mid-pytest. Raise the cap to 30 minutes to give feature branches headroom. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
…roundtrip (invoke-ai#9162) * fix(workflows): preserve core_metadata recall fields across workflow roundtrip The `core_metadata` node is configured with pydantic `extra='allow'` and carries recall metadata (`loras`, `controlnets`, `ipAdapters`, `t2iAdapters`, plus model-specific extras like `z_image_seed_variance_*`, `dype_preset`, `ref_images`). When a generated image's graph was loaded into the workflow editor and saved, those values were being dropped: * The four `*MetadataField` collection types were not registered as StatefulFieldType, so their inputs fell through to `zStatelessFieldInputInstance`, whose `value` is `z.undefined().catch(undefined)` — silently coercing the array away. * Extra keys not declared in the OpenAPI schema were dropped earlier still, in `graphToWorkflow`, because `template.inputs[key]` was undefined and the field was skipped with a warning. Result: the regenerated image had no recall metadata and "Recall all parameters" reported no LoRA / no variance settings — even though the backend executed the workflow correctly (`lora_selector` and the `z_image_seed_variance_enhancer` node were intact via their edges). This change: * Registers `LoRAMetadataField`, `ControlNetMetadataField`, `IPAdapterMetadataField`, `T2IAdapterMetadataField` as stateful field types with passthrough zod values. * Adds a synthetic `MetadataExtraField` so undeclared keys on `extra='allow'` nodes round-trip through the workflow editor. * `graphToWorkflow` synthesizes an extra-field template for keys not in the node template, scoped to nodes that accept extras. * `buildNodesGraph` forwards extra values verbatim when running the workflow. * `fieldValidators` and `InputFieldGate` no longer treat undeclared inputs on extra-accepting nodes as errors / unexpected fields. Adds regression tests covering the LoRA roundtrip, the extras roundtrip, and the parseSchema template type for `core_metadata.loras`. Fixes invoke-ai#9151 * fix(workflow): scope MetadataExtraField catch-all to extra-accepting nodes The MetadataExtraField input instance (value: z.any()) was added to the global field-instance union, so it matched any malformed input value on any node during workflow parsing (inputs are parsed without their template). Stale connection-only values were preserved instead of coerced to undefined and could leak into the backend graph via buildNodesGraph. Scope extras to node types that accept them (pydantic extra='allow', e.g. core_metadata): - Remove MetadataExtraField from the global stateful input-instance and value unions; add a dedicated zFieldInputInstanceWithExtras union - Parse node inputs in zInvocationNodeData based on node type: extra- accepting nodes use the with-extras union, all others the strict union - Removing z.any() from zStatefulFieldValue also restores proper typing for StatefulFieldValue/FieldValue/FieldInputInstance (were collapsing to any) - validateWorkflow: don't warn about undeclared extras on extra-accepting nodes (fixes spurious "loaded with warnings" on image recall) - nodeUpdate: preserve extras across template version bumps Add tests for the scoping boundary and the validateWorkflow behavior. * fix(workflow): scope metadata pass-through instances to extra-accepting nodes Follow-up to the MetadataExtraField scoping. The concrete metadata pass-through instances (LoRA/ControlNet/IPAdapter/T2IAdapter MetadataField) use zMetadataPassthroughValue (array(record(string, any)) | nullish), which is greedy enough to match a stale array-of-objects value on any field. They were still in the global zStatefulFieldInputInstance union, which zInvocationNodeData uses for every non-extra node - and inputs are parsed without their template. So a stale value on a connection-only input (e.g. img_resize.metadata) could survive parsing as a metadata instance and leak into the backend graph via buildNodesGraph. Move the four metadata pass-through instances out of the global union into the scoped zFieldInputInstanceWithExtras union (used only for nodes that accept extras, i.e. core_metadata). The FieldInputInstance type is now derived from a dedicated union that includes the metadata shapes (so builders type-check) but excludes the MetadataExtraField z.any() catch-all. Add a buildNodesGraph regression test (stale array-of-records on a connection-only input is coerced away and not serialized) plus parse-level coverage; update loraMetadataRoundtrip tests to use the scoped union. --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* feat(model): add Wan 2.2 image generation support (Phases 0-2)
Foundation + TI2V-5B MVP + A14B dual-expert MoE for Wan 2.2 image
generation. Wan was trained on video but is competitive with leading
open-source image models when run at num_frames=1; this commit wires
that path into InvokeAI.
Phase 0 — Foundation:
- BaseModelType.Wan + WanVariantType {T2V_A14B, TI2V_5B}
- SubModelType.Transformer2 for the dual-expert MoE
- MainModelDefaultSettings per variant
- step_callback Wan branch (16-channel preview; 48-channel TI2V-5B
falls back to slicing first 16 channels until proper factors land)
- Frontend enums + node colour
Phase 1 — TI2V-5B Diffusers MVP:
- Main_Diffusers_Wan_Config probe (variant from transformer_2/ +
vae/config.json::z_dim, with filename heuristic fallback)
- WanDiffusersModel loader (subclasses GenericDiffusersLoader)
- WanT5EncoderField, WanTransformerField (with dual-expert slots),
WanConditioningField, WanConditioningInfo
- New invocations: wan_model_loader, wan_text_encoder, wan_denoise,
wan_image_to_latents, wan_latents_to_image
- FlowMatchEulerDiscreteScheduler integration with on-disk config load
- RectifiedFlowInpaintExtension reused for inpaint
- 5D <-> 4D shape juggling: latents stay 4D in InvokeAI's pipeline,
re-add T=1 only inside the transformer call / VAE encode-decode
Phase 2 — A14B dual-expert MoE:
- Probe reads boundary_ratio from model_index.json
- Loader emits both transformer (high-noise) and transformer_low_noise
(low-noise expert at transformer_2/) for A14B
- _ExpertSwapper in wan_denoise drives GPU residency between experts:
high-noise for t >= boundary_ratio * num_train_timesteps, low-noise
below. Only one expert locked at a time so the cache can evict the
other - relies on existing CachedModelWithPartialLoad to handle
oversized models on lower-VRAM GPUs.
- guidance_scale_low_noise field for separate low-noise CFG override
Tests:
- 24 passing tests covering probe variant detection, default settings,
noise sampling, end-to-end denoise on a synthetic transformer (CPU),
dual-expert boundary swap, CFG branch
- 1 heavy-test placeholder gated by INVOKEAI_HEAVY_TESTS=1 for the
real-weights smoke test
Phase 3+ deferred: standalone VAE/encoder configs, GGUF, LoRA,
ControlNet, ref image, inpaint UI, frontend wiring, starter models.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(model): Wan 2.2 Phase 3 + tokenizer-load fix
Phase 3 adds standalone VAE and UMT5-XXL encoder configs so users can run
GGUF-quantized Wan transformers (Phase 4) without installing the full
~30 GB Diffusers pipeline.
VAE configs:
- VAE_Checkpoint_Wan_Config + VAE_Diffusers_Wan_Config (16-channel A14B
vs 48-channel TI2V-5B, distinguished by decoder.conv_in z_dim).
- 16-channel files share the AutoencoderKLWan architecture with Qwen
Image; disambiguated via filename heuristic ("wan" in name -> Wan,
otherwise -> Qwen Image). Mirror exclusion in QwenImage's probe.
- VAELoader gets a Wan branch that builds AutoencoderKLWan(z_dim=...)
via init_empty_weights, mirroring the QwenImage single-file pattern.
- Existing standard VAE probe excludes both QwenImage- and Wan-style
state dicts.
UMT5-XXL encoder:
- New ModelType.WanT5Encoder + ModelFormat.WanT5Encoder.
- WanT5Encoder_WanT5Encoder_Config probes the diffusers folder layout
(text_encoder/config.json with model_type=umt5, or flat layout with
config.json at root). Refuses full Wan pipelines.
- WanT5EncoderLoader handles both layouts and loads UMT5EncoderModel +
AutoTokenizer.
Component-source plumbing:
- WanModelLoaderInvocation now exposes wan_t5_encoder_model and
component_source pickers (mirrors QwenImage pattern). Resolution
order: standalone > main (if Diffusers) > component_source. Required
when the main model is a single-file format in Phase 4.
Bug fix in wan_text_encoder:
- Tokenizer was loading via AutoTokenizer.from_pretrained(<root>)
directly, which fails for nested layouts where files live in
<root>/tokenizer/. Now routed through the model cache so the
registered loaders handle layout differences correctly.
Frontend:
- New type guards (isWanVAEModelConfig, isWanT5EncoderModelConfig,
isWanMainModelConfig, isWanDiffusersMainModelConfig) and hooks/
selectors (useWanVAEModels, useWanT5EncoderModels,
useWanDiffusersModels). New zSubModelType / zModelType / zModelFormat
enum entries for transformer_2 and wan_t5_encoder.
Tests:
- 16 new tests covering z_dim detection, VAE checkpoint/diffusers
probes, the bidirectional Qwen-vs-Wan filename deferral, and the
UMT5 encoder probe (nested + flat + T5 + full-pipeline rejection).
- Total Wan test count: 41 passing, 1 heavy-test placeholder skipped.
- Full config test suite (63 tests) still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): unbreak frontend lint after Wan additions
Five issues turned up running `make frontend-lint`:
1. wan_denoise.py used `from __future__ import annotations`, which made
the `invoke()` return annotation a string ('LatentsOutput'). The
InvocationRegistry's `get_output_annotation()` returns the raw
annotation, so OpenAPI generation crashed with
`'str' object has no attribute '__name__'`. Removed the future-import
and added `Any` to the typing imports.
2. ModelRecordChanges.variant didn't list WanVariantType, so the
generated schema's install/update endpoints rejected `t2v_a14b` and
`ti2v_5b`. Added it.
3. Regenerated frontend/web/src/services/api/schema.ts from the live
backend so it now includes BaseModelType.wan, ModelType.wan_t5_encoder,
SubModelType.transformer_2, ModelFormat.wan_t5_encoder, the Wan
variants, all Wan invocation types and their conditioning/transformer
field types.
4. modelManagerV2/models.ts: added `wan_t5_encoder` to the category map,
`wan` to the base color/long-name/short-name maps, the two Wan
variants to the variant-name map, and `wan_t5_encoder` to the
format-name map.
5. ModelManagerPanel/ModelFormatBadge.tsx: added `wan_t5_encoder` to
FORMAT_NAME_MAP and FORMAT_COLOR_MAP.
`make frontend-lint` now passes cleanly (tsc, dpdm, eslint, prettier).
All 41 Wan Python tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chore(wan): drop unused FE exports flagged by knip
These were forward-compatibility wiring for Phase 9 (the FE graph
builder) that has no consumers yet; knip rightly flagged them. Removed
or de-exported. They'll come back when the graph builder lands and
needs them.
- common.ts: zWanVariantType drops `export` (still used internally by
zAnyModelVariant).
- types.ts: drop isWanMainModelConfig, isWanDiffusersMainModelConfig,
isWanVAEModelConfig (no callers). The remaining
isWanT5EncoderModelConfig is used by models.ts. WanT5EncoderModelConfig
type drops `export` (still used as the type guard's narrowing target).
- modelsByType.ts: drop the six unused useWan*/selectWan* hooks +
selectors and their type-guard imports.
`make frontend-lint` (tsc + dpdm + eslint + prettier + knip) now green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(wan): use *-Diffusers HF repo names in plan
The Wan-AI org publishes two flavours of each release:
* Wan-AI/Wan2.2-{TI2V-5B,T2V-A14B,I2V-A14B} ← upstream native
* Wan-AI/Wan2.2-{TI2V-5B,T2V-A14B,I2V-A14B}-Diffusers ← convertible
The native release has _class_name=WanModel in config.json and ships
weights flat at the repo root with no transformer/, vae/, text_encoder/
subdirs. It is not loadable by Diffusers' WanPipeline.from_pretrained.
Update plan doc to reference the -Diffusers repos throughout (probe
notes, starter-model entries) so the plumbing path matches what the
Diffusers loader actually expects.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): accept 0 as 'unset' sentinel for guidance_scale_low_noise
The frontend renders Optional[float] inputs with default 0 in the
numeric input rather than passing null/unset. Combined with ge=1.0,
this caused every wan_denoise invocation to fail Pydantic validation
with "Input should be greater than or equal to 1" until the user
manually entered a value (or knew to leave the field disconnected).
The validation error was rejected before invocation logging, so it
never showed up in the server log either - making the failure hard to
diagnose.
Relaxing the constraint to ge=0.0 and treating values below 1.0 as the
"fall back to primary Guidance Scale" sentinel. The user's natural FE
default (0) now works as expected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): correct preview dimensions and colors for TI2V-5B
Two bugs in the Wan branch of the diffusion step callback:
1. Wrong dimensions. The reported preview size hardcoded `* 8` for the
spatial downscale ratio, but TI2V-5B's Wan2.2-VAE uses 16x. A
1024x1024 target was being announced to the FE as 512x512.
2. Wrong colors. The previous fallback for 48-channel TI2V-5B latents
sliced the first 16 channels and applied the standard 16-channel
Wan-VAE projection. Those channel layouts are unrelated, so the
projection produced meaningless colors.
Adding the proper Wan2.2-VAE 48-channel RGB projection matrix (and
bias) from ComfyUI's Wan22 latent format, and selecting the right
matrix + spatial-scale by latent channel count: 16 → A14B (Wan VAE,
8x), 48 → TI2V-5B (Wan2.2-VAE, 16x).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): honor model's _class_name when building scheduler
TI2V-5B's scheduler_config.json declares _class_name=UniPCMultistepScheduler
with flow_shift=5.0. The previous code hardcoded
FlowMatchEulerDiscreteScheduler.from_pretrained(...), which silently
constructed a default-config FlowMatch instead of the UniPC the model
expects. The mismatched noise schedule manifests as soft / under-denoised
faces and global graininess in the final images.
Now: read scheduler_config.json, look up the named class on the diffusers
module, and instantiate that class via from_pretrained. UniPC and
FlowMatch share the same step()/set_timesteps()/sigmas/num_train_timesteps
interfaces, so the denoise loop works transparently for either.
A14B continues to use FlowMatchEulerDiscreteScheduler when its scheduler
config says so (its reference is FlowMatchEuler with shift=8.0). Falls
back to FlowMatchEulerDiscreteScheduler defaults when no on-disk config
is available.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): match diffusers WanPipeline tokenizer length and latent dtype
Two divergences from the Diffusers reference that were hurting image
quality (soft / grainy / distorted faces at default settings):
1. Tokenizer max_sequence_length was 226 in wan_text_encoder, but the
model was trained with 512-token sequences. The upstream native
config.json has text_len: 512, and Diffusers' WanPipeline.__call__
default is 512 (overriding _get_t5_prompt_embeds's stale 226 default).
Wan's cross-attention sees padded zeros past the prompt's actual
length but expects to be looking at a 512-position context window.
2. Latents were stored in bf16 throughout the denoise loop. Diffusers'
WanPipeline.prepare_latents explicitly uses dtype=torch.float32 and
only casts to the transformer's dtype right at the forward call:
latent_model_input = latents.to(transformer_dtype)
Storing in bf16 between steps accumulates ~40 steps of bf16
quantization on the scheduler's small per-step deltas. Now
latent_dtype = torch.float32 throughout, with a per-step cast for
the transformer forward pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chore(wan): add diffusers reference comparison script
scripts/wan_diffusers_reference.py runs a Diffusers-format Wan 2.2
checkpoint directly via WanPipeline.from_pretrained, with the same
arguments InvokeAI's wan_denoise uses. Use to A/B against InvokeAI
output when image quality is questionable.
Defaults to enable_model_cpu_offload so the script fits on 16 GB cards
where the full pipeline (transformer + UMT5-XXL + VAE) would otherwise
OOM. --offload {model,sequential,none} controls the strategy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(model): Wan 2.2 Phase 4 - GGUF transformer support
Adds single-file GGUF support for Wan 2.2 transformers, the path that
makes A14B usable on consumer GPUs (~7 GB/expert at Q4_K_M instead of
~28 GB at bf16).
Probe (configs/main.py):
- New helpers: _has_wan_keys (Wan vs Qwen/FLUX/Z-Image fingerprint via
condition_embedder.text_embedder.linear_1 + patch_embedding);
_detect_wan_gguf_variant (16ch -> A14B, 48ch -> TI2V-5B from
patch_embedding.weight.shape[1]); _detect_wan_gguf_expert (filename
heuristic for high_noise / low_noise / none).
- Main_GGUF_Wan_Config(base=Wan, format=GGUFQuantized, variant, expert).
Tolerates the ComfyUI 'model.diffusion_model.' / 'diffusion_model.'
prefixes via _has_wan_keys' multi-prefix scan.
- Registered in factory.py.
Loader (model_loaders/wan.py):
- WanGGUFCheckpointModel mirrors the QwenImage GGUF pattern:
gguf_sd_loader -> strip ComfyUI prefix -> auto-detect arch from state
dict shapes (num_layers, inner_dim, ffn_dim, text_dim, in_channels,
num_heads = inner_dim/128) -> init_empty_weights +
load_state_dict(strict=False, assign=True).
Loader invocation (wan_model_loader.py):
- New 'Transformer (Low Noise)' picker: optional second GGUF for the
A14B dual-expert MoE. Auto-swaps if the user wired the experts in
the wrong order. Warns when an A14B GGUF is loaded without a paired
low-noise expert (single-expert run, degraded quality).
- GGUF mains require either a standalone VAE+encoder or a Diffusers
Component Source (which can also supply boundary_ratio).
- Diffusers main path unchanged (still pulls both experts from
transformer/ + transformer_2/).
Tests (tests/.../test_wan_gguf_config.py):
- 14 tests across key fingerprint, variant detection, expert filename
heuristic, and the full probe (A14B high/low, TI2V-5B, GGUF rejection,
unrecognised state-dict rejection, explicit override).
Total Wan tests: 55 passing (no regressions). FE lint clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): support QuantStack-style GGUFs and standalone Diffusers VAE
The city96 Wan 2.2 GGUF repos have been removed from Hugging Face,
leaving QuantStack as the surviving distributor. QuantStack ships the
native upstream Wan key layout (text_embedding.0/2, self_attn/cross_attn,
ffn.0/2, head.head, head.modulation, ...) rather than the diffusers
naming city96 used; biases are stored as F16 rather than BF16; and the
standalone Wan VAE installs as a flat AutoencoderKLWan folder which the
generic loader rejects. Three fixes:
1. Probe now recognises both diffusers and native key layouts via a new
_is_native_wan_layout helper; _has_wan_keys accepts either text-proj
fingerprint.
2. GGUF loader converts native -> diffusers keys (mirroring diffusers'
convert_wan_transformer_to_diffusers) and unwraps non-quantized
GGMLTensors to plain tensors at compute_dtype. The unwrap is needed
because conv3d isn't in GGMLTensor's dispatch table, so the F16
patch_embedding bias would otherwise hit conv3d against bf16 latents.
3. VAELoader gains a VAE_Diffusers_Wan_Config branch that loads
AutoencoderKLWan directly; the generic path can't handle a flat
single-class folder when a submodel_type is provided.
Adds 12 tests covering the native layout (probe + converter + unwrap).
Verified end-to-end against Wan2.2-T2V-A14B-Q4_K_M from QuantStack:
1095 tensors round-trip key-for-key against WanTransformer3DModel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(model): Wan 2.2 Phase 5 - LoRA support
Probe + config (LoRA_LyCORIS_Wan_Config):
- Detects Wan LoRAs in three layouts: diffusers PEFT, native upstream PEFT
(ComfyUI), and Kohya (both naming variants).
- Anti-pattern guards prevent collisions with Anima (Cosmos DiT q_proj
convention), QwenImage (transformer_blocks), Flux (double/single blocks),
and Z-Image (diffusion_model.layers).
- Optional ``expert: "high" | "low" | None`` field; auto-detected from
filename (high_noise / low_noise / hyphenated / concatenated variants).
Key conversion (wan_lora_conversion_utils):
- Native upstream keys (self_attn/cross_attn, ffn.0/2) -> diffusers
(attn1/attn2, ffn.net.0.proj / ffn.net.2).
- Strips ``transformer.``, ``diffusion_model.``, ``base_model.model.transformer.``
prefixes from PEFT-style keys.
- Kohya layer names mapped through an explicit longest-match table.
- Output paths use diffusers naming so the LayerPatcher can resolve them
against WanTransformer3DModel parameter paths.
Loader integration:
- Adds BaseModelType.Wan branch to LoRALoader._load_model.
Invocation nodes (wan_lora_loader.py):
- WanLoRALoaderInvocation: single LoRA with auto/both/high/low target field.
- WanLoRACollectionLoader: list of LoRAs, auto-routed by each LoRA's
recorded expert tag.
- Output WanLoRALoaderOutput carries the WanTransformerField with updated
``loras`` / ``loras_low_noise`` lists.
Denoise integration:
- _ExpertSwapper now manages both the model_on_device context and the
LayerPatcher.apply_smart_model_patches context per expert. LoRA patches
are entered after device load and exited before device release, with
fresh iterators per swap.
- GGUF (quantized) experts request sidecar patching so GGMLTensor weights
aren't touched directly.
- Low-noise expert falls back to the primary loras list when
``loras_low_noise`` is empty (matches WanTransformerField semantics).
Tests: 81 new tests covering probe accept/reject across formats, anti-pattern
guards on competing architectures, converter round-trips for all three
layouts, invocation target resolution + routing + duplicate guards, and the
_ExpertSwapper lifecycle (lora context opens/closes in the right order
around the device swap, quantized flag forwards, no-LoRA path skips the
patch context, re-entering the same label is a no-op).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): probe Wan LoRA before Anima in the config union
Native-PEFT Wan LoRAs (lightx2v's Lightning, most ComfyUI-trained Wan
LoRAs) carry keys like ``diffusion_model.blocks.X.cross_attn.k.lora_A.weight``.
Anima's probe matches on the bare ``cross_attn``/``self_attn`` substring —
it does not require the Anima-specific ``_proj`` suffix nor any of the
``mlp``/``adaln_modulation`` Cosmos DiT markers — so these Wan LoRAs were
classified as ``BaseModelType.Anima`` because Anima happened to run first.
Reorder the LyCORIS section of ``AnyModelConfig`` so Wan probes first.
Wan's probe is strictly more restrictive (it rejects Anima's ``_proj``
attention suffix via the anti-pattern guard added in the previous commit),
so Anima LoRAs are still correctly classified after this reorder.
Existing users with mis-tagged installs need to delete the affected LoRA
records and reinstall.
Adds two regression tests: a union-ordering assertion, and a sanity check
that demonstrates Anima's probe *would* match Wan native keys if asked
directly — pinning the constraint that motivates the ordering.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chore(i18n): add Wan2.2 T5 Encoder model-manager label
The frontend source already references ``modelManager.wanT5Encoder``;
the locale key was added with a casing typo (``want5Encoder``). Fix
the key so the Wan T5 Encoder model type renders its display name
correctly in the model manager UI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(model): Wan 2.2 Phase 7 - reference-image (I2V) conditioning
Re-implementation after the first attempt — which used CLIP-vision
conditioning — was reverted. Wan 2.2 I2V-A14B does NOT use a CLIP-vision
encoder (the Diffusers repo ships ``image_encoder: [null, null]`` in
``model_index.json``); instead it conditions on a reference image by
VAE-encoding it and concatenating the resulting latents (plus a
first-frame mask) to the noise latents along the channel dim. The I2V
transformer therefore has ``in_channels=36`` (16 noise + 16 ref-image
latents + 4 mask) vs ``in_channels=16`` for T2V.
Taxonomy:
- Re-adds ``WanVariantType.I2V_A14B``.
Probes:
- Diffusers: ``_detect_wan_variant`` reads ``transformer/config.json::in_channels``;
36 → I2V_A14B, 16 → T2V_A14B (both share the dual-expert layout).
- GGUF: ``_detect_wan_gguf_variant`` recognises ``in_channels=36`` from the
patch_embedding tensor shape and emits I2V_A14B.
Backend extension (``backend/wan/extensions/wan_ref_image_extension.py``):
- ``preprocess_reference_image`` resizes + normalises to a 5D pixel tensor.
- ``encode_reference_image_to_condition`` VAE-encodes the image and stacks
a 4-channel first-frame mask on top, producing the
``[1, 20, 1, H/8, W/8]`` condition tensor the denoise loop consumes.
- Mirrors diffusers ``WanImageToVideoPipeline.prepare_latents`` with
``num_frames=1`` and ``expand_timesteps=False``.
Invocation node (``wan_ref_image_encoder.py``):
- "Reference Image - Wan 2.2": image + VAE + width/height pickers.
- Output ``WanRefImageConditioningField`` carries the condition tensor
name plus the dimensions used (so the denoise step can validate dim
parity).
Denoise integration:
- ``WanDenoiseInvocation`` gains an optional ``ref_image`` field.
- Variant gate: rejects ref_image on T2V_A14B and TI2V-5B with a clear
error before doing any work.
- Dimension gate: rejects ref-image width/height mismatch vs denoise.
- At every transformer call, concatenates the 20-channel condition
tensor to the 16-channel noise latents along the channel dim before
passing to the transformer (giving the 36-channel input I2V expects).
Tests: 14 new across the probe, the extension, and the denoise loop.
The synthetic ``_ZeroTransformer`` test stand-in now mirrors the real
I2V transformer's ``in_channels=36, out_channels=16`` asymmetry by
slicing its zero output back to 16 channels when the input is 36-wide.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): derive GGUF out_channels from proj_out shape (I2V support)
The GGUF loader was setting ``out_channels = in_channels`` which is wrong for
Wan 2.2 I2V-A14B: that variant has ``in_channels=36`` (16 noise + 16 ref-image
latents + 4 first-frame mask, concatenated by the denoise loop) but
``out_channels=16`` since the transformer only predicts the noise component
back. Loading an I2V GGUF would build a transformer with the wrong proj_out
shape and crash:
RuntimeError: Error(s) in loading state_dict for WanTransformer3DModel:
size mismatch for proj_out.weight: copying a param with shape
torch.Size([64, 5120]) from checkpoint, the shape in current model is
torch.Size([144, 5120]).
(144 = 36 * 4, 64 = 16 * 4 — patch_size=(1, 2, 2) → prod=4)
Read out_channels directly from the ``proj_out.weight`` shape in the state
dict. This is correct for all three Wan 2.2 variants without needing to know
the variant in advance.
Also tighten the num_layers fallback: T2V_A14B and I2V_A14B share 40 layers;
only TI2V-5B has 30. The fallback is rarely hit in practice (the per-block
count comes from the state dict scan), but the previous code would have
defaulted I2V_A14B to 30 layers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(model): make Anima LoRA probe mutually exclusive with Wan
InvokeAI's ``Config_Base.CONFIG_CLASSES`` is a Python ``set``, so iteration
order during model probing is non-deterministic across process restarts.
First-match-wins ordering in ``AnyModelConfig`` is documentation only — it
has no effect on which config is iterated first.
Anima's previous probe accepted any state dict containing the substring
``cross_attn`` or ``self_attn``, which collides with Wan's native LoRA key
layout (``diffusion_model.blocks.X.cross_attn.q.lora_down.weight``). Both
probes accepted Wan native LoRAs (including lightx2v's Lightning T2V and I2V
distillations), and the ``matches.sort_key`` tiebreaker only disambiguates
by ModelType, not within LoRA configs. So which config "won" depended on
dict hash order — sometimes Wan, sometimes Anima.
The previous mitigation reordered the AnyModelConfig union to put Wan
before Anima. That worked by luck and was inherently fragile.
Tighten Anima's probe to require Cosmos-DiT-exclusive subcomponents:
``mlp``, ``adaln_modulation``, or ``_proj``-suffixed attention names
(``q_proj``/``k_proj``/``v_proj``/``output_proj``) — none of which appear
in any Wan LoRA. Wan native uses bare ``.q``/``.k``/``.v``/``.o`` on
``self_attn``/``cross_attn``, and ``ffn.N``/``ffn.net.N`` instead of ``mlp``.
The new strict detectors live alongside the original loose ones so the
Anima conversion utility (which runs after probing) still works.
Regression tests in ``test_wan_lora_probe_independence.py`` cover:
- I2V Lightning V1 (the bug-triggering LoRA), T2V Lightning V2, Wan Kohya
and Wan diffusers PEFT layouts — Wan probe accepts, Anima probe rejects.
- Anima PEFT and Kohya layouts — Anima accepts, Wan rejects.
- A meta-test that runs every LoRA config in CONFIG_CLASSES against the
Lightning state dicts and asserts exactly one accepts — this catches
ANY future probe collision, not just Wan vs Anima.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): defer expert model loading in _ExpertSwapper to avoid cache thrash
The swapper used to take pre-loaded ``LoadedModel`` handles at construction:
high_info = context.models.load(self.transformer.transformer)
low_info = context.models.load(self.transformer.transformer_low_noise)
swapper = _ExpertSwapper(high_info=high_info, low_info=low_info, ...)
With dual ~9 GB A14B GGUF experts plus the ~10 GB UMT5-XXL encoder competing
for the same RAM cache, the LRU policy frequently dropped one expert by the
time the denoise loop swapped into it. The model manager then emitted
[MODEL CACHE] Locking model cache entry ... but it has already been
dropped from the RAM cache. This is a sign that the model loading
order is non-optimal in the invocation code (See ... #7513).
and reloaded the weights from disk (~1.2s extra per swap).
Refactor the swapper to take the ``ModelIdentifierField`` plus the
``InvocationContext`` and call ``context.models.load(model_id)`` lazily
inside ``get()``. Each swap obtains a fresh handle, the LRU window is
small, and the warning goes away.
Config metadata (used to compute ``is_quantized``) is read upfront via
``context.models.get_config()`` — that's metadata, not weights, so it
doesn't put pressure on the cache.
Tests: existing swapper lifecycle tests refactored to use a fake context
whose ``models.load`` is logged. A new ``test_lazy_load_per_swap_not_upfront``
pins the regression — it asserts ``models.load`` is NOT called at swapper
construction, only at first get() per expert.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(wan): add Phase 8 inpaint regression tests
The denoise_mask wiring + RectifiedFlowInpaintExtension integration in
wan_denoise.py was put in place during Phase 2/3 alongside the rest of
the denoise loop. Phase 8 of the plan was about ensuring this path
worked and is locked in by tests.
Three new tests under TestWanDenoiseInpaint:
1. test_preserved_region_matches_init_exactly: builds a half/half mask
(left = preserve, right = regenerate in user-side convention), runs
full denoise with the synthetic zero-output transformer, and asserts
the preserved half of the final latents equals the init exactly while
the regenerated half does not. Pins the mask-inversion + per-step
merge behavior.
2. test_inpaint_requires_init_latents: a mask without init latents must
raise a clear ValueError — the merge has nothing to weld back to.
3. test_no_mask_path_is_unchanged: regression that adding the inpaint
extension didn't perturb the non-inpaint codepath (with init latents
+ denoising_start=0.5 but no mask, the loop just runs img2img).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(frontend): add I2V_A14B to Wan variant zod enum + manager label
Phase 7 added the I2V_A14B backend variant. The frontend's zod enum
(features/nodes/types/common.ts:zWanVariantType) and the model manager's
variant-label map (features/modelManagerV2/models.ts) were still on the
two-variant list, so:
- ModelIdentifierField inputs with ui_model_variant filters on Wan
couldn't list I2V models.
- The model manager UI showed a raw 'i2v_a14b' string instead of the
human label.
Phase 9 (full linear-view wiring — type guards, hooks, params slice,
graph builder, tab UI) is in progress on a follow-up commit; this lands
the two small enum fixes first so the I2V probe / install paths work
correctly end-to-end with the existing FE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(wan): Phase 9 piece #1 - linear-view T2V txt2img graph builder
Adds the minimum frontend wiring needed to generate Wan 2.2 images from
the linear view:
- buildWanGraph.ts (new): text-to-image graph (model_loader →
text_encoder × 2 → denoise → l2i). Diffusers main model only —
transformer, VAE and UMT5 encoder all resolve from the same repo, so
no Wan-specific params slice fields are required yet. CFG-skip
branch when guidance_scale ≤ 1.0.
- useEnqueueGenerate / useEnqueueCanvas dispatchers: route
base === 'wan' to buildWanGraph.
- graph/types.ts: add wan_l2i / wan_i2l / wan_denoise / wan_model_loader
to the relevant node-type unions.
- addTextToImage / addImageToImage: include wan_denoise / wan_l2i so
width/height are wired correctly and the txt2img helper accepts the
Wan l2i node.
- isMainModelWithoutUnet: include wan_model_loader (Wan has no UNet,
same as the other modern bases).
- metadata.py: add wan_txt2img / wan_img2img / wan_inpaint to the
generation_mode enum (img2img / inpaint pieces land next).
- schema.ts: regenerated to pick up the metadata enum + new
Wan invocations.
Pieces left in Phase 9: params slice (standalone VAE / T5 / GGUF
low-noise / LoRA / ref-image fields + selectors), img2img + I2V + inpaint
branches in the graph builder, and Wan-specific UI components.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(wan): Phase 9 piece #2 - GGUF support and CFG-Low control in linear view
Adds the three Wan-specific params + UI controls that gate GGUF workflows
plus a separate low-noise CFG slider for A14B users.
Params slice:
- wanTransformerLowNoise (the second-expert GGUF for A14B)
- wanComponentSource (Diffusers Wan model providing VAE + UMT5-XXL
when the main is a GGUF)
- wanGuidanceScaleLowNoise (optional separate CFG for the low-noise
expert; null = fall back to the primary CFG)
Plus a `selectIsWan` selector for accordion gating.
UI components:
- ParamWanModelSelects.tsx (Advanced accordion): two model pickers —
Transformer (Low Noise) filtered to Wan GGUF mains, and VAE/Encoder
Source filtered to Wan Diffusers mains. Mirrors the
ParamQwenImageComponentSourceSelect structure.
- ParamWanGuidanceScaleLowNoise.tsx (Generation accordion): slider +
number input with an "auto" indicator when cleared. Default 3.5
matches the diffusers reference 4.0 / 3.0 split.
Wiring:
- Generation accordion: ParamWanGuidanceScaleLowNoise shown when base
is wan, scheduler excluded for wan (same pattern as Anima/Qwen).
- Advanced accordion: ParamWanModelSelects shown when base is wan, and
Wan excluded from the SD-family VAE/CFG-rescale blocks.
- buildWanGraph.ts: forwards the three new params to the model loader
and denoise nodes (transformer_low_noise_model, component_source,
guidance_scale_low_noise) and adds them to the graph metadata.
Hooks/types:
- useWanDiffusersModels + useWanGGUFModels in modelsByType.ts.
- isWanDiffusersMainModelConfig + isWanGGUFMainModelConfig type guards.
- Three new locale strings (wanComponentSource, wanTransformerLowNoise,
wanGuidanceScaleLowNoise[Auto]).
GGUF workflow now works end-to-end in the linear view: pick a Wan GGUF
main, set Transformer (Low Noise) to the paired second-expert GGUF, set
VAE/Encoder Source to any Diffusers Wan repo (TI2V-5B is convenient at
~12 GB) — generate produces an image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): UX polish on the Wan linear-view controls
Bundles four small fixes applied during a usability review of the Wan
linear-view section (piece #2):
1. **Filter Main vs Transformer (Low Noise) dropdowns by expert tag.**
The Wan GGUF probe records each file's ``expert`` field
(``"high"`` / ``"low"`` / ``"none"``) via filename heuristic.
- ``MainModelPicker``: hides ``expert === 'low'`` Wan GGUFs so users
can't accidentally wire a low-noise expert as the primary main.
- Transformer (Low Noise) picker (``useWanGGUFLowNoiseModels``):
shows ``expert === 'low'`` Wan GGUFs only.
Diffusers Wan mains and TI2V-5B aren't affected — they don't carry
the ``expert`` field on their config schema. The backend's auto-swap
safety net stays in place.
2. **Match the primary CFG slider's range.** The Wan low-noise CFG
slider was constrained to 1–10 while the primary CFG ranges 1–20.
With the diffusers reference 4/3 split, the low-noise slider thumb
sat noticeably further right than the primary — visually misleading.
Both sliders now share the 1–20 range with marks at [1, 10, 20].
3. **Label fits the form column.** "CFG (Low Noise)" → "CFG (Low)" so
the slider fits cleanly next to its label instead of overlapping.
4. **Indicator state for the low-noise CFG slider.** Replaced the inline
"(auto)" / "(same as cfg)" text — which kept overlapping the slider
regardless of how short the label got — with an X-only reset button
that's only visible when the user has set an explicit value. Absence
of the X conveys auto/fallback state without any text overhang.
5. **Friendlier Transformer (Low Noise) placeholder.** "Second-expert
GGUF for A14B (pair with the high-noise main)" → "Add for full
detail" — concise nudge for users who haven't paired the second
expert yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(wan): Phase 9 piece #3 - linear-view img2img branch
Adds Wan 2.2 image-to-image to the linear view, mirroring the Qwen Image
pattern. The mode switches on the canvas state — pure-prompt runs go
through addTextToImage as before; canvas runs with an init image go
through addImageToImage which wires a fresh wan_i2l (Image to Latents -
Wan 2.2) node between the init image and the denoise's `latents` input,
honoring the existing denoise_start slider.
buildWanGraph:
- Drops the txt2img-only guard, branches on generationMode.
- img2img: spins up a wan_i2l node and hands it to addImageToImage
alongside the existing denoise / l2i / modelLoader (as vaeSource).
- inpaint / outpaint still fail loudly — pieces #4-#6.
graphBuilderUtils.getDenoisingStartAndEnd:
- Adds 'wan' to the simple-linear case (denoising_start = 1 -
denoisingStrength). Note: Wan's flow-matching schedule is "sticky"
on the init compared to SDXL — users will likely need denoisingStrength
≥ 0.7 to see substantial change, matching the user-found 0.15-0.3
denoising_start sweet spot from earlier img2img testing. We may
revisit this with an exponent rescale (like FLUX uses) if the
response curve feels off.
addImageToImage:
- Adds 'wan_i2l' to the i2l-node-type union so the Wan i2l can be
threaded through the shared helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): add wan_denoise to addImageToImage/addInpaint/addOutpaint type checks
Three sibling graph-helper utilities had the same modern-base list as
addTextToImage did, and the buildWanGraph img2img branch tripped one of
them at canvas-Generate time:
error [generation]: Failed to build graph
{name: 'Error', message: 'Wrong assertion encountered'}
The else-branch in each helper assumes 'denoise_latents' (the SD1.5/SDXL
legacy path) and asserts that — failing for any modern base not listed
above the branch. addTextToImage was already updated in Phase 9 piece #1;
this catches the parallel cases that the img2img/inpaint/outpaint flows
go through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(wan): Phase 9 piece #4 - linear-view inpaint and outpaint branches
Wires Wan 2.2 inpaint and outpaint through the existing addInpaint /
addOutpaint helpers. The backend's RectifiedFlowInpaintExtension was
plumbed into wan_denoise.py back in Phase 8 (commit ab54617173); this
just connects the FE.
buildWanGraph:
- generationMode === 'inpaint' → spin up a wan_i2l, call addInpaint
with denoise + l2i + modelLoader (used as both vaeSource and
modelLoader since the Wan model loader carries the VAE).
- generationMode === 'outpaint' → parallel branch with addOutpaint.
addInpaint:
- i2l-node-type union now includes 'wan_i2l' (the addImageToImage and
addOutpaint type unions already do — different union shapes).
metadata.py:
- generation_mode literal adds "wan_outpaint" alongside the existing
wan_txt2img / wan_img2img / wan_inpaint entries.
isMainModelWithoutUnet already includes wan_model_loader (Phase 9 piece
create_gradient_mask when Wan is the main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(wan): Phase 9 piece #5 - linear-view I2V branch (raster as reference image)
Wan 2.2 I2V-A14B models condition on a reference image whose VAE-encoded
latents are concatenated to the noise along the channel dim each step
(in_channels=36 on the I2V transformer). In the linear view this maps
cleanly onto the existing canvas raster layer: pick an I2V model, drag
an image to raster, generate.
buildWanGraph:
- Fetch the modelConfig early so the variant gate (i2v_a14b vs the
rest) can drive the branch shape instead of being a post-hoc check.
- I2V + txt2img: fail loudly ("Switch to the canvas tab and drag an
image to the raster layer"). I2V models won't produce useful output
without a reference, and the backend would crash trying to
concatenate a missing condition tensor.
- I2V + img2img: pull the raster image via the canvas compositor,
wire it through a wan_ref_image_encoder (which VAE-encodes it and
builds the 4-mask + 16-latent condition tensor backend-side), then
feed the result into denoise.ref_image. Denoise runs from fresh
noise (denoising_start=0, no init_latents) — the ref image is
cross-attention/concat conditioning, not a noise-trajectory anchor.
- I2V + inpaint/outpaint: fail clearly. Combining ref-image
conditioning with a denoise mask is conceptually possible but the
backend interaction hasn't been validated end-to-end.
metadata.py:
- Adds "wan_i2v" to the generation_mode literal so the metadata field
on I2V renders correctly.
T2V flows (txt2img / img2img / inpaint / outpaint) are unchanged for
non-I2V Wan variants (T2V-A14B and TI2V-5B).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): enforce multiple-of-16 dimensions to match transformer patch grid
Wan 2.2's transformer has ``patch_size=(1, 2, 2)``: it patch-embeds with
stride 2 then un-patches by 2. Combined with the VAE's 8x spatial scale,
canvas H/W must be a multiple of ``8 * 2 = 16`` — not just 8 — for the
patch round-trip to land exactly. Otherwise the latents and noise
prediction disagree by one in the spatial dim and the scheduler step
fails:
RuntimeError: The size of tensor a (147) must match the size of
tensor b (146) at non-singleton dimension 3
(here latent_w=147 → patch_w=73 → un-patched_w=146 ≠ 147)
This was silent for T2V at 1024x1024 (already a multiple of 16) but
fired for I2V at non-multiple-of-16 canvas sizes.
Fixes:
- ``optimalDimension.getGridSize``: Wan moves from the default 8 case to
the multiple-of-16 case (alongside flux / sd-3 / qwen-image / z-image
which have the same patch arithmetic). The canvas bbox UI now snaps
Wan dimensions to multiples of 16.
- ``wan_denoise.py`` and ``wan_ref_image_encoder.py``: bump width/height
``multiple_of`` from 8 to 16. Defense-in-depth — workflow-editor
users won't be able to send a non-16-aligned dim either.
Existing backend tests (23 passing) still hold — 1024 is divisible by 16
so the test fixtures didn't exercise the off-by-one path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): show negative prompt box in Wan linear-view
Wan was missing from SUPPORTS_NEGATIVE_PROMPT_BASE_MODELS, so the
linear-view negative-prompt input was hidden even though the Wan denoise
node already wires negative conditioning when CFG > 1
(buildWanGraph.ts:67-75). Adds 'wan' to the list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(wan): Phase 9 piece #6 - Wan LoRA collection in linear view
Adds Wan LoRA wiring to buildWanGraph, mirroring the Qwen Image pattern.
The shared LoRASelect / LoRAList UI in the linear view already filters
LoRAs by the selected main model's base, so Wan LoRAs surface
automatically when a Wan main is picked — no UI changes needed.
addWanLoRAs (new):
- Filters state.loras.loras to enabled Wan LoRAs.
- For each LoRA: spawns a ``lora_selector`` node and threads it
through a single ``collect`` collector.
- Routes the collector into a ``wan_lora_collection_loader`` which
sits between modelLoader and denoise — modelLoader.transformer →
loader, then loader.transformer → denoise (rerouting the original
modelLoader → denoise edge).
- Emits per-LoRA metadata so PNG metadata + workflow restore work.
The dual-expert routing (high-noise vs low-noise vs untagged) is
handled entirely on the backend by ``WanLoRACollectionLoader`` based on
each LoRA's recorded ``expert`` tag (set by the probe from the filename
heuristic in piece #5 of Phase 5). The FE just hands over the bag of
LoRAs; no per-list FE plumbing needed.
buildWanGraph:
- Calls addWanLoRAs(state, g, denoise, modelLoader) after the base
transformer edge is in place. The helper is a no-op when no Wan
LoRAs are enabled, so it's safe to call unconditionally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(wan): detect LoRA variant and filter by main model
Wan 2.2 A14B (inner_dim=5120) and TI2V-5B (inner_dim=3072) LoRAs are not
interchangeable — applying one against the wrong main model crashes the
layer patcher with a tensor-shape error (e.g. A14B Lightning on TI2V-5B
mains produced ``shape '[3072, 3072]' is invalid for input of size 26214400``).
Probe Wan LoRAs' inner-dim at install time and record the family on a new
``variant`` field (``a14b`` / ``5b`` / null). The LoRA picker in the linear
view hides incompatible variants when the user selects a main, and the
graph builder filters any still-enabled mismatches at submit time with a
warning. Untagged LoRAs (probe couldn't identify) pass through so they
aren't silently hidden.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(wan): ref-image panel, GGUF readiness, and auto-default sources
Wan 2.2 I2V now uses the global Reference Images panel (same UX as Qwen
Image Edit and FLUX.2 Klein) instead of pulling the conditioning image
from a canvas raster layer. Adds:
- WanReferenceImageConfig zod type + isWanReferenceImageConfig guard;
integrated into the ref-image discriminated union, settings panel,
layer hooks, and validators.
- 'wan' added to SUPPORTS_REF_IMAGES_BASE_MODELS, but the panel only
shows for the i2v_a14b variant (T2V and TI2V-5B don't consume ref
images, so the panel is hidden for them).
- buildWanGraph I2V branch reads the first enabled wan_reference_image
from refImagesSlice; the canvas-raster-as-ref path is removed. I2V
now only supports txt2img mode (canvas img2img/inpaint/outpaint
assert with a clear message).
GGUF Wan readiness check: GGUF mains carry only the transformer, so the
loader needs a Diffusers Component Source (or standalone VAE + UMT5-XXL
encoder) to resolve the VAE and text encoder. Without one, enqueue is
now blocked with a clear reason. The low-noise A14B partner expert
remains optional (loader falls back to the high-noise expert when it's
missing).
Adds standalone Wan VAE and Wan T5 Encoder selectors to the Advanced
accordion (Qwen pattern). Wires them as vae_model / wan_t5_encoder_model
on the wan_model_loader node — backend priority is standalone > diffusers
main > component source.
Auto-default on Wan selection (so GGUF users don't have to fiddle with
Advanced): when the new main is a Wan GGUF, fill the Component Source,
standalone VAE, and standalone T5 encoder with first available matches
if not already set. Component Source is matched by variant family
(A14B GGUF prefers an A14B Diffusers; TI2V-5B prefers a TI2V-5B
Diffusers) since the two families use different VAE channel counts
(16 vs 48); within A14B, T2V and I2V share VAE/encoder so they're
interchangeable as a source. Runs on every Wan selection (including
Diffusers -> GGUF switches), only fills empty slots.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(wan): add Wan 2.2 starter models and bundle
Wan 2.2 starter pack (selected when the user picks the Wan 2.2 bundle)
brings up the minimal-cost path to running A14B T2V end-to-end:
- Standalone UMT5-XXL encoder and A14B VAE (so GGUF mains don't need
a full Diffusers download for their VAE/encoder sources).
- T2V A14B Q4_K_M and Q8_0 GGUF expert pairs (high + low noise).
- T2V Lightning V1.1 Seko rank-64 LoRA pair (4-step inference).
Additional Wan 2.2 starter models browseable from the model manager:
- Full Diffusers T2V A14B, I2V A14B, and TI2V-5B.
- I2V A14B Q4_K_M and Q8_0 GGUF expert pairs + Lightning V1 LoRA pair.
- TI2V-5B Q4_K_M and Q8_0 GGUFs + the 48-channel TI2V-5B VAE.
Each "high noise" GGUF lists its low-noise partner plus the shared VAE
and UMT5-XXL encoder as dependencies, so installing one of them pulls
in everything the loader needs. QuantStack's HighNoise/LowNoise file
naming and lightx2v's high_noise_model/low_noise_model.safetensors are
both picked up by the existing filename heuristic in the GGUF probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(wan): add Wan 2.2 hardware requirements
Adds Wan 2.2 A14B (T2V/I2V) and TI2V-5B rows to the hardware
requirements table with rough VRAM/RAM guidance per quantization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(wan): recall low-noise transformer, component source, and standalone VAE/T5
Wan-specific metadata fields embedded by the graph builder
(wan_transformer_low_noise, wan_component_source, wan_vae_model,
wan_t5_encoder_model, wan_guidance_scale_low_noise) had no recall
handlers in features/metadata/parsing.tsx, so recalling an image's
parameters would leave these fields empty. Adds a handler for each
that dispatches the matching paramsSlice action and renders a row in
the metadata viewer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(wan): add default Wan 2.2 T2V and I2V workflows
Ships two default workflows in the library, tagged so they appear in
"Browse Workflows" under the wan2.2 / text to image / image to image
tags:
- Text to Image - Wan 2.2: full T2V/TI2V-5B graph (model loader,
positive + negative encoders, denoise, l2i). Exposes the five
model slots, prompts, steps, dual CFG, and dimensions.
- Image to Image - Wan 2.2: I2V A14B graph that adds a
wan_ref_image_encoder. Exposes the reference image input plus
the standard fields.
Both follow default-workflow rules: IDs prefixed with default_,
meta.category = "default", and no references to user-installed
resources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(videos): Phase 1 - backend video storage, records, REST API
Adds a parallel video pipeline alongside the existing image pipeline so the
gallery can host MP4 alongside PNGs. Implements:
- New service modules (parallel to image equivalents):
video_records/ record store + sqlite impl
video_files/ disk file store (mp4 + first-frame webp thumb)
videos/ orchestrating service
board_video_records/ board <-> video association
- migration_32 creates `videos` and `board_videos` tables
- /api/v1/videos/ router: upload, list, get DTO, /full (with HTTP Range
so HTML5 <video> seek/scrub works), /thumbnail, /metadata, star/unstar,
delete, batch delete, board add/remove
- LocalUrlService.get_video_url and SimpleNameService.create_video_name
- imageio[ffmpeg] dep for video encode (used in later phases)
- Wires all four new services into InvocationServices, dependencies.py,
api_app.py, and three test fixtures
Verified end-to-end against an in-memory db + tmp output dir: upload,
probe, save (file + thumbnail + record), DTO build, list, delete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(videos): Phase 2 - polymorphic gallery list endpoint
Adds /api/v1/gallery/items/ and /api/v1/gallery/items/names returning a
unified time-sorted stream of images + videos so the frontend can render
them interleaved with a single virtualized query.
- gallery_common: GalleryItem discriminated union (kind + name + shared
fields + nullable video duration/fps), GalleryItemRef, names result
- gallery_default: SqliteGalleryService implements UNION ALL across the
images and videos tables, applying identical filters (origin/category/
is_intermediate/board_id/search) to each half; pagination via outer
ORDER BY + LIMIT/OFFSET; counts are summed across the two halves
- URLs are resolved at row -> DTO conversion time so each item routes to
the correct /api/v1/images or /api/v1/videos endpoint
- Wired into InvocationServices, dependencies.py, api_app.py, and the
three test fixtures
Existing /api/v1/images endpoints are unchanged so any non-gallery
consumers (queue, recall, metadata workflows) continue to work as-is.
Verified e2e: 2 images + 2 videos inserted in alternating order, both
list_items and list_item_names return the correct interleaved order;
category filter narrows to a single kind; starring an item bumps it to
the top when starred_first=True.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(videos): Phase 3 - frontend RTK endpoints + MP4 upload routing
Adds the typed API surface and upload integration so videos can be
uploaded through the same gallery upload button that handles images.
Schema: re-ran pnpm typegen against the running backend to pick up
VideoDTO, VideoRecordChanges, GalleryItem, GalleryItemKind,
GalleryItemRef, GalleryItemNamesResult and the two new paginated
result types.
RTK Query (services/api/endpoints/videos.ts) - parallel to images.ts:
listVideos, getVideoDTO, getVideoMetadata, getVideoNames, uploadVideo,
deleteVideo / deleteVideos, changeVideoIsIntermediate, starVideos /
unstarVideos, addVideoToBoard / removeVideoFromBoard. Imperative helpers
(getVideoDTO, getVideoDTOSafe, uploadVideo, uploadVideos) and the
useVideoDTO convenience hook ride alongside, mirroring the image side.
Tag types and invalidation: added Video / VideoList / VideoMetadata /
VideoNameList / BoardVideosTotal / GalleryItemList / GalleryItemNameList
to the api root. Board-affecting mutations now invalidate the polymorphic
gallery list/name caches so videos and images stay coherent once the
gallery wiring lands in Phase 4. Added a sibling
getTagsToInvalidateForVideoMutation helper.
Upload UX: useImageUploadButton.tsx's dropzone now accepts video/mp4,
video/webm, video/quicktime alongside the existing image MIMEs. The
drop handler splits files into image/video sets and routes each through
its own mutation; a new onUploadVideo callback parallels the existing
onUpload. Existing image-only callers pass through unchanged.
Polymorphic gallery query endpoints + the useGalleryItemDTO hook will
land with Phase 4 where they have actual consumers; the schema types
they'll need are already in place under @knipignore tags.
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass; live curl against the running dev server
uploads an MP4 and serves both the webp thumbnail and the MP4 with
a working HTTP Range response (206 + Content-Range).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(videos): Phase 4 - mixed gallery grid with video play badge
Videos now appear in the same gallery grid as images, interleaved by
created_at. Video thumbnails get a centered play-button badge so they
read as videos at a glance; everything else (selection, virtualization,
search, paged/virtual gallery views, keyboard nav) is unchanged.
Approach: selection state stays `string[]` of names. The kind is
recovered from the filename extension (.mp4 = video, anything else =
image), which is reliable because the backend's SimpleNameService
always emits `<uuid>.png` for images and `<uuid>.mp4` for videos. This
sidesteps a 32-file cross-cut from changing the selection shape to a
discriminated union, and selection is persist-denylisted so no
migration is needed.
Frontend:
- new isVideoName helper in features/gallery/store/types
- new endpoints/gallery.ts (deferred from Phase 3): useGetGalleryItemNamesQuery
- new ImageGrid/GalleryItemPlayBadge: centered triangular badge over thumbnail
- new ImageGrid/GalleryItemVideoStarIconButton: video-typed star toggle
- new ImageGrid/GalleryVideoItem: counterpart to GalleryImage; reuses
galleryItemContainerSX, GalleryItemSizeBadge (width/height-only stand-in),
selection handling (single/shift/ctrl/cmd); alt-click falls through to a
normal select since comparison is image-only
- use-gallery-image-names now calls the polymorphic gallery names endpoint
and exposes a mixed flat name list (existing callers - paged grid, search,
navigation hotkeys - get the same shape)
- useRangeBasedImageFetching partitions visible names by extension; images
bulk-fetch via the existing getImageDTOsByNames mutation, videos dispatch
individual getVideoDTO queries (no batch endpoint yet)
- GalleryImageGrid's ImageAtPosition dispatches on isVideoName to render
GalleryImage or GalleryVideoItem; star hotkey dispatches to the right
star/unstar mutation based on kind
- pruned the now-unused useGetImageNamesQuery / isImageName exports
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass; live curl of /api/v1/gallery/items returns
57 polymorphic items with video duration populated and image duration
null, /api/v1/gallery/items/names returns matching {kind, name} refs.
The useGalleryItemDTO hook is intentionally deferred to Phase 5 where
the polymorphic viewer is its first real consumer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(videos): Phase 5 - inline video player in the image viewer
Selecting a video now renders a polymorphic preview inside the existing
viewer panel: thumbnail with a centered play button by default; clicking
play swaps in an HTML5 <video controls autoplay>. Switching to a
different item drops the video element back to idle (auto-pauses) and
selecting an image again returns to the normal image preview.
New components (features/gallery/components/ImageViewer/):
- VideoPlayButtonOverlay: large centered play button with hover/shadow,
used over the thumbnail in the idle state.
- CurrentVideoPreview: idle/playing state machine. Resets on
video_name change. The <video> src points at /api/v1/videos/i/.../full
which supports HTTP Range, so seek/scrub work natively in the browser.
New hook:
- common/hooks/useGalleryItemDTO: polymorphic DTO resolver that
dispatches between useImageDTO and useVideoDTO based on filename
extension (isVideoName). Centralizes the kind-dispatch the viewer
and toolbar both need.
Wiring:
- ImageViewer dispatches on galleryItem.kind to render CurrentImagePreview
or CurrentVideoPreview. The compare-image DnD drop target is hidden when
a video is selected (comparison is image-only).
- ImageViewerToolbar hides the image-specific action row
(CurrentImageButtons - load workflow, recall metadata, edit, etc.) and
the metadata viewer toggle when a video is selected. The general-purpose
ToggleProgressButton stays.
Out of scope (per the plan): video deletion from the viewer (use gallery
hover icons), video-specific metadata viewer, comparison-mode support
for videos.
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(videos): accept MP4 (and other video) drops on the fullscreen dropzone
The gallery-wide drag-and-drop target lives in FullscreenDropzone, not
in useImageUploadButton (which only powers the upload button). It had
its own hardcoded image-only zod allowlist that rejected MP4 files
with "File type / extension is not supported".
- Broaden the zod refines to accept video/mp4, video/webm,
video/quicktime, video/x-matroska and the matching extensions
- Add isVideoFile helper, split dropped files into image/video sets,
and route each set through its own uploader (uploadImages /
uploadVideos). Both update their respective RTK caches and
invalidate the polymorphic gallery list/names.
- Skip the canvas-paste fast-path for single-video drops — the canvas
doesn't host videos as layers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(videos): right-click context menu on video items
Adds a three-item context menu (delete, change board, download) on
right-click / long-press of any gallery video item. Mirrors the image
context menu's singleton-portal architecture so re-renders stay cheap.
New files:
- features/gallery/contexts/VideoDTOContext: small React context that
scopes the active video DTO to the menu items (parallels
ImageDTOContext).
- features/gallery/components/ContextMenu/MenuItems/
ContextMenuItemDeleteVideo: window.confirm + deleteVideo mutation.
Videos can't be referenced from canvas/nodes/refs, so the image
modal's usage analysis is unnecessary; a one-step confirm matches
the "minimal" scope.
ContextMenuItemDownloadVideo: reuses the existing useDownloadItem
hook against videoDTO.video_url / video_name.
ContextMenuItemChangeBoardVideo: dispatches videosToChangeSelected
and opens the (now polymorphic) ChangeBoardModal.
- features/gallery/components/ContextMenu/VideoContextMenu: singleton
pattern lifted from ImageContextMenu — registers gallery video
elements via a Map; right-click looks up the target node and opens
the menu at the cursor.
Extended files:
- features/changeBoardModal/store/slice: added video_names alongside
image_names plus a videosToChangeSelected action. The two arrays are
mutually exclusive — setting one clears the other.
- features/changeBoardModal/components/ChangeBoardModal: now dispatches
the matching video board mutations (add/removeVideoToBoard, plural
endpoints don't exist yet so videos move one at a time — the menu
acts on a single selection so this is a one-iteration loop).
- features/gallery/components/ImageGrid/GalleryVideoItem: registers
itself with useVideoContextMenu.
- app/components/GlobalModalIsolator: mounts the singleton.
Verified: pnpm lint (knip, dpdm, eslint, prettier, tsc) all green;
pnpm test 1103/1103 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(wan): Phase 6 - Wan 2.2 T2V/I2V workflow nodes
Adds two new invocation nodes that produce MP4 videos from a Wan 2.2
A14B transformer + VAE, plus the supporting plumbing.
New invocations:
- WanVideoDenoise (wan_video_denoise) — multi-frame counterpart to
WanDenoise. Same per-step logic (CFG, MoE expert swap at the
boundary timestep, LoRA patching, scheduler dispatch) — reuses
_ExpertSwapper, _resolve_variant, and the scheduler/LoRA helpers
from wan_denoise. Difference: the noise tensor has a real temporal
dim built from num_frames, and the I2V condition is built across
all latent frames (frame 0 conditioned, rest zero). Defaults match
the Wan 2.2 reference: 832x480 / 81 frames / 40 steps / CFG 5.0
(high) + 4.0 (low). Inpaint / img2img are out of scope for this
first cut. TI2V-5B is rejected; T2V/I2V A14B only.
- WanLatentsToVideo (wan_l2v) — VAE-decodes 5D latents to RGB frames
via AutoencoderKLWan (T_pixel = (T_lat - 1) * 4 + 1), then encodes
an MP4 with imageio[ffmpeg] (libx264, yuv420p for browser
compatibility). The temp file is moved into outputs/videos/ via
context.videos.save().
Backend shared pieces:
- make_noise gains num_latent_frames (default 1, backward compatible).
- Added num_latent_frames_for(num_frames, scale=4) helper.
- New encode_reference_image_to_video_condition mirrors diffusers'
WanImageToVideoPipeline.prepare_latents with last_image=None and
expand_timesteps=False: pads the reference image with zero
pixel-frames, VAE-encodes the full pseudo-video, normalises, and
builds the 4-channel temporal-rearranged first-frame mask. Verified
numerically: 21 latent frames for num_frames=81, first latent
frame's 4 mask channels = 1, rest = 0.
- The existing single-frame encoder is left untouched.
Schema / context:
- New VideoField primitive (parallel to ImageField) and VideoOutput
invocation output (width/height/num_frames/fps/duration/video).
- New VideosInterface on InvocationContext with .save(source_path,
width, height, duration, fps, ...) returning VideoDTO. Mirrors
ImagesInterface — falls back to WithBoard / WithMetadata mixins
and embeds the queue item's workflow/graph as a JSON sidecar.
- WanRefImageConditioningField now carries num_frames so the denoise
nodes can sanity-check the I2V condition. WanRefImageEncoder bumps
to v1.1.0 and gains num_frames=1 input (use 81+ for video I2V; the
encoder dispatches between the single- and multi-frame helpers).
- Image WanDenoise now rejects multi-frame conditions with a clear
message pointing at WanVideoDenoise.
Verified: pnpm lint (5/5) green; pnpm tests (multiuser auth 122/122
+ broader suite via prior runs); numerical shape checks for noise
and ref-image condition; end-to-end smoke via VideoService.create.
A restart of the InvokeAI server is required to pick up the new
invocations in the workflow editor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(wan): add Wan 2.2 T2V and I2V starter video workflows
Two new default workflows for the workflow editor 'Browse' modal:
- 'Text to Video - Wan 2.2' — model loader -> two text encoders ->
wan_video_denoise -> wan_l2v. Exposes prompt, model picks, CFG
(high + low), dimensions, frames, fps, and steps.
- 'Image to Video - Wan 2.2' — same shape plus a wan_ref_image_encoder
feeding the denoise node's ref_image input. Exposes the reference
image and the frames field on the ref-image node (must match the
denoise node's frames — there is a clear validation error if they
diverge, but the starter has them in sync at 81).
Both default to the Wan 2.2 reference settings: 832x480, 81 frames @
16 FPS (~5 s), 40 steps, CFG 5.0 (high expert) + 4.0 (low expert),
seeded by a rand_int. Pass the existing _sync_default_workflows
validator (id starts with default_, meta.category=default).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(wan): startup crash from stringified VideoOutput annotation
run_app.py validates every invocation's return-type annotation against
the output-class registry. wan_latents_to_video.py had a stray
'from __future__ import annotations' which made the `invoke()` return
annotation a string ('VideoOutput') at runtime. The registry mismatch
triggered the unregistered-output warning path, which itself crashed
on output_annotation.__name__ because the annotation was a str:
AttributeError: 'str' object has no attribute '__name__'
The other Wan invocations don't use future annotations — drop the
import to match. Verified post-fix: api_app import populates 95
output classes, wan_l2v annotation resolves to the real VideoOutput
class and is in the registry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(wan): add Wan 2.2 Lightning T2V starter workflow
Same graph as 'Text to Video - Wan 2.2' but with two Apply LoRA - Wan
2.2 nodes chained between the model loader and the denoise node, and
defaults retuned for the Lightning distillation: 4 steps and CFG 1.0
on both experts (CFG=1 skips the negative-conditioning forward pass
entirely, ~20x f…
…nvoke-ai#9304) * feat(krea2): add Krea-2-Turbo model + LoRA support (WIP) Integrate Krea-2-Turbo (krea/Krea-2-Turbo) text-to-image per NEW_MODEL_INTEGRATION.md: Krea2Transformer2DModel (single-stream MMDiT) + Qwen3-VL text encoder (12-layer hidden-state tap, 4D prompt_embeds) + reused Qwen-Image VAE + FlowMatchEulerDiscrete scheduler. Backend: - taxonomy: BaseModelType.Krea2, ModelType/ModelFormat.Qwen3VLEncoder, Krea2VariantType (Turbo = "krea2_turbo" to avoid Z-Image collision) - config probes: Main_Diffusers/Checkpoint_Krea2, Qwen3VLEncoder, LoRA_LyCORIS_Krea2 (text_fusion/time_mod_proj signature; excluded from the Qwen-Image probe to avoid double-match) - loaders for the diffusers pipeline + standalone Qwen3-VL encoder, with runtime workarounds for the HF model's version mismatches (AutoTokenizer, extra_special_tokens={}, rope_parameters->rope_scaling) - native sampling (pack/unpack, position_ids, linear-mu shift) and hand-written Euler denoise loop; reuses qwen_image l2i/i2l - invocations: model_loader, text_encoder, denoise, lora_loader, plus two ecosystem enhancers (conditioning rebalance, seed variance) - LoRA conversion for diffusers PEFT (lora_transformer- prefix) Frontend: - 'krea-2' base + qwen3_vl_encoder type/format across model maps, buildKrea2Graph, addKrea2LoRAs, graph-builder denoise/base lists, optimal dimension 1024, regenerated schema.ts Fixes: - estimate transformer working memory in krea2_denoise so the cache reserves activation headroom and offloads more model under partial loading; fixes fp8 + LoRA OOM at 1024 (model was placed before LoRA patches were applied, leaving no room for their activations) WIP: requires diffusers main (>=0.39 dev) for Krea2Transformer2DModel; pyproject.toml temporarily pins diffusers to git main. * fix(krea2): support single-file VAE/encoder mix-and-match end-to-end Allow non-diffusers Krea-2 transformers (GGUF/fp8) to run with standalone single-file VAE + Qwen3-VL encoder, fixing several blockers found in testing. - buildKrea2Graph: drop the hard "requires Diffusers-format" assert; instead require both a VAE and a Qwen3-VL encoder to be selected when the transformer is not diffusers (mirrors readiness.ts). - Qwen3-VL encoder remap: handle both single-file key conventions — implicit (model.layers.*) and explicit (model.language_model.*). The old blind model.* -> language_model.* turned the bf16 file's keys into language_model.language_model.* (398 meta tensors -> "Cannot copy out of meta tensor" crash). Both files now load 0 missing / 0 unexpected / 0 meta. - Qwen3-VL tokenizer/config: broaden the offline-cache fallback from OSError to Exception so a partial HF cache (config present, vocab missing) re-fetches instead of dying with TypeError. - Qwen3-VL encoder fp8: keep an fp8 source checkpoint fp8-resident with per-layer upcast (storage float8_e4m3fn, compute bf16) instead of dequantizing to bf16. Halves resident VRAM (~8.9GB -> ~4.4GB), avoiding partial-load thrashing alongside a large transformer. Auto-enabled for fp8 sources on CUDA; bf16 files stay bf16. - Qwen-Image VAE: a native-layout qwen_image_vae single file is classified with the Anima base and loaded as AutoencoderKLWan, but the qwen l2i/i2l nodes need AutoencoderKLQwenImage. Add backend/krea2/vae_compat.py::as_qwen_image_vae to reinterpret a Wan VAE as AutoencoderKLQwenImage (state dicts are identical, 194/194 keys); both qwen VAE nodes use it. Idempotent for real QwenImage VAEs. * fix(krea2): re-apply Wan→QwenImage VAE adapter after upstream merge An upstream merge reintroduced the AutoencoderKLQwenImage isinstance asserts in the qwen VAE nodes (without the import → F821) and dropped the adapter in the i2l path. A native-layout qwen_image_vae single file is classified with the Anima base and loaded as AutoencoderKLWan, so the asserts fail at runtime. - qwen_image_latents_to_image: drop the reintroduced pre-device assert (the as_qwen_image_vae adapter inside model_on_device already handles the class). - qwen_image_image_to_latents: restore the as_qwen_image_vae import + adapter call, remove both asserts. - estimate_vae_working_memory_qwen_image only reads tensor shape + element size, so it runs correctly on either VAE class before the adapter. * fix(graph): make isMainModelWithoutUnet a type guard incl. krea2_model_loader krea2_model_loader was added to MainModelLoaderNodes but not to isMainModelWithoutUnet, and the guard wasn't a type predicate — so it never narrowed modelLoader. OutputFields of the loader union collapses to the common 'vae' field, making g.addEdge(modelLoader, 'unet', ...) in addInpaint/addOutpaint fail to type-check ('unet' not assignable to 'vae'). Redefine the guard as a type predicate keyed on the inverse (only main_model_loader/sdxl_model_loader expose a unet), so every transformer-based loader is treated as unet-less automatically and the negated branch narrows to the unet-bearing loaders. * test(krea2): add Krea2VariantType type-test to satisfy knip zKrea2VariantType was only used within common.ts (in zAnyModelVariant) and never referenced externally, so knip flagged it as an unused export. Every sibling variant enum avoids this by being asserted in common.test-d.ts; add the missing Krea2VariantType assertion, which both uses the export and verifies the manual zod enum matches the generated S['Krea2VariantType']. * build: pin diffusers to 0.39.0 (first stable release with Krea-2) diffusers 0.39.0 is the first stable release containing Krea2Pipeline / Krea2Transformer2DModel (plus the Qwen-Image VAE and Qwen3-VL text encoder Krea-2 relies on). Replace the temporary git-main dependency with the pinned release and update the lockfile's diffusers entry (version, sdist, wheel, specifier) to the official PyPI 0.39.0 artifacts. * feat(krea2): metadata recall, starter models, and config-probe tests Close the remaining gaps against docs/new-model-integration for Krea-2. Metadata recall (§7): buildKrea2Graph now records the standalone VAE, Qwen3-VL encoder, and both conditioning enhancers (seed-variance + rebalance) to image metadata, and parsing.tsx adds the matching recall handlers (base-guarded to 'krea-2'), so a Krea-2 image's VAE/encoder/enhancer settings restore on recall — important for reproducing single-file/GGUF generations. Starter models: add Krea-2 Raw (Base variant, full pipeline), Krea-2 Turbo GGUF Q4_K_M / Q8_0 (vantagewithai/Krea-2-Turbo-GGUF) with Qwen-Image VAE + Qwen3-VL encoder dependencies, and a standalone Qwen3-VL 4B encoder (Qwen/Qwen3-VL-4B- Instruct). The VAE dependency reuses the existing diffusers qwen_image_vae starter; krea2_turbo gains its explicit Turbo variant. Tests: add config-probe unit tests for Krea-2 variant detection (name heuristic, _has_krea2_keys, GGUF/checkpoint/diffusers variant, default settings) and for the single-file Qwen3-VL encoder probe (visual-tower vs. text-only Qwen3). 31 tests. * Chore Ruff * fix(krea2): validate denoise inputs and VAE compatibility * test(krea2): add loader, denoise, graph, listener, recall and starter-model coverage Backend: - test_krea2_state_dict_utils.py: cover the pure loader transforms (prefix strip, native<->diffusers conversion, scaled-fp8 dequant, Qwen3-VL key remap) and _reject_incomplete_load parametrized over the single-file/GGUF/encoder call sites (rejects meta-tensor partial loads, names the missing params) - test_krea2_denoise.py: _prepare_cfg_scale (broadcast/length/type), the per-step cfg list vs. img2img-clip regression, _validate_inputs happy path, and _get_noise determinism/shape - test_starter_models.py: Krea-2 bundle registration, diffusers+GGUF+standalone membership, and GGUF entries declaring their VAE + Qwen3-VL dependencies Frontend: - modelSelected.test.ts: Krea-2 standalone-component defaulting (auto-select on GGUF, anima-VAE fallback, no-overwrite, diffusers clears overrides, clear on switch away) - parsing.test.tsx: Krea2 VAE/encoder + enhancer recall gating (parses only for krea-2, never clobbers otherwise) - buildKrea2Graph.test.ts: CFG negative-conditioning gating, enhancer node insertion/chaining, non-diffusers standalone-model assertion, metadata - ImageMetadataActions.test.tsx: require all eight Krea2 recall handlers Fix: exclude krea-2 from the generic VAEModel metadata handler (it has a dedicated Krea2VAEModel handler), matching the existing z-image/flux2 exclusions. * fix(ui): wire Krea metadata recall actions * fix(krea2): address adversarial review findings * fix Krea-2 review findings * fix(krea2): use per-prompt position ids for the CFG uncond pass The rotary position ids (text tokens + image grid) were built once from the positive prompt's length and reused for the negative pass. When the negative prompt tokenizes to a different length than the positive one, the rotary embedding ends up a different length than the uncond query sequence and the transformer crashes in apply_rotary_emb ("tensor a (N) must match tensor b (M)"). Build a separate neg_position_ids from the negative prompt's length and pass it to the uncond transformer call. txt2img with CFG off (distilled Turbo) is unaffected — only the cond pass runs there. Adds a regression test that drives differing positive/negative prompt lengths and asserts len(position_ids) == text_len + image_tokens for each pass. * fix(krea2): resolve deep-review findings across encoder, loaders, LoRA and tokenization HIGH: - qwen3_encoder: _has_qwen_vl_visual_tower now also matches the nested model.visual.* layout (mirroring _is_qwen3_vl_encoder_state_dict), so a single-file Qwen3-VL 4B encoder no longer matches BOTH the text-only Qwen3 and the Qwen3-VL configs. The nondeterministic tie-break could register it as the wrong type and hide it from Krea-2's encoder dropdown, hard-blocking the single-file/GGUF install path. MEDIUM: - krea2_text_encoder: tokenize (prefix+prompt) and the assistant-turn suffix separately and concatenate, so prompts over the token budget keep the suffix (append-after-truncate) instead of having it silently cut off. LOW: - main: _get_krea2_variant_from_name lets "turbo" win and only matches "raw"/"base" as whole tokens, so Turbo files like "krea2_turbo_baseline_q4.gguf" are not read as Base. - krea2_lora_conversion_utils: raise a descriptive ValueError (not a bare KeyError) when a PEFT layer has lora_A without a matching lora_B. - factory: read config.json as UTF-8 so a non-ASCII config is not mis-treated as unrecognized (and the model dir wrongly rejected) under a cp1252 locale. - krea2 loader: _reject_incomplete_load also inspects named_buffers(), so a checkpoint missing a persistent buffer fails at load time rather than mid-inference. Tests: Qwen3-VL dual-match rejection, long-prompt suffix preservation, addKrea2LoRAs reroute, rebalance gains/validation, seed-variance determinism/out-of-place, variant filename heuristic, incomplete-LoRA error, UTF-8 config dir, meta-buffer rejection. * fix(krea2): reshape native final-layer modulation + honor scheduler shift config - loader: last.modulation.lin (native/GGUF) is reshaped to (2, hidden) to match diffusers Krea2FinalLayer.scale_shift_table, not just renamed. assign=True would otherwise install a flat 1-D parameter (which the meta-only completeness guard cannot catch), failing at inference on the primary GGUF/native path. Verified the final table is (2, hidden) and the per-block tables are (6, hidden) against the installed Krea2Transformer2DModel. - denoise: the resolution-aware timestep shift (mu) now reads base_shift/max_shift/ base_image_seq_len/max_image_seq_len from the loaded scheduler's config, falling back to the Krea-2 defaults, so a Raw checkpoint shipping a customized scheduler_config.json is sampled with its own shift parameters. Adds a converter test asserting last.modulation.lin -> final_layer.scale_shift_table is reshaped to (2, hidden). * fix(krea2): resolve remaining review findings * test(krea2): cover converter tensor shapes and scheduler-config mu path Add the two regression guards the loader/denoise fixes were missing: - Validate _convert_krea2_native_to_diffusers against the real Krea2Transformer2DModel (built on the meta device from KREA2_TRANSFORMER_CONFIG). Every scale_shift_table is sized from the actual module dims and asserted after conversion, pinning final_layer.scale_shift_table to (2, hidden) and each per-block table to (6, hidden). The stub-based boundary tests could not catch a wrong-shaped converted tensor, since load_state_dict(assign=True) installs any shape and _reject_incomplete_load only checks the meta device, not shapes. - Exercise the resolution-aware mu branch in Krea2Denoise._run_diffusion (shift=None, undistilled/Base config) so it is no longer dead: assert the mu passed to set_timesteps is derived from the loaded scheduler's base_shift/max_shift/base_image_seq_len/max_image_seq_len, and falls back to the Krea-2 defaults for absent keys. * fix(krea2): resolve final adversarial review findings * fix(krea2): calibrate seed variance to embedding std; fix randomize slider The Seed Variance enhancer added noise at an absolute magnitude (strength=20), so its effect depended on the embedding scale. Conditioning Rebalance multiplies the embeddings by up to ~20x, so with rebalance off the same noise overwhelmed the signal and prompt following collapsed (reported by lstein). Calibrate the noise to the embedding's standard deviation instead — the same approach the Z-Image Seed Variance enhancer already uses — so a given strength behaves consistently regardless of embedding scale. strength is now a std multiplier in [0, 2] (default 0.1); 0 or randomize_percent 0 is a no-op. Also fix the Randomize Percent slider: with sliderMin=1 and a coarse step of 5 the grid was anchored at 1 (1, 6, 11, 21, 26, ...) and never hit round tens. Anchor at 0 with a coarse step of 10, and relax the backend floor to ge=0.0. * Chore knip * fix(krea2): sync metadata recall ranges; accept diffusion_model LoRA layout Follow-up to the seed-variance recalibration: the metadata recall parsers still used the old ranges, so recalling an image dispatched state the backend rejects. - Krea2SeedVarianceStrength recall now parses 0..2 (the std-multiplier range), not 0..100 — recalling the old absolute value 20 no longer produces invalid state that buildKrea2Graph forwards to a failing generation. - Krea2SeedVarianceRandomizePercent recall now allows 0 (the disabled value), matching the slider, param state, and invocation. - LoRA_LyCORIS_Krea2_Config accepts a transformer-only LoRA using the diffusion_model.transformer_blocks.* layout under an explicit Krea-2 override; the converter already handles the diffusion_model. prefix. Adds range boundary tests for both recall parsers and diffusion_model.* LoRA accept/reject tests. * fix(krea2): reject orphan LoRA halves, invalid rebalance weights, incompatible VAE Three install/queue-time guards so malformed inputs are rejected up front instead of failing mid-generation: - LoRA identification now requires every lora_A/B (or lora_down/up) weight to have its partner half. A valid layer plus a dangling half previously installed and then crashed during LoRA conversion; both the explicit-override and the automatic-detection paths now validate completeness. - Krea-2 Conditioning Rebalance weights are validated as exactly 12 finite numbers before generation: in readiness (blocks the queue), in metadata recall (rejects instead of dispatching invalid state), and in the input field (isInvalid). Mirrors Krea2ConditioningRebalanceInvocation._parse_weights. - as_qwen_image_vae now requires the Qwen-Image geometry (16 latent channels, 8x spatial, no patchification) and rejects Wan 2.2's 48-channel / patchified VAE before encode/decode, rather than failing on 16-vs-48 normalization. Adds LoRA orphan-pair tests, rebalance-weight validator + recall tests, and Wan VAE geometry accept/reject tests. * Chore ruff + pnpm fix * fix(krea2): tighten LoRA validation + fix DoRA/alias conversion bugs Addresses five install/convert-time issues so malformed Krea-2 LoRAs and rebalance weights are rejected up front (or converted correctly): - Explicit Krea-2 override now rejects an orphaned lora_A/B (or lora_down/up) half anywhere in the state dict, not just under the approved prefixes — a transformer_blocks pair plus a dangling text_fusion half previously installed and then crashed during conversion. - Krea-2 LoRA detection now requires a complete weight pair; a file with only dora_scale (no A/B weights) is rejected instead of failing later on load. - Rebalance weights are restricted to decimal/scientific notation, rejecting the hex/binary/octal literals (0x10, 0b10, 0o10) that JS Number() accepts but the backend's Python float() rejects. - The converter now recognizes the standard PEFT/Diffusers DoRA magnitude key lora_magnitude_vector.weight, mapping it to dora_scale so a valid DoRA adapter loads as a DoRALayer instead of being split into a bogus layer. - Conflicting transformer./diffusion_model. aliases that normalize to the same target layer now raise explicitly instead of silently overwriting one. Adds tests for each case. * test(krea2): cover adversarial validation cases * feat(krea2): support native (ComfyUI) Krea-2 LoRAs; fix Anima misdetection Native Krea-2 LoRAs (e.g. sliders) name modules differently from InvokeAI's diffusers Krea2Transformer2DModel: diffusion_model.blocks.N with attn.wq/wk/wv/ wo/gate, mlp.{down,gate,up}, and a txtfusion stage. These were misidentified as Anima (whose strict detector matched the bare blocks.N.mlp.*) and, even when forced to Krea-2, could not be applied because the converter only understood the diffusers PEFT layout. - Add a verified 1:1 native->diffusers key remap in the Krea-2 LoRA converter (blocks->transformer_blocks, attn.wq/wk/wv->to_q/to_k/to_v, attn.wo->to_out.0, attn.gate->to_gate, mlp->ff, txtfusion->text_fusion). Every native module maps onto a real Linear in the diffusers model (checked against all 512 keys of a real slider LoRA). DoRA magnitude survives the remap. - Extend Krea-2 LoRA detection (config + converter) to recognize the native signature (txtfusion, or the gated attention attn.wq + attn.gate). - Tighten the Anima strict detector to require the Anima-specific mlp.layer_N / mlp_layerN naming instead of a bare mlp, so a native Krea-2 LoRA is no longer false-matched as Anima. No Anima/Wan regressions. Adds native remap, DoRA-through-remap, diffusers-untouched, and native identification tests. * fix(krea2): complete native LoRA normalization * fix(krea2): use memory-efficient attention to fit VRAM (was OOM/hang) Krea-2's transformer uses grouped-query attention (48 query / 12 KV heads) and its stock processor calls scaled_dot_product_attention with enable_gqa=True. PyTorch only supports enable_gqa on the math SDPA backend, which materializes the full O(seq^2) score matrix: ~6.75 GB per attention at 1280x720 (3600 tokens) and ~40 GB at 2560x1440. On builds without flash attention (e.g. Windows) there is no fused fallback, so generation either OOMs or the model cache offloads the transformer to RAM and the forward pass crawls. - Add Krea2MemoryEfficientAttnProcessor: expands the KV heads (repeat_interleave) so enable_gqa is not needed, and runs under the memory-efficient SDPA kernel (O(seq) memory, supports the padding mask). Numerically equivalent to the stock processor; measured ~6.75 GB -> ~1.41 GB per block at 3600 tokens. Installed on the transformer in krea2_denoise before the denoise loop. - Recalibrate _estimate_working_memory: with O(seq) attention the activation footprint is small and ~linear, so the previous ~2.6 MiB/token (O(seq^2)) figure no longer applies. The new estimate reserves realistic headroom (~8.5 GB at 2560x1440 instead of an impossible ~36 GB), so the idle Qwen3-VL encoder is evicted and the fp8 transformer stays resident on a 24 GB card. Adds processor equivalence tests (GQA and non-GQA) and a working-memory bound regression test. * fix(krea2): reject mixed-layout key collisions instead of silently overwriting Both Krea-2 key normalizers (native->diffusers transformer keys, ComfyUI single-file Qwen3-VL encoder keys) mapped each source key to one target key and wrote it straight into the output dict. A malformed mixed-layout checkpoint that carries both a native key and its already-normalized alias (e.g. blocks.0.attn.wq.weight and transformer_blocks.0.attn.to_q.weight, or a bare layers.1.weight and its model.-prefixed twin) collapses both onto one target key, and the surviving tensor depended on dict iteration order. Route every write through a shared _put_unique_key helper that raises an actionable RuntimeError naming both colliding source keys, so such a checkpoint fails at load time instead of silently dropping a tensor. Add order-independent collision regression tests for both normalizers. --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: JPPhoto <jpollack@jpollackphoto.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Optimize iterator graph materialization * Update generated OpenAPI schema * Optimize iterator graph expansion and memory * Fix graph adjacency cache lifecycle * Refactored `try...finally` behavior * Optimize saved workflow graph restoration
* feat(app): parallel multi-GPU session execution
Run one generation session per configured GPU concurrently, with a tiled
progress preview. Multi-user isolation is unchanged. Backed by five seams:
- Per-thread device context (TorchDevice.set/get/clear_session_device);
choose_torch_device() consults it first, so all device-selecting call sites
resolve to the calling worker's GPU with no per-node changes.
- Per-device model caches: build_model_manager builds one ModelCache per
generation device; ModelLoadService.ram_cache resolves by current thread
device; ram_caches fans out clear/drop/shutdown.
- Atomic concurrent dequeue: a dequeue lock makes select+claim atomic so
concurrent workers never claim the same item (works on FIFO; round-robin
from #9086 slots in later).
- Worker pool: one _SessionWorker per device, each pinning torch.cuda.set_device
and its session device, with its own runner and cancel event; cancellation
routes via an {item_id -> worker} lookup. Single-device installs keep the
exact legacy single-worker behavior. Profiling disabled when >1 worker.
- New config `generation_devices`; unset = legacy single-worker mode.
Frontend: the canvas staging area already tiles per queue item; the main
ImageViewer now tracks progress per session and renders a tile grid
(ProgressImageTiles) when more than one session is active.
Also adds a lock to ObjectSerializerForwardCache for concurrent access.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(tests): restore global device after multi-GPU cache routing test
test_model_load_device_routing mutated the process-wide get_config()
singleton (device = "cuda:0") to exercise the per-thread cache routing,
but never restored it. The leaked CUDA device was then picked up by a
later test (test_model_load::test_loading) via choose_torch_device(),
which crashed with "Torch not compiled with CUDA enabled" on the
CUDA-less CI runner. Add an autouse fixture to save/restore device and
clear any pinned session device.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(ui): regenerate openapi schema and frontend types for generation_devices
Regenerate openapi.json (make frontend-openapi) and the frontend
schema.ts types (make frontend-typegen) so they include the new
generation_devices config field, fixing the openapi-checks and
typegen-checks CI jobs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): regenerate openapi.json with uv to match CI generator
`make frontend-openapi` used a bare `python` from a different environment
that emitted the CacheStats @dataclass docstring as a schema description.
CI generates the schema via `uv run`, which does not, so openapi-checks
failed on the diff. Regenerate with the uv-locked environment to drop the
stray description while keeping the generation_devices field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(model-manager): serialize model construction against VRAM moves to prevent meta-device corruption
Parallel multi-GPU session workers could intermittently crash with "unrecognized
device meta" (denoise) or "Cannot copy out of meta tensor; no data!" (l2i), because
model loading relies on process-global, non-thread-safe monkey-patches.
accelerate.init_empty_weights() (used directly by the loaders and implicitly by
diffusers' default low_cpu_mem_usage=True in from_pretrained) swaps
torch.nn.Module.register_parameter globally for the duration of a load, routing every
newly-registered parameter to the meta device. The model cache's VRAM load/unload runs
nn.Module.load_state_dict(assign=True), whose assign path does setattr -> __setattr__ ->
register_parameter. When one worker's VRAM move overlapped another worker's from_pretrained,
the move's real weights got hijacked onto meta and blew up on the next .to(device).
Introduce MODEL_LOAD_LOCK, a write-preferring readers-writer lock:
- write lock = model construction (_load_and_cache, load_model_from_path), exclusive.
- read lock = VRAM load/unload (ModelCache.lock(), repair_required_tensors_on_device).
VRAM transfers across GPUs still overlap each other; they only block while a construction
holds the write lock. The lock is always acquired before any per-cache lock to keep a
consistent order and avoid an AB-BA deadlock with the writer's make_room/put.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): fix outpainting crash caused by model download collisions
* fix(backend): make DiskImageFileStorage thread-safe for parallel sessions
Image.open() is lazy: it reads the header but defers pixel decoding (and
holds the file handle open) until the first .load()/.copy()/.convert(). The
opened object was cached and the same object handed to every caller, so in
multi-GPU parallel mode two session-processor worker threads could call
.copy() on it concurrently and race on the shared file handle and decoder
state. This surfaced as "broken data stream when reading image file" and
"AssertionError: self.png is not None" during inpainting with batch >1.
Force the decode (image.load()) before the object enters the cache so the
cached object is safe for concurrent reads, and guard the cache structures
(__cache / __cache_ids) with a lock since they are now mutated from multiple
threads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(ui): stack per-session progress bars during parallel generation
The generation progress bars (under the Invoke button and the Viewer tab)
both read a single global $lastProgressEvent atom, which every session
overwrites. With parallel multi-GPU sessions this made the bar jump back
and forth between sessions.
Track progress per queue item id and render one bar per in-flight session,
stacked vertically, each removed as its session reaches a terminal state.
- stores.ts: add $progressEvents (map keyed by item_id),
$activeProgressEvents (sorted), and set/clear helpers.
- setEventListeners.tsx: populate per-item progress on invocation_progress;
clear per item on terminal status; clear all on connect/disconnect/queue
cleared.
- ProgressBar.tsx: render a vertical stack of bars (one per active session)
with a single-bar fallback for the idle / model-loading window; add
containerProps so dockview tabs can position the stack.
- Dockview tab call sites: move positioning into containerProps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): make $progressEvents module-local to satisfy knip
$progressEvents is only referenced within stores.ts (via the
$activeProgressEvents computed and the set/clear helpers), so exporting
it tripped knip's unused-exports check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): cap stacked tab progress bars to fit below the tab label
With 4 GPUs the stacked per-session progress bars grew past the bottom
strip of the dockview tab and overlapped the "Viewer" label.
Add a fitHeightPx prop: in fit mode the stack is capped to the available
strip (10px below the ~40px tab's centered label) and the bars flex to
share it, shrinking below their natural height only once they no longer
fit. With 1-2 sessions the bars keep their familiar thin height; with 3+
they scale down to stay within the strip. The sidebar bar is unaffected
and continues to stack at natural height (it has the vertical room).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(config): support "auto" generation_devices to use all GPUs by default
generation_devices now accepts "auto" (the new default), which expands to
every visible CUDA device — so multi-GPU parallel generation works out of
the box without manually listing devices. On GPU-less systems "auto"
resolves to the single cpu/mps device, preserving serial behavior.
- config_default.py: type is now Union[Literal["auto"], list[str]],
default "auto"; validator accepts "auto" or a list of device strings.
- devices.py: add TorchDevice.get_generation_devices(), the single resolver
that expands "auto", normalizes, and deduplicates.
- session_processor / model_manager: both consumers use the resolver
instead of iterating the raw config value (which would have iterated the
characters of the "auto" string).
- Regenerated docs/src/generated/settings.json.
- Tests for the resolver (auto-with/without-CUDA, dedup, empty).
An explicit single-device list (e.g. [cuda:0]) or an empty list opts out
of parallelism.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(frontend): typegen+openapi
* docs(multi-gpu): add configuration information
* chore(frontend): typegen + openapi again
* feat(settings): add Generation Devices selector to Settings dialog
Add a badges UI in the Generation section of the Settings dialog for
choosing which devices `generation_devices` should use, modeled on the
Log Namespaces toggle UI.
Backend:
- New `GET /api/v1/app/generation_device_options` endpoint listing the
selectable devices (cuda:N with GPU names, or the sole mps/cpu fallback).
- Add `generation_devices` to the runtime-config update allowlist with
validation rejecting invalid device strings and explicit nulls.
Frontend:
- New SettingsGenerationDevices component with active/inactive badges.
"Auto (all GPUs)" is exclusive; removing the last explicit device
reverts to auto. Admin/multiuser gated; notes restart requirement.
- Wire into the Generation section; regenerate schema; add en strings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(settings): boldface the restart notice on Generation Devices
Split the restart sentence into its own string and render it bold so
users notice that device changes require restarting InvokeAI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(settings): show GPU name in Generation Devices badges
Render device badges as "cuda:0 (RTX 3090 #1)" so identical cards can be
told apart. Strips the "NVIDIA GeForce" vendor prefix and adds a 1-based
"#N" suffix only when multiple cards share a name. The full device name
remains available as the badge tooltip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(frontend): openapi
* feat(multi-gpu): surface per-session GPU number in logs and UI
Help users track which CUDA device is processing each session:
- Model-load log: "Loaded model ... onto cuda device #N in ..s"
- Denoise progress bars: "Denoising (#N)" across all architectures
(SD1.5/SDXL, FLUX, FLUX2, Z-Image, Anima, SD3, CogView4)
- Progress preview circle: GPU number centered in the ring, via a new
`device` field on InvocationProgressEvent (resolved from the worker's
thread-local session device)
- Session Queue: new "GPU #" column between STATUS and TIME, backed by a
`device` column on session_queue (migration_32) recorded when a worker
claims an item
Adds TorchDevice.get_session_device_label()/get_session_device_index()
helpers and a frontend getCudaDeviceIndex() parser (with tests). Shows the
number on CUDA only; CPU/MPS show nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(multi-gpu): show per-device names in startup log and progress circles
- Startup log lists each generation device with its GPU number and id,
e.g. "Using torch device: [AMD Radeon PRO W7900 #1 (cuda:0), ...]".
Single-device setups keep the bare device name.
- Canvas progress circles now show the CUDA device index in the center,
matching the viewer panel.
- Progress-circle tooltips show the device name and number on hover.
- Both are hidden when only a single GPU is available.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(model-cache): share one CPU copy of model weights across per-GPU caches
In multi-GPU mode the model manager builds one ModelCache per generation device,
each with storage_device="cpu" and its own RAM-resident copy of every model. A model
loaded on N GPUs therefore occupied N copies in RAM, and each cache sized itself
against max_cache_ram_gb independently, so RAM use during the text/reference-image
encoding phases skyrocketed and the system swapped — worst when two images rendered
at once.
This deduplicates the CPU-resident weights and makes RAM accounting global.
- SharedCpuWeightsStore: process-/manager-global, refcounted store of one canonical
CPU state_dict per model key. The first device to load a key registers its weights;
subsequent devices adopt the canonical tensors and re-point their module's params at
them (load_state_dict(assign=True)), freeing the duplicate. Weights live once in RAM
regardless of GPU count; freed only when the last device releases. Per-device modules
are kept (params are device-shuffled in place, so two GPUs need two modules), but
their CPU-resident params alias the shared tensors.
- RamBudget: single system-wide RAM authority. Splits RAM into shared (counted once via
the store) and non-shared (per-instance). ModelCache eviction now runs against the
global, deduplicated total and re-checks availability each iteration, since evicting a
model another device still holds frees no RAM. build_model_manager wires one store +
one budget into all device caches; the cap is max_cache_ram_gb as a true system-wide
limit, else the sum of per-cache heuristics. Passing ram_budget=None preserves the
prior local accounting.
- LoRA/patch safety: direct LoRA patching did an in-place copy_ on the weight, which
would corrupt the now-shared canonical tensor (and taint keep_ram_copy even with one
GPU) when patching a CPU-resident weight. Switched to an out-of-place add (memory-
equivalent) so the canonical tensor is never mutated; fixed the FluxControlLoRA
expansion path to target the module's live parameter. Sidecar patching and
FreeU/Seamless (which patch forward methods) were already safe.
Validated on 2x AMD W7900 / ROCm: correct inference on both GPUs from one shared copy
(full + partial load + Q8_0 GGUF quantized), concurrent load/unload without corruption,
and LoRA isolation across devices. ~40 new tests; existing suites unchanged.
Adds scripts/multigpu_ram_driver.py to drive concurrent dual-GPU generations via the
queue API and measure peak RSS / leak drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(session-queue): cancel all in-progress items in bulk-cancel APIs (multi-GPU)
With one session-processor worker per device, multiple queue items can be in_progress
at once. cancel_by_batch_ids(), cancel_by_destination() and cancel_by_queue_id() excluded
in_progress rows from their bulk UPDATE and then canceled only the single get_current()
item (LIMIT 1), so on multi-GPU the other running items kept consuming a GPU and could
still produce output after the user requested cancellation.
Each running item must be canceled via _set_queue_item_status(), which emits the
QueueItemStatusChangedEvent that the processor maps to the worker running that item_id and
uses to set its cancel event. Add _cancel_in_progress_matching() to cancel every in-progress
item matching the same filter (with user-id scoping preserved) and call it from all three
bulk-cancel methods. The returned `canceled` count now includes canceled in-progress items.
Adds regression tests that dequeue two items onto separate devices and assert every bulk
cancel API moves all matching in_progress items to canceled and emits a cancel event for
each (and that user-scoped cancel leaves another user's in-progress item running).
Reported by JPPhoto in review of #9263.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(multi-gpu): address review findings (cancel race, bulk delete, device guards, refcount leak)
Fixes from the code review of PR #9263:
- Cancellation could be silently lost around dequeue: the per-iteration
worker.cancel_event.clear() ran AFTER dequeue + gc.collect() + logging, so a cancel
arriving in that window was set by the status handler and then wiped. Move the clear to
before dequeue, and after claiming an item re-check (cancel_event + a fresh DB status read
via _is_queue_item_terminal) and skip running if it is already terminal, closing both race
windows. The runner's stale queue_item.status check could not catch this.
- delete_by_destination only stopped one in-progress item (get_current) before deleting all
matching rows, leaving other GPU workers running (and then failing to update a deleted row).
Cancel every matching in-progress item via _cancel_in_progress_matching first.
- generation_devices validation: a bare non-"auto" string (e.g. "cuda:0") was iterated
character-by-character; an empty list silently fell back to one device. Reject both with a
clear message.
- get_generation_devices now fails fast on a CUDA device that does not exist (index past
device_count, or CUDA unavailable) instead of starting a worker that errors cryptically at
first allocation.
- Shared-weights wrappers: if the canonical re-point (load_state_dict assign=True) threw after
acquire(), the reference was leaked (the wrapper never entered the cache). Compute size
metadata first, make acquire the last step, and release on failure.
Adds tests for each: post-dequeue terminal guard, delete_by_destination cancellation,
generation_devices validation, absent-device rejection, and acquire-released-on-repoint-failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ci): ruff format + make CPU-incompatible device test mock CUDA
- Apply ruff 0.11.2 formatting to the files flagged by `ruff format --check`.
- The new fail-fast guard in get_generation_devices() (reject a CUDA device that
doesn't exist) made the pre-existing test_get_generation_devices_explicit_list_is_deduplicated
fail on CPU-only CI runners, since it passes a cuda list with no CUDA present. Mock
torch.cuda.is_available/device_count in that test (matching the existing pattern in this
file) so it validates dedup on any runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(multi-gpu): stop RAM blowup/swapping during concurrent generations
Three RAM fixes for multi-GPU (and one that helps single-GPU too), addressing
transient spikes to ~100% RAM and swapping during text-encode/transformer loads:
1. Cap the global RAM-cache budget at a safe fraction of system RAM. When
max_cache_ram_gb is unset, the budget was the *sum* of the per-device cache
heuristics, so N GPUs each claiming ~50% of RAM summed to ~N*50% and starved
the OS. Now clamp the sum to ModelCache.calc_system_ram_headroom_bytes()
(50% of RAM - 2GB baseline, floored at 4GB). Promote the sizing magic numbers
to named constants shared by the per-device heuristic and the global cap.
2. Adopt already-resident CPU weights across devices at load time. When a second
device loads a model another device already holds, deep-copy a registered
meta-weight structural clone and assign the shared canonical weights, instead
of re-reading the model from disk and materializing a full transient second
copy. Loader-agnostic (one mechanism in ModelLoader, no per-loader code):
works for diffusers, single-file checkpoint, GGUF and transformers models,
and preserves registered hooks (e.g. fp8 layerwise-cast). Best-effort with a
meta-tensor self-check and fallback to a normal disk load on any failure.
Skipped on single-device installs.
3. Dequantize FLUX.2 FP8 checkpoints straight to bf16. _dequantize_fp8_weights
materialized the whole model in float32 (~36GB for 9B) before a later cast to
bf16; now the multiply is done in float32 but stored bf16 per-weight, so the
model is never held in float32. Numerically identical; halves the cold-load
transient (helps single-GPU too).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(qwen-image): reserve VAE working memory so decode/encode don't OOM
The Qwen Image VAE encode/decode invocations called model_on_device() without a
working-memory estimate, unlike every other VAE family (SD/SDXL/SD3/CogView4/FLUX).
So the model cache reserved only its small default working memory, never offloaded
a large resident transformer (the VAE weights themselves are tiny), and the VAE's
forward-pass activations then OOM'd VRAM — e.g. a ~40GB Qwen Image Edit transformer
left ~1GB free while decode needed ~5GB. Reproduces single-GPU; unrelated to the
multi-GPU RAM work.
Add estimate_vae_working_memory_qwen_image() (same per-output-pixel scaling as the
other estimators, handling the 5D Qwen latents) and pass it from both the i2l
(encode, used for reference images in Image Edit) and l2i (decode) nodes, so the
cache offloads the transformer before the VAE runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flux2): tile reference-image VAE encode to avoid VRAM OOM
The FLUX.2 VAE encoder's mid-block self-attention scales quadratically with the
input's spatial size, and on ROCm scaled_dot_product_attention falls back to a
materialized attention matrix. Encoding a reference image (kontext) at full size
therefore allocated ~15GB in a single attention call at 1024px — and hundreds of
GB at the 2024px reference cap — OOMing VRAM regardless of how much other model
memory was freed.
Tile the reference-image encode to bound per-tile attention. The VAE's default
tile size equals its sample_size (1024), whose per-tile attention still OOMs, so
force a 512px tile (with a matching latent tile size derived from the config).
Save/restore the VAE's tiling config since it is a shared, cached instance, so the
final image decode does not inherit these settings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(multi-gpu): query execution device for VRAM-in-use accounting
ModelCache._get_vram_in_use() called torch.cuda.memory_allocated() with no device
argument, while _get_vram_available() reads memory_allocated(execution_device).
The formula relies on those two canceling. In multi-GPU mode each worker calls
torch.cuda.set_device for its own GPU, so the process-current device flips between
workers; the no-argument call can then read a different (e.g. idle) GPU's
allocation, breaking the cancellation and inflating "available" VRAM toward the
card total. The cache then believes there is room and never offloads, so VRAM
offloading effectively ignores device_working_mem_gb in multi-GPU. Single-GPU was
unaffected (current device always equals the execution device).
Query self._execution_device in both _get_vram_in_use() and the cache-state debug
log. Add a regression test asserting the per-cache execution device is used.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(qwen-image): calibrate VAE working-memory estimate to the 3D-conv decode peak
The Qwen Image VAE is a 3D-conv (video) VAE whose decode allocates large conv3d
feature maps. A ~1MP decode was measured to peak at ~17 GiB of VRAM — far above
what the generic 2200/1100 SD/FLUX constants reserved (~4.6 GiB), so the cache
concluded the decode "fit" alongside the resident 20GB transformer + 15GB text
encoder, never offloaded them, and OOMed. The offload only frees ~(working_mem -
free) bytes, so the reservation must both cover the real peak and be large enough
to trigger the offload of models the decode doesn't need.
Raise the Qwen decode/encode constants (13000/6500) to match the measured peak.
It's linear in output pixels, so it over-reserves past ~1.5MP (where the decode
can exceed the card even after offloading) — that case is covered by
force_tiled_decode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(qwen-image): honor force_tiled_decode in the l2i node
The Qwen Image latents-to-image node hardcoded vae.disable_tiling(), ignoring the
global force_tiled_decode setting that the SD/SDXL l2i node honors. Wire it up the
same way so users can opt into tiled VAE decode for very large outputs that exceed
VRAM even after the transformer/text encoder are offloaded. Off by default, so
normal-size decodes are unchanged (full-frame, no tile blending).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ui): stop progress disk flashing during indeterminate phases
The preview-panel progress circle re-renders on every InvocationProgressEvent. The
parent passes a fresh progressEvent object each event, so the CircularProgress
re-rendered constantly; during the indeterminate phases (everything except
denoising) that restarted its CSS spin animation each time, which looked like the
disk flashing. (Determinate denoising was unaffected because the value genuinely
changes per step.)
Split the circle into a memoized, ref-forwarding subcomponent keyed on its visual
props (isIndeterminate, value, device label) so message-only updates no longer
re-render it and the spin animation stays continuous. The Tooltip still anchors to
it via the forwarded ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(multi-gpu): offload text encoders to idle GPUs
Adds `offload_text_encoders_to_idle_gpus` (default on): when more than one
generation device is configured and a GPU is idle, a session's text/prompt
encoder runs on the idle GPU instead of the one running its denoise pipeline.
This avoids evicting the denoise model from VRAM to make room for the encoder,
and lets a cached encoder be reused across generations. Under full load (no
idle GPU) behavior is unchanged.
Mechanism:
- New GENERATION_DEVICE_POOL arbiter (backend/util/device_pool.py) with a
per-device exclusive-use lock. A native session blocking-acquires its own
device's lock for the whole run; an encoder node try-borrows an idle device's
lock for the duration of the node. This makes a borrowed encoder and a native
session mutually exclusive on a GPU -- preventing the shared-encoder
corruption that produced garbled images -- and is deadlock-free (borrows are
non-blocking; a session only ever blocks on its own device).
- DefaultSessionRunner re-pins the worker thread to the borrowed device for the
whole encoder node; conditioning is stored on the CPU and the denoiser picks
it up on its own GPU afterward.
- Nodes opt in via @invocation(idle_gpu_offloadable=True), mirroring the
existing `bottleneck` ClassVar marker. Applied to the text/prompt encoder
nodes (compel + sdxl/refiner, flux, sd3, qwen-image, anima, cogview4, flux2
klein, z-image, flux_redux).
Inspired by #9310; supersedes it.
Tests: device-pool lock semantics, two concurrency regression tests asserting a
session and a borrow never use a GPU at the same time, the runner offload
context-manager behavior, and a marker-wiring check.
Docs: invokeai-yaml.mdx (config setting) and creating-nodes.mdx (how to support
the feature in a node).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(multi-gpu): adopt GGUF weights across devices to stop RAM spikes
_build_meta_shell built meta placeholders with torch.empty_like, which
GGMLTensor.__torch_dispatch__ rejects (NotImplemented for aten.empty_like).
It threw on the first parameter, hit the silent except, and returned None —
so GGUF models (e.g. a Q8_0 transformer) never registered a shell and the
second GPU re-loaded the full model from disk, stacking a ~20GB transient on
the retained copy and spiking RAM to ~70%.
Fall back to a plain meta placeholder (logical shape/dtype) when empty_like
isn't implemented by a tensor subclass; verified the adopted GGMLTensor shares
the quantized storage, so it's one RAM copy across devices. Peak drops ~66→~46GB.
Log shell-build failures at debug so a future un-adoptable family is diagnosable
instead of silently double-loading.
Also restore log_memory_usage's per-cold-load RAM logging (the capture method
had no callers), slimmed to baseline→transient-peak process RAM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(multi-gpu): tie device #N label to cuda index, not filtered position
The backend device summary computed the disambiguating #N suffix by
enumerating the filtered generation_devices list, so disabling a device
(e.g. cuda:1) renumbered the survivors. The frontend labels over the full
device set, so the two disagreed. Compute the suffix over all available
devices instead, keeping the label stable and consistent with the frontend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(multi-gpu): flash restart reminder when generation devices change
Reword the Generation Devices caption to "Restart InvokeAI for changes to
take effect." and flash that same warning as a toast on every successful
change, so the restart requirement is hard to miss.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(queue): device-affinity dequeue to reduce model reload thrash on multi-GPU
When a GPU worker dequeues, prefer — among the fairness-chosen user's
equal-priority pending items — one whose models are already resident in that
device's cache. Cross-device model reloads cost tens of seconds for large
models; picking a warm item instead cuts thrash when a user queues a mix of
models.
Guardrails (from adversarial review):
- Round-robin user choice and priority tiers are never overridden; the swap
pool is limited to the candidate's user and priority.
- The swap window is capped at AFFINITY_MAX_LOOKAHEAD past the candidate's
item_id, bounding both cold-item deferral and per-dequeue scan cost.
- Explicitly configured session_queue_mode=FIFO opts out of reordering.
- Resident keys are snapshotted before the dequeue lock, and
ModelCache.cached_model_keys() acquires its lock non-blockingly, so a
long-running VRAM transfer can never stall other workers' dequeues.
- Path-keyed cache entries (load_model_from_path) are excluded so a Windows
drive letter can't poison substring scoring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(qwen): restore legacy key remapping for single-file VL encoders under transformers 5.x
The single-file Qwen2.5-VL encoder loader relied on
Qwen2_5_VLForConditionalGeneration._checkpoint_conversion_mapping to translate
ComfyUI's legacy key layout (visual.*, model.layers.*) to the modern one
(model.visual.*, model.language_model.*). transformers 5.x ships that mapping
empty — the conversion moved into from_pretrained's weight-converter machinery,
which our manual load_state_dict path bypasses — so the vision tower was left
on the meta device and loading failed with "Meta tensors remain".
Fall back to the equivalent hardcoded mapping when the class attribute is
empty or absent. Verified against qwen_2.5_vl_7b_fp8_scaled.safetensors:
loads all 8.29B params with no meta tensors remaining.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: address multi-GPU review findings from PR #9263 review
- Shared CPU weights: drop_model() now invalidates the model's canonical
entries in SharedCpuWeightsStore, so a rebuild on another device can
never adopt pre-settings-change weights still aliased by a locked
(stale-marked) entry. release() is identity-checked so a stale
holder's eviction cannot decrement a newly registered canonical.
update_model_record holds MODEL_LOAD_LOCK.write_lock() (off the event
loop) across the multi-cache drop to exclude in-flight loads.
- Runtime config API: generation_devices is now fully validated at the
route boundary — empty lists and unavailable devices (e.g. cuda:99)
return 422 without mutating or persisting config, using the same
TorchDevice resolution as startup.
- Cache stats: /v2/models/stats aggregates per-device caches instead of
reporting only the API thread's default cache.
- Config/docs contract: session_queue_mode description now documents
device-affinity reordering in single-user multi-GPU mode (and that
explicit FIFO disables it), and that user rotation outranks priority
across users in round_robin mode. Multi-GPU docs no longer claim
generation_devices: [] is valid, and describe shared-RAM weight
deduplication instead of per-GPU duplication.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: address multi-GPU review findings (RAM accounting, stats aggregation, MPS validation)
- SharedCpuWeightsStore.invalidate() now retires still-referenced entries
instead of dropping them from accounting, so RamBudget keeps counting
retired weights until the last locked holder releases them. Prevents
admitting models past max_cache_ram_gb while a replacement and a stale
copy are both resident.
- /models/stats aggregation takes max of cache_size and high_watermark
across per-device caches (they share one global RamBudget, so summing
over-reported an N-GPU system ~N times); event counters are still summed.
- TorchDevice.get_generation_devices() rejects 'mps' when MPS is
unavailable, so the runtime_config API 422s instead of persisting a
device that fails at first tensor op.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(frontend): lint:prettier
* fix: address JPPhoto's 2026-07-21 review (12 items)
Backend:
- layer_patcher: hold MODEL_LOAD_LOCK.read_lock() across patch application so
FLUX Control LoRA shape expansion (register_parameter) cannot overlap a
concurrent model construction's process-global init_empty_weights patch
- flux_redux/flux_denoise: store Redux conditioning on CPU (it may be produced
on a borrowed idle GPU) and assign the .to() result when consuming it
- model_cache/ram_budget: coordinate eviction across device caches — when a
cache's own stack is exhausted and the global budget is still short, peers
evict their unlocked entries (non-blocking lock, deadlock-free), so
max_cache_ram_gb holds even when RAM is retained only by an idle device
- session_queue: 'except current' operations protect the workflow-call chain of
EVERY in-progress item, not one arbitrary get_current() row
- session_queue: _cancel_in_progress_matching tolerates rows deleted by a
concurrent clear between its id SELECT and the per-item cancel
- session_processor: the post-dequeue cancel guard cancels the freshly claimed
item when skipping it (a stale cancel_event must not abandon it in_progress)
- session_processor: _clone_session_runner refuses to downgrade
DefaultSessionRunner subclasses or share custom runners across workers
- session_processor: an offloaded encoder's cache activity is attributed to the
running session's CacheStats (borrowed cache's stale stats pointer swapped
for the borrow duration)
- events: progress events report the queue item's persisted device, not the
thread-local (temporarily borrowed) one
- devices/config docs: generation_devices 'auto' defers to an explicitly
pinned legacy 'device:' setting so upgrades don't start workers on every GPU
Frontend:
- ImageViewer context: a terminal status only clears the shared progress
event/image globals when that item owns them (multi-GPU: canceling item A no
longer blanks item B's live preview)
- SettingsGenerationDevices: device tags are keyboard-operable (tabIndex +
Enter/Space activation)
Each fix has an exposure test per the review's suggestions.
* chore: regenerate openapi.json (auth on get_generation_device_options)
* fix(backend): avoid MODEL_LOAD_LOCK self-deadlock when patching a LoRA on a cold cache
apply_smart_model_patches() held MODEL_LOAD_LOCK.read_lock() across its patch
loop, but callers pass a lazy generator (e.g. flux_text_encoder._t5_lora_iterator)
that constructs each LoRA via context.models.load() on demand. A cold-cache load
takes MODEL_LOAD_LOCK.write_lock(); since the lock is non-reentrant and
write-preferring, acquiring the write lock while this same thread already holds the
read lock deadlocks (write waits for readers==0, but the consuming thread is that
reader). The generation hung silently right after the encoder/tokenizer load,
whenever a LoRA was applied and not already cached.
Materialize the patch iterable before taking the read lock so every LoRA
construction takes (and releases) the write lock first; the read lock then covers
patch application only, which is its actual purpose (FLUX Control LoRA shape
expansion calls register_parameter and must exclude concurrent construction).
Compatible with wan_denoise's per-call iterator factory, and unrelated to the SD
UNet path, which loads the LoRA before calling the singular patcher (no lock held).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address JPPhoto's four merge blockers from the 2026-07-22 review
1. model_cache: a peer whose lock is contended during cross-cache eviction
no longer leaves the shared RAM budget exceeded indefinitely.
evict_unlocked_for_peer returns None on contention; the requester records
a reconcile request on each skipped peer, and the synchronized-decorator
hook honors it as soon as the peer's current operation releases the lock
(outermost frame only — the RLock may be held reentrantly). The pending
flag stays set until the budget is actually satisfied, so overshoot held
by locked entries reconciles when their unlock releases the lock.
2. session_processor: a stale cancellation event from the previous item no
longer cancels the freshly claimed, unrelated item. The post-dequeue
guard now treats the DB status as the authority: a terminal row is
skipped; a set cancel_event with a non-terminal row is a stale signal
(a genuine cancel writes the row terminal BEFORE emitting) and is
cleared, with a post-clear terminal re-check closing the clear's own
race window. A shutdown-raced claim is still canceled so it isn't
abandoned in_progress.
3. flux2_klein_text_encoder: conditioning is detached and moved to CPU
before context.conditioning.save(), matching flux_text_encoder and
flux_redux — the node is idle_gpu_offloadable, and GPU-resident
embeddings would pin VRAM on a borrowed device after its pool lock is
released.
4. session_queue clear: user-scoped clearing no longer assumes one current
item. clear() cancels every in-progress item in scope via
_cancel_in_progress_matching (each item's own status-changed event
signals the worker running exactly that item) before deleting rows —
same pattern as delete_by_destination; the router's arbitrary
get_current() check (which could 403 the owner or cancel another user's
item) is removed; and _on_queue_cleared honors the event's user_id so a
scoped clear cannot stop other users' workers and abandon their rows.
Each fix carries the regression test JPPhoto specified: contended-peer
budget reconcile, stale-event-runs-item (plus the mid-clear race and
shutdown cases), CPU-backed Klein conditioning, and Alice/Bob concurrent
clear isolation at both the service and the event-handler layer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: regenerate OpenAPI schema for the clear endpoint docstring
The merge-blocker fix 68edb02127 reworded the clear route's docstring,
which is the OpenAPI operation description — openapi.json and schema.ts
must follow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): close lost-wakeup race in deferred RAM-budget reconcile
The deferred reconcile request was recorded pre-admission and honored only
by the peer's next lock release. Two interleavings could strand the shared
RAM budget above its cap indefinitely:
- Lost wakeup: the busy peer releases its lock (running its reconcile hook
while the flag is still unset) before request_budget_reconcile() sets the
flag; if the peer then stays idle, no future release honors the request.
- Pre-admission clearing: a peer's reconcile could run between the request
and the new model being counted, see the budget as satisfied, and clear
the flag before the admission pushed usage over the cap.
Fix both by (1) moving the reconcile request to the end of put(), after the
new model is counted, so peers always evaluate the true budget state, and
(2) having request_budget_reconcile() attempt the reconcile inline with a
non-blocking lock acquire: either the peer's lock is free now and the
reconcile runs immediately, or it is still held and the eventual release
hook — which runs strictly after the flag is set — performs it.
The prior regression test masked the race by touching cache_b.stats after
the request; it now emulates the production release hook in the holder
thread and asserts reconciliation with no subsequent cache access, and a
new test forces the lost-wakeup interleaving by delaying the request until
the peer's operation has fully finished.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): close remaining RAM-budget reconcile gaps
Addresses the three lingering issues from review of the deferred
budget-reconcile mechanism:
1. Manual lock releases bypass the reconcile hook. cached_model_keys()
and evict_unlocked_for_peer() acquire/release _lock without the
synchronized decorator, so a reconcile request whose inline attempt
failed on their held lock was stranded when they released. Both now
run the same reconcile hook after their manual release (non-blocking,
preserving cached_model_keys' no-stall guarantee and avoiding the
hold-A-block-on-B deadlock shape in evict_unlocked_for_peer).
2. clear() can wipe a concurrent request. A reconciler observing a
satisfied budget could clear the pending flag just after a peer's
admission (already counted, budget negative) set it, and the peer's
inline attempt then saw the flag unset and returned — leaving the
budget exceeded with no pending request. The reconcile now runs as a
loop with a single guarded clear site: because admissions are counted
before the flag is set, a negative budget re-check immediately after
the clear proves a request may have been wiped; the flag is restored
and reconciliation continues. This covers both former clear sites
(satisfied early-out and post-eviction).
3. No reconcile trigger when the admitting cache itself holds the
overshoot. put() requests reconciles from peers only, so when the
exceeded budget was held by the admitting cache's own locked entry,
no pending request existed anywhere and the eventual unlock ran its
hook with the flag unset. unlock() now records a reconcile request on
its own cache whenever it completes with the shared budget exceeded,
so the entry that just became evictable triggers the reconcile.
Supporting change: put() admitting a model while a peer's reconcile
request is already pending must not let its own release hook evict the
just-admitted entry before the loader's immediately-following get()
(that would break the in-flight load with an IndexError). CacheRecord
gains an awaiting_first_use grace flag, set on admission and cleared on
first get()/lock(), which the asynchronous eviction paths (budget
reconcile, peer-requested eviction) skip. The local make_room path
ignores it: cold loads are serialized under MODEL_LOAD_LOCK, so it can
never see another loader's entry inside the put()->get() window, and
this bounds the flag's lifetime if a load errors out in between.
Each new regression test was verified to fail against the previous
implementation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): bound the admission grace and keep cached_model_keys stall-free
Addresses the three issues from JPPhoto's 2026-07-27 review:
1. Prefetched submodels can no longer shield the budget forever. The SD
single-file loader's proactive submodel put()s are now admitted with
prefetch=True (no post-admission grace), since nothing ever get()s or
lock()s them. As a backstop, put() sweeps stale grace flags from prior
loads — cold loads are serialized under MODEL_LOAD_LOCK, so any flag
still standing at the next admission belongs to a dead load (errored
before get(), or LoadedModel dropped before lock()) and is cleared.
2. The grace now survives get() and ends at lock(). get() is synchronized,
so clearing the flag inside it let get()'s own release hook run a
pending reconcile and evict the very record it had just selected —
detaching a live model from the cache and its RAM accounting before the
caller could lock it. load_default also retrieves immediately after
put() so no failure in between can orphan a graced record.
3. cached_model_keys()'s manual-release hook hands a pending reconcile to
a short-lived background thread instead of running it inline:
reconciliation evicts models and calls gc.collect(), which would break
the method's no-stall contract and pause session dequeue.
Each new regression test verified to fail with its mechanism reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(model cache): release abandoned admission grace
* fix(model cache): keep grace release off the collecting thread
release_first_use_grace() is invoked from a weakref.finalize callback, so it
runs at an arbitrary decref/garbage-collection point in an arbitrary thread.
Making it @synchronized therefore made ModelCache._lock — and, through the
decorator's release hook, a full budget reconcile — reachable from anywhere.
That inverts the lock order RamBudget documents as impossible. The hook's
_reconcile_budget_if_pending reads RamBudget.available() ->
SharedCpuWeightsStore.total_bytes_in_use(), both plain non-reentrant locks. A
thread inside SharedCpuWeightsStore.acquire() holds the store lock while summing
tensor sizes, an allocation loop that trips generational GC; if that collection
reclaims an abandoned wrapper belonging to another device's cache, the release
hook re-enters the store lock the thread is already holding and the thread
deadlocks against itself, still holding it. Every other cache then blocks on its
next _delete_cache_entry -> release_shared_weights(). Reproduced on a two-cache
budget: the collecting thread wedges in total_bytes_in_use() and never returns.
The same hook also ran evictions, gc.collect() and empty_cache() inline in
whatever unrelated thread happened to drop the reference — including the API
event loop — undoing the no-stall contract cached_model_keys() was just given.
Do no locking work in the callback: hand the release to a short-lived background
thread, exactly as cached_model_keys() does with its own pending reconcile. The
thread may wait on the cache lock and do the slow work; the collecting thread
returns immediately.
The existing abandoned-wrapper test now polls for the (asynchronous) release.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): harden the deferred grace release
Follow-ups from an adversarial review of 55a6fc496d:
- Thread.start() can raise RuntimeError under thread/process limits. A
weakref.finalize callback gets no retry (weakref retires it before invoking
it) and its exceptions go to sys.unraisablehook, so the release was silently
lost and the record kept shielding an idle cache. Fall back to clearing the
flag inline under a non-blocking acquire, which takes no store or budget lock
and so still cannot deadlock the collecting thread. No reconcile on that path
by design: a pending request stays set for the next cache operation.
- The regression test's outcome was a pure function of the ambient allocation
count: nothing pinned the cycle between its creation and the collector thread,
so an automatic gen-0 pass landing in the setup reclaimed it on the main
thread and the test passed vacuously (or tripped its own setup assertions).
Under an allocation-shifting plugin it failed at 6 of 12 offsets. Disable
automatic gc across the setup so only the explicit collect reclaims the cycle;
the same sweep is now 12 of 12 passing, and the test still fails against
7ffee4db04 with the expected re-entrancy report.
- Correct the docstring: Thread.start() waits for the child to bootstrap, so the
guarantee is "no lock waits", not "returns immediately".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): queue deferred cache work
* fix(model cache): stop the deferred worker pinning records and dying silently
Follow-ups from an adversarial review of c384ea187d, which replaced the
per-release background threads with one long-lived worker per cache.
- The worker's `work` local stays bound while it blocks in the next get(), so
the last-processed CacheRecord — and transitively its model's CPU weights —
was pinned until some unrelated item happened to be queued behind it. That is
worse than an ordinary leak: _release_first_use_grace's release hook can evict
that very record, removing it from the cache AND subtracting its bytes from
the RamBudget, so the budget under-reported a model that was still resident
and the next admission over-committed. Reproduced on a two-cache budget: after
eviction plus an explicit gc.collect(), both the record and its module were
still alive; queueing one more item freed them. The per-call threads this
replaced did not have the bug — Thread._bootstrap_inner deletes _args on exit.
Clear the reference in a finally before looping back.
- `if self._deferred_work_thread.ident is None` is a "was it ever started"
check, not a liveness check: ident is never cleared and a Thread cannot be
restarted. A worker lost to an unexpected error (a logging handler that
raises, os.fork(), or shutdown() before the first put(), which leaves its
_DEFERRED_STOP queued for the thread that put() then starts) was gone for the
life of the process, silently disabling every later grace release and budget
reconcile — the failure this mechanism exists to prevent. Create a fresh
thread whenever the previous one has exited, and never after shutdown().
- Only the worker drains the queue, but both dispatch sites enqueued
unconditionally. cached_model_keys() runs on every dequeue
(session_queue_sqlite._get_device_resident_model_keys), so an idle-device
cache that never admitted a model — and so has no worker — accumulated one
queued reconcile per dequeue forever; post-shutdown the same held for both
sites, stranding CacheRecords in a queue nothing would drain. Route both
through _dispatch_deferred, which drops the item when no worker is running.
Dropping loses nothing: such a cache has nothing to evict, and put() re-runs a
pending reconcile through the synchronized release hook when it admits one.
- shutdown()'s early return meant a keep-alive timer re-armed by a
post-shutdown put() was never cancelled by a later shutdown(). Don't arm
timers on a shut-down cache.
Tests: the two *_thread_start_failure_* tests installed their monkeypatch after
put() had already started the worker, and c384ea187d removed the only
Thread.start() from those paths — the patched raise was unreachable, so both
passed without exercising their premise. Retargeted to what they actually
verify (the finalizer and the lookup must not block, and the reconcile still
happens). Four regression tests added; each fails against c384ea187d.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): stop the deferred worker outliving and pinning its cache
Second round of adversarial-review follow-ups on the deferred-work thread.
- The worker held a bound method, so a running thread — reachable from
threading._active — kept its ModelCache alive, and with it every CacheRecord
and every model's CPU weights. A cache released without shutdown() was
therefore immortal, which is the opposite of what RamBudget's weakref registry
is built for. Measured against the parent commit 7ffee4db04: five caches
dropped without shutdown() left 5/5 caches and 5/5 models resident and five
worker threads running, where the parent left 0/5 and no threads. The previous
`ident is None` guard had accidentally bounded this (a cache whose worker died
could never re-acquire a pinning thread); reviving the worker removed that
bound, so the fix has to remove the strong reference itself. The worker is now
a module-level function taking a weakref, and a weakref.finalize pushes
_DEFERRED_STOP when the cache is collected so the parked thread exits instead
of leaking one thread per abandoned cache.
- Thread.start() is called from put(), which runs under both the cache lock and
MODEL_LOAD_LOCK's write lock while completing a load. Under thread/pid
exhaustion (RLIMIT_NPROC, a container's pids.max) its RuntimeError escaped and
failed a generation whose model had already been fully constructed — to lose
an optimization that _dispatch_deferred is explicitly designed to survive the
absence of. Log and carry on; the next admission retries.
- _dispatch_deferred justified dropping work with "a cache without a worker has
never admitted a model". That was false: put() after shutdown() is reachable
in production, because Invoker.stop() stops model_manager before
session_processor, so an in-flight generation can admit a model after every
cache has been shut down — and thread exhaustion reaches the same state
without a teardown to bound it. Such a record was admitted with the first-use
grace, then permanently shielded from both asynchronous eviction paths with
its bytes still charged to the shared budget. Make the claim true instead of
rewording it: put() grants the grace only when a worker is running to release
it. lock() still clears it on the normal path, so nothing changes when the
worker is healthy.
Tests: the post-shutdown half of the drop test never set the pending flag, so
cached_model_keys() short-circuited before reaching the dispatch guard and the
assertion held unconditionally. Poll the budget in the pin test rather than
reading it the instant the key disappears (_delete_cache_entry pops before it
releases the weights). Two regression tests added; all seven of this series'
new tests fail against c384ea187d.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): release abandoned RAM budget
* fix(model cache): pin LoRA patches during use
* fix(model cache): harden the LoRA pin path and close its review gaps
Review fixes for the LoRA-pinning commit (1142430f87):
- Fix the CI failure it shipped: test_krea2_text_encoder's fake
LoadedModel lacked model_in_ram(), and the encoder's LoRA iterator now
calls it. The fake now models the pin (with depth tracking), and the
test asserts the patch spec carries a working pin.
- Stop a keep-alive Timer.start() failure from leaking a permanent pin.
@record_activity runs after lock_in_ram() has incremented the lock
count but before model_in_ram()'s unlock-pairing try block is entered,
so a RuntimeError under thread/pid exhaustion would pin the record
(and its shared-budget bytes) for the life of the process. The timer
is an optimization: log and continue instead.
- Give lock_in_ram() the same already-dropped-record diagnostic as
lock()/unlock(), so a pin on a detached record produces a matching
lock-side message (issue 7513).
- Pin the LoRA cache record in LoRAExt.patch_unet (the modular-denoise
path) while its tensors are read during direct patching. This was the
one remaining producer that dropped its LoadedModel handle at load
time, leaving the record evictable by a peer cache mid-patch.
- Close test vacuities: the pin-retention test in test_layer_patcher
could not detect a dropped cache_pins.close() (the ExitStack would be
collected silently), and the cache-side pin test only covered a warm
record, leaving lock_in_ram's grace-clearing dead code under test.
Added pin-release assertions for the normal, body-raise,
restore-raise, and mid-materialization-raise paths (all verified to
fail with close() neutered), a cold-record grace/finalizer test, and
a Timer-failure regression test (verified to fail pre-fix).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(model cache): close the remaining raise-after-lock leak and widen the LoRAExt pin
Self-review fixes for d63c1f37e0:
- The synchronized-decorator's post-release reconcile hook runs inside
the caller's frame after the method body, so a raise there (e.g.
TorchDevice.empty_cache on a sick CUDA context after an eviction)
escaped lock_in_ram()/lock() after the lock count was incremented —
the same permanent-pin leak as the Timer.start() case, via a
different path. The reconcile is deferrable housekeeping: swallow and
log; the pending flag is only cleared once the budget is satisfied,
so the next lock release retries. Regression test verified to fail
pre-fix.
- LoRAExt.patch_unet's pin now spans the yielded scope, not just the
patch application: despite force_direct_patching=True, fp8-storage
modules are routed to sidecar patching (float8 weights cannot be
patched in place), which stores a live reference to the cached
patch's layers inside the UNet for the whole denoise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com>
Co-authored-by: JPPhoto <jpollack@jpollackphoto.com>
Brings in 18 upstream commits: video generation (invoke-ai#9163), Krea-2 (invoke-ai#9304), Ideogram 4 (invoke-ai#9303), Wan 2.2, pressure-dependent brush opacity (invoke-ai#9232), GGUF T5 / Qwen3 tokenizer support, and a queue-status performance refactor (invoke-ai#9355). Nine files conflicted. Seven were the fork's wildcard_records service landing on the same lines as upstream's videos/gallery services — both sides kept. session_queue_sqlite.get_queue_status took upstream's lightweight 4-column current-item query (which avoids deserializing the session graph on every status poll) and carries the fork's origin_prefix filter into it, so an origin-scoped caller still cannot see another scope's current item. Added a negative-path test for that: the existing origin-scope test only covered a matching item, so it would have passed with the filter dropped. test_session_queue_status_event_isolation lost the get_current monkeypatch race test, per upstream — the single-snapshot read makes the race structurally impossible, and the monkeypatch would have become a silent no-op. diffusers moves 0.37 -> 0.39 and imageio[ffmpeg] / psutil are new, so this needs `uv sync`.
Upstream made `/images/i/{name}/full` and `/thumbnail` authenticated in
multiuser mode. `<img>` cannot send a bearer header, so they now accept the
path-scoped HttpOnly cookie that login sets instead — which means every
thumbnail in webv2 goes blank on a multiuser backend without this.
apiFetch now states `credentials: 'same-origin'` rather than relying on the
fetch default, so the login Set-Cookie is pinned by a test instead of by
coincidence. `init` still wins, leaving cross-origin callers room for
'include' (which also needs Access-Control-Allow-Credentials, so requesting
it unconditionally would break those setups rather than fix them).
Session restore additionally re-issues the cookie via POST /auth/media-cookie.
That path is the one case holding a valid JWT with no cookie — the session may
predate the cookie or have had it cleared — and it fails silently: every other
API call works while the gallery renders blank. The call is best-effort, since
broken media is a degraded gallery, not a signed-out session.
Boards gained `video_count` / `cover_video_name`, and date boards are now derived from the polymorphic gallery service, so a board or date can be non-empty on videos alone while `image_count` reads 0. Board covers resolve through either name — both are static WebP thumbnails, so the cover renders the same either way. The uncategorized pseudo-board is assembled client-side and so has no DTO to read `video_count` from; it gets its own total rather than a hardcoded 0. Image deletion now reports per-name outcomes: the backend stopped aborting a batch on the first failure, so a request can partly succeed. Only the names it actually deleted are evicted from the caches — treating the whole request as successful would hide a still-present image until the next full refresh — and a partial failure is reported instead of a false "Deleted image". Adds video media path/URL helpers alongside the image ones.
Registers the three bases upstream added, plus the qwen3_vl_encoder and wan_t5_encoder model types/formats. BASE_GENERATION drives SupportedGenerateBase, so adding the entries made the graph-builder and canvas-encode registries compile-time-required — which is the intended forcing function. All three use a 16px grid. Wan's comes from its transformer patch-embedding with stride 2 over the VAE's 8x scale; without it the scheduler step fails on a latents-vs-noise spatial mismatch. Krea-2 decodes with the Qwen-Image VAE, so its graph and canvas encode reuse that family's l2i/i2l nodes. Negative conditioning is attached only above cfg_scale 1, since Krea-2-Turbo ships with CFG off and the model takes no negative input there. The rebalance and seed-variance enhancers default off and chain in that order between the encoder and denoise. Ideogram 4 routes the prompt through its caption builder and has no negative conditioning input at all. Its steps/guidance/mu come from the sampler preset, so unset overrides are omitted from the node rather than sent as null — sending null would override the preset with nothing. It also has no image-to-latents node, so it is deliberately absent from CANVAS_I2L_NODE_TYPES and canvas generation reports it as unsupported. Wan emits a single-frame image (wan_l2i, not wan_l2v) and always supports a negative prompt. The low-noise A14B expert stays optional: without it the high-noise expert covers the whole schedule at lower quality. Two rules needed explicit handling. Krea-2 rebalance weights are free text forwarded to the backend parser, so a malformed string is caught before it can fail a queued item. And Wan LoRA variants (`a14b` / `5b`) name the same families as main variants (`t2v_a14b` / `i2v_a14b` / `ti2v_5b`) with different strings, so plain variant equality would have rejected every Wan LoRA while letting a genuinely mismatched one through — the families are not interchangeable and crash the layer patcher on a tensor-shape mismatch. Also narrows getActiveCompatibleLoras to Pick<GenerateSettings, 'loras'>, which is all it reads. That deleted a synthetic 25-field GenerateSettings literal in the upscale graph whose values were meaningless, along with the now-dead dimensions lookup feeding it.
Upstream split its single pressureSensitivity flag into pressureAffectsWidth and pressureAffectsOpacity. webv2 already had pressure-sensitive width via perfect-freehand thinning; this adds the opacity half. No persisted-state migration is needed here, unlike upstream's Redux slice: webv2's brush options are in-memory engine state seeded from DEFAULT_BRUSH_OPTIONS, so nothing on disk carries the old key. The renderer needed real work. strokeSession fills the whole stroke into a scratch surface at full alpha and composites it into the layer exactly once at the stroke opacity — that single composite is what stops a slow drag, which revisits the same pixels dozens of times, from compounding into a dark blob. Alpha varying along the stroke cannot be one globalAlpha on one composite. So the stroke is filled as bands: contiguous runs of samples at one quantized alpha (pressureBands.ts). Each band replaces its own footprint — destination-out its outline, then source-over at the band's alpha — instead of blending into it. Replacing preserves the no-compounding guarantee, and where bands overlap the later one wins, which is the pressure being applied now. Bands are quantized to 16 levels because each costs an outline computation and two fills; per-sample bands would be far more work for a difference below what reads on a soft edge. Consecutive bands share a sample so their outlines abut rather than leaving a hairline seam at the level change. Enabling it forces a full-region scratch refill per frame, since a band straddling the changed strip would be half-punched by a partial refill and lose its earlier half. That cost is why the flag defaults off, matching upstream. Mask strokes and the eraser stay pressure-opaque: a mask is an all-or-nothing alpha stencil, and a partially transparent one would silently attenuate denoise strength; a pressure-faded erase reads as a failed erase rather than a soft one. Non-finite pressure floors to the minimum band alpha rather than propagating — clamping alone let NaN reach globalAlpha, where a non-finite value is ignored and the band would have painted at full alpha, the opposite of a light touch. Upstream's konva/pressure.ts does not port; it is written against Konva's renderer. The two switches are ToggleDots (aria-labelled buttons), so unlike sibling Chakra switches sharing a Field.Root they cannot collide on a hidden-input id.
The chunk source-owner sets are compared by equality, so the four modules added by the model-family and pressure work fail both gates until re-recorded. Verified as additions rather than regressions before re-recording: - editor initialRawBytes +10550 (brotli +2130) for ~700 lines of new UI and stroke-band logic; every numeric byte limit still passed on its own. - otherRawBytes +1454, which matches the +1450 bytes the new i18n keys added to en.json — every route fetches it, which is why the overage appeared on all of them identically at the same value. - Timing metrics moved by a few ms, within run-to-run variance. Also routes GenerateModelFamilyFields at the @platform/ui subpaths rather than the barrel. The barrel was at its direct-importer budget (156) and the new component made 157. Depending on Field and Select directly is what that budget is asking for — the module now invalidates only when those two change instead of on any barrel change — so this is the remediation, not a raised cap.
The measurement script writes plain JSON.stringify output, which oxfmt rejects — and format:check is part of `pnpm run lint`, so an unformatted baseline fails the lint gate after every re-record.
krea2_model_loader declares ui_model_base=[QwenImage, Anima] for its VAE slot, but the picker filter, the retain-on-model-switch check, and the graph builder all matched base 'qwen-image' only. The same physical Qwen-Image VAE is registered under either base depending on which family it was installed for, so a VAE installed for Anima was hidden from the dropdown — and would have been dropped by the builder even if selected another way. Setting such a model's base to 'any' would not have worked around it either: isVaeForBases tests includes(model.base), so 'any' fails a ['qwen-image'] check just as 'anima' did. All three sites now share one isKrea2Vae filter so they cannot drift again, with tests pinning that the picker filter and the validator agree — a narrower picker than validator hides a VAE that validation then demands. Audited the other slots added alongside Krea-2 against their nodes: Wan's VAE (base wan), Wan T5 encoder (type only), low-noise expert (base wan, type main), component source (diffusers wan main) and Krea-2's Qwen3-VL encoder (type only) all already match wan_model_loader / krea2_model_loader.
webv2 carried fp8_storage in MainModelDefaultSettings but rendered no control, so the setting was unreachable — v1 has DefaultFp8Storage.tsx and webv2 had nothing. Adds it to the per-model default-settings section. Availability mirrors the backend's _should_use_fp8 rather than v1's UI condition, so the toggle is never offered where it would be silently ignored: Z-Image (diffusers' layerwise casting hits a dtype mismatch between skipped and hooked modules), LoRA and ControlLoRA (patched into a base model rather than run as their own forward pass, so the hooks never fire), and VAEs (fp8 measurably degrades decode). v1 only checks Z-Image; the ControlLoRA exclusion is load_default.py's, whose comment says the UI is expected to hide it. The field deliberately has no body control. _should_use_fp8 acts only on `fp8_storage is True`, so an explicit false and an absent value behave identically — the row's existing enable switch already distinguishes the two states that differ, and a second switch inside the card would imply a distinction that does not exist. FieldSpec.control is now optional to express that. Also collapses two copies of DefaultSettingsModel into one exported type. ModelDetail declared its own structural duplicate, which is why adding `base` to the section's version failed to typecheck against the selector feeding it. Note this does not address the Krea-2 OOM: fp8_storage is applied after load_state_dict, well past the fp32 dequantization spike that kills the process.
…r FLUX / FLUX.2 / SD3 / SDXL / Z-Image / Qwen-Image (invoke-ai#9281) * feat(pid): vendor PiD decoder backend (phase A of integration) Adds a vendored subset of NVIDIA's PiD (Pixel Diffusion Decoder) at invokeai/backend/pid/ as the foundation for upcoming FLUX / FLUX.2 / SD3 / Z-Image PiD decode nodes plus a future PiD-based 4x upscale node. Upstream: https://github.com/nv-tlabs/PiD (Apache 2.0). Vendor scope: * _src/{networks,models,modules}: PidNet, PixDiT_T2I, LQProjection2D, PidModel, PidDistillModel, PixelDiTModel, GeneralConditioner. * _ext/imaginaire: minimal Imaginaire framework subset (lazy_config, model, utils/{log,misc,distributed,device,count_params}). * configs/, tokenizers/, checkpointer/, trainer.py, visualize/, _demo_*, from_*, easy_io/, S3/wandb training helpers were intentionally excluded. Dependency stripping (no new hard deps introduced): * loguru, termcolor -> stdlib logging shim * iopath PathManager -> stdlib pathlib stub * fvcore Registry -> minimal stdlib Registry * lazy_config/lazy.py: yaml/dill/cloudpickle/detectron2 save/load paths replaced with a minimal LazyCall stub * lazy_config/instantiate.py: omegaconf DictConfig/ListConfig branches removed; configs are plain dict / LazyCall mappings * megatron, pynvml, boto3/wandb imports are try/except-guarded or local to functions and stay inert in our inference path All pid.* imports rewritten to invokeai.backend.pid.*; SPDX-Apache-2.0 headers retained on vendored files; attribution and detailed list of local modifications added in LICENSE-PiD.txt. The pre-trained PiD checkpoints distributed by NVIDIA remain under NSCLv1 (non-commercial); this commit only vendors code. Smoke test: PidNet, PidModel, PidDistillModel, GeneralConditioner import cleanly; LazyCall -> instantiate round-trip resolves to the expected nn.Module. ruff check passes. * feat(pid): wire PiD + Gemma-2 into model-manager and add decode nodes Adds the model-manager plumbing and workflow nodes needed to use the vendored PiD decoder (phase A) end-to-end with FLUX, SD3 and Z-Image. Model manager (Phase B + B.5): * taxonomy: ModelType.PiDDecoder, PiDDecoderVariantType (Res2k_Sr4x / Res2kTo4k_Sr4x), ModelType.Gemma2Encoder + ModelFormat.Gemma2Encoder, both added to AnyVariant + variant_type_adapter. * configs/pid_decoder.py: per-backbone PiD configs (FLUX / FLUX.2 / SD3) with state-dict probing on 'lq_proj' substring and backbone/variant detection from the official NVIDIA filenames. * configs/gemma2_encoder.py: Gemma-2 directory probing on Gemma2ForCausalLM architecture + tokenizer files. * AnyModelConfig union updated. * model_loaders/pid_decoder.py: loads .pth / .safetensors, strips the upstream 'net.' prefix, supports torch.load(weights_only=True). * model_loaders/gemma2_encoder.py: SubModelType.{Tokenizer, TextEncoder} dispatch; returns the causal LM's inner Gemma2Model (transformers 4.56's get_decoder() returns None for Gemma2). Decode pipeline (Phase C): * backend/pid/decode.py: build_pid_net + load_pid_decoder (per-backbone PixDiT_T2I hyperparams derived from PiD's pid_sr4x base + per-experiment overrides), encode_caption_for_pid (chi-prompt + Gemma encoding, mirrors PixelDiTModel._encode_text_raw), and a PiDDecoder wrapper with a reimplemented few-step distill sampler (no autocast / no distributed / no PixelDiTModel init paths from upstream). Invocations (Phase 6.x): * Gemma2EncoderField + PiDDecoderField in invocations/model.py. * gemma2_encoder_loader / pid_decoder_loader: thin ModelIdentifierField pickers that emit the corresponding fields. * z_image_pid_decode (pilot), flux_pid_decode, sd3_pid_decode: caption encode -> Gemma offload -> PiD state dict load -> PidNet construct -> decode. Per-backbone latent denormalisation (FLUX1 ae_params, SD3 hardcoded 1.5305/0.0609, Z-Image piggybacks on FLUX VAE). End-to-end validated with the released PiD_res2k_sr4x_official_flux_distill_4step.pth checkpoint and gemma-2-2b-it: PidNet rebuilds at exactly 456 keys / 1.36B params, sampler runs at ~5 GB VRAM peak (Gemma dominates), output shape and range match. FLUX.2 PiD decode is deliberately deferred: it needs BN-based latent denormalisation and 32->128 channel packing, and we have no FLUX.2 checkpoint to validate against yet. * feat(pid): end-to-end PiD pixel-diffusion decoder integration Adds the NVIDIA PiD decoder as a 4x super-resolution alternative to the regular VAE/RAE decode path. Includes model-manager configs and loaders for both the PiD checkpoints and the Gemma-2 caption encoder they require, plus four invocations: latent-in decode for FLUX / SD3 / Z-Image and an image-in pid_upscale node. - Decode pipeline keeps PidNet params in fp32 and uses bf16 autocast only for matmuls; caption embeddings have outliers that overflow bf16 RMSNorm. - encode_caption_for_pid forces tokenizer padding_side="right" (Gemma defaults to left, PiD trained with right) and returns the attention mask as bool so it stays compatible with SDPA. - Z-Image reuses the FLUX-trained checkpoint and reads scale/shift from the VAE config at runtime (PiD upstream notes they are checkpoint-specific). - TextLLM config now excludes Gemma2ForCausalLM so it falls through to the dedicated Gemma2 encoder config instead of being misclassified. - Frontend: new model_type / model_format / variant enums, type guards and category metadata; schema.ts regenerated via pnpm typegen. * Chore Ruff * Chore Typegen * Chore Knip * fix(pid): remove unused vendored models/utils.py (broken easy_io import) * feat(pid): identify decoder backbone from weight shapes, not filename Read latent channel count from lq_proj.latent_proj.0.weight (FLUX.2=128, FLUX.1/SD3=16) as the primary discriminator; fall back to filename/dir name only to disambiguate the architecturally identical FLUX.1/SD3 pair. Fixes FLUX.2 checkpoints (model_ema_bf16.pth) not being recognised, and correctly rejects unsupported backbones (RAE/dinov2, 768ch). Fix Flux2 docstring 32->128. * feat(ui): PiD decode (Fit mode) for FLUX text-to-image Add a "PiD Decode" mode select (Off / Fit / Native) to the FLUX advanced settings with PiD decoder + Gemma-2 encoder pickers. In Fit mode the FLUX graph swaps the VAE decode for a PiD 4x super-resolution decode and downscales back to the requested size. Adds params state (pidMode, decoder, encoder, steps) with a v3->v4 migration, model hooks, readiness checks, and graph guards for the not-yet-wired Native and non-txt2img paths. * feat(ui): PiD Native 4x mode for FLUX text-to-image Make the generation dimension helpers PiD-aware via an optional pidScale: in Native mode the user-facing dimensions are the 4x target (grid 64, optimal 2048), generation runs at target/4, and PiD's 4x output is used directly with no downscale. Thread pidScale through the params dimension reducers and the optimal-dimension/grid-size selectors, resync dimensions when toggling Native, and wire the Native path in the FLUX graph builder. Add working_mem_bytes for PiD Decode * feat(ui): PiD Fit decode for FLUX image-to-image Extract the PiD decode chain into buildPidDecodeChain (loaders + decode + fit-downscale, no denoise setup) so it can substitute for the VAE decode across generation modes. Widen addImageToImage's l2i param to ImageOutputNodes (it only consumes .image) and wire the PiD chain into the img2img branch in Fit mode. Native stays txt2img-only (a 4x result can't composite onto the bbox); inpaint/outpaint remain gated off for now. * feat(ui): PiD Native 4x decode for FLUX image-to-image Add addPidImageToImageNative: the canvas bbox is the 4x target, so the init image is downscaled to bbox/4, denoised at that resolution, and PiD decodes straight back up to the full bbox with no post-decode downscale - preserving all PiD detail while still compositing cleanly onto the region. Wire it into the img2img branch of buildFLUXGraph (native vs fit vs off) and drop the native-txt2img-only guard. Make the canvas FLUX grid check PiD-aware so a native bbox must be a multiple of 64 (16 * 4) for bbox/4 to land on the grid. * feat(ui): add informational popover to PiD Decode setting Explain PiD usage on hover, mirroring the DyPE popover: what the decoder is (NVIDIA Pixel Diffusion Decoder, 4x SR, needs a PiD decoder + Gemma-2 encoder), Fit vs Native modes, the 2K / 2K-to-4K target resolutions, that Steps can be lowered, and that Scale Before Processing must be off. Links to nv-tlabs/PiD. * feat(models): add PiD decoder + Gemma-2 encoder to starter models Register NVIDIA's PiD FLUX decoders (2K and 2K-to-4K presets, from nvidia/PiD) and the Efficient-Large-Model/gemma-2-2b-it caption encoder as starter models so they can be installed from the Model Manager. The Gemma-2 encoder is wired as a dependency of each decoder (and offered standalone). * feat(pid): add FLUX.2 Klein PiD 4x-SR decode support Add a flux2_pid_decode node that packs the stored FLUX.2 latent (32ch @ H/8) into PiD's 128ch @ H/16 layout before decoding; FLUX.2's BatchNorm denormalization is already applied in flux2_denoise, so no scalar denorm is needed (optional vae input reads identity constants). Generalize the frontend PiD decode chain (decodeNodeType, optional vaeSource) and wire the isFlux2 graph path for txt2img/img2img (Fit & Native). Base-aware PiD gating/decoder-filter, FLUX.2 readiness checks, and two nvidia/PiD FLUX.2 starter decoders (2K, 2Kto4K). Standard FLUX PiD path unchanged. * feat(pid): add SD3 PiD 4x-SR decode support Wire the existing sd3_pid_decode node into the SD3 graph builder (txt2img and img2img, Fit & Native) with a PiD guard, base-aware gating/decoder-filter (sd-3), and SD3 readiness checks. Add two nvidia/PiD SD3 starter decoders (2K, 2Kto4K). Harden the PiD config probe against the 16-channel FLUX.1/SD3 ambiguity: when the checkpoint's directory name is silent (the HF single-file download renames it), trust an explicit base override so SD3 checkpoints are not misidentified as FLUX.1. Also benefits Qwen. FLUX / FLUX.2 identification is unchanged. * feat(pid): add SDXL PiD 4x-SR decode support Build the full SDXL PiD backend stack: _PER_BACKBONE[SDXL] (4ch/down8), PiDDecoder_Checkpoint_SDXL_Config with a 4-channel latent-map entry, factory union + loader registration, and a new sdxl_pid_decode node (reads the VAE's scaling_factor/shift at runtime; SDXL fallbacks 0.13025/0.0). 4-channel latents are unambiguous, so no directory-name disambiguation is needed. Generalize the shared PiD decode chain to support SD-family denoise: denoise_latents has no width/height, so thread an optional noise node for sizing and round to the model's native grid (8 for SDXL, 16 for FLUX). Wire buildSDXLGraph (txt2img + img2img, Fit & Native) with the VAE as the decode's scaling source, base-aware gating/readiness, and a starter decoder (SDXL 2Kto4K only). PiD + SDXL refiner is blocked for now via a graph guard and a readiness reason. FLUX/FLUX.2/SD3 paths are unchanged. * feat(pid): add Z-Image PiD 4x-SR decode support Wire the existing z_image_pid_decode node into the Z-Image graph builder (txt2img and img2img, Fit & Native) with a PiD guard and readiness checks. Z-Image shares FLUX.1's 16-channel VAE and has no PiD checkpoints of its own, so it reuses the FLUX decoder: the decoder filter maps z-image -> flux, showing FLUX PiD decoders when a Z-Image model is active. The Z-Image VAE is passed to the decode node so it reads the real scaling_factor / shift instead of the fallback constants. No backend, schema, or starter-model changes. FLUX/FLUX.2/SD3/SDXL paths are unchanged. * feat(pid): add Qwen-Image PiD 4x-SR decode support Build the full Qwen-Image PiD backend stack: _PER_BACKBONE[QwenImage] (16ch/down8), PiDDecoder_Checkpoint_QwenImage_Config (added to the 16-channel latent map + filename heuristic), factory union + loader registration, and a new qwen_image_pid_decode node. Unlike the scalar-scaling bases, the Qwen-Image VAE normalizes per channel (latents_mean / latents_std) and stores a 5D video-style latent, so the node denormalizes per-channel (z * std + mean, read from the VAE config) and drops the singleton temporal frame before decoding - matching qwen_image_l2i. Wire buildQwenImageGraph (txt2img + img2img, Fit & Native) with the Qwen-Image VAE as the decode's normalization source, base-aware gating/readiness, and a starter decoder (Qwen-Image 2Kto4K only). The 16-channel FLUX/SD3/Qwen ambiguity is handled by the existing trusted- base-override probe hardening. FLUX/FLUX.2/SD3/SDXL/Z-Image paths are unchanged. * Chore Ruff * Add Docs * fix(pid): green up frontend tests and knip for the PiD branch - graph-builder tests: set pidMode 'off' in the FLUX / Qwen-Image / SDXL+SD3 param fixtures so the PiD guard doesn't fire on an undefined pidMode and call the (unmocked) size helpers - paramsSlice migration test: expect _version 4 (v3→v4 adds the PiD fields) - remove the unused setPidSteps action and selectPidSteps selector flagged by knip; the pidSteps state field stays at its default of 4 * Chore openapi + typegen * docs: regenerate invocation-context data for offload_from_vram * fix(pid): resolve PiD decoder review findings (state, readiness, steps, race) Address all reviewer findings on the PiD decoder feature: - Guard offload_model_from_vram with @synchronized, matching its sibling drop_model, to prevent a race when the VRAM working set is mutated concurrently during model swaps. - On base change (modelChanged), clear a PiD decoder that is incompatible with the new main model's decoder base at the root, respecting the Z-Image -> FLUX decoder reuse so a still-valid decoder is kept. - On switching to a base without PiD support, reset pidMode to 'off' and refit the dimensions so no hidden 4x native grid survives. - Extend the scaled-grid bbox readiness validation to SD3, SDXL and Z-Image, mirroring the existing FLUX.2 native-grid check. - Add a PiD Steps control (slider + number input, 1-8, default 4) with a pidStepsChanged action and selectPidSteps selector, so the documented step count is actually configurable and flows into the graph. Add readiness and paramsSlice tests covering the scaled-grid blocking and the base-change decoder/pidMode behavior * fix(pid): address review — cap steps at 4, honor encoder device, decoder/base guards Merge blockers: - PiD steps 5-8 produced duplicate timesteps: the student schedule has only 4 transitions (a 5-point list), so sub-sampling to >4 steps rounded distinct indices onto the same point and wasted network forwards on repeated timesteps. Cap the public range at 4 across the backend fields (le=4), the UI slider/input (max=4), and the pidSteps zod schema (int, 1-4), and harden _get_t_list with a strictly-decreasing assertion as a safety net. - CPU-only Gemma encoders crashed on CUDA hosts: each PiD invocation passed the global compute device to caption encoding, pushing the tokenizer output to CUDA while a cpu_only encoder stayed on the CPU. Encode on the encoder's actual device instead (next(encoder.parameters()).device), honoring model_on_device(). - Gemma 2 could no longer be configured as a generic TextLLM: the specialised- architecture exclusion rejected Gemma2ForCausalLM unconditionally. Only defer to the encoder config during automatic classification; keep an explicit type=text_llm request valid (the generic causal-LM loader supports it). Follow-ups folded in: - Reject incompatible Gemma 2 sizes before execution: PiD's caption projection is fixed at Gemma-2-2b's 2304-dim hidden state, so the encoder config now rejects 9B (3584) / 27B (4608) up front instead of failing deep in inference. - Validate the PiD decoder's base against each base-specific decode node: the base-agnostic loader let the Nodes editor wire any decoder into any node. Add assert_pid_decoder_matches_base and call it in all seven decode nodes, preserving the Z-Image-reuses-FLUX-decoder case (its node backbone is FLUX). Add regression tests: the distill schedule (strictly decreasing 1-4, safety net trips at 5) and decoder/base validation; the Gemma2 hidden-size gate; and TextLLM classification (auto-defers, explicit type still matches, plain causal LMs match). The expand-prompt pipeline always sent a dedicated "system" role message, which some chat templates (notably Gemma) reject with "System role not supported", 500-ing prompt expansion for those models. When applying the chat template fails with a system-role error, fold the system prompt into the first user turn and retry instead of failing. Adds regression tests for both the fallback and the normal (system-supported) path. * Chore fix * fix(pid): keep large Gemma2 as TextLLM, raise (not assert) schedule guard, compute_device, narrow pid_upscale VAE Address the latest review on the PiD PR: - Merge blocker: automatic classification sent Gemma 2 9B/27B to Unknown. The PiD Gemma2 encoder config rejects their non-2304 hidden size, and TextLLM deferred *every* Gemma2ForCausalLM, so neither matched. TextLLM now defers only the size the encoder config accepts (2304 = Gemma-2-2b); larger variants stay TextLLM. - Schedule safety net used assert, which is stripped under `python -O`, leaving _get_t_list(num_steps=5) returning a duplicate schedule. Raise ValueError instead so the guard holds in optimized runtimes; the regression test now asserts ValueError and passes under `python -O`. - All seven PiD caption paths derived the Gemma device from the first parameter, which is wrong under partial loading (first param on CPU, later modules on CUDA). Use the cache contract's LoadedModel.compute_device instead. - pid_upscale advertised Z-Image / 16-channel VAEs but delegates to the FLUX-only vae_encode. Narrow the field description and validate the VAE is a FLUX AutoEncoder up front (a diffusers AutoencoderKL now fails with a clear error instead of a stripped-assert failure inside vae_encode). Update the TextLLM/Gemma2 tests (per-size config-level + a factory-level check that 2304 -> Gemma2Encoder and 3584/4608 -> TextLLM) and the schedule test (ValueError, green under python -O). * chore: regenerate OpenAPI schema and frontend types * feat(pid): accept single-file GGUF Gemma-2-2b as the PiD caption encoder The PiD Gemma encoder was directory + HuggingFace only, so a llama.cpp GGUF (e.g. gemma-2-2b-it-Q4_K_M.gguf) could not be used. Add GGUF support: - Gemma2Encoder_GGUF_Config: identifies a single .gguf file, reads the GGUF metadata and requires general.architecture == "gemma2" and <arch>.embedding_length == 2304 (Gemma-2-2b), rejecting 9B/27B as the directory config does. - Gemma2EncoderGGUFLoader (format gguf_quantized): loads via transformers from_pretrained(<dir>, gguf_file=<name>) — transformers dequantizes gemma2 GGUFs and reads the tokenizer from the GGUF metadata — then exposes the Gemma2Model decoder, matching the directory loader. PiD encodes the caption once and offloads the encoder, so dequantizing at load is acceptable. - Register the config in the AnyModelConfig union. No frontend change: the PiD encoder picker filters by type=gemma2_encoder, so the GGUF variant appears automatically. Verified end-to-end against a real q4_k_m file: it classifies as Gemma2Encoder_GGUF_Config and loads to a Gemma2Model producing 2304-dim hidden states. Adds config identification tests (match, 9B/27B rejected, non-gemma2 rejected, non-.gguf rejected). * Chore Ruff * fix(models): stop Qwen3 GGUF config from matching Gemma-2 GGUF encoders A Gemma-2 GGUF satisfies the generic Qwen3 GGUF key heuristic (token_embd.weight + blk.* keys), so it matched both Qwen3Encoder_GGUF_Config and the intended Gemma2Encoder_GGUF_Config. On a fresh install the Gemma config happened to win, but re-identification could pick Qwen3, mis-classifying the model. Add _has_gemma2_keys (Gemma uses blk.*.post_attention_norm / post_ffw_norm, which a Qwen3 encoder never has — Qwen3 has attn_q_norm/attn_k_norm instead) and reject such state dicts in both Qwen3 encoder configs' _validate_looks_like_qwen3_model (GGUF and checkpoint), mirroring the existing T5 / Qwen-VL exclusions. The Gemma config already rejects Qwen3 GGUFs via the general.architecture metadata, so the two are now mutually exclusive and identification is deterministic. Add regression tests: _has_gemma2_keys detection and that the Qwen3 GGUF config rejects a Gemma-keyed state dict. * test(models): assert re-identified Gemma GGUF drops the stale Qwen3 variant The Gemma2 GGUF encoder config has no `variant` field, so re-identifying a model previously mis-detected as a Qwen3 GGUF (which carries a variant) drops it — the serialized record has no variant key and replace_model overwrites it away. Assert this explicitly in the Gemma GGUF identification test. * fix(pid): point 2K-to-4K starter decoders at NVIDIA's v1.5 replacements NVIDIA deprecated the FLUX / FLUX.2 / Qwen-Image `res2kto4k_sr4x` PiD decoders and moved them to `checkpoints_deprecated/`, replacing them with the recommended `v1pt5_res2kto4k_sr4x` checkpoints. Our starter models still pointed at the old `checkpoints/` paths, which now 404 on install. Repoint the three affected 2K-to-4K starters (FLUX, FLUX.2, Qwen-Image) to the v1.5 successors and note the upgrade in their descriptions. The 2K (`res2k_sr4x`) decoders and the SD3 / SDXL 2K-to-4K decoders are not deprecated and are unchanged. Base and variant are still sent as explicit overrides, so config identification is unaffected by the new directory name (res2kto4k -> Res2kTo4k_Sr4x). * feat(pid): load Gemma-2 GGUF encoders natively (keep weights quantized) The GGUF Gemma encoder used transformers' from_pretrained(gguf_file=...), which dequantizes every weight at load — so a quantized Gemma cost the same VRAM as the unquantized model. Load it via InvokeAI's GGMLTensor path instead: read the config from GGUF metadata, map llama.cpp tensor names to Gemma2Model, and keep the 2D projection weights as GGMLTensor (dequantized on demand by the model cache). Materialize only the embedding and the RMSNorm weights, subtracting 1 from the norms (llama.cpp folds +1 in; Gemma2RMSNorm re-adds it), and assert nothing is left on meta. Verified: hidden states match the fully-dequantized loader within quantization tolerance. Adds key-mapping tests and a local load/compare test. NVIDIA's v1.5 decoders use a different network (lq_hidden_dim=1024, PiT injection) that build_pid_net (512-dim legacy) cannot load, causing a size-mismatch crash. - Point the FLUX/FLUX.2/Qwen 2K-to-4K starters back at the legacy checkpoints (moved to checkpoints_deprecated/) that the current network loads. - Reject a checkpoint whose lq_proj hidden dim is not the supported 512 at identification time, instead of accepting it and failing inside the decode. - Enumerate all supported backbones (add SDXL, Qwen-Image) in the loader title and correct the variant enum docs (not every backbone ships both presets). Full v1.5 architecture support is planned as a follow-up. Adds PiD decoder identification tests (legacy accepted, 1024-dim v1.5 rejected). * Chore openapi + typegen * Docs update --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: JPPhoto <jpollack@jpollackphoto.com>
Loading a ComfyUI MXFP8 Krea-2 transformer crashed in _dequantize_scaled_fp8 with "The size of tensor a (3072) must match the size of tensor b (384)". The Krea-2 loader carried its own copy of the scaled-fp8 dequantization, and it had drifted behind the Qwen-Image one in three ways: 1. Block-wise scales were multiplied in raw. Microscaling stores one scale per 32 values along the last dim, so a (6144, 6144) weight carries a (6144, 192) scale, which cannot broadcast. The scale needs repeat_interleave first. This is the crash: all 256 quantized tensors in a real Krea-2 MXFP8 checkpoint are block-scaled, none scalar. 2. E8M0 scales were treated as multipliers. MXFP8 stores the scale as a uint8 biased power-of-two exponent, so the multiplier is 2**(scale - 127). Using the raw byte does not crash — it inflates weights by ~10^6. Measured on the real checkpoint: std 0.055 decoded correctly versus std ~14000 raw. Fixing only the shape would have replaced a crash with silent garbage output, which is worse. Dispatched on dtype, since a linear scale is never an integer. 3. Dequantization went via .float(). That materialises 4 bytes/param before a separate downcast, so a 12.5 GB fp8 transformer peaked at 50 GB of host RAM and was OOM-killed on a 32 GB machine. Multiplying in the target dtype halves it. fp8 has 3 mantissa bits and bf16 shares float32's exponent range, so nothing meaningful is lost. Both encodings occur in one install — the transformer is uint8 block-wise E8M0, the Qwen3-VL encoder float32 scalar — so the dispatch is load-bearing. Extracted to backend/quantization/scaled_fp8.py with both loaders delegating, since two divergent copies is what caused this. Verified against the real checkpoint: all 256 tensors dequantize, shapes preserved, finite, weight std 0.015-0.081 across the transformer. Also fixes the encoder call site, missed when the signature changed and only reachable at generation time. Neither single-file path has a functional test, so a source-level guard asserts both call sites pass a dtype; mypy would catch it but the module has too many pre-existing errors to gate on.
…9152) * feat: add System Prompts library for Expand Prompt button - Add system_prompts SQLite table (migration 32) seeded with 6 curated default prompts adapted from FLUX.2, HunyuanImage 3.0, Qwen-Image, Z-Image and HiDream - Add CRUD service layer + REST router at /api/v1/system_prompts - Add RTK Query endpoints, management modal (list/create/edit/delete) and a system-prompt picker in the Expand Prompt popover - Persist last picked system prompt + text-LLM model via Redux * feat(system-prompts): scope CRUD to owner/admin for multi-user installs - Migration 32 now adds user_id + is_public columns and seeds the 6 default prompts as user_id='system', is_public=TRUE; - Storage layer gains optional user_id scoping on get_many/update/delete, and create requires user_id + is_public - Router uses CurrentUserOrDefault: list scopes to own+public, GET returns 403 for foreign private prompts, PATCH/DELETE require owner or admin - Frontend adds useCanEditSystemPrompt hook, hides edit/delete on prompts the user does not own, shows System/Shared badges in the list, and exposes a 'Share with everyone' toggle in the form when multiuser is on * Add Default System Prompt as DB row * fix(system-prompts): unbreak migration import + cover ownership in tests - Critical: migration_32.py had a 7-space indent on the second cursor.execute(), raising IndentationError on import and blocking server startup. Re-indent and restore the ALTER TABLE backfill block lost in the previous edit. - Medium: drop the heavy import of invokeai.backend.text_llm_pipeline from the migration (which would pull torch+transformers into the migrator import path). Inline DEFAULT_SYSTEM_PROMPT verbatim and rename the seeded row to "Default"; the value still mirrors text_llm_pipeline.DEFAULT_SYSTEM_PROMPT. - Medium: add 7 storage-layer tests covering own/public/admin scoping and the no-mutate guarantees on non-owner update/delete, plus 9 router tests with JWT auth covering 401/403/404 paths, owner is_public flip, and admin override. - Conftest and the existing workflows-multiuser test fixtures now wire a real SqliteSystemPromptRecordsStorage so InvocationServices construction succeeds with the new required parameter. * Chores Ruff + typegen * test(system-prompts): wire system_prompt_records in multiuser_authorization fixture The new required InvocationServices parameter broke 122 unrelated tests in tests/app/routers/test_multiuser_authorization.py because that file builds its own InvocationServices. Add SqliteSystemPromptRecordsStorage to its fixture the same way the workflows-multiuser fixture and the global conftest were updated in the previous commit. * feat(system-prompts): add Text LLM (with System Prompt Preset) workflow node Adds a sibling node to TextLLMInvocation that takes a SystemPromptField (a DB-backed preset reference) instead of a free-text system prompt. Selecting a preset in the workflow editor pulls its content from the System Prompts library at run time. The original TextLLMInvocation is unchanged, so users keep the free-text option and can pick the appropriate node per workflow. - New SystemPromptField primitive in app/invocations/fields.py - Shared _run_text_llm helper extracted from TextLLMInvocation; both nodes use it - Frontend wires SystemPromptField as a new stateful field type analogous to StylePresetField (zod schemas, type guards, builders, slice action, color, Combobox renderer backed by useListSystemPromptsQuery) - Pytest covers both behaviours: DB lookup happens with the configured id and forwards the resolved content; SystemPromptNotFoundError short-circuits the pipeline call so the LLM is not invoked * Chore Ruff + Typegen * chore: regenerate openapi schema for system prompts endpoints * Chore fix Path * test: pass system_prompt_records to InvocationServices in merged-in tests Main's image-move and workflow-call tests construct InvocationServices directly and predate the required system_prompt_records service, so they failed after the merge. Add the argument at the three construction sites. * fix(tests): add missing video/gallery services to system prompts test fixture The mock_services() fixture in test_system_prompts_multiuser.py predates the video generation merge, which added five required InvocationServices args (videos, video_files, video_records, board_video_records, gallery). All nine tests in the file errored at setup, failing every python-tests CI job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(system-prompts): address review feedback on invoke-ai#9152 - TextLLMWithPresetInvocation now enforces the same access rules as the REST API before resolving a preset. The record store is unscoped, so a user could previously read another user's private prompt by enqueueing a graph that references its id -- the content becomes the LLM's system message and is recoverable from the output. Mirrors call_saved_workflow's ownership check. - Bump the migration id (and module name) to 2026_07_10_create_system_prompts. The migrator runs each id once, so the ADD COLUMN backfill for dev databases created from an earlier revision of this branch was unreachable; those DBs hit "no such column: user_id" on every list and create with no self-repair. The earlier id was never released, so a new id is free. Also corrects the module docstring: INSERT OR IGNORE is not what keeps deleted defaults deleted. - delete() raises SystemPromptNotFoundError when nothing was deleted, and the router maps that to 404. Single-user installs skipped the existence check, so DELETE reported 200 for ids GET 404s on and deleting a row twice succeeded twice -- contradicting the PR's own QA contract. This also removes the update()/delete() asymmetry. - Drop the stale manual type augmentation in endpoints/systemPrompts.ts; schema.ts already carries user_id/is_public. Nits: move systemPrompts after stylePresets in en.json; drop the boolean index idx_system_prompts_is_public; document the SystemPromptField id-portability limitation in the node docstring. Tests: node-level permission tests for the escalation path and the allowed cases; single-user router tests for the delete contract; migration tests for the backfill, idempotency and id/module-name consistency. * feat(system-prompts): add Krea 2 expansion prompt, fix node/backfill visibility Seed the Krea 2 prompt-expansion system message (krea-ai/krea-2, docs/expansion.txt) as an eighth default, verbatim from upstream. Also address review feedback on invoke-ai#9152: Drop the `is_default` clause from TextLLMWithPresetInvocation's ownership check. SYSTEM_PROMPT_DEFAULT_USER_ID ("system") is not only the seeded defaults' owner but also the synthetic user id every request carries in single-user mode, so the clause made every prompt created before an install switched to multiuser readable by anyone via a graph, while GET /system_prompts/i/{id} correctly 403s on it. The seeded defaults are is_public=TRUE, so is_public already covers them and the node's rule is now identical to the router's by construction. Re-share the seeded defaults in the multiuser backfill. ADD COLUMN stamps is_public=FALSE onto pre-existing rows and the seed is INSERT OR IGNORE, so the defaults stayed private and get_many (own OR public) returned an empty list for every non-admin. Scoped to the seeded ids and to the backfill branch so a default a user deliberately made private is never re-shared. Correct the SYSTEM_PROMPT_DEFAULT_USER_ID docstring and the user_id field description, which described the id as meaning "built-in default" - the reading the bypass was built on. Tests: the parametrized node case asserted a private "system"-owned prompt was allowed; corrected and paired with a regression test for the single-user -> multiuser upgrade path. The backfill test now seeds the defaults first and asserts they end up public, plus a test that a re-run leaves privatization alone. --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update to Transformers 5.1.0 * remove extra stuff * feat: ERNIE-Image integration (incl. diffusers 0.38 + transformers 5) Adds Baidu ERNIE-Image and ERNIE-Image-Turbo as a new BaseModelType, mirroring the FLUX.2 / Z-Image integration pattern. Both models share the ErnieImageTransformer2DModel architecture (3072 hidden, 24 layers, 24 heads) and an AutoencoderKLFlux2 VAE; they differ only in default inference settings (50 steps + CFG 4.0 vs 8 steps + CFG 1.0 for Turbo). Built on top of PR invoke-ai#8859 (transformers 5.1+) and additionally bumps diffusers 0.36.0 -> 0.38.0, which is the first release containing the ErnieImagePipeline and ErnieImageTransformer2DModel. Backend - BaseModelType.ErnieImage, ModelType.PromptEnhancer, two new SubModelTypes (pe, pe_tokenizer) for the bundled prompt enhancer - Main_Diffusers_ErnieImage_Config + Main_Checkpoint_ErnieImage_Config with state-dict-based detection (x_embedder + text_proj + adaLN_modulation) - Diffusers loader registered for ERNIE-Image; uses upstream subdir conventions, loads transformer / vae / text_encoder / tokenizer plus optional pe / pe_tokenizer - New invokeai/backend/ernie_image/ with sampling utilities (2x2 patchify, BN normalize/denormalize, sigma schedule, padded text packing) and a rectified-flow denoise loop supporting Euler/Heun/LCM - Five invocations: model_loader, text_encoder (with prompt-enhancer toggle), denoise, vae_encode, vae_decode - ErnieImageConditioningInfo + ConditioningField/Output + pickle allowlist - ERNIE_IMAGE_SCHEDULER_MAP reusing the FlowMatch* scheduler classes - New generation modes ernie_image_{txt2img,img2img,inpaint,outpaint} - Starter models for baidu/ERNIE-Image and baidu/ERNIE-Image-Turbo + STARTER_BUNDLES entry Frontend - Regenerated services/api/schema.ts to expose the new node types - Type unions extended (ImageOutput / LatentToImage / ImageToLatents / DenoiseLatents / MainModelLoaderNodes) - ParamsState gains ernieImageScheduler + ernieImageUsePromptEnhancer, with reducers, selectors, and selectIsErnieImage - buildErnieImageGraph (txt2img/img2img/inpaint/outpaint) wired into useEnqueueCanvas - ERNIE entries added to MODEL_BASE_TO_{COLOR,LONG_NAME,SHORT_NAME} and prompt_enhancer to MODEL_TYPE_TO_LONG_NAME - ParamErnieImageScheduler and ParamErnieImagePromptEnhancer rendered conditionally in GenerationSettingsAccordion - All add{TextTo,ImageTo,Inpaint,Outpaint}Image type guards extended to accept ernie_image_denoise; isMainModelWithoutUnet ditto Diffusers 0.38 fix-out - hotfixes.py: import LoRACompatibleConv directly; the lazy-module __getattr__ no longer exposes diffusers.models.lora as an attribute Verification - pytest tests/ -m "not slow": 619 passed, 0 failed - pnpm lint:tsc / lint:eslint / lint:prettier: clean - pnpm test:no-watch: 563 passed, 0 failed - Manual smoketest pending: requires baidu/ERNIE-Image weights and a GPU (8B parameters; CPU not practical) Out of scope (follow-up phases) - ControlNet, IP-Adapter, LoRA support for ERNIE-Image - Single-file checkpoint loading (defensive scaffolding only) - Metadata recall handlers in the gallery side panel * Chore Ruff + Typegen * Chore ruff * fix(ernie-image): timestep scale, live preview, missing graph case, and UI cleanup - Pass timesteps in [0, num_train_timesteps] to the transformer instead of [0, 1]; the diffusers Timesteps embedding expects the unnormalised range, which produced mosaic-pattern garbage instead of an image. - Unpatchify predicted-x0 before the denoise step callback and route through sd_step_callback so the canvas shows a live preview during sampling (uses FLUX.2's RGB factors -- same AutoencoderKLFlux2 / 32 latent channels). - Add the ernie-image case to useEnqueueGenerate (Generate tab); was only wired up in useEnqueueCanvas, so plain text-to-image failed with "No graph builders for base ernie-image". - Move the Prompt Enhancer toggle from the Generation accordion into a dedicated ERNIE-Image block in the Advanced accordion; hide the rest of the SD-style advanced controls (CLIP skip, CFG rescale, seamless, color comp., separate VAE) since none apply to ERNIE-Image. - Detect ERNIE-Image-Turbo by name in MainModelDefaultSettings.from_base so installs (starter or manual) get steps=8, cfg_scale=1.0 instead of the standard 50/4.0. - Pin compel to Cstannahill/compel5@chore/transformers5-diffusers-smoke for transformers>=5 compatibility (PR damian0815/compel#129). * Merged missed * Chore Ruff * Chore Fix UV lock * fix(ernie): restrict ERNIE-Image to text-to-image only ERNIE's denoise node has no denoise_mask input, so masked modes are unsupported. Drop img2img/inpaint/outpaint from the ERNIE graph builder, exclude ernie_image_denoise from a new MaskableDenoiseNodes type used by addInpaint/addOutpaint, and align addImageToImage unions with the node type aliases. Fixes tsc failures on the ernie-image branch. * fix(ernie): restrict ERNIE-Image to text-to-image only ERNIE-Image's denoise node has no denoise_mask input and no mask logic, so masked modes (inpaint/outpaint) are impossible and image-to-image is dropped as well. Frontend: - buildErnieImageGraph now builds txt2img only; asserts on other modes - add MaskableDenoiseNodes (DenoiseLatentsNodes minus ernie_image_denoise) and use it in addInpaint/addOutpaint - drop ernie_image_vae_encode from ImageToLatentsNodes; align addImageToImage unions with the LatentToImage/ImageToLatents aliases - regenerate schema.ts Backend: - delete the ernie_image_vae_encode invocation (i2l, only used by the removed image-input modes) - drop ernie_image_{img2img,inpaint,outpaint} from GENERATION_MODES Fixes the tsc failures on the ernie-image branch. * Chore OpenApi * fix(model_manager): remove duplicate _has_anima_keys shadowing ComfyUI-bundled prefix support A duplicate _has_anima_keys definition (older, net.-only) shadowed the complete version that also recognizes the `model.diffusion_model.` ComfyUI-bundled prefix, causing Anima identification to reject bundled checkpoints. * Chore Openapi * fix(ernie-image): exclude prompt enhancer from fp8, tighten turbo detection Address review findings on invoke-ai#9115: - Exclude SubModelType.PromptEnhancer and PromptEnhancerTokenizer from fp8 layerwise casting. The prompt enhancer is a causal LM driven by generate() — one full forward per generated token — so casting made the whole LM round-trip bf16<->fp8 per token, on top of fp8 rounding a model whose entire job is text quality. - Match turbo detection on the install directory's leaf name instead of the whole path string. An in-place install records an absolute path, so an ancestor directory like /mnt/turbo-nvme/ silently gave the base model Turbo's 8 steps and CFG 1.0. - Pass local_files_only=True on all ERNIE-Image from_pretrained calls and load tokenizers bare, matching the sibling loaders. - Cap the prompt enhancer's max_new_tokens at 1024. Driving it off model_max_length hangs the graph if the tokenizer config omits it and transformers substitutes its int(1e30) sentinel. Adds regression tests for the fp8 exclusion and the turbo path matching. * fix(ernie-image): honor denoising window, noise init latents, harden PE gate Self-review follow-ups on invoke-ai#9115: - Honor denoising_end. Every FlowMatch scheduler appends its own terminal 0 sigma, and passing the window minus its last entry let that zero stand in for the requested end sigma - so denoising_end < 1.0 ran a full denoise in fewer, coarser steps. Hand the scheduler the whole window and truncate its appended zero instead, which also keeps the scheduler's own `shift` applied to the terminal sigma. - Reject a denoising window that rounds down to a single sigma. It yielded zero steps, so the loop returned its input untouched and the graph decoded raw noise with no error at all. - Blend image-to-image init latents with noise at the first sigma, and reject denoising_start > 0 when no latents are provided. Both cases previously lied to the model about where the sample sits on the rectified-flow path. The shape check now covers batch and spatial dims, and its message no longer points at a VAE-encode node that does not exist. - Require both `pe` and `pe_tokenizer` to be present AND declared in model_index.json before offering the prompt enhancer. get_hf_load_class resolves submodels from that file, so a directory-only check let a partial install pass the gate and then hard-fail the generation - by default, since the toggle is on. - Pass the real generation dimensions to the prompt enhancer instead of leaving it on its 1024x1024 defaults. Adds regression tests for each, including a graph-builder test suite for ERNIE (which had none). Added ERNIE-Image + Krea 2 Raw into README.md under Supported Model. * Readme.me update with ERNIE-Image-Turbo * fix(ernie-image): seed the stochastic scheduler, blend at the shifted sigma Round-4 review follow-ups on invoke-ai#9115: - Pass a seeded generator into scheduler.step. FlowMatchLCMScheduler is stochastic - it re-noises the sample every step - so with generator=None it drew from the global RNG. The seed field only controlled the initial latent, so the same seed produced a different image on every run and seed recall from gallery metadata could not reproduce an LCM generation. Seeded from `seed ^ 0xFFFFFFFF` (as denoise_latents.py does) so the step noise stays decorrelated from the initial noise, and from a CPU generator so it is device-independent like the initial noise. - Blend image-to-image init latents at the scheduler's post-shift first sigma instead of the raw schedule value. get_schedule emits raw linspace values and the scheduler applies `shift` in set_timesteps, so the blend built the sample at one sigma and then told the first model call it was at another. The blend moves into denoise(), which is the only layer that knows the shifted sigma. - Add an `add_noise` field mirroring z_image_denoise. The only node that can currently feed `latents` is another ernie_image_denoise, whose output already sits at the handoff sigma - re-noising it broke the very multi-stage handoff the denoising_end fix enables. - Give the prompt enhancer the original size rather than the intermediate scaled render size. Regenerates openapi.json / schema.ts for the new field. Adds regression tests for the seed contract (same seed identical, different seeds different, euler unaffected) and for the post-shift blend. --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
…m-main-2026-07-29 # Conflicts: # invokeai/app/api_app.py # invokeai/app/services/invocation_services.py # invokeai/frontend/web/openapi.json # invokeai/frontend/web/src/services/api/schema.ts # tests/conftest.py
joshistoast
requested review from
JPPhoto,
Pfannkuchensack,
blessedcoolant and
lstein
as code owners
July 31, 2026 00:56
github-actions
Bot
added
root
python
python-tests
ci-cd
backend
api
services
frontend
invocations
docs
python-deps
labels
invoke-ai#9383) * feat(model-manager): identify UNet-only SDXL LoRAs (e.g. slider LoRAs) Self-attention-only SDXL LoRAs (e.g. Civitai "slider" LoRAs) patch only the UNet and contain no cross-attention (attn2) or text-encoder (lora_te*) keys. lora_token_vector_length() reads the base's context dimension from exactly those keys, so it returns None for such LoRAs and identification fails with "unrecognized token vector length None". Add a structural fallback: SDXL's UNet has a deep transformer stack (up to 10 transformer blocks) in its lower-resolution attention blocks, so transformer_blocks indices reach >= 2, whereas SD1.x/SD2.x only ever have a single transformer block (index 0) per attention. _state_dict_looks_like_sdxl_unet_lora() detects SDXL from that structure alone. The regex is anchored on the UNet down/up/mid-block + attentions grouping so it won't false-positive on DiT LoRAs (FLUX/Qwen/Z-Image) that also use transformer_blocks. The fallback only runs after the normal token-vector detection returns None, so existing LoRAs are unaffected. Wired into both the LyCORIS and Diffusers base configs. * fix(model-manager): anchor SDXL UNet-only LoRA heuristic on lora_unet_ prefix The structural fallback for UNet-only SDXL LoRAs matched on block names alone, which missed kohya sd-scripts' Stability-AI naming (input_blocks_8_1, middle_block_1, output_blocks_0_1) and wrongly claimed diffusers/PEFT LoRAs (unet.….lora_A.weight). The latter identified as SDXL and then crashed in convert_sdxl_keys_to_diffusers_format() with "Unrecognized SDXL LoRA key prefix" mid-generation, where on main they had installed as inert Unknown models. Anchoring on ^lora_unet_ aligns the heuristic with exactly the key set the SDXL loader can convert, and adding the Stability block names covers both kohya naming conventions. Also: assert the rejection reason in the SD1/SD2 test, cover the LoRA_Diffusers_* path, and document the >= 2 transformer-block threshold's known miss for LoRAs confined to the 2-block attentions. * fix(model-manager): anchor SDXL UNet-only LoRA heuristic on lora_unet_ prefix The structural fallback for UNet-only SDXL LoRAs matched on block names alone, so diffusers/PEFT LoRAs (unet.….lora_A.weight) were wrongly claimed as SDXL. Those then crashed in convert_sdxl_keys_to_diffusers_format() with "Unrecognized SDXL LoRA key prefix" mid-generation, where on main they had installed as inert Unknown models. Anchoring on ^lora_unet_ aligns the heuristic with exactly the key set the SDXL loader can convert. Also adds kohya's Stability-AI block names (input_blocks_N_N, middle_block_N, output_blocks_N_N) alongside the diffusers ones, so both sd-scripts naming conventions are covered. Tests now build the reported LoRA's fixture from its real block layout (all 840 keys of civitai.com/models/1105685), assert the rejection reason in the SD1/SD2 test, cover the LoRA_Diffusers_* path, and document the >= 2 transformer-block threshold's known miss for LoRAs confined to the 2-block attentions. --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
The .superpowers/sdd/ task briefs and reports from the mixed-media gallery work were committed across 10+ commits on this branch. They are working notes, not repository content. .claude/ was already ignored but .superpowers/ was not, which is how they leaked in. Ignore it and drop the 12 tracked files; the directory stays on disk untouched.
b173759 restored this fork's cache_used to the per-device /models/stats aggregate. Upstream's multi-GPU work also added two eviction paths that _delete_cache_entry outside any synchronized method — evict_unlocked_for_peer and _reconcile_budget_if_pending — and neither re-synced the stats the way put/unlock/make_room/drop_model do. The result was the same symptom b173759 fixed, through a different door: an evicted cache kept reporting its pre-eviction cache_used and in_cache until its device next ran a session, and because the aggregate takes max(cache_used) across caches, one stale idle cache pinned the reported usage for the whole system — reading as a cache that never releases memory. in_cache is summed, so it over-reported outright, and cleared never counted peer evictions at all. Both loops now sync under the lock and add their evictions to cleared. put() also syncs after the peer-reconcile loop rather than before it, since a reconcile there can free RAM inline and cache_used reads the global budget. cache_used had no multi-cache test coverage, which is why this slipped through; both paths are now covered.
Two fork-side helpers in the queue router predated fields that upstream's video-generation and multi-GPU work added, so the merge left them a step behind. strip_missing_image_results only inspected ImageField, so completed history kept advertising deleted *videos* — the exact 404 hydration loop the helper exists to prevent. It now resolves VideoField against video_records too, caching each kind separately so an image and a video sharing a name are not conflated. sanitize_queue_item_for_user redacts 20 fields but not the new device column, so a non-admin could see which GPU another user's job was placed on. Redacted for the same reason as origin/destination/priority. The webv2 badge already renders nothing for a null device, so no frontend change is needed.
Numeric migrations are keyed by migration_<to_version>. This fork's migration_33 creates the projects table; upstream's migration_33 creates the image-subfolder move tables, which this fork carries as migration_34. Two different migrations, one id. A database that was ever opened by an upstream build therefore has migration_33 recorded, and this fork skips its own projects migration forever. migration_34 is no help — its DDL is all IF NOT EXISTS, so it silently no-ops. The user ends up with a working install whose project queries fail with 'no such table: projects'. Reproduced end to end before writing the fix. Repaired with a dated migration, which cannot collide with an upstream number: it runs once on every database, creates the table and trigger if they are missing, and no-ops otherwise. migration_32 also differs from upstream's but only in comments, so 33 is the only real collision. Future fork migrations should use dated ids so this class of problem cannot recur.
Brings in ernie image/turbo (invoke-ai#9115) and UNet-only SDXL LoRA identification (invoke-ai#9383), closing the remaining gap with upstream. Only openapi.json and schema.ts conflicted; both are generated, so they were resolved by regenerating from the merged backend rather than by hand-merging.
Upstream's invoke-ai#9152 added a System Prompts library to the legacy web app; webv2 called /utilities/expand-prompt with no system prompt at all, so the feature existed on the backend and in one frontend only. The popover now carries a picker over the prompts the account can see, and behind a manage toggle the list that creates, edits and deletes them. Structure follows the prompt-templates feature next door: a pure core module for ownership and selection, a data module for transport, and a catalog hook. Two things are deliberately not copies of upstream: - Ownership is decided by is_public and the owner id, never by comparing against the user id "system". That id also belongs to every prompt created in single-user mode, so treating it as built-in would make an install that later enabled multiuser show its own prompts as read-only. - The selection is resolved on read rather than stored back. The id outlives the record — another tab can delete it, and a shared prompt disappears when its owner unshares it — so a stale id falls back to the first visible prompt, which is also what an unselected picker uses. The four new modules join the prompt-templates UI in the eager editor chunk, which is what the source-owner gate flagged; the browser baseline is re-recorded for +682 bytes (+0.04%) with no owners removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v7/mainwith upstreammainthrough09baa8d3e0, including Video Generation, current model/runtime fixes, Krea 2 multi-conditioning, and the System Prompts library.Related Issues / Discussions
QA Instructions
cd invokeai/frontend/webv2 && pnpm check:releasepnpm lint:tscpassedManual real-backend Wan generation was not run locally; the mixed-media API, playback, Range, mutation, upload, and accessibility paths are covered by the automated suites.
Merge Plan
Use a merge commit rather than squash-merging so upstream ancestry is preserved and the next upstream sync remains tractable.
Checklist
What’s Newcopy (if doing a release after this PR)