Skip to content

Fix text CFG context for interleaved image conditions - #6

Open
arshiaizd wants to merge 1 commit into
EPFL-VILAB:mainfrom
arshiaizd:fix/preserve-text-cfg-context
Open

Fix text CFG context for interleaved image conditions#6
arshiaizd wants to merge 1 commit into
EPFL-VILAB:mainfrom
arshiaizd:fix/preserve-text-cfg-context

Conversation

@arshiaizd

@arshiaizd arshiaizd commented Aug 26, 2026

Copy link
Copy Markdown

Summary

  • preserve the text-unconditional CFG context when an image-like condition follows text
  • append image conditions to both the full and text-free caches instead of replacing the text-free cache with a copy of the fully conditioned cache
  • replay the same Torch RNG state for both cache updates so stochastic VAE encoding is identical and the paired operation consumes RNG once
  • apply the same behavior to chained intermediates, segmentation conditions, and mirrored second-pass conditioning

Problem

For an input order such as [caption, depth], _feed_conditions first created the correct text-free cache before appending the caption. When depth was subsequently appended, it assigned cfg_text_context = deepcopy(gen_context). At that point gen_context already contained both caption and depth, so the text-conditioned and text-unconditional branches became identical. As a result, changing cfg_text_scale had no effect for this ordering.

Simply updating the two caches independently would introduce another inconsistency because VAE conditioning samples a stochastic posterior. The helper in this change replays the RNG state so both branches receive the same sampled image latent while preserving the external random stream.

Validation

Validated locally for both caption-then-image and image-then-caption ordering, identical stochastic samples across CFG branches, single RNG consumption, and Python syntax.

@arshiaizd
arshiaizd force-pushed the fix/preserve-text-cfg-context branch from cdbcc5a to 34f861d Compare August 26, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant