Why does the conditioning image only go through VAE and not ViT

[ssj9596]

I noticed that in Generator Mode, the conditioning image is only passed through the VAE and is not fed through the ViT for high-level semantic understanding. I’m curious about the reasoning behind this design choice. Were any ablation studies conducted to validate it? Thank you!

[Xuanmeng-Zhang]

Thank you for the thoughtful question. In Generator Mode, the VAE maps the conditioning image into the same latent space as the video frames, helping preserve fine-grained information such as spatial layout, color, and texture. In contrast, ViT representations typically emphasize higher-level semantic features and may discard pixel-level details important for faithful visual conditioning. We therefore use ViT features primarily in reasoner (VLM), where they provide semantic tokens aligned with language. This design is also consistent with approaches such FLUX.1 Kontext.

We conducted related ablation experiments during development, and our observations supported this design choice, although we did not include those results in the paper. The effect can also be readily verified through a straightforward comparison between VAE-only conditioning and conditioning with ViT features.