SumomoLee/qwen-image-anime-decft

Qwen-Image VAE — anime decoder fine-tune (v1)

Decoder-only fine-tune of the Qwen-Image VAE (qwen_image_vae.safetensors, ComfyUI/Wan layout, 194 keys, bf16, original sha256 a70580f0…3e023d1f). Only decoder tensors were changed. The encoder, quant_conv and the latent normalisation are bit-identical to the original, so the latent space is unchanged: any diffusion model, LoRA or latent cache built on the Qwen-Image VAE keeps working. Drop-in replacement (load it as the VAE).

file	sha256	notes
qwen_image_anime_decft_v1.safetensors	4cb48db5319bd813a1a2622ef76a95f23d27d3f73a1203ba5b4057903b220ac5	recommended: fidelity-oriented
qwen_image_anime_decft_v1_sharp.safetensors	a7f894a8bf576f6fee78510afe786a7a7d9e96e7e6e4f028579a43e3f1cee82b	perceptually sharper (+ FDL loss): LPIPS −54 % vs original, PSNR −0.5 dB

Format checks (both files): same key set, dtypes, shapes and metadata as the original; 0 non-decoder tensors changed; 102 decoder / conv2 tensors changed; temporal-only time_conv tensors untouched; for 5-D causal-conv weights only the last temporal slice (the one used for single images) was rewritten. Loads with stock ComfyUI.

What it fixes

The original Qwen-Image decoder shows a regular 1–2 px checkerboard / stripe lattice, most visible in flat and dark regions and on gradients. v1 removes it (lattice amplitude down ~50× on reconstructions, ~10× on diffusion-generated latents, close to the measurement noise floor) while improving fidelity and colour accuracy.

Metrics

bf16 weights and compute, output quantised to 8 bit, held-out images never used in training. "coh2" = phase-coherent amplitude of the period-2 lattice in the high-pass residual (/255; noise floor ≈0.012–0.019); "fft2" = Nyquist peak-to-neighbourhood power ratio.

Reconstruction (2,000 held-out images: 1,000 anime + 1,000 photos)

decoder	PSNR (anime / photo)	LPIPS-alex	ΔE76 (8× down)	error coh2	error fft2
original Qwen-Image	36.88 (37.63 / 36.12)	0.0310	0.445	0.171	104
v1	37.31 (38.15 / 36.48)	0.0248	0.328	0.0034	1.3
v1_sharp	36.38	0.0142	0.351	0.0039	1.3
Wan2.1 decoder (reference)	35.66	0.0209	0.531	0.033	6.9

Diffusion-generated latents (256 generic prompts from a DiT trained on the Qwen-Image latent space, Euler 30 steps, CFG 4.5; 30 at 1536²/2048²)

decoder	coh2	fft2	lattice in dark flat areas
original	0.237	37	0.152
v1	0.023	1.8	0.040
Wan2.1	0.110	13.8	0.019

Text images (200 held-out AnimeText images)

decoder	PSNR	LPIPS	ΔE
original	29.38	0.0273	0.383
v1	29.51	0.0185	0.328

Other checks: fp16 / fp32 decodes match bf16 (v1 37.34 dB in both); no NaN/Inf in fp16 or bf16 on any test latent (including zero and solid-colour latents); encoded solid colours and gradients reproduce within 2–4/255 in the interior, gradient PSNR 52–55 → 56–57 dB. Known small issue: one corner pixel of a saturated solid image can deviate by up to 8/255 (original: 5/255).

Training

• Frozen: encoder, quant_conv, latent mean/std. Initialised from the original Qwen-Image decoder.
• Data: 153,778 images — 65k high-quality anime (1024² buckets), 49.8k anime (1536² buckets, stratified by flat colour / dark / gradient / text / face / lineart / random), 30k photos, 9k AnimeText. Sampling share: anime 84 %, photo 10 %, text 6 %.
• Generated-latent robustness: 40 % of samples use degraded latents — one-step x0 predictions of a diffusion transformer trained on this latent space, from real latents noised at σ∈[0.02, 0.18] — paired with the original image. 8–10 % are synthetic flats, gradients and thin lines encoded by the frozen encoder.
• Crops: latent-aligned, 256–512 px with 32 px decoded context, 768 px in the last ~1k steps; from round 2 on 30–40 % of crops snapped to image borders/corners.
• Losses: Charbonnier L1 1.0; LPIPS-vgg 0.1; colour anchor (L1 on 8× average-pooled output) 1.0; period-2/4 lattice-fold penalty on the error 1.0; spectral-norm PatchGAN (hinge, LeCAM 0.1, adaptive weight 0.1, 0.05 in round 3); v1_sharp only: FDL (VGG19) 0.001.
• Optimiser: AdamW (0.9, 0.99), cosine to 0.1×, EMA 0.999, bf16 autocast.
• Schedule: stage 1 — top-2 up stages + head (2.66M params), lr 3e-5, 6,000 steps; round 2 — whole decoder (25.3M params), lr 1e-5, 4,000 steps, border crops; round 3 — whole decoder, lr 5e-6, 4,000 steps (v1 = EMA step 4,000; v1_sharp = same with FDL). 4×B300 per run, ~0.27 s/step, 6 crops of 512² per GPU.