Restoring Diversity in Distilled Image and Video Generators by Probing Activations

Horoz, Furkan; Kara, Ozgur; Oktay, Kaan; Akan, Adil Kaan; Kara, Atakan; Bhattad, Anand; Rehg, James Matthew

Restoring Diversity in Distilled Image and Video Generators by Probing Activations

Furkan Horoz¹ Ozgur Kara¹ Kaan Oktay² Adil Kaan Akan² Atakan Kara² Anand Bhattad^3,† James Matthew Rehg^1,†

¹University of Illinois Urbana-Champaign ²fal ³Johns Hopkins University
Preprint, 2026
^†Equal advising.

arXiv (soon) Code 🤗Models BibTeX

Diversity back in few-step generators. Few-step students keep (or even exceed) their teachers' activation variance but squeeze it into fewer directions, starting at one intermediate block of the first denoising step. ReDiverse puts the missing directions back with a short fine-tune (ReDiverse-FT), with no training at all (ReDiverse-TF), or with both.

Abstract

Distilling diffusion models into few-step student models substantially reduces sampling latency, but induces severe diversity collapse: samples generated from different random seeds show minimal variation. In this work, we investigate the root cause of this failure by probing the internal representations of student models relative to their teachers. We find that distilled students maintain total activation variance comparable to their teachers, but concentrate it into substantially fewer directions. This directional collapse is localized to an identifiable intermediate block at the initial denoising step. Consequently, diversity collapse stems from a misallocation of activation variance rather than an inherent loss of generative capacity. Guided by this finding, we propose ReDiverse to reinstate the missing variance directions. At training time, ReDiverse-FT fine-tunes a lightweight adapter that regresses student activations onto the teacher's, reviving the teacher's feature variance characteristic. At inference time, ReDiverse-TF introduces a training-free mechanism that steers activations along precomputed teacher principal directions. Across image and video distillation benchmarks, ReDiverse consistently restores sample diversity while preserving generation quality, further supported by the user study.

Video Results

A 4-step video student repeats one composition for a prompt. ReDiverse, started from the same noises, changes the subject, the setting and the camera. Each tile is one initial noise: slide from the 4-step student to ReDiverse, or press Sweep, to see both clips of that noise.

Image Results

From collapsed to diverse: each tile is one initial noise. Slide from the 4-step student to ReDiverse, or press Sweep, and every sample changes from the student's version to ReDiverse's version of the same noise. Click a tile to view it at full resolution.

Method Comparison

Every method starts from the same noises, one seed per column. Image baselines: DP-DMD, a re-distilled student whose first step is regressed toward the teacher, and Noise Optimization, an optimization-based reference that searches for initial noises maximizing a DINOv2 diversity score at test time. Video baseline: DFD, which post-trains the student with real videos.

Same Variance, Fewer Directions

We give the same noises to a teacher and its 4-step student, run their first denoising step, and look at the activations of every block across noises. Three quantities describe that cloud: its total variance, its participation ratio (how many directions the variance effectively spans), and the concentration (variance per direction). Early blocks match. At an intermediate block, the collapsed block ℓ, the student keeps or even amplifies the total variance, but spans far fewer directions, and the collapse propagates to the output.

Variance characteristic along the depth, SDXL and Wan 2.1 enlarge

Participation ratio, total variance and concentration of teacher and student at every block of SDXL and Wan 2.1

Participation ratio, total variance and concentration of teacher and student activations at every layer, for (a) an image model (SDXL) and (b) a video model (Wan 2.1). Total variance and concentration are min–max normalized to [0, 1] over teacher and student (SDXL's in log space). The vertical line marks the collapsed block.

Student / teacher ratios at the collapsed block ℓ

Student	Participation ratio	Total variance	Concentration
DMD2 (SDXL)	0.40	1.63	4.76
TDM (SD3)	0.49	1.03	2.68
lightx2v (Wan 2.1)	0.60	1.41	2.54
Turbo (MiniMax-H3)	0.37	2.92	8.08

Every student spans fewer directions than its teacher (ratio below 1) while keeping or raising the total variance.

The Lost Directions Are Still Decodable

We move the first-step activation at the collapsed block along a top principal direction, with growing strength. Moving the teacher along its own direction changes framing, pose and scene. Injecting that same teacher direction into the student produces the same structural changes, with no change to the student's weights. Moving the student along its own direction only changes appearance, such as coat pattern and lighting.

The directions lost in distillation are the ones that set structure, and the student's later blocks can still render them. Diversity collapse is a misallocation of variance, not a loss of capacity.

Steering SDXL along teacher and student directions

Method

ReDiverse restores the missing directions at the collapsed block ℓ, in the student's first step only. The two parts can be used alone or together.

ReDiverse-FT. Teacher and student start from the same noise and prompt. A LoRA on the blocks up to ℓ regresses the student's step-1 activation onto the teacher's activation at step τ. Each prompt comes with 16 noises, so matching noise by noise transfers the teacher's variation; the LoRA is active only at the first step.

ReDiverse-TF. Training-free. For each prompt, PCA over the teacher's step-τ activations from N = 25 noises gives directions u_k and spreads s_k. Each sample adds Σ_k α_k s_k u_k, α_k ~ U[−2, 2], to the student's activation at ℓ in step 1.

Results

Sample diversity, DINO (×100, higher is more diverse)

Method	SDXL	SD3	Wan 2.1	MiniMax-H3
Student (4 steps)	59.4	48.4	31.6	36.5
ReDiverse-TF	60.1	53.8	34.6	42.2
ReDiverse-FT	64.6	53.5	46.8	44.1
ReDiverse	65.0	60.1	52.2	50.9
Teacher (multi-step)	66.3	61.6	51.6	52.5

Mean pairwise DINO distance between samples of a prompt on GenEval (553 prompts × 100 seeds) and VBench (300 prompts × 10 clips). All rows except the teacher sample in four steps. ReDiverse closes most of the gap to the multi-step teacher. Prompt alignment stays close to the student (CLIPScore within 0.5 points on images), while reward-model scores drop on images; the paper reports the full set of diversity and quality metrics and a user study.

BibTeX

@article{horoz2026rediverse,
  title   = {Restoring Diversity in Distilled Image and Video Generators by Probing Activations},
  author  = {Horoz, Furkan and Kara, Ozgur and Oktay, Kaan and Akan, Adil Kaan and Kara, Atakan
             and Bhattad, Anand and Rehg, James Matthew},
  journal = {arXiv preprint},
  year    = {2026}
}