---
title: "We introduce a new ''rule'' for understanding diffusion models: Selective Underfitting. It..."
canonical_url: "https://typefully.com/t/xCWbsZ1"
markdown_url: "https://typefully.com/t/xCWbsZ1.md"
agent_guide_url: "https://typefully.com/ai-agents.md"
post_delimiter: "***"
robots: noindex
source: typefully_shared_draft
share_id: "xCWbsZ1"
---

We introduce a new ''rule'' for understanding diffusion models: Selective Underfitting. 

It explains:
🚨 How diffusion models generalize beyond training data
🚨 Why popular training recipes (e.g., DiT, REPA) are effective and scale well

Co-led with @kiwhansong!
(1/n)

![CleanShot 2025-10-06 at 02.16.02@2x.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/203122/f81ce7de-af37-4ddf-ba43-8939ffe5437e.png)

***

Diffusion models face an apparent paradox; if training were perfect, they would reproduce the training data, yielding no novel samples. This is because they are trained to approximate the empirical score, i.e., the score with respect to finite training data. (2/n)

***

Prior work, e.g., Kamb & Ganguli, Niedoba et al., argues that diffusion models underfit the empirical score due to inductive bias—global underfitting.
Instead, we find underfitting is selective: better models fit the score well in a certain region but underfit outside it. (3/n)

***

We distinguish two regions: supervision region vs. extrapolation region. As a model scales, error between the learned score and the empirical score decreases in the supervision region (X underfitting, blue) but increases in the extrapolation region (O underfitting, red). (4/n)

![CleanShot 2025-10-06 at 02.46.38@2x.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/203122/196cb5bb-84c5-43b9-b46f-1c7b10399e8d.png)

***

These two regions can be further characterized as follows:

- Supervision region: where noisy training samples are concentrated (high probability, blue shells).
- Extrapolation region: where the model is actually queried during inference (for most time steps, red region). (5/n)

***

In the supervision region, the learned and empirical scores are sufficiently close that the model reproduces the training images when sampling starts from there. These quantitative and qualitative results verify selective underfitting. (6/n)

![CleanShot 2025-10-06 at 03.23.50@2x.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/203122/762974fb-0fa2-4ac4-bb75-3387ea299913.png)

***

Therefore, we claim that diffusion models operate by 
(1) approximating the empirical score within the supervision region
and 
(2) extrapolating it to the extrapolation region, which is ultimately used at inference time. (7/n)

***

[Generalization] The supervision region's size strongly affects a model’s ability to generalize. Enlarging this region degrades generalization. This architecture-agnostic result (SiT, U-Net) reveals a new axis beyond architectural bias for diffusion model generalization. (8/n)

![CleanShot 2025-10-06 at 03.17.35@2x.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/203122/0efde464-4535-490c-9fd7-a736672e3a4d.png)

***

[Analyzing FID] We decompose FID into the following two components via a novel scaling law:
(1) How well the empirical score is fitted in the supervision region (supervision loss)
(2) How effectively this supervision loss translates to FID
(extrapolation efficiency)
(9/n)

![CleanShot 2025-10-06 at 03.29.27@2x.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/203122/7169c22b-3ef4-401d-8238-0497312e8841.png)

***

This framework explains why REPA and DiT work:
- REPA boosts extrapolation efficiency with minimal change to supervision loss.
- Transformers trail U-Nets in extrapolation efficiency but excel in FLOPs → supervision loss, yielding better compute efficiency (FLOPs → FID).
(10/n)

![CleanShot 2025-10-06 at 03.42.23@2x.png](https://d3e0luujhwn38u.cloudfront.net/original/img/original/203122/03e5be5e-8602-4cf6-93d4-849e49a322cf.png)

***

We further argue that recent diffusion training methods, such as REPA and ReDi, align the score network’s output with the representation, e.g., DINO, improving generative performance by better handling underfitting (extrapolation) regions. (11/n)

***

Takeaways:
(1) Diffusion models don’t underfit everywhere, but selectively in distinct regions.
(2) The relative sizes of these regions affect generalization.
(3) Decomposing FID into interactions between two regions explains why several diffusion recipes work well. (12/n)

***

Our lens of selective underfitting opens the door to new questions: how does a model extrapolate toward a favorable score that yields good perceptual samples? Can we design training recipes that better balance supervision and extrapolation? (13/n)

***

paper: [https://arxiv.org/abs/2510.01378]
website: [https://selective-underfitting.github.io/] (14/n)

***

Last but not least, this work would not have been possible without my co-first author @kiwhansong0, senior lead authors @vincesitzmann and @sitanch, and our amazing advisors @du_yilun and @ShamKakade6. (15/n)

***

Huge thanks to @kiwhansong0 — my best friend since high school and also now an amazing research partner. Even as an undergrad at MIT, his insights into research and academia are remarkably professional and have taught me so much. Can’t wait for our next project together! (16/n)