In this blog, we explore the architectural design of diffusion models and consistency models in depth. We began with the fundamental U-Net architecture that underpins most diffusion models, explaining how its encoder–decoder structure with skip connections enables multi-scale denoising.
We saw that the use of ResNet blocks and time-step embeddings allows these networks to be both deep and conditioned on the diffusion process, and we discussed how self-attention layers were integrated to model long-range interactions in images.
Building on this, we examined how Transformers have entered the diffusion scene: from hybrid attention-augmented U-Nets to fully transformer-based diffusion backbones such as DiT, which leverage the scalability of the Transformer architecture to achieve state-of-the-art results (wpeebles.com).
Moving to conditioning, we outlined the techniques by which diffusion models incorporate external information. Simple approaches (adding label embeddings or using FiLM conditioning) were related to analogous designs in GANs such as BigGAN (medium.com), while more advanced methods (notably cross-attention for text-to-image) were shown to greatly enhance the expressiveness of the model (runwayml.com).
We described ControlNet as a significant architectural extension enabling spatial conditioning by adding a parallel, trainable branch to a frozen diffusion model (ar5iv.labs.arxiv.org). This allowed diffusion models to accept structured inputs (edges, poses, etc.) and was a prime example of how new capabilities can be introduced by clever architectural additions without retraining from scratch.
In specialized domains, we saw that architectural variants such as EGNN-based networks provide equivariance and are essential for tasks such as molecule generation (ar5iv.labs.arxiv.org), highlighting that the choice of architecture must respect the symmetries or structure of the data for optimal results. We briefly noted that for other data types (audio, video), appropriately tailored architectures (1-D conv networks, 3-D conv or attention over time) are used, but they share common themes of residual learning and multi-scale processing.
Finally, we delved into approaches that reduce the number of diffusion steps. While these are not new architectures per se, methods including progressive distillation (arxiv.org) and consistency models (proceedings.mlr.press) effectively modify how the network is used or trained to achieve fast sampling. We explained that one can view consistency models as pushing the architecture to its extreme — performing the entire generative transformation in one network pass — which blurs the line between diffusion models and traditional one-shot generative models.
It’s remarkable that the very same architectural backbone (e.g. a U-Net) can one day be used in a 1000-step simulation and, after some retraining, also serve as a one-step direct generator, all depending on the objective and constraints applied (arxiv.org).
Throughout the chapter, we have maintained a focus on architectural components and their roles: convolution vs. attention, residual connections, how conditioning information threads through the network, etc. By consolidating or removing components that are less central to current state-of-the-art (for instance, we treated some older architectures such as basic autoencoders or non-residual nets only in passing), we highlighted the elements that repeatedly show up in cutting-edge models: U-Nets (or Transformers) with residual and attention mechanisms, conditioning interfaces (from simple embedding addition to sophisticated hyper-networks e.g., ControlNet), and scalability through latent space modeling (VQ-VAEs enabling LDMs).
The narrative arc of this chapter shows a clear progression: starting from a single network successfully denoising at multiple scales, then adding ways for the network to be guided (first by class labels, then by text, then by images/conditions), then optimizing the network’s usage to be more efficient. Each progression required new architectural ideas or refinements.
In conclusion, diffusion model architecture has proven to be modular and extensible. One can plug in different backbones (CNN or Transformer), attach different “heads” or conditioning paths, or even chain multiple models (as in distillation) and the generative performance only grows. The fluency of the writing in this chapter hopefully made these technical concepts approachable, with code snippets and equations clarifying how these modules function within the overall diffusion framework.
As diffusion models continue to evolve, we anticipate further architectural innovations, perhaps integrating diffusion with retrieval systems, more adaptive transformer structures, or neurosymbolic components, but the building blocks discussed here will likely remain highly relevant. Generative models are an interplay of architecture, training, and inference, and understanding the architecture provides a solid foundation to grasp the full picture of how diffusion and consistency models achieve the feats that were once the domain of GANs or other approaches. With this foundation, a reader is well-equipped to study more advanced topics such as diffusion model solvers, score-based models in continuous time, or the latest hybrid architectures that push generative AI forward.