Diffusion Models

Image Generation

Diffusion Transformer (DiT)

• FLUX.1 [Code]

  • Black Forest Labs

  • Text-to-image generation

  • Models

    • FLUX.1-dev: https://huggingface.co/black-forest-labs/FLUX.1-dev

    • FLUX.1-schnell: https://huggingface.co/black-forest-labs/FLUX.1-schnell

• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv:2403.03206) [arXiv] [Blog]

  • Stability AI

  • Stable Diffusion 3 (SD3)

  • Multimodal Diffusion Transformer (MMDiT)

  • Models

    • Stable Diffusion 3 Medium: https://huggingface.co/stabilityai/stable-diffusion-3-medium

• Scalable Diffusion Models with Transformers (ICCV 2023) [arXiv] [Paper] [Code] [Homepage]

  • UC Berkeley & NYU

  • DiT

UNet

• Kolors: Effective Training of Diffusion Model for Photorealistic Text-to-Image Synthesis [Technical Report]

  • Kuaishou Kolors

  • Text-to-image generation

  • Model: https://huggingface.co/Kwai-Kolors/Kolors

• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (arXiv:2307.01952) [arXiv]

  • Stability AI

  • Models

    • https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0

    • https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0

• High-Resolution Image Synthesis with Latent Diffusion Models (CVPR 2022) [Paper] [arXiv] [Code]

  • LMU Munich & Runway ML

  • Latent Diffusion Models (LDMs)

  • Models

    • Stable-Diffusion-v1-5: https://huggingface.co/runwayml/stable-diffusion-v1-5

      • Initialized with the weights of the Stable-Diffusion-v1-2 checkpoint and subsequently fine-tuned on 595k steps at resolution 512x512.

Video Generation

• Stable Video 4D (SV4D)

  • Stability AI

  • Model: https://huggingface.co/stabilityai/sv4d

    • Generate 40 frames (5 video frames x 8 camera views) at 576x576 resolution, given 5 reference frames of the same size.

• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (arXiv:2311.15127) [arXiv] [Blog]

  • Stability AI

  • Stable Video Diffusion (SVD)

  • Text-to-video and image-to-video generation

  • Models

    • https://huggingface.co/stabilityai/stable-video-diffusion-img2vid

      • Generate 14 frames at resolution 576x1024 given a context frame of the same size.

    • https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt

      • Fine-tuned from the SVD-img2vid.

      • Generate 25 frames at resolution 576x1024 given a context frame of the same size.

Acronyms

• LLM: Large Language Model