Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Looped Transformers

Looped Transformers repeatedly apply shared Transformer blocks to refine hidden states, increasing effective computational depth without adding a new set of weights at every step.

Comparison of a k-layer Transformer, a shared k-layer block looped L times, an unshared kL-layer Transformer, and middle-block looping.
A looped Transformer reuses the same k-layer block for L iterations; middle looping repeats only the middle block.

Foundations

  • Looped Transformers are Better at Learning Learning Algorithms (ICLR 2024) [Paper] [arXiv] [Code]

    • UW-Madison

    • Trains a shared Transformer block to emulate iterative learning algorithms for in-context regression and other data-fitting tasks.

    • Matches standard Transformers on the studied tasks with less than 10% of their parameter count.

  • Looped Transformers as Programmable Computers (ICML 2023) [Paper] [arXiv]

    • UW-Madison & Princeton

    • Constructs a constant-depth looped Transformer that executes programs encoded in its input, including memory operations, function calls, and conditional branches.

    • Demonstrates calculator, linear-algebra, and in-context backpropagation programs with a frozen Transformer.

  • Universal Transformers (ICLR 2019) [Paper] [arXiv]

    • Amsterdam & DeepMind & Google Brain

    • Introduces Universal Transformer (UT): self-attentive recurrence across depth with shared weights and parallel processing across sequence positions.

    • Adds per-position halting with Adaptive Computation Time (ACT); improves algorithmic and language tasks and establishes Turing completeness under stated assumptions.

Reasoning, Generalization, and Mechanisms

  • Reasoning with Latent Thoughts: On the Power of Looped Transformers (ICLR 2025) [Paper] [arXiv]

    • Google Research & TTIC

    • Shows that a shallow Transformer looped multiple times can approach deeper untied models on synthetic and language-model reasoning tasks.

    • Proves that looped models can simulate chain-of-thought steps through latent computation, separating effective depth from parameter count.

  • Looped Transformers for Length Generalization (ICLR 2025) [Paper] [arXiv] [Code]

    • UW-Madison & MIT & UC Berkeley

    • Trains looped Transformers with adaptive step counts on tasks expressible as repeated operations in RASP-L, a learnable subset of the Restricted Access Sequence Processing language.

    • Improves extrapolation to unseen input lengths on arithmetic and algorithmic tasks.

  • Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024) [Paper] [arXiv]

    • MIT & Google Research

    • Proves that population-loss minimizers of linear looped Transformers implement multi-step preconditioned gradient descent for in-context linear regression.

    • Establishes gradient-flow convergence through a gradient-dominance condition, addressing learnability beyond expressivity.

Recurrent-Depth Language Models and Latent Reasoning

  • Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (NeurIPS 2025) [Paper] [arXiv] [Code]

    • ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems & Tübingen AI Center & UMD & LLNL

    • Trains Huginn, a 3.5B-parameter recurrent-depth language model, on 800B tokens without specialized reasoning supervision.

    • Scales inference computation by iterating a latent block; studies adaptive per-token compute, key-value (KV) cache sharing, and speculative decoding.

  • Scaling Latent Reasoning via Looped Language Models (arXiv:2510.25741) [arXiv] [Homepage]

    • ByteDance Seed & UCSC & Princeton & Mila & UdeM & PKU & CMU & UPenn & Conscium & Manchester & M-A-P

    • Introduces Ouro / LoopLM, combining recurrent latent computation with an entropy-regularized objective for learned depth allocation during pretraining.

    • Scales training to 7.7T tokens and uses controlled experiments to separate knowledge manipulation from knowledge capacity.

Extensions to Diffusion Models

  • Looped Diffusion Transformer (arXiv:2609.40305) [arXiv] [Code]

    • SenseTime Research & THU & NTU

    • Looped-DiT reuses Transformer blocks within each denoising step, combining intermediate-loop supervision with self-modulating attention.

    • Compares extra recurrent depth with extra denoising steps under fixed inference budgets and evaluates text-to-image generation at matched parameters and compute.

Last updated