Transformer Variants: From Encoder-Decoder to State Space Models

Status: public · Confidence: medium (0.8) · Basis: verified_sources

## TL;DR

Transformer variants adapt the original attention-based architecture to different data types and scaling constraints. Major branches include encoder-decoder language models, encoder-only models, decoder-only models, vision Transformers, sparse or linear attention variants, and newer state-space alternatives.

## Core Explanation

The original Transformer used self-attention and cross-attention in an encoder-decoder architecture. Later variants changed the architecture for language understanding, autoregressive generation, vision tasks, and long-context efficiency. Vision Transformer treats image patches as tokens, while Swin Transformer adds hierarchical shifted windows. Efficient Transformer research studies ways to reduce the quadratic cost of standard attention on long sequences.

## Further Reading

- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
- [An Image is Worth 16x16 Words](https://arxiv.org/abs/2010.11929)
- [Swin Transformer](https://arxiv.org/abs/2103.14030)
- [Efficient Transformers: A Survey](https://doi.org/10.1145/3530811)

## Related Articles

- [State Space Models: Mamba, Linear-Time Sequence Modeling, and Alternatives to Transformers](../state-space-models.md)
- [3D Human Modeling: Parametric Body Models, Mesh Recovery, and Digital Avatars](../3d-human-modeling.md)
- [AI Art and Creativity: Generative Models and Authorship](../ai-art-and-creativity.md)