Efficient Self Attention With Local Window Constraints

via “efficient block-local attention with spatial locality bias”

* ⭐ 04/2022: [Hierarchical Text-Conditional Image Generation with CLIP Latents (DALL-E 2)](https://arxiv.org/abs/2204.06125)

Unique: Uses learnable 2D relative position biases within fixed-size windows to encode spatial locality, enabling efficient local attention with explicit geometric inductive bias — distinct from absolute positional encodings and from attention without position bias

vs others: More efficient than full self-attention for high-resolution images while maintaining stronger spatial locality than global attention, and provides better inductive bias for vision tasks than position-free local attention

via “efficient self-attention with local window constraints”

* ⭐ 07/2022: [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors... (Swin UNETR)](https://link.springer.com/chapter/10.1007/978-3-031-08999-2_22)

Unique: Implements shifted window attention where consecutive transformer blocks use offset window partitions (e.g., shifting by half window size), creating a checkerboard pattern that enables information flow between adjacent windows without computing full global attention. This architectural pattern reduces complexity while maintaining effective receptive field growth across layers.

vs others: Achieves 3-4x faster inference than global attention ViT variants on 224×224 images while maintaining comparable accuracy, and uses 50% less peak memory during training compared to full self-attention implementations.