Grid Local Attention With Shifted Window Boundaries

via “grid-local attention with shifted window boundaries”

* ⭐ 04/2022: [Hierarchical Text-Conditional Image Generation with CLIP Latents (DALL-E 2)](https://arxiv.org/abs/2204.06125)

Unique: Applies orthogonal axis decomposition with shifted windows on transposed dimensions, creating true 2D receptive field expansion through two sequential attention passes rather than single-axis shifting — enables global context with linear complexity

vs others: Achieves better global context coverage than single-axis Swin Transformer with comparable efficiency, and provides more structured receptive field growth than sparse attention patterns