MaxViT: Multi-Axis Vision Transformer (MaxViT)

MaxViT implements a dual-axis attention mechanism that decomposes full 2D spatial attention into sequential block-local and grid-local attention passes, reducing computational complexity from O(N²) to O(N) while maintaining receptive field coverage. The architecture alternates between local window attention (attending within fixed spatial blocks) and shifted-window attention (attending across block boundaries), enabling efficient modeling of both local texture and global semantic relationships in images without requiring full quadratic attention matrices.

MaxViT constructs a hierarchical pyramid of feature maps across multiple depths by progressively downsampling spatial dimensions while increasing channel capacity, using multi-axis attention at each level. Token aggregation occurs through overlapping patch embedding at different scales, enabling the model to capture features from fine-grained local patterns to coarse semantic structures. This design mirrors CNN-style feature pyramids but maintains transformer's flexibility for variable input resolutions and global context.

MaxViT implements block-local attention by partitioning spatial dimensions into non-overlapping windows and computing attention only within each window, with learnable relative position biases that encode spatial locality. This reduces attention computation from O(HW × HW) to O(window_size²) per block, enabling quadratic attention within local neighborhoods while maintaining linear overall complexity. Position biases are parameterized as learnable 2D embeddings that bias attention scores based on relative spatial offsets.

MaxViT complements block-local attention with grid-local attention computed on transposed feature maps, where spatial dimensions are permuted to create orthogonal attention patterns. Shifted window boundaries (similar to Swin Transformer) are applied to enable cross-block communication without explicit global attention. This dual-axis approach ensures that every token can attend to both local neighbors and spatially distant tokens through the combination of two orthogonal attention passes, effectively creating a receptive field larger than individual window sizes.

MaxViT uses overlapping patch embeddings at the input stage and between hierarchical levels, where patches are extracted with spatial overlap rather than non-overlapping tiling. This approach preserves boundary information and reduces aliasing artifacts that occur with non-overlapping patches. Embeddings are computed via learned linear projections that map overlapping spatial regions to token embeddings, enabling smooth feature transitions across patch boundaries and better preservation of fine-grained spatial structure.

MaxViT progressively increases channel dimensions as spatial resolution decreases across the hierarchy, using learned linear projections to expand feature dimensionality at each downsampling step. This design maintains computational balance across levels by trading spatial resolution for channel capacity, ensuring that each hierarchical stage has sufficient representational capacity. Channel expansion ratios are typically 2× per level, implemented via efficient projection layers that can be fused with attention operations.

MaxViT serves as the visual encoder backbone in DALL-E 2, processing images into feature representations that align with CLIP's vision-language embedding space. The hierarchical features from MaxViT are projected into CLIP's latent space, enabling joint vision-language understanding where visual features are semantically aligned with text embeddings. This integration allows the model to leverage both visual and textual information for downstream tasks like text-to-image generation, with the MaxViT encoder providing efficient multi-scale visual understanding.

MaxViT supports variable-resolution inputs through dynamic padding strategies that adapt to input dimensions while maintaining alignment with window and patch sizes. The model pads images to multiples of the combined window and patch sizes, then tracks padding information to enable accurate feature map reconstruction. This design allows efficient batch processing of images with different resolutions without requiring fixed input sizes, enabling flexible deployment across diverse image sources.