CodeFormer

Restores degraded or low-quality facial images using a transformer-based architecture with codebook-based generative priors. The system decomposes restoration into content tokens (structural information) and quality tokens (texture/detail), enabling recovery of fine facial features from heavily compressed, blurry, or artifact-laden inputs. Uses a multi-scale feature extraction pipeline with cross-attention mechanisms to align degraded input features with learned high-quality facial priors stored in a learned codebook.

Extracts hierarchical facial features from degraded input images at multiple scales (coarse structure → fine details) and aligns them with learned high-quality facial priors through cross-attention mechanisms. The architecture uses progressive feature refinement, where coarse features guide fine-grained restoration, preventing misalignment and structural distortion. Implements spatial attention to focus restoration effort on facial regions (eyes, mouth, nose) most sensitive to quality degradation.

Maintains a learned codebook of high-quality facial feature representations (discrete latent codes) trained on clean facial image distributions. During restoration, degraded input features are mapped to nearest codebook entries, and high-quality features are synthesized by interpolating or selecting from the codebook. This approach constrains the restoration to plausible facial variations, preventing hallucination of unrealistic features. The codebook is trained via vector quantization, enabling discrete latent space search.

Provides a Gradio-based web interface for uploading degraded facial images and viewing restoration results in real-time. The interface handles image upload, preprocessing (face detection, alignment), model inference, and side-by-side comparison visualization. Gradio manages HTTP request/response handling, file storage, and browser rendering without requiring local installation. The interface includes sliders or toggles for controlling restoration intensity or quality parameters.

Detects facial regions in input images using a pre-trained face detector (likely MTCNN, RetinaFace, or similar), extracts bounding boxes, and crops/aligns the face region for restoration. The detector handles multiple faces, extreme poses, and occlusions with configurable confidence thresholds. Extracted face regions are normalized (resized, centered) before feeding to the restoration model, ensuring consistent input dimensions and reducing computational overhead.

Decomposes the restoration task into two parallel streams: content tokens (capturing facial structure, identity, pose) and quality tokens (capturing texture, fine details, surface properties). This decomposition allows the model to preserve identity while selectively enhancing quality, preventing over-smoothing or hallucination. Content tokens are extracted from the degraded input and refined using priors; quality tokens are synthesized from the codebook. The two streams are recombined to produce the final restored image.