LivePortrait

Transforms a static portrait image into an animated video by applying facial motion control derived from a reference video or motion sequence. Uses deep learning-based facial landmark detection and motion transfer to map head pose, eye gaze, and expression changes from a source onto the target portrait while preserving identity and photorealism. The system operates through a multi-stage pipeline: facial analysis → motion extraction → neural rendering with identity preservation constraints.

Extracts facial motion, head pose, and expression parameters from a source video and applies them to a target portrait or video, enabling motion reuse across different identities. The system performs temporal facial landmark tracking across video frames, computes motion deltas (rotation, translation, expression coefficients), and applies these transformations to the target through a neural renderer that maintains target identity while adopting source motion patterns.

Detects and tracks facial landmarks (eyes, nose, mouth, jaw, face contour) across video frames in real-time, computing temporal consistency through Kalman filtering or optical flow constraints. Outputs 2D or 3D landmark coordinates and head pose (pitch, yaw, roll) that serve as input for downstream motion transfer or animation tasks. Uses lightweight CNN or transformer-based detectors optimized for inference speed on consumer GPUs.

Analyzes facial expressions and emotional states in a source face, encodes them as expression coefficients (Action Units or latent emotion vectors), and applies these expressions to a target face while preserving target identity. Uses a disentangled representation where expression and identity are learned in separate latent spaces, enabling independent manipulation. The system leverages facial action unit (FACS) decomposition or learned emotion embeddings to ensure anatomically plausible expression transfer.

Estimates and manipulates head pose (pitch, yaw, roll) and eye gaze direction independently, enabling precise control over where a portrait 'looks' and how its head is oriented. Uses 3D face model fitting or learned pose regression to extract pose parameters, then applies inverse kinematics or neural rendering to reorient the face and eyes without distorting facial features. Supports both continuous pose interpolation and discrete pose targets.

Processes multiple videos sequentially or in parallel, extracting motion parameters (landmarks, pose, expression) from each frame and aggregating results into structured datasets. Implements frame-level parallelization where independent frames are processed concurrently on GPU, with results cached to disk to enable resumable processing of long videos. Outputs motion parameters in standardized formats (JSON, CSV) compatible with downstream animation or training pipelines.

Provides a browser-based UI built with Gradio that enables users to upload images/videos, adjust motion control parameters (pose, expression, motion intensity), and preview results in real-time without coding. Implements client-side parameter validation and server-side inference orchestration, with WebSocket streaming for progressive video output rendering. Supports drag-and-drop file upload, parameter sliders for continuous control, and preset templates for common animation styles.

Accepts heterogeneous input combinations (portrait image + motion video, video + expression parameters, multiple videos for motion blending) and automatically aligns them temporally and spatially for downstream processing. Implements input validation, format conversion, and preprocessing pipelines that normalize different input modalities to a common representation. Supports frame rate conversion, resolution scaling, and temporal interpolation to handle mismatched input specifications.

Provides parametric control over the magnitude and style of applied facial motion through scaling factors and blending weights. Enables users to dial motion intensity from 0% (no motion) to 100%+ (exaggerated motion) by scaling landmark displacement vectors, and to blend between different motion styles (e.g., subtle vs. expressive) through interpolation in motion latent space. Supports preset motion styles (e.g., 'professional', 'energetic', 'subtle') that adjust multiple parameters simultaneously.