Mastering Diverse Domains through World Models (DreamerV3)

DreamerV3 learns a compact world model that predicts future states in a learned latent space, then uses this model to plan and train policies through imagination without requiring environment interaction for every gradient step. The architecture uses a variational autoencoder (VAE) to compress observations into a latent representation, a recurrent state-space model to predict latent dynamics, and a decoder to reconstruct observations. Policy and value functions are trained on imagined trajectories generated by rolling out the world model, dramatically reducing sample complexity compared to model-free RL.

DreamerV3 learns a single world model that captures visual dynamics common across multiple tasks, then trains separate task-specific policy heads that leverage the shared latent representation. The world model is trained on a mixture of trajectories from different tasks without explicit task conditioning, discovering task-invariant visual features (object motion, physics) that transfer across diverse objectives. Task-specific policies are trained through imagination using the shared world model, enabling rapid adaptation to new tasks with minimal additional data.

DreamerV3 is extended in the GLAM framework to ground large language models (LLMs) in interactive environments through online RL. The approach uses an LLM to generate high-level task descriptions or reward functions, which are then used to train RL agents in simulated or real environments. The agent learns a world model of the environment and uses it to optimize policies that maximize the LLM-specified rewards. This enables LLMs to interact with and learn from environments without explicit programming of reward functions or environment dynamics.

DreamerV3 handles both continuous (robotic control) and discrete (Atari games) action spaces through a unified policy parameterization in the learned latent space. The policy network outputs action distributions (Gaussian for continuous, categorical for discrete) that are sampled during imagination rollouts. The world model's dynamics function is action-agnostic, treating actions as inputs to the recurrent state predictor without architectural changes, enabling seamless switching between control modalities.

DreamerV3 trains policies by rolling out imagined trajectories in the learned latent space, computing policy gradients without environment interaction. The process involves: (1) sampling initial latent states from the world model's prior, (2) rolling out the policy in imagination for H steps, (3) computing returns using the value function, and (4) backpropagating policy gradients through the imagined trajectory. The world model is frozen during policy optimization, enabling efficient amortization of world model computation across multiple policy updates.

DreamerV3 introduces symlog (symmetric logarithm) scaling to handle rewards spanning 10+ orders of magnitude without task-specific normalization. The symlog function applies log scaling to large-magnitude rewards while preserving linear scaling for small rewards, enabling a single value function and reward prediction head to handle both sparse rewards (e.g., game scores of 0-1000) and dense rewards (e.g., continuous control with rewards in [-1, 1]). This is applied to both reward prediction in the world model and value function targets, eliminating the need for per-task reward normalization.

DreamerV3 trains the world model and policy jointly using a unified loss function that combines reconstruction, dynamics, and policy objectives. The world model learns to compress observations into a latent space that is simultaneously useful for predicting future states and for learning control policies. The policy and value function are trained on imagined rollouts from the world model, creating a feedback loop where policy performance informs which latent features are most useful for control. This joint training is enabled by a shared encoder/decoder architecture and careful balancing of loss weights.

DreamerV3 uses a variational autoencoder (VAE) to compress high-dimensional visual observations (e.g., 64x64 RGB images) into a compact latent representation (typically 32-256 dimensions). The encoder network maps observations to a Gaussian distribution in latent space, while the decoder reconstructs observations from latent samples. The VAE is trained with a reconstruction loss (L2 or L1) and a KL divergence regularizer that encourages the latent distribution to match a standard normal prior. This compression enables efficient world model learning and policy optimization in the latent space.

DreamerV3 models environment dynamics using a recurrent state-space model where a GRU (gated recurrent unit) network predicts the next latent state given the current latent state and action. The GRU maintains a hidden state that captures temporal dependencies and long-range correlations in the environment dynamics. The model is trained to minimize prediction error on one-step-ahead latent state predictions, enabling efficient amortization of dynamics learning across multiple rollout steps. The recurrent structure enables the model to learn complex temporal patterns (e.g., object momentum, delayed effects) without explicit temporal convolutions.

DreamerV3 trains a value function using a two-headed critic architecture where one head predicts the value of the current state (critic) and another predicts the target value for bootstrapping (target). Both heads are trained on imagined trajectories using symlog-scaled returns computed from the world model's reward predictions and the target value function. The two-headed design enables stable bootstrapping without explicit target networks or replay buffers. The value function is trained with a Huber loss to reduce sensitivity to outliers in the imagined returns.

DreamerV3 supports online RL where the world model is continuously updated with new environment interactions, enabling the agent to adapt to changing environments or learn from new data. The process involves: (1) collecting environment interactions using the current policy, (2) adding new transitions to a replay buffer, (3) updating the world model on a mixture of old and new data, and (4) optimizing the policy on imagined rollouts from the updated world model. This enables the agent to discover and adapt to environment changes without retraining from scratch.