Symbolic Discovery of Optimization Algorithms (Lion)

Discovers novel optimization algorithms through symbolic regression and genetic programming by searching the space of mathematical expressions. The system uses tree-based symbolic representations to compose primitive operations (addition, multiplication, momentum terms, adaptive learning rates) into complete optimizer update rules, then evaluates candidates on benchmark optimization tasks to identify high-performing algorithms. This approach generates human-interpretable optimizer equations rather than black-box neural network policies.

Transfers knowledge from large-scale vision-language models (trained on web data) to robotic control by grounding language understanding in robot action spaces. The system leverages pre-trained multimodal representations to map visual observations and natural language instructions to robot motor commands, enabling robots to execute complex manipulation tasks described in language without task-specific retraining. This bridges the gap between internet-scale language-vision knowledge and embodied robotic control through action-grounded fine-tuning.

Evaluates and improves the generalization of discovered/learned optimizers by testing them on held-out optimization tasks with different loss landscapes, architectures, and problem structures. The system measures optimizer performance across diverse benchmarks (vision, language, reinforcement learning) to identify which discovered algorithms transfer well versus overfit to discovery-phase tasks. This capability enables filtering of discovered optimizers for real-world applicability and understanding of generalization boundaries.

Maps natural language descriptions to robot action sequences by learning joint embeddings of vision, language, and action modalities. The system encodes visual observations and language instructions into a shared representation space, then decodes this representation into executable robot actions through a learned action decoder. This enables the model to understand semantic relationships between language concepts and their corresponding motor behaviors, supporting compositional generalization to novel language-action combinations.

Discovers optimizers specialized for specific optimization problem classes by running symbolic regression on benchmark suites tailored to those domains. The system evaluates candidate optimizer expressions on representative tasks (e.g., training vision transformers, fine-tuning language models, RL policy optimization) and selects expressions that maximize convergence speed and final performance on those specific benchmarks. This produces domain-tuned optimizers that outperform general-purpose algorithms on their target problem class.