Retrospective Trajectory Optimization Via Policy Gradient Learning

via “agent behavior learning and policy optimization”

Hi HN,I’m Vincent from Aden. We spent 4 years building ERP automation for construction (PO/invoice reconciliation). We had real enterprise customers but hit a technical wall: Chatbots aren't for real work. Accountants don't want to chat; they want the ledger reconciled while they slee

Unique: Learns topology and routing policies from execution traces using ML, enabling data-driven optimization of agent networks without manual tuning

vs others: More sophisticated than heuristic-based evolution, but requires more data and expertise; less predictable than rule-based optimization

via “generative-reward-optimization-grpo-training”

Train transformer language models with reinforcement learning.

Unique: Implements unified reward+policy training where the model generates both outputs and rewards in a single forward pass, reducing pipeline complexity compared to RLHF while maintaining explicit reward signals through a learned reward head

vs others: More integrated than RLHF because it eliminates separate reward model training, while more explicit than DPO because it maintains interpretable reward scores that can be inspected and debugged

via “imagination-based policy optimization with latent rollouts”

* ⏫ 02/2023: [Grounding Large Language Models in Interactive Environments with Online RL (GLAM)](https://arxiv.org/abs/2302.02662)

Unique: DreamerV3 uses a two-headed value function (critic and target) trained on imagined trajectories with symlog scaling, enabling stable policy optimization without explicit target networks or replay buffers. The imagination rollout is differentiable end-to-end, allowing gradients to flow through the world model during policy updates (though the world model is typically frozen).

vs others: Achieves better sample efficiency than model-free RL (PPO, SAC) by training on imagined rollouts, while maintaining stability through careful value function design and avoiding the distribution shift issues that plague naive model-based approaches.

via “distributed policy gradient optimization across gpu clusters”

* ⭐ 02/2022: [Magnetic control of tokamak plasmas through deep reinforcement learning](https://www.nature.com/articles/s41586-021-04301-9%E2%80%A6)

Unique: Uses distributed PPO with asynchronous experience collection and synchronized gradient updates across GPU clusters, with careful load balancing to ensure all workers remain busy and communication overhead is minimized through efficient allreduce patterns

vs others: Achieves 10-50x faster wall-clock training time than single-GPU PPO by distributing environment rollouts across many workers while maintaining training stability through synchronized policy updates, compared to fully asynchronous methods that suffer from stale gradient problems

via “trajectory-conditioned solution generation with scoring feedback”

* ⏫ 10/2023: [Eureka: Human-Level Reward Design via Coding Large Language Models (Eureka)](https://arxiv.org/abs/2310.12931)

Unique: Encodes the full optimization history as in-context examples rather than using a learned surrogate model or explicit reward function. The LLM implicitly learns to recognize patterns in the trajectory (e.g., 'solutions with property X scored higher') and applies those patterns to generate the next candidate, enabling adaptation without explicit model updates.

vs others: Simpler and faster to implement than Bayesian optimization or neural surrogate models, while capturing richer semantic patterns than random search or grid search by leveraging the LLM's pre-trained understanding of solution quality.

via “reward shaping and curriculum learning for complex locomotion tasks”

* ⭐ 10/2022: [Discovering faster matrix multiplication algorithms with reinforcement learning (AlphaTensor)](https://www.nature.com/articles/s41586-022%20-05172-4)

Unique: Combines multi-component reward shaping with progressive curriculum learning, where task difficulty increases automatically as policy performance improves, enabling stable training toward complex locomotion objectives

vs others: Guides RL training toward natural, energy-efficient gaits by decomposing objectives into weighted reward components and progressively increasing difficulty, compared to sparse reward or single-objective approaches

via “proximal policy optimization (ppo) for language model policy optimization”

* ⭐ 03/2022: [Multitask Prompted Training Enables Zero-Shot Task Generalization (T0)](https://arxiv.org/abs/2110.08207)

Unique: Applies PPO with KL regularization to language generation, treating token selection as sequential decisions and using a learned reward model as the optimization signal. The KL penalty against the supervised fine-tuned model prevents reward hacking and maintains general language capabilities while optimizing for human preferences.

vs others: More stable and sample-efficient than vanilla policy gradient methods, and the KL regularization prevents the model from diverging too far from human-like language patterns while still optimizing for preferences, unlike unconstrained RL which can lead to reward hacking.

### Other Papers <a name="2023op"></a>

Unique: Applies policy gradient optimization directly to language model action logits using retrospective trajectory data, enabling agents to learn from their own execution history without external reward models or human feedback — a departure from supervised fine-tuning or RLHF approaches that require explicit human preferences

vs others: More sample-efficient than online RL methods because it reuses trajectories already generated during agent deployment, and more scalable than RLHF because it avoids human annotation bottlenecks by learning from task outcomes directly

via “policy improvement with offline-constrained actor-critic updates”

* ⏫ 03/2023: [Reward Design with Language Models](https://arxiv.org/abs/2303.00001)

Unique: RLPD applies KL-divergence constraints directly in the policy gradient update rather than as a separate regularization term, enabling tighter control over policy evolution and more principled constraint satisfaction compared to penalty-based approaches.

vs others: More stable than unconstrained policy gradient methods (SAC, PPO) when offline data is available, and more flexible than fully offline methods (CQL, IQL) because constraints are soft and can be relaxed as online evidence accumulates