Portfolio Optimization With Reinforcement Learning

via “reward model training with configurable loss functions”

Reinforcement learning from human feedback — SFT, DPO, PPO trainers for LLM alignment.

Unique: Supports multiple loss variants (Bradley-Terry, Elo, margin-based) with automatic hyperparameter suggestions based on dataset statistics, and includes built-in reward calibration utilities to estimate preference probabilities from scores

vs others: More flexible than monolithic reward models because it supports both regression and ranking objectives; better integrated with TRL's ecosystem than standalone reward modeling libraries because it shares data pipeline and chat template handling

via “reinforcement learning training with preference optimization”

2x faster LLM fine-tuning with 80% less memory — optimized QLoRA kernels for consumer GPUs.

Unique: Integrates preference optimization (DPO) with Unsloth's kernel optimizations and LoRA training, enabling efficient preference-based learning on consumer GPUs. Provides a unified framework for supervised and preference-based fine-tuning, whereas most frameworks treat them separately.

vs others: More accessible than full RL training because DPO doesn't require reward models or complex RL infrastructure, and more efficient than standard DPO because custom kernels optimize preference loss computation, whereas standard implementations use generic PyTorch operations.

via “agentic reinforcement learning training pipeline for agent optimization”

📚 《从零开始构建智能体》——从零开始的智能体原理与实践教程

Unique: Provides concrete patterns for implementing RL training loops for agents, including reward signal generation and trajectory collection, treating RL as an optional optimization layer rather than a requirement, enabling teams to start with prompt-based agents and add RL training as they scale

vs others: More sophisticated than pure prompt engineering but more practical than full policy learning from scratch; enables continuous improvement of agent behavior based on real-world performance

via “model fine-tuning and optimization with rl and prompt tuning”

Build and run agents you can see, understand and trust.

Unique: Integrates RL-based fine-tuning and prompt tuning as first-class optimization capabilities, allowing agents to improve their behavior through learning rather than requiring manual prompt engineering or model retraining

vs others: More integrated than LangChain's optimization support because fine-tuning and prompt tuning are built into the framework; more practical than AutoGen's optimization because it provides concrete RL and prompt tuning implementations

via “agent behavior learning and policy optimization”

Hi HN,I’m Vincent from Aden. We spent 4 years building ERP automation for construction (PO/invoice reconciliation). We had real enterprise customers but hit a technical wall: Chatbots aren't for real work. Accountants don't want to chat; they want the ledger reconciled while they slee

Unique: Learns topology and routing policies from execution traces using ML, enabling data-driven optimization of agent networks without manual tuning

vs others: More sophisticated than heuristic-based evolution, but requires more data and expertise; less predictable than rule-based optimization

via “portfolio optimization with constraint-aware agent reasoning”

FinRobot: An Open-Source AI Agent Platform for Financial Analysis using LLMs 🚀 🚀 🚀

Unique: Implements portfolio optimization through agent reasoning over constraints rather than pure mathematical optimization, enabling explainable allocation decisions and constraint satisfaction verification

vs others: Produces explainable portfolio recommendations with constraint justifications, whereas pure optimization approaches generate allocations without reasoning about why constraints are satisfied

via “configurable-rl-algorithm-implementation-with-ppo-and-grpo-variants”

The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.

Unique: Decouples reference model and critic network management from the main training loop, enabling efficient computation of KL penalties and advantage estimates without duplicating model weights in GPU memory. Asynchronous training orchestration allows rollout workers to continue collecting trajectories while the trainer processes previous batches, reducing idle time.

vs others: More flexible than TRL's PPO implementation because it supports multiple algorithm variants and explicit reference model management; more specialized than general RL frameworks like RLlib because it's optimized specifically for language model training with agentic workflows.

via “reinforcement-learning-training-with-dpo-and-ppo”

Web UI for training and running open models like Gemma 4, Qwen3.6, DeepSeek, gpt-oss locally.

Unique: Integrates DPO and PPO training directly with Unsloth's kernel optimizations, reusing the same attention and quantization kernels as supervised fine-tuning, and provides a unified training API that handles preference data formatting, reward computation, and policy updates without requiring external RL frameworks

vs others: Faster than trl library's standalone implementations because it leverages Unsloth's kernel optimizations for forward/backward passes, and more integrated than separate RL frameworks because it shares model loading, quantization, and export pipelines with supervised training

Professional-grade stock market analysis and predictions powered by AI, accessible directly through Claude Desktop. **Key Features:** • 10-day price predictions - 79.86% directional accuracy (validated on 12,901 predictions) • Market regime detection - Bull/bear/sideways classification • AI-powered

Unique: Utilizes a dynamic reinforcement learning approach that adapts to changing market conditions, providing tailored portfolio management strategies.

vs others: Offers a more adaptive and intelligent optimization process compared to static portfolio management tools.

via “generative-reward-optimization-grpo-training”

Train transformer language models with reinforcement learning.

Unique: Implements unified reward+policy training where the model generates both outputs and rewards in a single forward pass, reducing pipeline complexity compared to RLHF while maintaining explicit reward signals through a learned reward head

vs others: More integrated than RLHF because it eliminates separate reward model training, while more explicit than DPO because it maintains interpretable reward scores that can be inspected and debugged

via “reward function design and shaping for complex multi-objective tasks”

* ⭐ 02/2022: [Magnetic control of tokamak plasmas through deep reinforcement learning](https://www.nature.com/articles/s41586-021-04301-9%E2%80%A6)

Unique: Combines potential-based reward shaping with multi-objective weighting to balance lap time, safety, and race position, using domain knowledge about racing physics to structure rewards that guide learning without over-constraining agent behavior or creating conflicting gradient signals

vs others: Achieves better policy robustness than single-objective rewards (lap time only) by explicitly balancing safety and race performance, and better sample efficiency than inverse RL approaches by leveraging domain knowledge to structure rewards directly

via “reward shaping and curriculum learning for complex locomotion tasks”

* ⭐ 10/2022: [Discovering faster matrix multiplication algorithms with reinforcement learning (AlphaTensor)](https://www.nature.com/articles/s41586-022%20-05172-4)

Unique: Combines multi-component reward shaping with progressive curriculum learning, where task difficulty increases automatically as policy performance improves, enabling stable training toward complex locomotion objectives

vs others: Guides RL training toward natural, energy-efficient gaits by decomposing objectives into weighted reward components and progressively increasing difficulty, compared to sparse reward or single-objective approaches

via “proximal policy optimization (ppo) for language model policy optimization”

* ⭐ 03/2022: [Multitask Prompted Training Enables Zero-Shot Task Generalization (T0)](https://arxiv.org/abs/2110.08207)

Unique: Applies PPO with KL regularization to language generation, treating token selection as sequential decisions and using a learned reward model as the optimization signal. The KL penalty against the supervised fine-tuned model prevents reward hacking and maintains general language capabilities while optimizing for human preferences.

vs others: More stable and sample-efficient than vanilla policy gradient methods, and the KL regularization prevents the model from diverging too far from human-like language patterns while still optimizing for preferences, unlike unconstrained RL which can lead to reward hacking.

via “retrospective trajectory optimization via policy gradient learning”

### Other Papers <a name="2023op"></a>

Unique: Applies policy gradient optimization directly to language model action logits using retrospective trajectory data, enabling agents to learn from their own execution history without external reward models or human feedback — a departure from supervised fine-tuning or RLHF approaches that require explicit human preferences

vs others: More sample-efficient than online RL methods because it reuses trajectories already generated during agent deployment, and more scalable than RLHF because it avoids human annotation bottlenecks by learning from task outcomes directly

via “portfolio-optimization-modeling”

via “ai-driven-portfolio-optimization”

via “portfolio optimization and rebalancing recommendations”

Unique: Finster likely integrates ML-predicted returns directly into the optimization objective rather than using historical averages, and includes compliance-aware constraints (ESG filters, regulatory position limits) natively in the solver formulation

vs others: Combines ML-driven return predictions with constrained optimization to respect institutional constraints, whereas traditional robo-advisors use static allocation rules or simple mean-variance optimization with historical inputs