Implicit Reward Model Extraction From Language Model Log Probabilities

via “reward model training for reinforcement learning from human feedback (rlhf)”

Shanghai AI Lab's multilingual foundation model.

Unique: InternLM provides pre-trained reward models that can be fine-tuned on domain-specific preferences, reducing training time compared to training from scratch; integrates with XTuner for efficient fine-tuning

vs others: More accessible than building custom reward models from scratch; comparable to OpenAI's reward modeling approach but with full transparency and ability to customize for specific domains

via “implicit reward model extraction from language model log-probabilities”

* ⏫ 06/2023: [Faster sorting algorithms discovered using deep reinforcement learning (AlphaDev)](https://www.nature.com/articles/s41586-023-06004-9)

Unique: Mathematically proves that language model log-probability ratios encode reward information, eliminating the need for a separate reward model while maintaining theoretical grounding in reward-based RL frameworks

vs others: More interpretable than black-box RLHF reward models because the reward function is directly derived from model probabilities; more efficient than training separate reward models because no additional training is required

via “reward model training from pairwise human preference comparisons”

* ⭐ 03/2022: [Multitask Prompted Training Enables Zero-Shot Task Generalization (T0)](https://arxiv.org/abs/2110.08207)

Unique: Uses a language model itself as the reward model rather than a separate scoring function, enabling the reward model to understand semantic nuances in instructions and outputs. The pairwise comparison approach is more data-efficient than absolute scoring and better captures relative preferences.

vs others: More semantically sophisticated than hand-crafted reward functions or simple metrics, and more data-efficient than absolute rating scales because pairwise comparisons provide stronger training signals for preference learning.

via “token probability and logit inspection for interpretability”

Python bindings for the llama.cpp library

Unique: Direct access to llama.cpp's logit computation without post-processing, enabling inspection of raw model outputs before sampling, useful for implementing custom decoding strategies or analyzing model behavior

vs others: More detailed than OpenAI API which only returns top-k alternatives, and lower latency than Hugging Face Transformers because logits are computed in the same inference pass

via “language model policy parameterization with action logit extraction”

### Other Papers <a name="2023op"></a>

Unique: Directly uses language model logits as the policy without a separate policy network, enabling end-to-end optimization of the language model for both generation quality and task success — this is distinct from approaches that train separate policy heads on top of frozen language models

vs others: More parameter-efficient than separate policy networks because it reuses the language model's existing capacity, and more interpretable because action selection is grounded in language model semantics

via “reward design with language model guidance”

* ⏫ 03/2023: [Reward Design with Language Models](https://arxiv.org/abs/2303.00001)

Unique: RLPD integrates LM-based reward design as a first-class component with automatic validation against offline data, whereas prior work treats reward engineering as a separate manual step. This enables end-to-end specification of RL tasks from natural language to learned policies.

vs others: More flexible than hand-crafted rewards because LMs can express complex multi-objective specifications, and more reliable than pure inverse RL because rewards are validated against ground-truth offline trajectories before deployment