Multi Dimensional Preference Annotation Across Llm Responses

via “multi-dimensional preference annotation across llm responses”

64K preference dataset for RLHF training.

Unique: Explicitly decomposes preference feedback into four independent dimensions (helpfulness, honesty, instruction-following, truthfulness) rather than collapsing into a single reward signal, allowing models to learn trade-offs and enabling analysis of which behaviors matter most for different use cases. This architectural choice enables training models that can balance competing objectives rather than optimizing for a single monolithic preference.

vs others: More granular than single-axis preference datasets (like HHRLHF) because it captures orthogonal dimensions of quality, enabling researchers to study and optimize for specific behavioral trade-offs rather than assuming all preferences align on one axis.