Response Quality Variance Quantification Across Model Families

64K preference dataset for RLHF training.

Unique: Includes responses from models with intentionally different capability levels (GPT-4 vs Llama-7B), enabling quantification of quality variance and identification of prompts where models diverge. This variance is preserved in the dataset rather than normalized away, supporting analysis of preference learning robustness to quality variation.

vs others: More informative than preference datasets with responses from similar-capability models because it captures quality variance across the capability spectrum, enabling analysis of whether preference learning methods are robust to variation in response quality or sensitive to specific model pairs.