Train Test Split Stratification And Benchmark Reproducibility

via “train-test split with language-stratified sampling”

6M functions across 6 languages paired with documentation.

Unique: Implements language-stratified sampling to ensure balanced representation of all 6 languages in train/test splits, preventing models from overfitting to high-resource languages (Python, Java) at the expense of low-resource languages (Ruby, PHP). This design choice directly influenced how subsequent code datasets (e.g., CodeSearchNet's successors) structure their splits.

vs others: More rigorous than random train/test splits because it ensures language distribution is preserved, enabling fair evaluation of multi-language models and preventing spurious performance gains from language-specific biases.