Language Pair Validation And Coverage Detection

via “code translation task evaluation with language-pair validation”

Multilingual code evaluation across 17 languages.

Unique: Validates code translation by executing both source and target code against identical unit tests and comparing outputs, ensuring functional equivalence rather than syntactic similarity. Uses language-specific compiler mappings to handle the complexity of 17 different compilation environments and their idiosyncrasies.

vs others: More rigorous than BLEU-score-based translation metrics because it validates actual functional correctness through execution, and covers more language pairs (17 vs typical 2-4) with explicit compiler integration.