IFEval

IFEval evaluates LLM instruction-following by defining a library of 23+ verifiable formatting constraints (word count limits, keyword inclusion, bullet points, capitalization patterns, JSON structure requirements) that can be automatically checked against model outputs without human judgment. The evaluation framework parses constraint specifications, applies them to generated text using regex, string matching, and structural parsing, then computes pass/fail metrics across a dataset of 541 instructions with varying constraint complexity.

IFEval parses human-readable constraint specifications embedded in instructions (e.g., 'Your response must be between 100-200 words' or 'Include the keyword IMPORTANT') into structured constraint objects with normalized parameters. The parser extracts constraint type, bounds, keywords, and formatting rules using regex and string matching, then validates constraint syntax and resolves ambiguities (e.g., 'at least 5 bullet points' → constraint type: bullet_points, min: 5).

IFEval ensures reproducible evaluation by implementing deterministic constraint checkers that produce identical results across runs, without randomness or non-deterministic behavior. The evaluation pipeline is stateless and does not depend on external services or non-deterministic operations, enabling bit-for-bit reproducible results when evaluating the same model outputs against the same constraints.

IFEval evaluates model outputs against multiple constraints simultaneously, computing pass/fail scores for each constraint independently and aggregating them into instruction-level and dataset-level metrics. The evaluation engine applies constraint checkers in sequence (word count validator, keyword matcher, structural parser for JSON/bullet points, etc.), tracks which constraints pass/fail, and generates detailed failure reports identifying which specific constraints caused instruction-following failures.

IFEval validates word count constraints by tokenizing model output using whitespace splitting, counting tokens, and comparing against specified bounds (minimum, maximum, or exact word count). The validator handles edge cases like punctuation attachment to words, contractions, and hyphenated words using standard whitespace tokenization, then reports pass/fail and actual vs. required word counts.

IFEval validates keyword constraints by searching for required keywords in model output using case-insensitive substring matching, and verifying that excluded keywords are absent. The validator supports multiple keywords per constraint, handles partial word matches (e.g., 'important' matches 'importantly'), and reports which keywords were found/missing and their positions in the output.

IFEval validates structural formatting constraints by parsing model output for specific patterns: bullet points (lines starting with '-', '*', or numbers), JSON structure (valid JSON parsing), capitalization rules (first letter capitalization, all-caps words), and paragraph structure. The validator uses regex patterns and structural parsing to detect formatting compliance, reporting which structural requirements were met or violated.

IFEval aggregates per-constraint scores into instruction-level metrics (% constraints passed) and dataset-level metrics (mean accuracy, per-constraint success rates, failure distributions). The aggregation engine computes pass rates for each instruction (all constraints must pass for instruction to pass), groups failures by constraint type, and generates summary statistics and detailed reports identifying which instructions and constraints are most problematic.

IFEval includes a curated dataset of 541 instructions with manually annotated formatting constraints, covering diverse domains (writing, analysis, coding, creative tasks) and constraint types (word count, keywords, structure, capitalization). The dataset provides ground-truth constraint annotations, enabling reproducible evaluation of instruction-following across a diverse range of tasks and constraint complexities.

IFEval's constraint checking architecture is modular, allowing researchers to define new constraint types by implementing a constraint checker class with a validation method. The framework provides a base constraint class and registration mechanism, enabling custom constraints (e.g., 'response must be a valid email address', 'output must contain exactly 3 paragraphs') to be added without modifying core evaluation logic.

IFEval provides a model-agnostic evaluation interface that accepts model outputs as text, independent of how the model was generated (API, local inference, batch processing). The evaluation pipeline takes instruction-output pairs and constraint specifications, applies constraint checkers, and computes metrics without requiring model-specific integration or API access, enabling evaluation of any LLM that can generate text.