FinQA

Evaluates AI systems' ability to perform chained mathematical operations (addition, subtraction, multiplication, division, comparisons) across structured tables and unstructured text extracted from real SEC filings. The dataset provides ground-truth answers requiring 2-5 sequential computational steps, enabling benchmarking of quantitative reasoning pipelines that must parse financial data, identify relevant values, and execute correct operation sequences without intermediate errors.

Provides ground-truth financial context by embedding questions within actual SEC filing excerpts and structured financial tables from S&P 500 companies' earnings reports. The dataset preserves original document structure and financial terminology, enabling evaluation of whether AI systems can correctly interpret domain-specific concepts (revenue recognition, GAAP vs non-GAAP metrics, segment reporting) before applying mathematical operations. Supports fine-tuning and in-context learning approaches that require authentic financial language and formatting.

Requires systems to extract and integrate numerical values from both structured tables and unstructured text within the same question context. The dataset forces handling of data heterogeneity: values may appear as formatted numbers in tables (with thousands separators, currency symbols), as written numbers in text ('five million dollars'), or as percentages in different notations. Systems must normalize, validate, and cross-reference values across formats before performing calculations, testing robustness to real-world financial data inconsistencies.

Provides standardized evaluation framework with 8,281 question-answer pairs enabling reproducible benchmarking of AI systems' financial reasoning capabilities. The dataset includes train/validation/test splits with consistent evaluation metrics (exact match accuracy, numerical tolerance thresholds), enabling fair comparison across different model architectures, training approaches, and baseline systems. Supports leaderboard-style evaluation and tracks model performance progression on a well-defined, publicly available benchmark.

Each question in the dataset is annotated with the explicit sequence of mathematical operations required to reach the correct answer, enabling analysis of reasoning complexity and intermediate step accuracy. The annotation structure captures operation types (addition, subtraction, multiplication, division, comparison), operand identification, and step dependencies, allowing systems to be evaluated not just on final answer correctness but on reasoning process quality. Supports training approaches that explicitly model reasoning chains and enables error analysis at the operation level.

Questions span diverse financial metrics (revenue, earnings, margins, ratios, cash flows, balance sheet items) requiring systems to understand metric semantics, relationships, and calculation methods. The dataset implicitly tests whether systems can distinguish between related but distinct metrics (e.g., gross profit vs operating income vs net income) and understand their roles in financial analysis. Enables evaluation of financial domain knowledge depth beyond simple keyword matching, testing whether systems grasp accounting principles underlying metric definitions.

Questions require comparing financial metrics across time periods (year-over-year, quarter-over-quarter) and across entities (company comparisons, segment analysis), testing systems' ability to handle temporal context and multi-entity reasoning. The dataset includes questions requiring identification of relevant time periods, extraction of values from different fiscal periods, and computation of changes or ratios across time. Enables evaluation of whether systems understand financial reporting calendars, fiscal year conventions, and temporal relationships in financial data.