pandera

Pandera enables developers to define reusable validation schemas using a declarative API that maps to pandas DataFrames, Series, and Index objects. Schemas are Python objects (DataFrameSchema, SeriesSchema) that encapsulate column definitions, data types, nullable constraints, and custom validators. Validation is performed by calling the .validate() method, which returns the validated DataFrame or raises a SchemaError with detailed failure information including row/column locations and constraint violations.

Pandera validates individual DataFrame columns against specified data types (int, float, string, datetime, categorical, etc.) and nullable constraints using a Column object that wraps pandas dtype checking. The validation engine uses pandas' dtype inference and comparison to ensure columns match expected types, and supports coercion (e.g., converting strings to datetime) via the coerce parameter. Custom dtype validators can be registered to handle domain-specific types or complex validation logic.

Pandera can generate schemas from Python dataclasses and Pydantic models, enabling developers to define data structures once and use them for both type checking and DataFrame validation. The schema generation engine inspects dataclass fields and Pydantic model definitions to infer column types, nullable constraints, and validators. This enables tight integration between type-checked Python code and DataFrame validation.

Pandera allows developers to attach custom validation functions to columns and DataFrames using the Check class, which wraps callable validators (lambdas, functions, or methods) that operate on Series or scalar values. Validators can be applied element-wise (to each value) or row-wise (to entire rows), and support groupby operations for conditional validation (e.g., 'validate that sales > 0 only for active regions'). The validation engine applies these checks after type validation and reports failures with row indices and values that triggered the violation.

Pandera includes a SeriesSchemaStatistics class that enables validation of statistical properties of Series data, such as mean, std, min, max, and quantiles. Developers can define expected ranges for these statistics and Pandera will compute them during validation, comparing actual values against expected bounds. This is useful for detecting data drift or anomalies in production pipelines where the distribution of values should remain stable over time.

Pandera supports validation of DataFrames with multi-level indices (MultiIndex) and hierarchical column structures through the Index class, which can be composed into schemas. Developers can define constraints on index levels (e.g., level 0 must be unique, level 1 must be sorted) and validate them alongside column constraints. The validation engine checks index properties and reports failures with level-specific information.

Pandera provides a schema inference API (infer_schema function) that automatically generates a DataFrameSchema or SeriesSchema by analyzing a sample DataFrame or Series. The inference engine examines data types, nullable patterns, and optionally computes statistics to populate schema constraints. Inferred schemas can be exported as Python code or YAML, enabling developers to use them as starting points for manual refinement or to document expected data structures.

Pandera supports defining and loading schemas from YAML files or Python dictionaries, enabling schema-as-configuration workflows. Developers can write schemas in YAML format with column definitions, constraints, and validators, then load them using the io.from_yaml() function. Schemas can also be exported to YAML for documentation or version control. This enables non-technical stakeholders to review and modify schemas without writing Python code.

Pandera integrates with the Hypothesis library to enable property-based testing of data validation schemas. Developers can use the @check_output decorator to automatically generate test data that matches a schema and verify that validation passes. This enables testing of schema definitions themselves and ensures that schemas correctly describe the data they're meant to validate. Hypothesis generates edge cases and random data to stress-test schemas.

Pandera supports lazy validation mode where all validation errors are collected and reported together rather than failing on the first error. Developers can call .validate(lazy=True) to accumulate errors across all columns and rows, then inspect the SchemaError object to see all failures at once. This is useful for data quality reporting where stakeholders want to see all issues in a dataset rather than fixing them one at a time.

Pandera provides experimental support for validating Polars DataFrames (a faster, memory-efficient alternative to pandas) through a polars-specific schema API. Developers can define PolarsSchema objects that work similarly to DataFrameSchema but are optimized for Polars' columnar architecture and lazy evaluation. Validation leverages Polars' native type system and expression API for efficient constraint checking.