Parea AI

Wraps LLM provider clients (OpenAI, Anthropic, LiteLLM) using language-specific decorators (@trace in Python, functional wrappers in TypeScript) that automatically capture all LLM API calls, inputs, outputs, latency, and cost data without modifying application code. Integrates with framework SDKs (LangChain, DSPy, Instructor) to trace nested LLM calls across the entire execution chain. Evaluation functions are registered at decoration time and executed asynchronously post-call, enabling real-time quality assessment without blocking inference.

Provides a web-based Prompt Playground that allows developers to create multiple versions of the same prompt and test them against the same input dataset in parallel, displaying outputs side-by-side with metrics (latency, cost, evaluation scores). Supports prompt templating with variable substitution, model selection (OpenAI, Anthropic, etc.), and parameter tuning (temperature, max_tokens). Experiment runner executes all variants against a dataset and aggregates results, enabling statistical comparison of prompt quality without manual iteration.

Enables comparison of cost and quality across different models and providers within the same experiment. Calculates cost per call based on model and token counts, and aggregates cost metrics alongside quality metrics in experiment results. Supports filtering and sorting experiments by cost-per-quality ratio, enabling identification of cost-optimal prompt/model combinations. Cost data is automatically updated as provider pricing changes, ensuring accurate cost tracking over time.

Supports team-based access to Parea platform with role-based permissions (roles not documented, but implied to include viewer, editor, admin). Team members can be invited to workspaces and assigned roles that control access to prompts, datasets, experiments, and observability data. Supports team-level settings and audit logging (audit logging not explicitly documented). Free tier limited to 2 members, Team tier supports 3 members base + $50/additional member (up to 20 total).

Provides Python and TypeScript SDKs with programmatic APIs for running experiments, retrieving results, and integrating Parea into CI/CD pipelines. Developers can call `p.experiment(...)` to run experiments programmatically, retrieve results as structured data, and make decisions based on experiment outcomes (e.g., deploy only if quality threshold is met). Results are returned as Python dicts/dataclasses or TypeScript objects, enabling integration with custom analysis or deployment logic.

Allows developers to define custom evaluation functions in Python or TypeScript that score LLM outputs against arbitrary criteria (correctness, tone, length, semantic similarity, etc.). Metrics are registered in the SDK and executed automatically on traced LLM calls, with results stored and aggregated in dashboards. Supports both deterministic metrics (regex matching, length checks) and LLM-based metrics (using another LLM to evaluate outputs). Evaluation results are queryable and filterable in the web UI, enabling drill-down analysis of which prompts/models perform best on specific criteria.

Captures all LLM API calls in production and staging environments, logging inputs, outputs, model, latency, token counts, and cost per call. Aggregates data into dashboards showing cost trends, latency percentiles, error rates, and quality metrics over time. Supports filtering by prompt version, model, user, or custom tags to drill down into specific subsets of traffic. Cost calculation is automatic based on provider pricing (OpenAI, Anthropic, etc.) and updated as pricing changes. Enables detection of performance regressions, cost anomalies, and quality degradation in production.

Provides a dataset management system where developers can upload, version, and organize test datasets (CSV, JSON, or via SDK) used for prompt evaluation and experimentation. Datasets are stored in Parea and can be reused across multiple experiments and prompt variants. Supports dataset versioning to track changes over time, and enables filtering/slicing datasets by tags or conditions. Datasets are linked to experiment runs, creating an audit trail of which data was used to evaluate which prompts.

Provides a web-based interface for human reviewers to evaluate LLM outputs, provide feedback, and assign quality scores. Reviewers can see LLM outputs alongside original inputs and evaluation metrics, and can add comments or ratings. Feedback is stored and linked to the original trace, enabling analysis of human vs. automated evaluation agreement. Supports assignment of review tasks to team members with role-based access control. Human feedback can be used as ground truth for evaluating prompt variants or training custom evaluation models.

Manages prompt versions and enables deployment to production environments. Developers can create new prompt versions in the Prompt Playground, test them against datasets, and deploy to production. Deployments are versioned, allowing rollback to previous versions if quality degrades. Supports canary deployments (rolling out to a percentage of traffic) and A/B testing in production. Deployed prompts are accessed via API or SDK, with version selection handled by Parea (e.g., 'latest', 'stable', or specific version ID).

Provides a unified SDK interface for calling multiple LLM providers (OpenAI, Anthropic, LiteLLM) without changing application code. Supports provider-agnostic prompt templates and automatic routing to different models based on cost, latency, or availability. Integrates with LiteLLM for multi-provider support, enabling fallback to alternative providers if primary provider is unavailable. Traces all calls regardless of provider, enabling cost and quality comparison across providers.

Provides native integrations with popular LLM frameworks (LangChain, DSPy, Instructor) to automatically trace calls within those frameworks without modifying framework code. Integrations hook into framework callbacks or middleware to capture LLM calls, tool use, and chain execution. Enables tracing of complex multi-step workflows (chains, agents, pipelines) with full visibility into intermediate steps and decision points. Traces are aggregated into a single execution tree, showing the full flow from input to final output.

Provides a programmatic experiment runner (available in Python and TypeScript SDKs) that executes all prompt variants against a dataset and aggregates results with statistical metrics. The runner parallelizes variant execution, collects evaluation scores, and computes aggregate statistics (mean, std dev, pass rate, etc.). Results are stored and linked to the experiment, enabling historical comparison of experiments. Supports filtering results by evaluation metric to identify which variants perform best on specific criteria.