@inngest/ai

Provides a unified TypeScript interface layer that abstracts over heterogeneous AI provider APIs (OpenAI, Anthropic, Gemini, Grok, Azure OpenAI) with compile-time type safety. Uses provider-specific adapter classes that normalize request/response formats and handle provider-specific quirks, allowing developers to swap providers without changing application code. Each adapter implements a common interface contract that maps to Inngest's event-driven execution model.

Implements function calling (tool use) across providers with different schema formats by normalizing tool definitions into a canonical schema format, then translating to provider-specific representations (OpenAI's function_calling format, Anthropic's tool_use, etc.). Handles provider differences in how they declare parameters, return types, and tool selection logic. Automatically marshals function results back into the LLM context for multi-turn tool-use workflows.

Provides batch processing capabilities for high-volume LLM requests, leveraging provider-native batch APIs (OpenAI Batch API, Anthropic Batch API) to reduce costs and latency. Automatically groups requests into batches, submits them to providers, and polls for results. Integrates with Inngest's event system to track batch status and emit events when batches complete. Supports cost optimization strategies like batching similar requests together and prioritizing cheaper models for batch processing.

Implements caching of LLM requests and responses with optional semantic deduplication (detecting similar prompts that would produce similar outputs). Uses configurable cache backends (in-memory, Redis, Inngest event store) and supports cache invalidation strategies. Automatically deduplicates requests based on exact match (fast) or semantic similarity (slower but catches paraphrased prompts). Integrates with Inngest's event system to track cache hits/misses and enable cost analysis.

Enables extraction of structured data (JSON, typed objects) from LLM responses by specifying output schemas and delegating to provider-specific structured output mechanisms (OpenAI's JSON mode, Anthropic's structured output, Gemini's schema constraints). Automatically validates responses against the declared schema and provides type-safe access to extracted fields. Handles provider differences in how they enforce schema compliance and error recovery when responses don't match the schema.

Provides streaming support for LLM responses (token-by-token output) with automatic integration into Inngest's event system. Streams are buffered and can be emitted as Inngest events, allowing downstream workflow steps to process partial results in real-time. Handles provider-specific streaming protocols (Server-Sent Events, WebSocket) and normalizes them into a common stream interface. Manages backpressure and ensures streamed data is durably logged in Inngest's event store.

Automatically tracks token consumption (input/output tokens) for each LLM call and aggregates usage across providers with different pricing models. Provides cost estimation based on provider-specific pricing rates (updated periodically) and supports custom pricing configuration. Integrates with Inngest's event metadata to attach usage data to each workflow execution, enabling cost analysis and budgeting. Handles provider differences in how they report token counts (e.g., Claude's token counting API vs OpenAI's inline reporting).

Implements provider-aware retry logic that distinguishes between transient failures (rate limits, temporary outages) and permanent failures (invalid API key, model not found). Uses exponential backoff with jitter and provider-specific retry strategies (e.g., respecting Retry-After headers from OpenAI). Integrates with Inngest's built-in retry mechanism to ensure failed LLM calls are automatically retried as part of the workflow execution, with full durability guarantees. Provides configurable retry policies per provider or model.

Monitors token consumption relative to each model's context window limit and provides utilities to manage context (e.g., truncating message history, summarizing old messages). Automatically enforces token limits by rejecting requests that would exceed the model's maximum context size. Provides visibility into context utilization and warns when approaching limits. Supports dynamic context pruning strategies (e.g., remove oldest messages, summarize and replace) to keep conversations within bounds.

Provides a routing layer that selects the best model for a given task based on declared capabilities (e.g., vision support, function calling, structured output) and cost/latency constraints. Implements fallback chains where if the primary model fails or is unavailable, the request is automatically routed to a secondary model with similar capabilities. Supports cost-based model selection (e.g., prefer cheaper models for simple tasks) and latency-based selection (e.g., prefer faster models for real-time applications). Integrates with Inngest's event system to track model selection decisions.

Enables versioning of prompts as Inngest workflow artifacts, allowing different prompt versions to be tested against each other in A/B test configurations. Automatically routes requests to different prompt versions based on configurable split percentages or user cohorts. Tracks performance metrics (latency, cost, output quality) per prompt version and integrates with Inngest's event system to enable analysis of which prompts perform best. Supports rollback to previous prompt versions without code changes.

Integrates content moderation by leveraging provider-native safety features (OpenAI's moderation API, Anthropic's safety guidelines) and provides a unified interface for content filtering across providers. Automatically flags or blocks requests/responses that violate safety policies, with configurable severity levels and custom rules. Logs moderation decisions to Inngest's event store for auditing and compliance. Supports both input filtering (reject unsafe prompts) and output filtering (reject unsafe completions).