promptflow

Defines executable LLM application workflows as directed acyclic graphs (DAGs) using YAML syntax (flow.dag.yaml), where nodes represent tools, LLM calls, or custom Python code and edges define data flow between components. The execution engine parses the YAML, builds a dependency graph, and executes nodes in topological order with automatic input/output mapping and type validation. This approach enables non-programmers to compose complex workflows while maintaining deterministic execution order and enabling visual debugging.

Enables defining flows as standard Python functions or classes decorated with @flow, allowing developers to write imperative LLM application logic with full Python expressiveness including loops, conditionals, and dynamic branching. The framework wraps these functions with automatic tracing, input/output validation, and connection injection, executing them through the same runtime as DAG flows while preserving Python semantics. This approach bridges the gap between rapid prototyping and production-grade observability.

Automatically generates REST API endpoints from flow definitions, enabling flows to be served as HTTP services without writing API code. The framework handles request/response serialization, input validation, error handling, and OpenAPI schema generation. Flows can be deployed to various platforms (local Flask, Azure App Service, Kubernetes) with the same code, and the framework provides health checks, request logging, and performance monitoring out of the box.

Integrates with Azure ML workspaces to enable cloud execution of flows, automatic scaling, and integration with Azure ML's experiment tracking and model registry. Flows can be submitted to Azure ML compute clusters, with automatic environment setup, dependency management, and result tracking in the workspace. This enables seamless transition from local development to cloud-scale execution without code changes.

Provides CLI commands and GitHub Actions/Azure Pipelines templates for integrating flows into CI/CD pipelines, enabling automated testing on every commit, evaluation against test datasets, and conditional deployment based on quality metrics. The framework supports running batch evaluations, comparing metrics against baselines, and blocking deployments if quality thresholds are not met. This enables continuous improvement of LLM applications with automated quality gates.

Supports processing of images and documents (PDFs, Word, etc.) as flow inputs and outputs, with automatic format conversion, resizing, and embedding generation. Flows can accept image URLs or file paths, process them through vision LLMs or custom tools, and generate outputs like descriptions, extracted text, or structured data. The framework handles file I/O, format validation, and integration with vision models.

Automatically tracks all flow executions with metadata (inputs, outputs, duration, status, errors), persisting results to local storage or cloud backends for audit trails and debugging. The framework provides CLI commands to list, inspect, and compare runs, enabling developers to understand flow behavior over time and debug issues. Run data includes full execution traces, intermediate node outputs, and performance metrics.

Introduces a markdown-based file format (.prompty) that bundles prompt templates, LLM configuration (model, temperature, max_tokens), and Python code in a single file, enabling prompt engineers to iterate on prompts and model parameters without touching code. The format separates front-matter YAML configuration from markdown prompt content and optional Python execution logic, with built-in support for prompt variables, few-shot examples, and model-specific optimizations. This approach treats prompts as first-class artifacts with version control and testing support.

Provides native tool nodes for calling LLMs (OpenAI, Azure OpenAI, Anthropic, Ollama) with automatic connection management, token counting, and cost tracking. Each LLM tool accepts a prompt template, model name, and parameters, handles authentication via connection objects, and returns structured responses with token usage metadata. The framework abstracts provider-specific APIs behind a unified interface, enabling flows to switch LLM providers by changing configuration without code changes.

Enables developers to wrap Python functions as reusable tools via decorators (@tool) with automatic input/output schema generation from type hints, enabling them to be composed into flows as first-class nodes. The framework introspects function signatures, generates JSON schemas for validation, and handles tool invocation with dependency injection for connections and context. Tools can be packaged and shared across flows, with support for async functions and complex input types (lists, dicts, file paths).

Processes large datasets by executing flows against multiple input records from JSONL files, with automatic parallelization, result aggregation, and error handling. The batch executor reads JSONL line-by-line, maps each record to flow inputs, executes the flow in parallel (configurable worker count), and writes results to output JSONL with execution metadata (duration, status, error messages). This enables evaluation and testing of flows at scale without manual iteration.

Provides a framework for defining evaluation flows that compute quality metrics (accuracy, F1, BLEU, custom metrics) on flow outputs by comparing against ground truth or using LLM-based evaluators. Evaluation flows are standard flows that accept flow outputs and reference data, returning metric scores. The framework aggregates metrics across batch runs, enables comparison between flow versions, and visualizes metric trends over time. This enables data-driven optimization of LLM applications.

Automatically instruments flow execution with distributed tracing via OpenTelemetry, capturing all LLM API calls, tool invocations, and custom code execution as spans with timing, parameters, and outputs. The tracing system tracks token usage across all LLM calls, calculates costs, and exports traces to OpenTelemetry-compatible backends (Jaeger, DataDog, Azure Monitor). This enables observability of production LLM applications without code changes.

Provides a centralized connection management system that stores API keys, connection strings, and credentials securely (encrypted at rest), abstracting provider-specific authentication details. Connections are defined once and referenced by name in flows, enabling credential rotation without code changes and supporting multiple authentication methods (API keys, OAuth, connection strings). The system integrates with Azure Key Vault for enterprise credential management.

Provides CLI and SDK tools for testing flows locally with interactive debugging, including single-step execution, breakpoints, variable inspection, and execution visualization. Developers can run flows against test inputs, inspect intermediate values at each node, and visualize the execution DAG with actual data flowing through it. The framework caches node outputs to enable rapid iteration on specific nodes without re-executing upstream dependencies.