Gorilla

BFCL V4 evaluates 70+ LLMs (OpenAI, Anthropic, Google, Mistral, local models) on function-calling accuracy using a weighted scoring formula that allocates 40% weight to agentic multi-step tasks, 30% to multi-turn conversations, and 30% to single-turn accuracy. The evaluation framework uses specialized checker modules that compare model-generated function calls against ground truth, supporting both live API validation and offline schema-based verification across non-live, live, and irrelevance test categories.

Gorilla provides OpenFunctions models (v0, v1, v2) as Apache 2.0 licensed alternatives to proprietary function-calling models, accessible via OpenAI-compatible API endpoints at luigi.millennium.berkeley.edu:8000/v1. These models are fine-tuned specifically for accurate function invocation and support parallel execution of multiple function calls, streaming responses, and domain-specific adaptation through RAFT fine-tuning. The models handle JSON formatting, parameter validation, and multi-turn function-calling conversations natively.

BFCL includes evaluation of whether models correctly identify when a user request doesn't require any function call, preventing unnecessary or irrelevant function invocations. The irrelevance category tests scenarios where the best response is to decline calling a function or to respond with general knowledge instead. This accounts for 10% of BFCL V4 scoring and is critical for preventing agents from over-invoking tools.

Gorilla implements a model handler system that abstracts over different LLM providers (OpenAI, Anthropic, Google, Mistral, Cohere, DeepSeek, xAI, local models) with a unified interface. Each provider has a handler that translates between the provider's API format and Gorilla's internal representation, enabling seamless evaluation across 70+ models without provider-specific code. Handlers manage authentication, request formatting, response parsing, and error handling for each provider.

Gorilla includes a CI/CD pipeline for managing model versions, running automated evaluations on new model checkpoints, and releasing models to the public endpoint (luigi.millennium.berkeley.edu:8000/v1). The pipeline validates model quality, runs regression tests against prior versions, and gates releases based on performance thresholds. This enables rapid iteration on OpenFunctions models while maintaining quality standards.

RAFT is a dataset generation and fine-tuning pipeline that enables models to adapt to domain-specific APIs by combining retrieval of relevant API documentation with in-context learning. The system generates synthetic training data by retrieving API schemas, creating realistic function-calling prompts, and fine-tuning OpenFunctions models on this domain-specific data. This approach reduces hallucination when invoking proprietary or internal APIs that weren't in the model's training data, enabling accurate function calling on custom API sets without requiring massive labeled datasets.

GoEx is a Docker-based sandboxed execution environment that safely executes LLM-generated function calls with post-execution validation and rollback capabilities. The system intercepts function calls, executes them in an isolated container, validates outputs against expected schemas, and can undo state changes if validation fails or if the call was determined to be unsafe. This enables agents to take real actions (API calls, database writes) with safety guarantees, preventing hallucinated or malicious function calls from corrupting system state.

BFCL V4 includes specialized evaluation for agentic tasks that require multi-step reasoning, web search integration, and memory management across conversation turns. The evaluation framework provides test scenarios where agents must search the web for information, maintain context across multiple turns, and chain function calls together to solve complex problems. This is implemented as 40% of the overall scoring formula, reflecting the importance of agentic capabilities beyond simple function calling.

Gorilla maintains API Zoo, a community-curated repository of 1,600+ API schemas and documentation covering major services (Stripe, GitHub, Twilio, AWS, Google Cloud, etc.). This repository serves as the ground truth for function-calling evaluation and as training data for RAFT fine-tuning. The API Zoo is structured with standardized schema formats, enabling consistent evaluation across diverse APIs and providing a comprehensive dataset for training function-calling models.

Agent Arena is a competitive evaluation platform where agents are matched against each other on identical tasks, with results aggregated into ELO ratings similar to chess rankings. This enables direct comparison of agent capabilities beyond simple accuracy metrics, capturing relative performance differences and head-to-head matchups. The system tracks agent performance over time as models are updated, providing a dynamic leaderboard that reflects current state-of-the-art.

BFCL evaluates models on multi-turn conversations where function calls must be made in context of previous turns, requiring models to maintain conversation state and reference earlier information. This capability tests whether models can handle realistic agent scenarios where context accumulates across turns and function calls depend on previous results. Multi-turn evaluation accounts for 30% of the overall BFCL V4 scoring, reflecting its importance for practical agent applications.

BFCL includes live API testing where function calls are executed against real API endpoints (Stripe, GitHub, Twilio, etc.) and results are validated against actual API responses. This goes beyond schema validation to test whether generated function calls actually work with real services, catching hallucinations that pass schema checks but fail in practice. Live API testing accounts for 10% of BFCL V4 scoring and requires valid API credentials for each service.

BFCL includes offline validation where function calls are checked against JSON schemas without executing against real APIs. This tests whether models generate syntactically correct function calls with valid parameters, catching basic hallucinations like incorrect parameter names or types. Non-live validation is fast and doesn't require API credentials, making it suitable for rapid iteration and evaluation of many models. Non-live testing accounts for 10% of BFCL V4 scoring.

Gorilla is an advanced agent designed to enable large language models to accurately invoke over 1,600 API calls, significantly reducing hallucinations and enhancing reliable programmatic interactions.