Gorilla

BFCL V4 evaluates 70+ LLMs (API-based and locally-hosted) on function-calling accuracy using a weighted scoring formula that allocates 40% weight to agentic multi-step tasks, 30% to multi-turn conversations, and 30% to single-turn accuracy. The framework generates function-call responses from test prompts, then compares outputs against ground truth using specialized checker functions that validate JSON formatting, parameter correctness, and task completion semantics.

Gorilla provides Apache 2.0 licensed models (gorilla-openfunctions-v0/v1/v2) fine-tuned specifically for function calling, accessible via OpenAI-compatible endpoints at luigi.millennium.berkeley.edu:8000/v1. These models are trained on 1,600+ API documentation examples using RAFT (Retrieval-Augmented Fine-Tuning) and support parallel function execution, enabling agents to invoke multiple APIs concurrently without hallucination or parameter mismatches.

BFCL's live API evaluation (10% weight in V4) tests models on real function calls against actual APIs (not mocks), validating that generated calls work end-to-end. This includes calling real Stripe, GitHub, and other production APIs with test credentials, checking that responses match expected formats, and validating that side effects (e.g., created resources) are correct. Live evaluation catches issues that mock evaluation misses (API version mismatches, authentication failures, rate limiting).

Gorilla provides comprehensive logging and debugging infrastructure that captures detailed execution traces for every evaluation run, including model inputs, outputs, intermediate reasoning steps, and error messages. Logs are structured (JSON format) and queryable, enabling post-hoc analysis of why models failed on specific tasks. This infrastructure supports iterative debugging of prompts, model selection, and function schemas.

Gorilla includes a CI/CD pipeline for managing model versions, running automated evaluations on new model checkpoints, and releasing models to the public endpoint (luigi.millennium.berkeley.edu:8000/v1). The pipeline validates model quality, runs regression tests against prior versions, and gates releases based on performance thresholds. This enables rapid iteration on OpenFunctions models while maintaining quality standards.

RAFT (Retrieval-Augmented Fine-Tuning) is a dataset generation pipeline that creates domain-specific training data by retrieving relevant API documentation, generating synthetic function-calling examples, and filtering them through quality checks. It enables rapid adaptation of OpenFunctions models to custom APIs without manual annotation, using a retrieval-augmented approach to ensure generated examples match your API schema and documentation style.

GoEx is a Docker-based sandboxed execution environment that safely executes LLM-generated function calls with post-facto validation and undo capabilities. It intercepts function calls before execution, validates them against a security policy, executes them in an isolated container, and provides rollback mechanisms if validation fails or side effects are undesirable. This enables agents to take real actions (database writes, API calls) with safety guarantees.

API Zoo is a curated, community-maintained repository of 1,600+ API documentation entries in standardized JSON Schema format, covering popular services (Stripe, Slack, GitHub, AWS, etc.). It serves as the training corpus for OpenFunctions models and RAFT fine-tuning, and provides a standardized reference for function-calling evaluation. The repository is version-controlled and accepts community contributions, ensuring documentation stays current with API changes.

Agent Arena is an evaluation platform that runs agents head-to-head on identical tasks and assigns ELO ratings based on comparative performance. It enables researchers to compare agent architectures, model choices, and tool configurations in a tournament-style format, with ratings updated dynamically as new evaluations are added. This provides a more nuanced ranking than single-metric leaderboards.

The response generation pipeline is a unified interface for invoking 70+ LLMs (OpenAI, Anthropic, Google, Mistral, Cohere, DeepSeek, xAI, Llama, Qwen, etc.) with a model handler abstraction layer. Each model has a dedicated handler that manages API authentication, request formatting, response parsing, and error handling, enabling seamless evaluation across heterogeneous model families without code changes. The pipeline supports both API-based and locally-hosted models.

BFCL's multi-turn evaluation capability (weighted at 30% in V4) assesses how well models maintain context across conversation turns and generate appropriate function calls based on prior exchanges. The evaluation framework preserves conversation history, validates that function calls reference previous context correctly, and checks for consistency across turns. This enables assessment of agents that must remember prior API results and adapt subsequent calls.

BFCL V4's agentic evaluation (40% weight) tests complex multi-step tasks requiring web search, memory management, and reasoning across multiple API calls. Tasks include scenarios where agents must search the web for information, store results in memory, and use them in subsequent API calls. The evaluation framework provides mock web search and memory APIs, validates that agents use them appropriately, and scores based on task completion rather than individual function calls.

BFCL's irrelevance detection capability (10% weight in V4) evaluates whether models correctly refuse to invoke functions when the user query is unrelated to available APIs. The evaluation framework includes test cases where no function call is appropriate, and scores models on whether they correctly identify irrelevance and refuse to hallucinate function calls. This prevents agents from making spurious API calls that waste resources or cause unintended side effects.