ToolLLM

Automatically collects and curates 16,464 real-world REST APIs from RapidAPI with metadata extraction, categorization, and schema parsing. The system ingests API specifications, endpoint definitions, parameter schemas, and response formats into a structured database that serves as the foundation for instruction generation and model training. This enables models to learn from genuine production APIs rather than synthetic examples.

Generates high-quality instruction-answer pairs with explicit reasoning traces using a Depth-First Search Decision Tree algorithm that explores tool-use sequences systematically. For each instruction, the system constructs a decision tree where each node represents a tool selection decision, edges represent API calls, and leaf nodes represent task completion. The algorithm generates complete reasoning traces showing thought process, tool selection rationale, parameter construction, and error recovery patterns, creating supervision signals for training models to reason about tool use.

Maintains a public leaderboard that tracks model performance across multiple evaluation metrics (pass rate, win rate, efficiency) with normalization to enable fair comparison across different evaluation sets and baselines. The leaderboard ingests evaluation results from the ToolEval framework, normalizes scores to a 0-100 scale, and ranks models by composite score. Results are stratified by evaluation set (default, extended) and complexity tier (G1/G2/G3), enabling users to understand model strengths and weaknesses across different task types. Historical results are preserved, enabling tracking of progress over time.

A specialized neural model trained on ToolBench data to rank APIs by relevance for a given user query. The Tool Retriever learns semantic relationships between queries and APIs, enabling it to identify relevant tools even when query language doesn't directly match API names or descriptions. The model is trained using contrastive learning where relevant APIs are pulled closer to queries in embedding space while irrelevant APIs are pushed away. At inference time, the retriever ranks candidate APIs by relevance score, enabling the main inference pipeline to select appropriate tools from large API catalogs without explicit enumeration.

Automatically generates diverse user instructions that require tool use, covering both single-tool scenarios (G1) where one API call solves the task and multi-tool scenarios (G2/G3) where multiple APIs must be chained. The generation process creates instructions by sampling APIs, defining task objectives, and constructing natural language queries that require those specific tools. For multi-tool scenarios, the generator creates dependencies between APIs (e.g., API A's output becomes API B's input) and ensures instructions are solvable with the specified tool chains. This produces diverse, realistic instructions that cover the space of possible tool-use tasks.

Organizes instruction-answer pairs into three progressive complexity tiers: G1 (single-tool tasks), G2 (intra-category multi-tool tasks requiring tool chaining within a domain), and G3 (intra-collection multi-tool tasks requiring cross-domain tool orchestration). This hierarchical structure enables curriculum learning where models first master single-tool use, then learn tool chaining within domains, then generalize to cross-domain orchestration. The organization maps directly to training data splits and evaluation benchmarks.

Fine-tunes LLaMA-based models on ToolBench instruction-answer pairs using two training strategies: full fine-tuning (ToolLLaMA-2-7b-v2) that updates all model parameters, and LoRA (Low-Rank Adaptation) fine-tuning (ToolLLaMA-7b-LoRA-v1) that adds trainable low-rank matrices to attention layers while freezing base weights. The training pipeline uses instruction-tuning objectives where models learn to generate tool-use sequences, API calls with correct parameters, and reasoning explanations. Multiple model versions are maintained corresponding to different data collection iterations.

Executes tool-use inference through a pipeline that (1) parses user queries, (2) selects appropriate tools from the available API set using semantic matching or learned ranking, (3) generates valid API calls with correct parameters by conditioning on API schemas, and (4) interprets API responses to determine next steps. The inference pipeline supports both single-tool scenarios (G1) where one API call solves the task, and multi-tool scenarios (G2/G3) where multiple APIs must be chained with intermediate result passing. The system maintains API execution state and handles parameter binding across sequential calls.

Extends inference to open-domain settings where the full API set is not pre-specified by implementing a learned API retriever that ranks relevant APIs for a given query. The system embeds user queries and API specifications into a shared semantic space, then retrieves top-K relevant APIs using dense vector similarity. A separate Tool Retriever model (trained on ToolBench data) learns to rank APIs by relevance, enabling the inference pipeline to select from thousands of APIs without explicit enumeration. This enables deployment scenarios where new APIs can be added without retraining the main model.

Provides multiple inference algorithms that control how the model reasons about tool use and generates API call sequences. Depth-First Search (DFS) explores tool-use paths systematically, Chain-of-Thought (CoT) generates explicit reasoning steps before tool selection, and ReACT (Reasoning + Acting) interleaves reasoning with action execution. Each algorithm is implemented as a separate inference loop that controls prompt formatting, token generation, and response parsing. Users can select algorithms based on task complexity and latency requirements, with DFS typically producing more thorough exploration at higher latency, and CoT balancing reasoning quality with speed.

Exposes the inference pipeline through a web server (toolbench_server.py) that accepts HTTP requests with user queries and returns tool-use results. The server manages model loading, API credential handling, inference execution, and response formatting. It supports both synchronous request-response patterns for simple queries and asynchronous patterns for long-running multi-tool chains. The interface abstracts away model and API complexity, enabling non-technical users to interact with tool-use agents through a simple REST API or web UI.

Evaluates tool-use models by measuring the percentage of instructions successfully completed within a limited API call budget (typically 5-10 calls). For each instruction, the system executes the model's generated API call sequence against real APIs, captures responses, and determines success based on whether the final result matches the expected answer. Pass rate directly measures task completion capability and is computed separately for G1 (single-tool), G2 (intra-category multi-tool), and G3 (cross-domain multi-tool) instructions, enabling fine-grained capability analysis.

Evaluates tool-use models by comparing their performance against a reference baseline (typically ChatGPT-ReACT) using preference-based metrics. For each instruction, both the evaluated model and baseline generate solutions, and a preference judge (often GPT-4 or human annotators) determines which solution is better based on correctness, efficiency, and reasoning quality. Win rate is computed as the percentage of instructions where the evaluated model outperforms the baseline. This metric captures nuanced performance differences that binary pass rate cannot, and enables ranking models on a continuous scale.