ToolLLM

Systematically collects and catalogs 16,464 real-world REST APIs from RapidAPI with metadata extraction, schema parsing, and endpoint documentation. The collection pipeline normalizes API specifications into a structured format compatible with instruction generation and inference, enabling models to learn patterns across diverse API designs, authentication schemes, and parameter structures.

Generates diverse, realistic user instructions for both single-tool (G1) and multi-tool (G2 intra-category, G3 intra-collection) scenarios using template-based and LLM-assisted generation. The system creates instructions that require tool selection, parameter reasoning, and API chaining, organized into three complexity tiers that progressively increase reasoning requirements from isolated API calls to cross-collection orchestration.

Maintains a public leaderboard (toolbench/tooleval/results/) that tracks evaluation results for different ToolLLaMA model variants and inference algorithms across standardized evaluation sets. The leaderboard enables reproducible comparison of models, tracks progress over time, and provides normalized scores accounting for different evaluation conditions, facilitating transparent benchmarking of tool-use capabilities.

Trains a specialized API retriever component that learns to rank relevant APIs from the 16,464-catalog based on query semantics. The retriever uses embedding-based or learned similarity approaches to match user queries to APIs, enabling open-domain tool use without explicit API specification. Training uses query-API relevance labels from the instruction dataset, learning patterns of which APIs are useful for different types of queries.

Implements error handling mechanisms within the inference pipeline that detect API failures (timeouts, invalid parameters, rate limits, malformed responses) and trigger recovery strategies such as parameter re-generation, alternative tool selection, or graceful degradation. The system learns from DFSDT-annotated error recovery patterns during training, enabling models to adapt when APIs fail rather than terminating execution.

Organizes evaluation data into standardized formats (G1 single-tool, G2 intra-category multi-tool, G3 intra-collection multi-tool) with explicit versioning and metadata tracking. Each evaluation set includes instructions, ground truth answers, API specifications, and expected reasoning traces, enabling reproducible evaluation across different models and inference algorithms with clear documentation of dataset composition and evolution.

Generates ground-truth answers for instructions using Depth-First Search Decision Tree (DFSDT) methodology, which produces step-by-step reasoning traces showing tool selection decisions, API call construction, response interpretation, and error recovery. Each annotation includes the complete decision path, parameter choices, and intermediate results, creating supervision signals that teach models not just what tools to use but why and how to use them.

Implements two training strategies for adapting LLaMA-based models to tool use: full fine-tuning that updates all model parameters on ToolBench instruction data, and LoRA (Low-Rank Adaptation) fine-tuning that trains low-rank decomposition matrices while freezing base weights. Both approaches integrate DFSDT reasoning traces as training supervision, enabling models to learn tool selection, API parameter construction, and multi-step reasoning from the 16,464-API dataset.

Executes inference pipelines (qa_pipeline.py) that enable fine-tuned models to solve user queries by selecting appropriate APIs, constructing valid API calls with correct parameters, executing those calls, and interpreting results. Supports both single-tool scenarios (selecting one API per query) and multi-tool scenarios (chaining multiple API calls with intermediate result interpretation), with built-in error handling for API failures and parameter validation.

Enables inference on queries where the relevant APIs are unknown upfront by using a learned API retriever component (qa_pipeline_open_domain.py) that semantically matches user queries to relevant APIs from the 16,464-API catalog. The retriever ranks APIs by relevance using embeddings or learned similarity metrics, then passes top-K APIs to the inference pipeline, enabling the model to solve queries without explicit API specification.

Implements multiple inference algorithms that control how models reason about and execute tool use: Depth-First Search (DFS) explores tool chains exhaustively, Chain-of-Thought (CoT) generates explicit reasoning steps before tool selection, and ReACT (Reasoning + Acting) interleaves reasoning with tool execution. Each algorithm trades off between reasoning transparency, computational cost, and success rate on complex multi-tool tasks.

Provides a web server interface (toolbench_server.py) that exposes trained ToolLLaMA models as HTTP endpoints, enabling interactive queries, real-time API execution, and result streaming. The server handles concurrent requests, manages API credentials securely, enforces rate limiting, and provides logging/monitoring for production deployment of tool-use agents.

Evaluates tool-use models using a pass rate metric that measures the percentage of instructions successfully completed within a limited number of API calls (typically 5-10). An instruction passes if the model's final answer matches the ground truth or achieves the specified task goal, accounting for the trade-off between solution quality and API call efficiency. This metric directly measures practical tool-use capability rather than intermediate reasoning quality.

Evaluates tool-use models using preference-based metrics that compare model outputs to a reference model (typically ChatGPT-ReACT) through human or LLM-based judgment. Win rate measures the percentage of instructions where the evaluated model outperforms the reference, capturing relative capability differences and enabling fine-grained comparison of reasoning quality, tool selection accuracy, and error recovery beyond binary pass/fail metrics.