OpenBMB/ToolBench
ToolBench
ToolBench is an open platform for training, serving, and evaluating large language models that retrieve and call real-world APIs to complete user instructions. It provides an API-aware inference engine that selects relevant tools from a large corpus and generates sequences of tool calls to produce final answers, along with a custom API registration system that lets users add their own REST endpoints for the model to discover and invoke.
The platform includes a complete instruction-tuning pipeline for training models on curated tool-use data, a multi-tool execution engine that coordinates sequential and parallel API calls, and a preference-based quality assessment framework that uses a judge model to compare candidate action sequences. It also offers a tool-use evaluation benchmark that measures task completion rate and preference between different tool-calling strategies.
Beyond training and evaluation, ToolBench provides a web-based chat interface and REST endpoints for serving fine-tuned models with tool-augmented responses. The system supports API retrieval for open-domain queries, enabling models to handle tasks requiring any of thousands of tools, and allows users to register custom APIs with documentation and implementation code for model discovery during inference.
Features
- API Retrieval-Augmented Generations - Retrieves relevant APIs from a large corpus before inference to ground model responses in real-world tools.
- Tool-Use Training - Provides a complete instruction-tuning pipeline for training models on curated tool-use data.
- Tool-Use Instruction Tuning - Includes a complete instruction-tuning pipeline for training models on curated tool-use data.
- Instruction Tuning Pipelines - Provides a complete instruction-tuning pipeline for training models on curated tool-use data.
- API-Aware Inference Engines - Provides an API-aware inference engine that selects relevant tools from a large corpus and generates tool-calling sequences.
- Tool-Use Performance Evaluators - Measuring how well a model completes tasks using APIs, including task success rate and preference between candidate action sequences.
- Tool-Augmented Serving Frameworks - Provides a web-based chat interface and REST endpoints for serving fine-tuned models with tool-augmented responses.
- Tool Call Data Generators - Generates high-quality instruction-tuning examples that teach models to plan and execute API calls.
- Tool-Using Model Inference - Provides an inference engine that retrieves APIs and generates tool-call sequences to answer user queries.
- Tool-Use Serving Frameworks - Provides a web-based chat interface and REST endpoints for serving fine-tuned models with tool-augmented responses.
- API-Use Fine-Tuning - Fine-tunes language models on curated instruction-solution path datasets to enable real-world REST API calling.
- API Retrieval - Supports API retrieval for open-domain queries, enabling models to handle tasks requiring any of thousands of tools.
- API Retrieval Systems - Selects relevant APIs from a large corpus before inference to ground model responses in real-world tools.
- Custom API Registrations - Registers custom REST APIs with documentation and implementation so the model can discover and call them during inference.
- API Registration Systems - Ships a custom API registration system that lets users add their own REST endpoints for model discovery and invocation.
- API Registration Systems - Lets users register custom REST APIs with documentation and implementation for model discovery during inference.
- Custom API Registrations - Registers user-defined REST APIs with documentation and implementation for model discovery during inference.
- Multi-Tool Execution Engines - Coordinates sequential and parallel API calls across multiple tools to complete complex instructions.
- Custom API Inferences - Allows registering user-defined APIs with documentation and implementation for use during inference.
- Preference-Based Quality Assessments - Uses a ChatGPT-based judge to compare action sequences and determine preferred tool-use strategies.
- Action-Sequence Evaluation Frameworks - Ships a judge-model framework that compares tool-call sequences to assess quality and preference.
- Action Sequence Comparators - Ships a preference-based quality assessment framework that uses a judge model to compare candidate action sequences.
- Tool-Use Evaluation Benchmarks - Offers a tool-use evaluation benchmark that measures task completion rate and preference between different tool-calling strategies.
- Custom API Inferences - Extends the model's tool-use capability to user-defined APIs by providing their documentation and implementation code.
- Tool-Using Model API Servings - Exposes a fine-tuned model behind a REST endpoint that streams back tool-augmented responses.
- Tool-Use Accuracy Evaluators - Measures task completion rate for tool-using models, reporting the proportion of successfully completed instructions.
- Tool-Use Quality Comparators - Implements a ChatGPT-based judge that compares action sequences to determine preferred tool-use strategies.
- Web Chat Interfaces - Provides a web-based chat interface for interacting with a tool-using model.
- Tool-Using Model Servings - Runs a fine-tuned model behind a web server that streams back tool-augmented responses.
- Tool-Using Model Web UIs - Provides a chat interface where users interact with a tool-using model via a web UI.
- Agent Tool Learning - Facilitates mastering real-world APIs for tool-augmented agents.
- Language Model Development - Platform for training and evaluating LLMs for tool use.
- Tool Optimization - Facilitating mastery of real-world APIs.
- Tool Use And Integration - Facilitates training and evaluation for large-scale API tool use.
