Youtu Agent

Youtu Agent is an open-source framework for building, running, and evaluating autonomous agents powered by large language models. It provides the core infrastructure for creating agents that follow reasoning loops, use toolkits, and coordinate with other agents to solve complex tasks, all managed through YAML-driven configuration files.

The framework distinguishes itself through its support for multi-agent orchestration, where a planner agent decomposes tasks and coordinates specialized worker agents, and through its integration with the Model Context Protocol for connecting to external toolkits. It includes a sandboxed code execution environment that supports over 20 programming languages, browser automation capabilities for web research, and a trajectory-based performance distillation method that improves agent performance without fine-tuning model parameters.

Beyond core agent development, the framework offers a comprehensive evaluation pipeline with database-backed experiment tracking, configurable judging using language models or rule-based matching, and support for resuming interrupted evaluations. It provides tooling for defining custom reward functions, running multi-phase benchmarks, and comparing experiment results. The system also includes web-based interfaces for interacting with agents, Docker deployment options, and support for multimodal inputs including images and video.

Features

Multi-Agent Coordination Systems - Coordinates multiple sub-agents in a plan-and-execute workflow to generate SVG graphics from terminal or web UI.

Typed JSON Schemas - Provides a mechanism for defining tool schemas using typed JSON schemas for agent discovery and invocation.

Agent Workflow Orchestrations - Coordinates multiple independent agents in code to create complex Plan-and-Execute workflows.

Agentic LLM Frameworks - An agent framework for building autonomous agents with reasoning loops, toolkits, and multi-agent orchestration.

Agentic Reasoning Loops - Creates single agents that follow a Reason-Act loop to solve straightforward tasks step by step.

Agent Configurations - Drives agent behavior through configuration files and few-shot examples for standard out-of-the-box applications.

Tool Call Execution Loops - Implements the core tool call execution loop that drives agent interactions.

Autonomous Agent Definitions - Defines agent behavior and toolkits through a YAML-driven configuration system built on Pydantic and Hydra.

MCP Server Connections - Connects to Model Context Protocol toolkits by specifying transport type, connection parameters, and secrets.

Autonomous Agent Loops - Building and running LLM-powered agents with reasoning loops, toolkits, and YAML-driven configuration for task automation.

YAML-Defined Agents - Enables defining custom agents entirely through YAML configuration files specifying model, toolkits, and instructions.

Agent Evaluation Experiment Trackers - Loads YAML files that configure datasets, agents, and judging models to automate performance assessment.

Language Model Querying - Provides the core interface for sending prompts to language models and receiving text responses.

Agent Performance Evaluators - Benchmarking agent behavior against custom datasets with LLM-based judging, trajectory tracking, and configurable scoring pipelines.

Agent Prediction Evaluations - Evaluate a batch of agent predictions against expected results to determine correctness.

Model Tool Bindings - Provides a core mechanism for attaching tool definitions to language models for agentic execution.

Multi-Agent Orchestration - Coordinating multiple specialized agents through plan-and-execute workflows to decompose and solve complex tasks autonomously.

Multi-Agent Orchestrators - Defines planner, workers, and reporter within a single config to run coordinated multi-agent systems.

Multi-Agent Task Orchestrators - Orchestrates a Planner, Workers, and Reporter to decompose and solve complex tasks using a Plan-and-Execute strategy.

Simulated Environments - Provide agents with state and context through environments such as a shell for filesystem access or a browser for web interaction.

Standardized Benchmarks - Runs the evaluation pipeline on pre-built benchmark datasets such as WebWalkerQA or GAIA.

Pre-Built Agent Toolkits - Bundles pre-built tools for web search, file manipulation, code execution, and document analysis.

Code Execution Sandboxes - Runs user-supplied code in an isolated sandbox environment supporting over 20 programming languages.

Multi-Phase Benchmark Pipelines - Run a benchmark through preprocessing, rollout, judging, and statistics phases to measure agent performance.

Sandboxed Code Execution Environments - Provides an isolated sandbox for executing user-supplied code across over 20 programming languages.

YAML Configuration Files - Defines agent prompts, tools, and environments in structured YAML files for reproducibility.

Agent Configuration Files - Defines agents, tools, and environments through structured YAML files loaded into validated Pydantic models.

Model Context Protocol Integrations - Integrates external toolkits via the Model Context Protocol for standardized agent tool access.

Evaluation Pipelines - Provides a database-backed evaluation pipeline for tracking agent performance and experiment results.

Agentic Plan-And-Execute Workflows - Coordinates planner, worker, and reporter agents to decompose and execute complex tasks sequentially.

Evaluation Pipelines - Executes a complete evaluation workflow that rolls out an agent on a dataset and then judges its performance.

LLM Judge Accuracy Validators - Evaluates agent answers using a language model guided by prompts or fast rule-based matching for precise answers.

Agent Performance Benchmarks - Provides a standardized four-stage pipeline for benchmarking agent performance against datasets.

Offline Agent Evaluation Runners - Runs agents against test datasets, collects trajectories, and judges outputs against ground truth.

Web-Based Agent UIs - Wraps agents in interactive Gradio web interfaces for user-friendly command and control.

Agent Generators - Ships a meta-agent that automatically generates tools, YAML configs, and agent setups from task descriptions.

Programmatic Agent Launchers - Provides programmatic interfaces for launching agents from Python scripts.

Custom Reasoning Extensions - Allows extending core agent components like planners with domain-specific reasoning logic.

Agentic Search Tools - Provides a command-line search agent that answers questions by searching the web using a pre-configured toolkit.

Agent Instantiation - Loads agent configuration files and returns the correct agent instance based on its declared type.

Agent Tool Definitions - Automates creation of tool definitions to reduce manual boilerplate during agent development.

Programmatic Agent Spawning - Provides programmatic agent spawning capabilities for dynamic agent creation.

Tool Format Converters - Provides tool format conversion for OpenAI API compatibility.

Cloud Model Client Configurations - Sets default parameters like model name, API key, and base URL via environment variables or constructor arguments.

Web Search & Fetch Configurations - Integrates web search and content extraction services so the agent can retrieve and process online information.

Function-to-Tool Converters - Provides utilities for converting registered tools into formats compatible with the OpenAI Agents SDK.

Native Tool Call Parsers - Provides streaming tool call responses with incremental JSON fragments for low-latency interfaces.

Reward Functions - Creates custom verification logic scoring agent responses against ground truth using string matching or LLM judgment.

Large Language Model Configurations - Sets the primary text-generation model that the agent uses for reasoning and task execution.

Re-judging Tools - Reruns only the judgment phase of a completed agent rollout to reassess performance.

MCP Tool Connectors - Provides utilities for converting registered tools into the Model Context Protocol format.

Evaluator Model Configurators - Configures a separate language model to score and assess agent outputs during automated evaluation.

Evaluation Config Loaders - Loads named evaluation configurations from YAML files and returns them as typed Python objects.

Evaluation Database Backends - Switches evaluation storage from SQLite to external databases like PostgreSQL via environment variables.

Resumable Evaluation Runs - Saves every sample's state in a database so an experiment can stop and restart from the exact point it left off.

Multimodal Model Runners - Sets separate models for processing images and audio to enable the agent to handle visual and auditory inputs.

Agent Configuration Generators - Creates SimpleAgent configurations through an interactive meta-agent that asks about name, instructions, and tools.

Training-Free Performance Distillation - Improves agent performance by distilling experiential knowledge from execution trajectories without fine-tuning model parameters.

Agent-as-a-Tool Execution - Wraps parallel sub-agents inside custom tools so a single agent can delegate complex logic and parallelism.

Programmatic Agent Execution - Allows running custom agents defined in YAML by instantiating them from Python scripts.

HTTP API Tool Registrations - Provides a mechanism for registering custom HTTP APIs as tools for agent invocation.

Custom Benchmark and Framework Integration - Adds a new dataset by implementing a processor that prepares samples, judges responses, and calculates metrics.

Open Source Models - Supports building and running autonomous agents using open-source models without proprietary APIs.

Agent Execution Traces - Records tool calls and agent trajectories with a database-backed tracing system for performance analysis.

Tool Group Configurators - Provides a mechanism for grouping related tools into toolkits to simplify agent configuration.

Agent SDK Format Converters - Provides tool format conversion for the OpenAI Agents SDK.

Web Search Engines - Performs web searches using a configurable engine and returns results for a given query.

Browser Automation Tools - A tool for navigating web pages and extracting content to support automated research and data collection.

CLI Agent Runners - Launches interactive chat sessions with agents directly from the command line using provided scripts.

Builtin Toolkit Configurations - Activates and customizes built-in toolkits by specifying mode, enabling tools, and setting parameters.

Python Execution Sandboxes - Executes arbitrary Python code inside an isolated environment with configurable timeout and persistent variable sharing.

Training-Free GRPO Config Loaders - Loads named training-free GRPO configurations from YAML files and returns them as typed Python objects.

MCP Format Converters - Provides tool format conversion for the Model Context Protocol.

Training-Free GRPO Workflows - Executes training-free GRPO loops that generate rollouts and produce enhanced agent configurations for improved performance.

Code Executors - A secure code executor that runs polyglot code and shell commands in isolated environments for agent tool use.

Skill Files - Provides modular skill files that extend agent capabilities with structured instructions for complex tasks.

Agent Prediction Rollouts with Retries - Execute agent predictions on dataset samples asynchronously and retry up to a configurable number of times on failure.

Agent Execution Tracing - Captures agent execution traces using OpenTelemetry and Phoenix for debugging and observability.

Experiment Result Comparators - Compares metrics, trajectories, and success patterns side by side across multiple evaluation runs.

Agent Interaction Streams - Ships a Gradio-based web UI that streams agent reasoning steps and tool calls in real time.

Web Chat Interfaces - Starts a local web server with a chat-style UI to visualize and interact with an agent's conversation in real time.

Browser Automation - Controlling a web browser programmatically to navigate pages, extract content, and answer questions for automated research tasks.

Agent Frameworks - Agent framework optimized for open-source models.

AI Agents and Automation - Multi-modal intelligent agent framework by Tencent Cloud.

TencentCloudADPyoutu-agent

Features

Star history