Alibaba-NLP/WebAgent
19,549 Stars·1,495 Forks·Python·Apache-2.0·7 Aufrufetongyi-agent.github.io/blog/introducing-tongyi-deep-research↗
WebAgent
WebAgent is an autonomous web navigation agent and research system designed to browse the internet and synthesize information to answer complex queries. It functions as a reasoning orchestrator that navigates the web iteratively to perform deep research and extract structured data.
The project includes a reinforcement learning training pipeline that generates synthetic interaction datasets for model pre-training and fine-tuning. It employs token-level policy gradients to stabilize training in non-stationary environments and uses a dual-mode inference scaling mechanism to balance execution between standard and high-compute reasoning loops.
The system covers multi-modal content parsing for both live web pages and local documents such as PDFs. It also provides a benchmarking framework to evaluate research quality by comparing generated responses against ground-truth datasets.
Features
- Autonomous Web Research Agents - Provides an autonomous agent that crawls and synthesizes information from the web to generate deep research reports.
- Autonomous Web Research Loops - Implements iterative reasoning loops that autonomously search the web and extract content to answer complex queries.
- Autonomous Web Researchers - An autonomous system designed to navigate and synthesize information from the web without human intervention.
- Interaction Dataset Generation - Implements a pipeline to generate synthetic interaction datasets for model pre-training and fine-tuning.
- Token-Level Policy Gradients - Employs token-level policy gradients to stabilize reinforcement learning training in non-stationary environments.
- Reasoning Orchestrators - Orchestrates reasoning modes and execution loops to optimize the quality of generated research outputs.
- Reinforcement Learning Optimizers - Optimizes model policies using token-level gradients to improve performance in non-stationary web environments.
- Reinforcement Learning Training Pipelines - Provides a framework for managing the lifecycle of reinforcement learning training for web agents.
- Synthetic Data Generators - Automates the production of instruction-following interaction datasets for model training.
- Training Data Generation - Generates synthetic interaction datasets through automated web browsing for model pre-training and fine-tuning.
- Web Content Extractions - Extracts structured information and summaries from online web pages using language models.
- Quality Evaluators - Uses automated frameworks to assess the accuracy of generated research against gold-standard responses.
- Ground-Truth Scoring - Evaluates the accuracy of research responses by comparing them against gold-standard ground-truth datasets.
- Inference Scaling Mechanisms - Features a dual-mode inference scaling mechanism to balance standard and high-compute reasoning loops.
- Reasoning Compute Balancing - Implements a dual-mode inference scaling mechanism to balance execution between standard and high-compute reasoning loops.
- Multi-Modal Content Normalizers - Normalizes heterogeneous inputs from live web pages and local PDFs into a uniform representation for processing.
- Agent Performance Benchmarks - Provides a benchmarking framework to evaluate the efficacy and accuracy of the research agent's outputs.
- Web Data Extractors - Parses and structures information from both online web pages and local documents.
- Search and Research Agents - Frameworks for web navigation, information seeking, and deep research agents.
- Web and Environment Benchmarks - Benchmark for LLM web traversal.
