google/clusterfuzz
Clusterfuzz
ClusterFuzz is an automated platform that runs coverage-guided fuzzers at scale to find security and stability bugs in software. It orchestrates libFuzzer and AFL++ across distributed clusters of worker bots, collecting coverage feedback to guide input mutation and discover crashes. The platform provides a web-based dashboard for configuring fuzzing jobs, monitoring progress, and inspecting crash reports, with role-based access control to restrict sensitive features.
The system automates the full fuzzing lifecycle, from build pipeline integration and corpus management to crash triage and bug tracking. It deduplicates crashes by generating normalized stacktrace signatures, automatically files and closes bug reports on issue trackers, and identifies the exact code change that introduced a regression through binary search across revisions. Testcase minimization reduces crashing inputs to their smallest reproducible form, while coverage reporting generates per-fuzzer and aggregate summaries in JSON and HTML formats.
ClusterFuzz handles bot deployment across Linux, macOS, and Windows on Google Compute Engine or custom infrastructure, with cron-driven maintenance for tasks like corpus pruning and crash verification. It supports both coverage-guided and blackbox fuzzing, and provides a local development mode for testing configurations before production deployment. The platform emits structured logs for monitoring and tracks fuzzer performance metrics, crash statistics, and code coverage over time through its web interface.
Features
- Coverage-Guided Fuzzing - An automated platform that runs coverage-guided fuzzers at scale to find security and stability bugs in software.
- Automated Bug Report Management - Creates, triages, and closes bug reports automatically in issue trackers like Monorail and Jira.
- Crash and Monitoring - Provides a web interface to browse crash reports, statistics, and other fuzzing results.
- Fuzzing Orchestrators - Provisions worker machines that pull fuzzing tasks from the cluster and execute them in a managed environment.
- Fuzzing Job Definitions - Defines fuzzing jobs by specifying environment variables and build locations for target programs.
- Parallel Fuzzing Distributions - Manages a cluster of worker bots that execute fuzzing tasks across thousands of machines.
- Fuzzing Campaign Managers - Defines and configures fuzzing tasks with specific parameters, targets, and environments to run on a cluster.
- Web Dashboards - Provides a browser-based dashboard for viewing crashes, managing jobs, and inspecting results.
- Scalable Cluster Deployments - Runs coverage-guided fuzzing jobs on clusters of any size, scaling from a single machine to tens of thousands of VMs.
- Fuzzing Campaign Monitors - Checks bot activity, console output, and generated test cases to verify that fuzzing is running and producing crashes.
- Regression Range Identifiers - Identifies the exact commit range where a regression was introduced using archived builds.
- Regression Identifiers - Identifies the exact code change that introduced a crash by performing a binary search across revisions.
- Regression Isolators - Bisects source history to identify the exact commit that introduced a crash or bug.
- Build Pipeline Integrations - Automatically fetches the latest instrumented builds from cloud storage for fuzzing bots.
- Job Scheduling & Configuration - Creates and modifies job definitions that control how fuzzing tasks are executed.
- Web-Based Job Management - Provides a browser interface for defining and managing fuzzing jobs through environment variables and options.
- Coverage-Guided Fuzzers - Uploads a libFuzzer build and runs it across a cluster to find memory-safety bugs like buffer overflows.
- Job Type Definitions - Defines the execution parameters for fuzzing jobs, including binary names, arguments, and timeouts.
- Distributed Fuzzing Clusters - Distributes fuzzing jobs across clusters of any size, from a single machine to tens of thousands of VMs.
- Bug Introduction Range Identifiers - Identifies the exact commit range where a bug was introduced through binary search across revisions.
- Distributed Task Queues - Distributes fuzzing tasks across worker bots that pull work from a centralized queue and report results.
- Crash History Managers - Provides a browser-based dashboard to view, filter, and manage crash reports and fuzzing results.
- Crash Deduplication and Triaging - Groups crash reports into unique bugs by generating and matching normalized stacktrace signatures.
- Delta Debugging Minimizations - Reduces a crashing input to its smallest possible form while still triggering the same crash.
- Bug Reproduction Environments - Provides exact testcase, build, and environment details needed to reproduce a crash.
- Fuzzer Performance Metric Displays - Displays performance reports, improvement recommendations, and metadata for in-process fuzz targets.
- Fuzzer Performance Monitors - Monitors execution speed and crash counts over time with filters to compare fuzzers or view data as charts.
- Code Coverage Analysis - Stores and displays code coverage data from external builders, letting teams monitor which lines are exercised by fuzzers.
- Fuzzing Coverage Analyses - Tracks which lines of code are exercised during fuzzing to identify untested areas and guide test generation.
- Bot-Based Task Execution - Executes a unit of work on a bot, such as a fuzzing session or testcase minimization.
- Automated Bug Lifecycle Management - Automatically files, triages, and closes bug reports on issue trackers when crashes are found or fixed.
- Test Case Minimization - Reduces a crashing input to the smallest possible version that still triggers the bug, making it easier to debug.
- Fuzzing Result Dashboards - Displays discovered testcases and their stack traces on a dashboard for triage.
- Blackbox Fuzzing Configurations - Specifies the target application name, required and optional arguments, and per-testcase timeout for blackbox fuzzing jobs.
- Sanitizer Flag Overrides - Overrides default runtime flags for AddressSanitizer, MemorySanitizer, or UndefinedBehaviorSanitizer per job.
- Cron Scheduling - Uses scheduled cron jobs to trigger periodic maintenance tasks like corpus pruning and crash verification.
- Build Pipeline Integrations - Points a job to a Google Cloud Storage bucket so the system automatically picks the newest build archive for fuzzing.
- Fuzzing Bot Deployments - Deploys macOS bots on physical hardware with sanitizer options and libFuzzer using service account authentication.
- Fuzzing Bot - Adjusts fuzzing bot count by editing a cluster configuration file that an automated job reads to create or delete instances.
- Fuzzing Bot Cluster Templates - Defines fuzzing bot clusters on GCE via YAML configuration with zone, instance count, and template settings.
- Continuous Build Pipelines - Configures a continuous build system that compiles the target and delivers fresh binaries to the fuzzing cluster.
- Fuzzing Dashboards - Provides a browser interface to configure jobs, monitor fuzzing progress, and inspect crash reports.
- Google Cloud Fuzzing Deployments - Deploys fuzzing bots on Google Compute Engine with automated build pipelines and cloud storage integration.
- Fuzzing Job Configurations - Defines environment variables that control how a fuzzing job runs, including binary paths, sanitizer options, and target settings.
- Fuzzing Job Definitions - Defines a fuzzing task by selecting a coverage-guided or blackbox approach and configuring its parameters on the Jobs page.
- Fuzzing Bot Deployments - Deploys Linux bots optimized for fuzzing with sanitizer support and preemptible instances for cost efficiency.
- Cron Job Automation - Runs periodic maintenance and analysis tasks automatically via the App Engine cron scheduler.
- Fuzzing Bot Deployments - Deploys fuzzing bots on non-Google machines or other cloud providers using Docker images and service accounts.
- Fuzzing Bot Deployments - Deploys Windows bots supporting AddressSanitizer and libFuzzer with administrator password for remote access.
- Administrative Access Control - Sets administrative settings including credentials and user permissions for the fuzzing platform.
- Dictionary-Based Mutations - Uses predefined token dictionaries to guide mutation of structured input formats during fuzzing.
- Corpus Seeding - Uploads seed corpora at build time so fuzzers start testing from known inputs rather than scratch.
- Role-Based Access Control - Restricts access to crash reports, job configurations, and security-sensitive features based on user roles.
- Role & Permission Configurators - Allows administrators to assign users to roles and manage access levels through a configuration interface.
- Security Vulnerability Reporting - Restricts access to security-related bug reports and crash data to privileged users and administrators only.
- Crash Regression Bisections - Identifies the exact code change that introduced a crash through binary search across revisions.
- Automatic Crash Closure - Automatically closes bugs as verified when crashes stop occurring for a defined period.
- Crash Attribute Filters - Filters testcases by crash type, state, and security implications to narrow down relevant bugs.
- Crash Severity Classifiers - Classifies crashes by type like heap-buffer-overflow to indicate severity for triage.
- Crash Statistics Dashboards - Shows crash frequency and reproduction conditions like platform and fuzzer to aid debugging.
- Unreliable Crash Trackers - Tracks crashes without reliable testcases and replaces them with reproducible reports when found.
- Crash Regression Ranges - Determines the exact revision range in which a crash was introduced by testing builds across versions.
- Crash Reproducibility Checks - Tests whether a previously found crash still reproduces on the latest build or has been fixed.
- Crash Reproduction Runners - Ships a feature to run uploaded testcases against jobs to confirm crash reproduction.
- Testcase Archive Submissions - Accepts testcase archives for execution against jobs and returns crash details with regression ranges.
- Testcase Submission Interfaces - Provides a submission interface for uploading single testcases to verify externally reported bugs.
- Native Crash Trace Formatters - Displays sanitizer-generated stack traces to help developers locate the code path causing a crash.
- Bug Fix Verifiers - Verifies bug fixes by re-running testcases against the latest build and updating bug status.
- Corpus Minimization - Shrinks a fuzzing corpus to the smallest set of inputs that maintains the same code coverage.
- Custom Fuzzer Submissions - Packages a program that generates test cases from a corpus and submits it to the platform for automated execution.
- Blackbox Fuzzing - Feeds predefined or randomly generated inputs to a target without coverage feedback for input parsing testing.
- Code Coverage Reportings - Runs instrumented fuzzers against stored corpora and produces per-fuzzer and aggregate coverage summaries.
- Crash Reproduction Details - Provides reproduction steps and crash details to help developers locate and patch the root cause.
- Test Case Minimizers - Reduces crashing inputs to their smallest reproducible form for easier debugging.
- Coverage Artifact Uploads - Pushes coverage JSON files and HTML reports to a configurable cloud storage bucket for centralized access.
