# techempower/frameworkbenchmarks

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/techempower-frameworkbenchmarks).**

8,112 stars · 2,040 forks · C++ · NOASSERTION · archived

## Links

- GitHub: https://github.com/TechEmpower/FrameworkBenchmarks
- Homepage: https://www.techempower.com/benchmarks/
- awesome-repositories: https://awesome-repositories.com/repository/techempower-frameworkbenchmarks.md

## Topics

`benchmark` `framework` `frameworkbenchmarks` `performance` `suite`

## Description

TechEmpower FrameworkBenchmarks is an open-source project that provides a standardized, reproducible suite for measuring and comparing the performance of web frameworks across many languages. At its core, it defines a set of common server-side tasks—such as JSON serialization, database queries, and server-side template rendering—and executes them uniformly against hundreds of framework implementations to produce comparable throughput and latency metrics. The project is built around a multi-language benchmark harness that automates the full test lifecycle, from dependency installation and server startup through load generation and result collection, all within Docker containers to ensure consistency.

What distinguishes this project is its comprehensive, community-driven infrastructure for maintaining and evolving the benchmark suite. It features a commit-message-controlled CI pipeline that selectively runs tests only for changed frameworks, supports personal fork execution for faster feedback, and uses JSON-driven test discovery to enumerate and execute tests. The suite orchestrates a three-tier distributed benchmarking environment, separating application, database, and load-generator containers across networked machines to simulate production-like deployments. Results are aggregated into a single normalized composite score, and the project provides real-time log streaming and live benchmark data viewing for immediate debugging and comparison.

The project supports a wide range of benchmark types, including single and multiple database row retrieval, row updates, and server-side HTML template rendering. It includes mechanisms for registering new database backends, caching database query results, and tuning operating system parameters for optimal network performance. The suite also offers error monitoring that reports HTTP 500-series and socket-level errors during peak load tests, along with tools for comparing and diffing results between different benchmark runs.

Contributions are managed through a structured pull-request-based proposal system for both frameworks and databases, with template-based scaffolding to standardize new additions. The project provides automated deployment scripts for setting up the benchmark environment across multiple machines, and includes a verified framework implementation flag to indicate that a test has been reviewed by its official maintainer.

## Tags

### Testing & Quality Assurance

- [Web Framework Benchmarks](https://awesome-repositories.com/f/testing-quality-assurance/software-testing/testing-frameworks/test-frameworks/performance-testing-frameworks/web-framework-benchmarks.md) — Runs standardized, reproducible performance tests across many web frameworks to compare throughput and latency.
- [End-to-End Benchmark Lifecycles](https://awesome-repositories.com/f/testing-quality-assurance/automated-test-suites/package-lifecycle-testing/end-to-end-benchmark-lifecycles.md) — Automates the complete benchmark lifecycle from dependency installation through load testing and result collection. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Basic-Concepts))
- [Cross-Framework Benchmarks](https://awesome-repositories.com/f/testing-quality-assurance/cross-framework-benchmarks.md) — Executes a predefined set of performance tests across multiple web frameworks to compare throughput and latency. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Documentation))
- [Reproducible Benchmark Scripts](https://awesome-repositories.com/f/testing-quality-assurance/debugging-diagnostics/error-handling/benchmark-execution/reproducible-benchmark-scripts.md) — Executes a standardized test harness that produces consistent, repeatable performance results across framework versions. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Techempower))
- [Web Application Task Executions](https://awesome-repositories.com/f/testing-quality-assurance/debugging-diagnostics/error-handling/benchmark-execution/web-application-task-executions.md) — Executes fundamental server-side operations including JSON serialization, database access, and template composition for reproducible comparisons. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki))
- [Query Performance Benchmarks](https://awesome-repositories.com/f/testing-quality-assurance/performance-testing-analysis/performance-diagnostics/performance-measurement/query-performance-benchmarks.md) — Measures how quickly frameworks handle common database operations like querying and serializing results. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Techempower))
- [Benchmark Test Definitions](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/benchmark-test-definitions.md) — Declares framework test endpoints and runtime configuration in JSON files for harness discovery. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Codebase-Framework-Files))
- [CI Benchmark Pipelines](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/ci-benchmark-pipelines.md) — Automatically executes the full benchmark lifecycle from dependency installation through load testing and result collection using GitHub Actions.
- [Cross-Framework Performance Test Executions](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/cross-framework-performance-test-executions.md) — Executes a predefined set of performance tests across multiple web frameworks to produce comparable latency and throughput metrics. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Documentation))
- [Distributed Benchmark Suites](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/distributed-benchmark-suites.md) — Deploys a standardized test across three networked machines to measure web framework throughput and latency. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Benchmarking-Getting-Started))
- [Framework Performance Test Executions](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/framework-performance-test-executions.md) — Executes a suite of predefined performance tests against a web framework implementation and collects results. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Codebase-Overview))
- [Standardized Performance Test Executions](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/standardized-performance-test-executions.md) — Executes a standardized performance test against a web framework and verifies the results meet correctness requirements. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Benchmarking-Getting-Started))
- [Benchmark Result Comparison](https://awesome-repositories.com/f/testing-quality-assurance/agent-performance-benchmarks/benchmark-result-analysis/benchmark-result-comparison.md) — Provides tools for comparing and diffing performance results between different benchmark runs. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Codebase-Overview))
- [Distributed](https://awesome-repositories.com/f/testing-quality-assurance/debugging-diagnostics/error-handling/benchmark-execution/distributed.md) — Coordinates benchmark execution across separate application, database, and load generation servers.
- [Maintainer-Verified Implementations](https://awesome-repositories.com/f/testing-quality-assurance/functionality-verifiers/cross-implementation-parity-verifiers/maintainer-verified-implementations.md) — Flags framework test implementations as reviewed and vouched for by official maintainers. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Verified-Tag))
- [Benchmark Implementation Contributions](https://awesome-repositories.com/f/testing-quality-assurance/software-testing/testing-frameworks/test-frameworks/browser-and-ui-testing/browser-automation-frameworks/web-testing-frameworks/benchmark-implementation-contributions.md) — Accepts community contributions to add new web framework or language implementations following structured templates.
- [JSON-Based Test Discoverers](https://awesome-repositories.com/f/testing-quality-assurance/software-testing/testing-frameworks/test-frameworks/execution-and-infrastructure/test-execution-runners/test-enumerators/json-based-test-discoverers.md) — Uses JSON configuration files to declare framework endpoints and runtime settings for automated test discovery.
- [Performance Test Documentation](https://awesome-repositories.com/f/testing-quality-assurance/software-testing/testing-frameworks/test-frameworks/performance-testing-frameworks/performance-test-documentation.md) — Documents standardized test suite structure and metadata for reproducible benchmarks. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Development-Readme-Formats))
- [Benchmark Documentation](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/benchmark-documentation.md) — Records software versions, dependencies, and test sources for reproducible performance tests. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Development-Readme-Formats))
- [Selective Benchmark Executions](https://awesome-repositories.com/f/testing-quality-assurance/test-and-benchmark-execution/selective-benchmark-executions.md) — Runs individual benchmark test types like JSON serialization or database queries in isolation for targeted measurement. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Development-Testing-and-Debugging))

### Artificial Intelligence & ML

- [Throughput Comparisons](https://awesome-repositories.com/f/artificial-intelligence-ml/framework-comparisons/throughput-comparisons.md) — Runs standardized tests measuring requests per second and latency across web frameworks executing common tasks. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki))
- [Weighted Score Aggregators](https://awesome-repositories.com/f/artificial-intelligence-ml/evaluation-metrics/scoring-pipelines/feature-cross-scoring/criteria-based-scoring-engines/weighted-score-aggregators.md) — Aggregates weighted results from multiple benchmark test types into a single normalized composite score.
- [Normalized Performance Score Aggregators](https://awesome-repositories.com/f/artificial-intelligence-ml/evaluation-metrics/scoring-pipelines/feature-cross-scoring/criteria-based-scoring-engines/weighted-score-aggregators/normalized-performance-score-aggregators.md) — Aggregates performance results across all test types into a single weighted score, normalized by dividing by 1,000 for readability. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Composite-Scoring-for-Frameworks))

### Part of an Awesome List

- [Common Web Task Benchmarks](https://awesome-repositories.com/f/awesome-lists/ai/web-and-environment-benchmarks/common-web-task-benchmarks.md) — Ships a standardized suite of benchmarks for common web tasks like JSON serialization and database queries. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Benchmarks-Project))

### Content Management & Publishing

- [Plaintext Response Benchmarks](https://awesome-repositories.com/f/content-management-publishing/markdown-to-rich-text-parsers/plaintext-converters/plaintext-response-benchmarks.md) — Ships a plaintext response benchmark that measures raw request routing throughput across frameworks. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview))

### Data & Databases

- [JSON Response Serializers](https://awesome-repositories.com/f/data-databases/data-structures/structured-return-objects/json-response-serializers.md) — Provides a standardized JSON serialization benchmark that measures how quickly frameworks serialize objects into JSON responses. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview))
- [Benchmark Database Deployments](https://awesome-repositories.com/f/data-databases/database-management/benchmark-database-deployments.md) — Configures and deploys database instances required by the benchmark tests. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Codebase-Overview))
- [Multi-Backend Database Benchmarks](https://awesome-repositories.com/f/data-databases/database-performance-utilities/performance-benchmarking/multi-backend-database-benchmarks.md) — Tests measuring database read, write, and query performance across multiple database backends.
- [Web Framework Performance Comparisons](https://awesome-repositories.com/f/data-databases/database-performance-utilities/performance-benchmarking/web-framework-performance-comparisons.md) — Runs standardized benchmarks across many web frameworks to measure speed and scalability on tasks like JSON serialization and database access. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Documentation))
- [Multiple Row Retrievals](https://awesome-repositories.com/f/data-databases/database-schema-mapping/row-mapping-logic/row-record-access/tabular-row-storage/mock-row-queries/table-row-counts/multiple-row-retrievals.md) — Retrieves configurable numbers of random rows from a database table as JSON arrays. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview))
- [Single Row Fetches](https://awesome-repositories.com/f/data-databases/query-result-fetching/single-row-fetches.md) — Retrieves one random row from a database table using an ORM and returns it as a JSON object. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview))
- [Row Field Updates](https://awesome-repositories.com/f/data-databases/row-field-updates.md) — Reads random rows, modifies a field on each, and persists changes back to the database. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview))
- [Benchmark Database Registrations](https://awesome-repositories.com/f/data-databases/database-management-systems/database-engines/cloud-native-databases/database-registration/benchmark-database-registrations.md) — Registers new database backends via folder structure, Dockerfile, and Python class for benchmark provisioning. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Development-Adding-a-Database))
- [Database Result Caching](https://awesome-repositories.com/f/data-databases/database-result-caching.md) — Stores fetched database rows in an in-memory or local-process cache to reduce database load on subsequent requests. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview))
- [Unified Performance Score Aggregators](https://awesome-repositories.com/f/data-databases/real-time-data-aggregators/performance-metric-aggregators/unified-performance-score-aggregators.md) — Aggregates results across all test types into a single weighted score, giving a unified view of framework performance. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Composite-Scoring-for-Frameworks))

### Development Tools & Productivity

- [Automated Test Execution](https://awesome-repositories.com/f/development-tools-productivity/debugging-profiling-testing/test-execution-management/automated-test-execution.md) — Installs dependencies, launches frameworks and databases, verifies responses, and gathers results without manual intervention. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Basic-Concepts))
- [Test Scaffolding Generators](https://awesome-repositories.com/f/development-tools-productivity/code-generators/boilerplate-generators/test-scaffolding-generators.md) — Generates required file structure and configuration from predefined templates to standardize new framework additions.
- [Benchmark](https://awesome-repositories.com/f/development-tools-productivity/contribution-guidelines/benchmark.md) — Provides structured templates and guidelines for adding new framework implementations. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Development-Readme-Formats))
- [Community Benchmark Contributions](https://awesome-repositories.com/f/development-tools-productivity/open-source-benchmarking-tools/community-benchmark-contributions.md) — Community-driven project that accepts contributions to add new frameworks and databases to the benchmark suite.
- [Production-Like Configurations](https://awesome-repositories.com/f/development-tools-productivity/performance-optimization-tools/performance-benchmarks/production-like-configurations.md) — Configures each framework according to its production best practices to measure realistic performance. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Environment))

### DevOps & Infrastructure

- [CI Workflow Automations](https://awesome-repositories.com/f/devops-infrastructure/automation-orchestration/task-execution-frameworks/ci-workflow-automations.md) — Automatically runs the benchmark suite via GitHub Actions on commits to validate new frameworks and languages. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Development-Testing-and-Debugging))
- [Benchmark Pipeline Integrations](https://awesome-repositories.com/f/devops-infrastructure/ci-cd-pipelines/ci-pipeline-integrations/benchmark-pipeline-integrations.md) — GitHub Actions workflow that automatically runs benchmarks on pull requests and commits to validate changes.
- [Application Containerization](https://awesome-repositories.com/f/devops-infrastructure/container-orchestration/container-runtimes/runtime-configuration-interfaces/docker-socket-orchestrators/docker-target-configurators/docker-container-deployments/application-containerizers/application-containerization.md) — Builds self-contained Docker images that install each framework and run its server for reproducible benchmarking. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Codebase-Framework-Files))
- [Benchmark Containerization](https://awesome-repositories.com/f/devops-infrastructure/container-orchestration/container-runtimes/runtime-configuration-interfaces/docker-socket-orchestrators/docker-target-configurators/docker-container-deployments/benchmark-containerization.md) — Self-contained Docker images that package each framework with its runtime for reproducible benchmarking.
- [Multi-Service Benchmark Orchestrators](https://awesome-repositories.com/f/devops-infrastructure/container-orchestration/container-runtimes/runtime-configuration-interfaces/docker-socket-orchestrators/docker-target-configurators/docker-container-orchestrators/multi-service-benchmark-orchestrators.md) — Orchestrates separate Docker containers for application, database, and load generation across networked machines.
- [HTTP Workload Executions](https://awesome-repositories.com/f/devops-infrastructure/container-orchestration/workload-scheduling-scaling/batch-workload-execution/http-workload-executions.md) — Executes a suite of predefined HTTP workloads across multiple framework implementations to measure throughput and latency. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Basic-Concepts))
- [Distributed Benchmark Deployments](https://awesome-repositories.com/f/devops-infrastructure/distributed-benchmark-deployments.md) — Deploys benchmark execution across separate application, database, and load generation servers for realistic testing.
- [CI Change Analysis](https://awesome-repositories.com/f/devops-infrastructure/ci-change-analysis.md) — Triggers verification jobs only for frameworks whose files were modified, skipping unchanged ones to save time. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-GitHub-Actions))
- [Commit Message Gating](https://awesome-repositories.com/f/devops-infrastructure/event-driven-triggers/build-pipeline-triggers/commit-type-pipeline-triggers/commit-message-gating.md) — Implements commit-message-based gating to selectively run or skip benchmark tests for changed frameworks.
- [CI Scope Override Annotations](https://awesome-repositories.com/f/devops-infrastructure/release-automation/github-release-automators/release-note-generation/commit-message-overriders/ci-scope-override-annotations.md) — Controls benchmark test scope via commit message hooks like [ci skip] or [ci fw-only]. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-GitHub-Actions))

### Education & Learning Resources

- [Web Framework Benchmark Suites](https://awesome-repositories.com/f/education-learning-resources/research-academic-resources/cross-framework-benchmarking-suites/web-framework-benchmark-suites.md) — Standardized performance tests that measure throughput and latency across hundreds of web frameworks and languages.
- [Community Contributions](https://awesome-repositories.com/f/education-learning-resources/community-contributions.md) — Accepts community pull requests to add or update test implementations for new frameworks and languages. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Techempower))

### Networking & Communication

- [Serialization Speed Benchmarks](https://awesome-repositories.com/f/networking-communication/json-serialization/serialization-speed-benchmarks.md) — Tests the speed at which frameworks serialize objects into JSON format as part of the benchmark suite. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/About-Techempower))

### Software Engineering & Architecture

- [Multi-Language Benchmark Harnesses](https://awesome-repositories.com/f/software-engineering-architecture/executable-activity-definitions/test-harnesses/multi-language-benchmark-harnesses.md) — Automated test runner that installs dependencies, executes HTTP workloads, and collects results for any framework.
- [Environment Variable-Based Configuration](https://awesome-repositories.com/f/software-engineering-architecture/application-lifecycle-management/configuration-management/environment-variable-management/environment-variables/environment-variable-based-configuration.md) — Injects build-time and runtime parameters via environment variables to adapt framework behavior without code changes.
- [Benchmark Proposal Submissions](https://awesome-repositories.com/f/software-engineering-architecture/proposal-lifecycle-management/proposal-frameworks/proposal-submission-workflows/benchmark-proposal-submissions.md) — Accepts pull request proposals for adding new web frameworks or languages to the benchmark suite. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Suggested-Frameworks-Languages-and-Features))

### User Interface & Experience

- [Server-Side Template Rendering](https://awesome-repositories.com/f/user-interface-experience/dynamic-html-generators/server-side-template-rendering.md) — Fetches database records, adds entries, sorts them, and renders HTML pages using server-side templates. ([source](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview))

### Web Development

- [Database Performance Testing](https://awesome-repositories.com/f/web-development/web-infrastructure-deployment/database-backed-applications/database-performance-testing.md) — Measures framework performance on database operations including single and multiple row fetches, updates, and template rendering.