TechEmpower/FrameworkBenchmarksArchived

Features

• Web Framework Benchmarks - Runs standardized, reproducible performance tests across many web frameworks to compare throughput and latency.
• Throughput Comparisons - Runs standardized tests measuring requests per second and latency across web frameworks executing common tasks.
• Common Web Task Benchmarks - Ships a standardized suite of benchmarks for common web tasks like JSON serialization and database queries.
• Plaintext Response Benchmarks - Ships a plaintext response benchmark that measures raw request routing throughput across frameworks.
• JSON Response Serializers - Provides a standardized JSON serialization benchmark that measures how quickly frameworks serialize objects into JSON responses.
• Benchmark Database Deployments - Configures and deploys database instances required by the benchmark tests.
• Multi-Backend Database Benchmarks - Tests measuring database read, write, and query performance across multiple database backends.
• Web Framework Performance Comparisons - Runs standardized benchmarks across many web frameworks to measure speed and scalability on tasks like JSON serialization and database access.
• Multiple Row Retrievals - Retrieves configurable numbers of random rows from a database table as JSON arrays.
• Single Row Fetches - Retrieves one random row from a database table using an ORM and returns it as a JSON object.
• Row Field Updates - Reads random rows, modifies a field on each, and persists changes back to the database.
• Automated Test Execution - Installs dependencies, launches frameworks and databases, verifies responses, and gathers results without manual intervention.
• CI Workflow Automations - Automatically runs the benchmark suite via GitHub Actions on commits to validate new frameworks and languages.
• Benchmark Pipeline Integrations - GitHub Actions workflow that automatically runs benchmarks on pull requests and commits to validate changes.
• Application Containerization - Builds self-contained Docker images that install each framework and run its server for reproducible benchmarking.
• Benchmark Containerization - Self-contained Docker images that package each framework with its runtime for reproducible benchmarking.
• Multi-Service Benchmark Orchestrators - Orchestrates separate Docker containers for application, database, and load generation across networked machines.
• HTTP Workload Executions - Executes a suite of predefined HTTP workloads across multiple framework implementations to measure throughput and latency.
• Distributed Benchmark Deployments - Deploys benchmark execution across separate application, database, and load generation servers for realistic testing.
• Web Framework Benchmark Suites - Standardized performance tests that measure throughput and latency across hundreds of web frameworks and languages.
• Serialization Speed Benchmarks - Tests the speed at which frameworks serialize objects into JSON format as part of the benchmark suite.
• Multi-Language Benchmark Harnesses - Automated test runner that installs dependencies, executes HTTP workloads, and collects results for any framework.
• End-to-End Benchmark Lifecycles - Automates the complete benchmark lifecycle from dependency installation through load testing and result collection.
• Cross-Framework Benchmarks - Executes a predefined set of performance tests across multiple web frameworks to compare throughput and latency.
• Reproducible Benchmark Scripts - Executes a standardized test harness that produces consistent, repeatable performance results across framework versions.
• Web Application Task Executions - Executes fundamental server-side operations including JSON serialization, database access, and template composition for reproducible comparisons.
• Query Performance Benchmarks - Measures how quickly frameworks handle common database operations like querying and serializing results.
• Benchmark Test Definitions - Declares framework test endpoints and runtime configuration in JSON files for harness discovery.
• CI Benchmark Pipelines - Automatically executes the full benchmark lifecycle from dependency installation through load testing and result collection using GitHub Actions.
• Cross-Framework Performance Test Executions - Executes a predefined set of performance tests across multiple web frameworks to produce comparable latency and throughput metrics.
• Distributed Benchmark Suites - Deploys a standardized test across three networked machines to measure web framework throughput and latency.
• Framework Performance Test Executions - Executes a suite of predefined performance tests against a web framework implementation and collects results.
• Standardized Performance Test Executions - Executes a standardized performance test against a web framework and verifies the results meet correctness requirements.
• Server-Side Template Rendering - Fetches database records, adds entries, sorts them, and renders HTML pages using server-side templates.
• Database Performance Testing - Measures framework performance on database operations including single and multiple row fetches, updates, and template rendering.
• Weighted Score Aggregators - Aggregates weighted results from multiple benchmark test types into a single normalized composite score.
• Normalized Performance Score Aggregators - Aggregates performance results across all test types into a single weighted score, normalized by dividing by 1,000 for readability.
• Benchmark Database Registrations - Registers new database backends via folder structure, Dockerfile, and Python class for benchmark provisioning.
• Database Result Caching - Stores fetched database rows in an in-memory or local-process cache to reduce database load on subsequent requests.
• Unified Performance Score Aggregators - Aggregates results across all test types into a single weighted score, giving a unified view of framework performance.
• Test Scaffolding Generators - Generates required file structure and configuration from predefined templates to standardize new framework additions.
• Benchmark - Provides structured templates and guidelines for adding new framework implementations.
• Community Benchmark Contributions - Community-driven project that accepts contributions to add new frameworks and databases to the benchmark suite.
• Production-Like Configurations - Configures each framework according to its production best practices to measure realistic performance.
• CI Change Analysis - Triggers verification jobs only for frameworks whose files were modified, skipping unchanged ones to save time.
• Commit Message Gating - Implements commit-message-based gating to selectively run or skip benchmark tests for changed frameworks.
• CI Scope Override Annotations - Controls benchmark test scope via commit message hooks like ci skip or ci fw-only.
• Community Contributions - Accepts community pull requests to add or update test implementations for new frameworks and languages.
• Environment Variable-Based Configuration - Injects build-time and runtime parameters via environment variables to adapt framework behavior without code changes.
• Benchmark Proposal Submissions - Accepts pull request proposals for adding new web frameworks or languages to the benchmark suite.
• Benchmark Result Comparison - Provides tools for comparing and diffing performance results between different benchmark runs.
• Distributed - Coordinates benchmark execution across separate application, database, and load generation servers.
• Maintainer-Verified Implementations - Flags framework test implementations as reviewed and vouched for by official maintainers.
• Benchmark Implementation Contributions - Accepts community contributions to add new web framework or language implementations following structured templates.
• JSON-Based Test Discoverers - Uses JSON configuration files to declare framework endpoints and runtime settings for automated test discovery.
• Performance Test Documentation - Documents standardized test suite structure and metadata for reproducible benchmarks.
• Benchmark Documentation - Records software versions, dependencies, and test sources for reproducible performance tests.
• Selective Benchmark Executions - Runs individual benchmark test types like JSON serialization or database queries in isolation for targeted measurement.

Star history