1brc

The 1BRC (One Billion Row Challenge) is a Java performance benchmarking exercise that processes one billion temperature records from a text file to compute the minimum, mean, and maximum temperature per weather station. At its core, it is a large-scale data aggregation challenge designed to test how efficiently a Java program can parse and aggregate structured data from a plain text file, serving as both a programming exercise and a benchmark for Java performance optimization.

The project distinguishes itself through a collection of performance-oriented architectural patterns for high-throughput data processing. These include branchless temperature parsing using bitwise operations, CPU-core-local aggregation maps that eliminate lock contention, a custom primitive hash map with long keys and int values to minimize object overhead, and garbage-collection-aware allocation that pre-allocates all working data structures upfront. Additional differentiators include JIT-friendly loop unrolling, memory-mapped file I/O, parallel stream processing across file chunks, and direct memory access via sun.misc.Unsafe to bypass bounds checks.

The project also provides supporting capabilities for benchmarking and profiling, including synthetic dataset generation with configurable parameters for reproducible testing, CPU profiling with flamegraphs to visualize execution time distribution, and tools for measuring and optimizing Java code execution speed against the fixed data processing challenge. The repository includes utilities for generating benchmark data files and profiling application performance to identify bottlenecks.

Features

Data Aggregation Challenges - A programming exercise that processes one billion temperature records from a text file to compute per-station statistics.

Performance Benchmarks - Measuring and optimizing the execution speed of Java programs processing large datasets.

Aggregated Temperature Statistics - Reads a large text file of weather station temperature readings and computes the min, mean, and max per station.

Memory-Mapped File Access - Reads the input file by mapping it directly into virtual memory, avoiding traditional buffered reads for faster access.

Grouped Aggregations - Computing summary statistics like min, mean, and max across grouped data records.

Primitive - Uses a hand-optimised hash map with primitive long keys and int values to minimise object overhead and garbage collection.

Chunked File Processing - Splits the file into chunks processed concurrently by multiple threads, aggregating partial results before merging.

Text File Processing Benchmarks - A benchmark that tests how efficiently a Java program can parse and aggregate structured data from a plain text file.

Branch-Less Parsing Techniques - Parses temperature values using bitwise operations and integer arithmetic instead of branching, reducing CPU pipeline stalls.

Zero-Allocation Architectures - Pre-allocates all working data structures upfront and avoids object creation during the hot loop to eliminate GC pauses.

Thread-Local Aggregation - Assigns each processing thread its own aggregation map to eliminate lock contention, merging results only at the end.

Java Benchmarking Tools - A tool for measuring and optimizing Java code execution speed against a fixed data processing challenge.

Large File Processing - Reading and aggregating data from text files with billions of rows efficiently.

CPU Profilers - Identifying performance bottlenecks in Java code using flamegraphs and execution time analysis.

Direct Memory Access - Leverages Unsafe for direct memory operations on the mapped file, bypassing bounds checks for maximum throughput.

JIT-Friendly Loop Unrolling - Writes tight, manually unrolled loops that the JIT compiler can further optimise into efficient native machine code.

gunnarmorling1brc

Features

Star history