intel/hyperscan
Hyperscan
Hyperscan is a high-performance regular expression matching library that scans large volumes of data against thousands of patterns simultaneously. It accepts PCRE-compatible regular expressions and supports multi-pattern matching in a single pass, approximate matching within a configurable edit distance, and streaming mode for processing data that arrives in blocks. The library is designed for throughput-oriented scanning across block, streaming, and vectored inputs.
What distinguishes Hyperscan is its hybrid automata engine, which combines deterministic and nondeterministic finite automata to match many patterns at once. It offers a lightweight runtime that excludes the compiler, reducing dependencies for embedded deployments. Other differentiating capabilities include logical combination of patterns (match fires only when a boolean expression over matches is true), pattern prefiltering for unsupported patterns, serialization of compiled databases with version and platform checks, and a custom allocator interface for controlling memory allocation.
Hyperscan provides pattern compilation of PCRE-compatible expressions and literal strings, syntax validation, and an approximate prefiltering compilation mode. Scanning operations include block mode for discrete records, streaming mode with state tracking across blocks, and a callback interface reporting match start and end offsets. Memory management covers user-supplied allocation for databases, scratch space, and streams, with independent scratch space lifecycle. At load time, Hyperscan selects the most efficient CPU instruction set variant automatically, and includes a throughput benchmarking tool for evaluating scanning performance.
Features
- High-Performance Matchers - A high-throughput regex matching library for scanning large pattern sets across streaming, block, and vectored data.
- Pattern Matching Engines - Accepts PCRE-compatible regular expressions and supports approximate matching with edit distance constraints.
- Streaming Regex Matching - Matches patterns across streaming data blocks, tracking partial matches without recompilation.
- Block Mode Scanning Operations - Scans discrete records in block mode for faster pattern matching without streaming overhead.
- Hybrid DFA-NFA Engines - Combines DFA and NFA to match thousands of patterns simultaneously in a single scan pass.
- Multi-Pattern Scanners - Scans input simultaneously against thousands of compiled regular expressions using hybrid automata techniques.
- Pre-Compiled Engines - Compiles regular expressions into an optimized pre-compiled database for high-performance concurrent scanning.
- High-Performance Regex Scanners - Scans large data volumes for many regex patterns simultaneously using SIMD and automata techniques.
- Pattern Scanning Interfaces - Provides a callback interface for reporting match offsets during scanning of various data layouts.
- Extended Syntax Support - Accepts PCRE-compatible regular expressions, enabling easy migration from other regex engines.
- Approximate Matchers - Performs fuzzy pattern matching within a specified edit distance, trading performance for tolerance to variations.
- Stream and Block Pattern Scanners - Supports both streaming mode across data blocks and block mode for faster discrete input scanning.
- Approximate Matching Tools - Matches patterns within a configurable edit distance for approximate, fuzzy matching.
- Multi-Pattern Matching Algorithms - Matches thousands of regular expressions simultaneously in a single pass using hybrid automata.
- Compiler-Free Runtimes - Ships a lightweight runtime that excludes the compiler for reduced dependencies in embedded deployments.
- Binary Pattern Serialization - Serializes compiled pattern databases with version and platform checks for storage or transfer.
- CPU Instruction Optimizations - Selects the most efficient CPU instruction set variant at load time for maximum scanning performance.
- Logical Pattern Combinations - Supports logical combinations of patterns so a match fires only when a boolean expression over matches is true.
- Compiler-Free Runtimes - Ships a lightweight runtime that excludes the compiler for reduced dependencies in embedded deployments.
- Streaming Matchers - Matches regular expressions across data streams, tracking partial matches across multiple blocks.
- Streaming Regex Matchers - Matches patterns across data streams, tracking partial matches across multiple blocks.
- Approximate Prefiltering Compilation Modes - Provides approximate prefilter compilation for unsupported patterns to enable fuzzy matching via later exact confirmation.
- Approximate Regex Matchers - Moves pattern matching with a specified edit distance for fuzzy matching.
- Edit Distance Matchers - Enables fuzzy pattern matching within a configurable edit distance, trading throughput for tolerance to small variations.
- Pattern Match Combinators - Combines pattern matches with boolean logic so a match fires only when a logical expression over matches is true.
- Regular Expressions - High-performance library for simultaneous matching of multiple regular expressions.
