Rdatatable/data.table

Features

• External File Imports - Reads text files into memory by automatically detecting separators, column types, and quote-escaping rules.
• R Data Manipulation Libraries - Provides a high-performance framework for tabular data manipulation, aggregation, and relational joining within the R language.
• Tabular Data Manipulations - Performs high-performance data wrangling, including filtering, aggregation, and reshaping, using efficient memory management and reference semantics.
• Tabular Data Processors - Provides a high-performance tabular data processing framework for filtering, aggregating, and joining large datasets.
• Binary Search Filtering - Retrieves specific rows using indices or computations with options to return all, first, or last matches.
• Data Import and Export - Provides efficient import and export of delimited files using high-performance parsing and serialization.
• Data Pipeline Acceleration - Accelerates complex data reshaping and aggregation tasks using optimized C-based internal routines.
• Delimited Text Parsing - Parses delimited text files into memory with automatic detection of separators and column types.
• Data Reshaping Operations - Converts data between wide and long formats using melting and casting with pattern-based column selection.
• Dataset Joins - Combines multiple datasets using various join modes while minimizing memory overhead and supporting complex merging.
• Fast Delimited File I/O - Provides high-speed ingestion of large delimited text files with automatic type detection and decompression.
• File-Based Data Ingestion - Reads large files into memory at high speeds with automatic type detection and flexible parsing options.
• Group-By Aggregations - Computes expressions across subsets of data defined by grouping variables using a concise, flexible syntax.
• High-Performance Data Analysis - Orders rows using high-performance sorting algorithms to accelerate data processing tasks.
• In-Memory Data Processors - Provides a high-speed in-memory engine for filtering, grouping, and reshaping large-scale datasets.
• In-Place Data Modifiers - Modifies data structures in place without creating memory-intensive copies to improve performance during large-scale data processing.
• Variable In-Place Mutations - Supports in-place mutation of data structures to avoid expensive object copying during transformations.
• In-Place Data Mutators - Modify specific values or add new columns directly within the existing memory structure to avoid the performance cost of duplicating the entire dataset.
• Key-Based Binary Search Subsetting - Enables high-performance binary search subsetting by setting columns as keys to physically reorder data.
• Large-Scale Data Computation - Enables high-speed processing and aggregation of massive datasets with billions of rows.
• Logical Table Subsetting - Retrieves specific rows and columns based on logical conditions, keys, or variable names.
• Multi-Threaded Data Aggregations - Distributes grouped computation tasks across multiple CPU cores to handle billions of rows efficiently.
• In-Place - Updates data by reference to avoid expensive object copying and reduce memory overhead.
• Grouped Query Executions - Filters rows and computes expressions across specific groups within a single, efficient operation.
• Scalable Tabular Aggregations - Computes summary statistics and groupings across billions of rows using multi-threaded execution and memory-efficient processing.
• Secondary Indexes - Computes and stores secondary indices to accelerate data access without requiring full table reordering.
• Table Joining Operations - Combines multiple datasets using equi, non-equi, rolling, range, or interval join methods.
• Advanced Joins - Merges datasets using rolling, overlapping range, non-equi, or aggregate join logic.
• Tabular Data Frames - Implements an enhanced, memory-efficient tabular data structure that supports in-place modification and accelerated binary search subsetting.
• Tabular Data Wrangling - Provides high-performance tools for cleaning, transforming, and reshaping large tabular datasets.
• Tabular Row and Column Subsetting - Filters rows and selects specific columns using a concise syntax for fast data retrieval.
• In-Place Data Structures - Uses reference semantics and in-place modification to handle massive datasets with minimal memory overhead.
• Memory Efficiency Strategies - Provides high-performance, memory-efficient utilities for importing and exporting tabular data.
• Native C Implementations - Offloads critical data manipulation and aggregation routines to compiled C code for maximum execution speed.
• Radix Sorts - Orders rows based on one or more columns using a fast internal radix sort algorithm.
• Performance and Optimization - Accelerates grouping, rolling calculations, and transformations using high-performance internal execution paths.
• Reference-Based Table Combination - Concatenates tables side-by-side using reference semantics to avoid memory-intensive object copying.
• 64-bit Integer Handling - Detects and preserves the precision of integers larger than 2^31 using a specialized 64-bit data type.
• Batch Value Updates - Updates specific values without duplicating the entire object during the modification process.
• Multi-Column Melting - Unpivots a dataset by converting multiple columns into row pairs or melting into multiple columns simultaneously.
• Column Management - Provides comprehensive tools for reordering columns and handling missing data during table manipulation.
• Conditional Value Replacements - Evaluates logical conditions to replace values within columns based on specified criteria.
• CSV Exporters - Export datasets to CSV files using multi-threaded processing to reduce write time.
• Memory-Mapped File Access - Uses memory-mapped file access to sample and infer data structures before loading.
• Type Inference - Samples file contents using memory-mapped access to determine the most efficient data types before loading.
• Data Filtering - Provides mechanisms for filtering data based on field conditions.
• Data Format Transformations - Converts tabular data between wide and long formats using optimized casting and melting operations.
• Table Format Translators - Transforms diverse data structures into a high-performance table format while maintaining internal element structures.
• Automatic Type Detection - Automatically infers column data types during the ingestion process to optimize memory usage.
• Batch Type Conversions - Performs batch type conversions across multiple columns simultaneously using pattern-based selection.
• Grouped Analysis - Calculates expressions and filters data across specific groups using a concise and efficient syntax.
• Pipeline Configuration Export and Import - Allows configuration of separators, missing value handling, and type coercion during data import and export.
• Data Indexing Structures - Organizes data structures using keys to enable fast retrieval and efficient filtering.
• Interval Overlap Joins - Merges two tables based on overlapping intervals or ranges.
• Non-Equi Joins - Matches rows using comparison operators like inequalities to compare numeric ranges or dates.
• Parallel Dataframe Operations - Utilizes multi-threading to speed up computationally intensive data processing tasks across large datasets.
• Range Data Extraction - Retrieves raw data objects associated with a specified range of values.
• Data Serialization and Parsing - Reads and writes structured files with automatic format detection, encoding support, and progress reporting.
• High-Performance CSV Exporters - Writes data to CSV files using a high-performance writer.
• Recursive Table Merging - Joins a sequence of tables from left to right using specified join types and keys.
• Import Filtering - Filters data by selecting or dropping columns during the reading process to optimize memory.
• Column-Level Import Projections - Selects specific columns by name or index during the initial read process to reduce memory consumption.
• Conditional Row Filters - Identifies and filters out records based on the absence of values within a specified set.
• Date and Time Libraries - Handles dates and times using integer-based classes for improved performance and easier manipulation.
• Temporal Component Extraction - Extracts and formats date and time components such as years, quarters, and weeks with optimized efficiency.
• Duplicate Row Filtering - Removes duplicate records from result sets or counts unique values across columns.
• Duplicate Row Identification - Locates repeated entries by returning logical vectors or the index of the first duplicate.
• Enhanced Data Frame Construction - Creates high-performance tabular structures from lists or direct function calls.
• Expression Parameterization - Substitutes variables, function names, and character values within filtering and computation arguments using a provided environment.
• Functional Query Extensions - Filters and transforms datasets using any expression or external package function within a query.
• Global String Caching - Uses global string caching to minimize memory usage for repetitive text data during import.
• Grouped Expression Executions - Runs arbitrary expressions and functions on subsets of data grouped by specific columns.
• Grouped Aggregations - Calculates summary statistics across groups using optimized functions like sum and mean to increase execution speed.
• Multi-Dimensional Aggregations - Summarizes data using grouping sets, cubes, and roll-ups with support for custom labels.
• Grouped Function Application - Executes custom calculations on subsets of data within each group for complex analytical workflows.
• High-Performance Tabular Coercion - Converts various data structures like matrices and lists into a high-performance tabular format with minimal overhead.
• Cartesian Join Prevention - Blocks joins that would result in an explosive number of rows to protect system memory.
• Value Filtering - Filters entries based on a predicate applied to their values within specified bounds.
• Long-to-Wide Reshaping - Converts data from a long format back to a wide format and applies aggregate functions to handle multiple observations.
• High-Performance Casting - Transforms data from long to wide formats using a fast, optimized implementation.
• Memory-Optimized Reshaping - Transforms data by aggregating values and spreading them across new columns for memory efficiency.
• Missing Data Removal - Drops rows containing missing values from a dataset using high-performance internal routines.
• Missing Value Imputation - Replaces missing entries in vectors using strategies like last observation carried forward.
• Coalescing Utilities - Fills missing data points by replacing them with the first available non-missing value from a set.
• Multi-Dimensional Grouping Sets - Calculates summaries using rollup, cube, and grouping set operations for multi-dimensional data analysis.
• Multi-Source Data Importers - Supports loading datasets from diverse sources including files, URLs, raw strings, and shell pipes.
• Optimized String Vector Matching - Finds the first occurrences of character strings using high-performance sorting algorithms.
• Multi-threaded Matrix Operations - Distributes grouped computation and sorting tasks across multiple CPU cores for parallel processing.
• Parallel Sorting - Orders datasets using multi-threaded radix sort algorithms for high-performance data alignment.
• Automatic Indexing - Accelerates subsequent queries by generating and saving indices during the first execution of a filter operation.
• Query Execution Optimizations - Applies automatic indexing and internal performance enhancements to accelerate filtering, grouping, and sorting.
• Query Performance Tuning - Optimizes data retrieval through automatic secondary indexing and pre-allocation of column slots.
• Regular Expression Data Extraction - Extracts specific patterns from text using named regular expressions to reshape data into structured formats.
• Advanced Relational Joins - Supports advanced relational operations including non-equi, rolling, and overlapping interval joins for complex dataset merging.
• Row Deletions - Removes specific rows from a table without creating a full copy to optimize memory usage.
• Group Optima Retrieval - Retrieves the specific row containing the maximum or minimum value for every distinct group.
• Group-Specific Row Extractions - Retrieves specific rows, such as the first or last entry, independently for every group in a dataset.
• Integer-Based Temporal Indexing - Uses integer-based storage for temporal data to accelerate sorting operations and minimize memory footprint.
• Integer-Based Date Classes - Provides specialized date and time classes that use integer storage for memory efficiency and faster arithmetic operations.
• String Pattern Filters - Filters tabular data based on string patterns and regular expressions.
• Time-Series Object Conversions - Converts tabular data into time-series objects by utilizing a temporal column as the primary index.
• Structured Data File Extractors - Analyzes file layouts to automatically detect field separators, headers, and row counts.
• Overlap Join Optimizations - Executes overlap joins and creates join tables to combine datasets efficiently.
• Join-Based Row Filtering - Identifies rows that only exist in one table or overlap between two tables without combining columns.
• Conditional Join Logic - Supports merging tables using relationships that change based on the specific characteristics of the data rows.
• Table Stacking - Merges multiple tables vertically into a single large dataset for high-speed processing.
• Column Structural Modifications - Modifies table columns directly within the existing memory structure to avoid expensive object copying.
• Tabular Duplicate Detection - Identifies and counts repeated records or unique entries within a data table.
• Dynamic Column Subsetting - Enables dynamic column subsetting using regular expressions or logical functions for flexible data retrieval.
• Time Series Analysis - Implements specialized tools for calculating rolling window aggregates and adaptive statistics on sequential data.
• Temporal Rounding - Adjusts temporal data to the nearest interval, such as hours or months, to facilitate grouped analysis.
• K-Nearest Value Search - Finds values with the minimum absolute difference to a target in sorted datasets.
• Unique Value Counting - Identifies distinct entries and calculates their occurrence frequencies in a dataset.
• Wide-to-Long Reshaping - Transforms wide-format data into long-format by collapsing multiple columns into key-value pairs.
• Fast Casting Implementations - Reshapes data from a long format to a wide format using fast casting operations.
• Multi-Column Pivoting - Expands multiple value columns from a long format into a wide format in a single operation.
• Rolling Statistics - Applies functions across a moving window of data to calculate trends and summaries.
• 64-bit Integer Types - Detects high-precision numbers exceeding standard limits and automatically assigns them to specialized sixty-four bit data types.
• Gzip File Read-Write Operations - Supports reading and writing of compressed tabular data formats like gzip and zip.
• File Read and Write Operations - Imports and exports delimited text files at high speeds to facilitate efficient data ingestion and persistence.
• Data Ranking Utilities - Assigns numerical ranks to data elements using various tie-breaking strategies for statistical analysis.
• Rolling Window Functions - Computes moving averages, sums, and other windowed metrics across sequential data.
• Adaptive Windows - Calculates rolling window aggregates where the window size varies based on observation intervals.
• Dynamic Window Sizing - Determines the width of a rolling window based on time-series indices.
• Conditional Element Assignment - Performs fast element-wise conditional checks and value assignments across large datasets.
• Multi-Condition Value Replacement - Evaluates a series of conditions and returns corresponding values based on the first true match.
• Large Dataset Explorers - Manipulates and aggregates massive data structures with high memory efficiency and speed to support large-scale data analysis workflows.
• Deep Copy Utilities - Enables the creation of fully independent, deep copies of datasets to prevent unintended side effects during in-place modifications.
• Parallel - Orders large datasets using a multi-threaded radix sort implementation for high-performance data alignment.
• Data Manipulation - High-performance data manipulation syntax.
• Numerical Libraries - Fast aggregation and manipulation of large datasets in R.
• Package and Dependency Management - High-performance data manipulation package for R.

Star history