hosseinmoein/DataFrame

Features

• Column-Major Storage - Uses column-major storage to maintain heterogeneous data in contiguous memory for high cache efficiency.
• In-Memory Data Stores - Provides a high-performance in-memory data store for heterogeneous tabular data using contiguous memory allocation.
• Tabular Data Frames - Implements high-performance in-memory structured grids for manipulating tabular data and performing matrix operations.
• Statistical Analysis - Provides a comprehensive framework for descriptive and inferential statistical analysis of datasets.
• Analysis Frameworks - Provides a collection of algorithms for computing descriptive statistics, hypothesis tests, and probability distributions.
• Data Manipulation - Ships an engine for slicing, joining, pivoting, and filtering tabular datasets with parallel execution.
• C++ Implementations - Provides a C++ library for managing heterogeneous tabular data in contiguous memory with indexed columns.
• Column Data Appending - Adds a single value or range of values to the end of an existing named column.
• Conditional Data Filters - Generates boolean vectors based on whether column elements satisfy user-defined predicates.
• Custom Column Functions - Applies custom user-defined functions across multiple columns of a tabular dataset.
• Data Concatenations - Appends rows from one data structure to another while handling missing columns via filling or filtering.
• Data Joins - Implements general purpose merging of data structures based on shared keys or indices.
• Data Manipulation Frameworks - Implements a comprehensive framework for slicing, joining, merging, grouping, sorting, and filtering structured tabular data.
• Data Serialization - Reconstructs data frames from binary buffers or strings via serialization.
• View Data Assignment - Provides read-only and read-write views of contiguous or disjoined data slices.
• Array View Creation - Implements memory-efficient data views for reading or modifying subsets without copying.
• Record Counting - Calculates the number of rows that satisfy a user-defined predicate across specified columns.
• Positional Data Extraction - Extracts subsets of data using a range of indices or a list of locations.
• File-Based Data Exports - Writes multidimensional data to files using CSV, JSON, and high-performance binary formats.
• Byte Alignment Optimizations - Allocates data on custom byte boundaries to enable SIMD instructions and prevent cache-line sharing.
• Incremental Data Appending - Allows adding new records across multiple named columns to grow the tabular dataset.
• Index Accessors - Accesses the internal container holding row or column labels to inspect the indexing structure.
• Index-Based Data Alignment - Uses a primary metadata column to synchronize rows across multiple heterogeneous columns during joins and merges.
• Index Resampling - Adjusts the sampling interval of the index and fills data when changing frequency.
• Index-Based Extraction - Retrieves subsets of data and indices based on specified ranges or location lists.
• Data Column Extraction - Creates new data structures containing only specified lists of columns.
• Extremum Value Detectors - Calculates the maximum or minimum values of a specific column while handling missing data.
• Missing Data Imputation - Fills gaps in datasets using interpolation or defined policies to ensure data continuity.
• Missing Value Imputation - Replaces missing values using static substitutes, forward/backward fills, or interpolation.
• Tabular Data Preprocessing - Implements tools for handling missing values, removing outliers, and normalizing continuous columns in structured data.
• Tabular Data Sorting - Implements efficient row ordering based on specified columns in ascending or descending directions.
• Time Series Analysis - Processes temporal data using smoothing, detrending, and seasonality detection to identify cycles and trends.
• Time Series Analysis Toolkits - Offers a toolkit for decomposing temporal trends and identifying seasonality in scalar and multidimensional datasets.
• Sequential Time Index Appenders - Appends values to the primary index column to extend the dataset's identifier sequence.
• Unique Value Counting - Identifies all unique entries in a column and calculates the frequency of each occurrence.
• Correlation Coefficient Calculators - Calculates mathematical relationships and statistical metrics such as Pearson correlation and rolling exponentially weighted correlations.
• Signal Processing - Transforms data into the frequency domain using Fast Fourier Transforms to analyze periodicity and remove noise.
• Descriptive Statistics Summaries - Computes a bundle of summary metrics including mean, standard deviation, and quantiles for distribution analysis.
• Multivariate Analysis - Provides dimensionality reduction, clustering, and matrix factorization to uncover structures in large datasets.
• Statistical Analysis Libraries - Provides a comprehensive toolset for calculating descriptive statistics and correlations across multidimensional datasets.
• Statistical Moment Calculation - Computes essential statistical moments including mean, variance, standard deviation, skew, and kurtosis using a visitor pattern.
• Parallel Computation Thread Pools - Distributes data processing tasks across a thread pool to accelerate the analysis of large datasets.
• Memory Alignment Utilities - Specifies byte alignment boundaries and custom allocators to optimize hardware-level vector instruction performance.
• Affinity Propagation Clustering - Divides a named column into clusters using the Affinity Propagation algorithm.
• Hampel Filter Pipelines - Implements the Hampel filter for removing outliers using a sliding window and median absolute deviation.
• C++ Machine Learning Libraries - Provides a high-performance C++ framework for dimensionality reduction, clustering, and vector similarity analysis.
• BIRCH Clustering - Divides columns into clusters using the BIRCH algorithm and K-Means to group similar points.
• Spectral Clustering - Divides scalar or multidimensional data into clusters using a spectral clustering algorithm.
• Density-Based Clustering - Groups data into clusters based on local density using the Mean-Shift algorithm.
• Cluster-Based Outlier Detection - Detects anomalies by measuring local density deviations relative to k-nearest neighbors.
• Independent Component Analysis - Decomposes multivariate datasets into additive independent subcomponents to identify hidden structures.
• K-Means Clustering - Divides named columns into a specified number of clusters using centroid-based partitioning.
• k-Nearest Neighbors Outlier Detection - Identifies outlier data points by measuring distance to the k-th nearest neighbor using kd-trees.
• Outlier Detection - Identifies anomalies in time-series data using sliding windows and absolute deviation.
• Linear Regression - Provides statistical modeling of linear relationships between two columns, including slope and intercept calculations.
• Cumulative Count Calculators - Produces a vector of running counts of non-missing data points across a column.
• Cumulative Extremum Calculators - Calculates the running maximum or minimum of a data column and returns the results as a vector.
• Cumulative Product Calculations - Computes a vector of running products for a given column with missing-value handling.
• Cumulative Sums - Computes a vector of running sums for a data column with options to preserve missing values.
• Range-Based Truncation - Removes rows from the index and columns that fall outside a specified range.
• Training Data Outlier Removers - Allows removing outlier rows based on the Inter-Quartile Range statistical method.
• FFT Outlier Removers - Detects and removes anomalous data points using Fast Fourier Transform analysis.
• Vector Similarity Search - Calculates mathematical closeness between vectors using metrics like Euclidean distance and Cosine similarity.
• Time and Date - Handles date and time data with nanosecond precision and support for multiple time zones.
• Anomaly Detection - Identifies unusual data points using statistical methods and replaces them via fill policies.
• Anomaly Detection Algorithms - Flags outlier values that exceed a defined number of standard deviations from the mean.
• Binary Serialization - Implements tools for converting datasets into compact binary formats for efficient storage and transmission.
• Categorical Encodings - Transforms categorical columns into binary numerical indicator columns using one-hot encoding.
• Column Data Merging - Merges contents of the same named column from multiple dataframes using a custom functor for logic.
• Multi-Column Processing - Passes references of the index and multiple named columns to a visitor functor for complex calculations.
• Column Value Aggregations - Provides column-based aggregation to compute total sums while optionally ignoring missing data.
• Column Value Enumerators - Retrieves a list of all distinct values from a specific column while preserving order.
• Dataset Comparators - Identifies differing values between two data structures with identical indices to produce a divergent set.
• Predicate-Based Joins - Supports merging two data structures using user-defined predicate functions to determine which rows to include.
• Numerical Scaling - Scales data using Z-score, min-max, or Euclidean normalization to standardize values across a dataset.
• String Converters - Provides functions to cast tabular data and indices into string representations for storage or transmission.
• Transformation Chains - Sequences modular functions into a transformation pipeline to process datasets incrementally.
• Mutable Data Views - Provides lightweight read-only or mutable views of data slices to enable subset manipulation without copying underlying memory.
• Upsert by Key - Modifies values in a dataset by matching indices from a source container, effectively performing an upsert.
• File-Based Data Import - Provides capabilities for loading heterogeneous data from CSV, JSON, and binary files into the data structure.
• Generated Columns - Creates new columns by computing values from existing data using a visitor functor.
• Group-By Aggregations - Aggregates data into a new structure based on equal values in one or more columns.
• Quantile-Based Row Filters - Extracts rows that fall above or below a specific quantile of a named column.
• Heterogeneous Data Loading - Populates the data structure with indices and columns of varying types by moving external vectors.
• K-Nearest Neighbor Retrieval - Implements algorithms to identify the k most similar data points using configurable distance functions.
• Long-to-Wide Reshaping - Provides the ability to rotate long-format data into wide matrix-like structures by converting column values into headers.
• Matrix Transposition - Provides the ability to swap rows and columns for datasets with a single data type.
• Missing Data Removal - Filters out rows containing missing values based on customizable thresholds.
• Policy-Driven Transformations - Employs configurable strategy objects to standardize how missing values, interpolation, and ranking logic are handled.
• Range-Based Column Filters - Identifies elements in a column that fall between two bound values and returns a boolean vector.
• Rolling Linear Regressions - Computes rolling means, slopes, and intercepts over specified periods to analyze temporal trends.
• Row Deletions - Supports removing data rows within specified index ranges to maintain dataset consistency.
• Date-Based Filters - Selects rows based on specific days of the month using a date-time index.
• String Pattern Filters - Filters column data by checking if items start or end with a specific string pattern.
• Clusterings - Groups time-series sequences into clusters based on pattern similarity using normalized cross-correlation.
• Time-Series Statistical Profiling - Evaluates time-series stationarity using KPSS and Augmented Dickey-Fuller tests to identify statistical properties.
• Time Bucketing - Aggregates data and indices into fixed intervals based on distance, count, or time.
• Statistical Bucketing - Aggregates data into fixed-size intervals and calculates summary statistics for each bucket.
• Time-Series Index Generators - Generates a vector of timestamps between two dates based on a specified frequency.
• Time Series Decomposition - Separates observed time series data into trend, seasonal, and residual components.
• Time Series Transformations - Provides tools for removing trends and seasonality from temporal sequences to prepare them for analysis.
• Vector Magnitude Calculators - Computes dot products, magnitudes, and Euclidean or Manhattan distances to determine vector similarity.
• SIMD-Aligned Data Structures - Allocates data on specific byte boundaries to support SIMD hardware acceleration and prevent cache-line sharing.
• Visitor-Based Algorithm Dispatch - Decouples data processing logic from storage by using visitor objects that traverse data points or sequences.
• Wide-to-Long Reshaping - Implements unpivoting of wide-format data into long-format by melting multiple value columns into a single pair.
• Window Functions - Executes custom operations over a sliding window of data across a column to produce sequenced results.
• Custom Analytical Algorithms - Enables the creation of custom visitor objects that process data points individually or as sequences to extend analysis.
• Autocorrelation Calculators - Determines the partial correlation of a stationary time series with its lagged values.
• Average Deviation Calculators - Computes mean or median absolute deviations around a center point to measure data dispersion.
• Standard Deviation Calculators - Implements numerically stable standard deviation and mean calculations with bias correction options.
• Autocorrelation Analysis - Computes the correlation of a data column with its own lagged values.
• Cross-Correlation Analysis - Computes a series of correlations between two time-series by shifting them across lag periods.
• Covariance Calculators - Calculates the variance-covariance or correlation matrix for specified columns with z-score normalization.
• Fourier Transforms - Implements fast Fourier transforms to convert signals between time and frequency domains for periodicity analysis.
• Fourier-Series Seasonality - Identifies repeating patterns by detrending data and applying Fast Fourier Transform to find dominant frequencies.
• Data Ranking Utilities - Assigns ranks to items within a sortable data vector using configurable tie-breaking strategies.
• Discrete Difference Calculations - Computes the discrete difference between shifted data points in a column.
• Distribution Comparison Tests - Calculates the Kolmogorov-Smirnov test statistic to determine if two samples share a distribution.
• Dynamic Time Warping - Calculates the distance between temporal sequences using Dynamic Time Warping to align signals.
• Exponential Moving Averages - Implements zero-lag moving averages based on de-lagged exponential moving averages.
• Generic Mean Calculations - Computes arithmetic, geometric, harmonic, quadratic, and weighted averages across data while optionally ignoring missing values.
• Functor-Based Array Mapping - Uses visitor functors to execute custom logic across named columns and indices sequentially or asynchronously.
• Goodness-of-Fit Tests - Assesses how well a sample dataset fits a theoretical normal distribution using the Cramer-von Mises test.
• T-Test Mean Comparisons - Calculates t-statistics and degrees of freedom to compare means between paired or unpaired samples.
• High-Performance and Parallel Computing - Provides a high-performance execution model that distributes computationally intensive data tasks across a thread pool.
• Inter-Quartile Range Outlier Detectors - Identifies anomalies by calculating the Inter-Quartile Range and returning indices of out-of-bounds points.
• Principal Component Analysis - Implements principal component analysis to reduce dimensionality by extracting structural patterns from matrices.
• Matrix Decompositions - Provides singular value decomposition to factorize matrices into singular vectors and values.
• Median Calculators - Finds the median value of a data column in linear time using a selection-based approach.
• Exponentially Weighted Statistics - Computes moving statistics by applying a decay factor that prioritizes recent observations.
• Statistical Significance Testing - Performs Chi-Squared tests to determine if differences between observed and expected frequencies are statistically significant.
• Power Transformations - Transforms scalar or multidimensional data into a normal distribution using Box-Cox power transformation formulas.
• Product Calculations - Provides general utilities for calculating the product of all values within a data column.
• Quantile and Percentile Calculators - Calculates dataset quantiles using configurable policies for linear interpolation or midpoint averaging.
• Rank-Based Statistical Tests - Compares two independent groups using rank-based analysis via the Mann-Whitney U test.
• Rate of Change Calculations - Computes the rate of change between a dataset and its moving average for scalar and multidimensional data.
• Population and Sample Variance Calculators - Computes both population and sample variance with options for numerical stability and bias correction.
• Z-Score Calculators - Computes the z-score of a sample column against a population column to measure deviation.
• Standard Error Calculators - Computes the standard error of the mean for scalar or multidimensional data columns.
• Time Series Smoothing - Detrends and smooths time series data using oscillator-type filters to identify cycles in datasets.
• Tabular Duplicate Removers - Implements utilities to identify and eliminate duplicate records from tabular datasets.
• Expanding Window Functions - Executes functions over a data vector using a rolling window that expands based on specified increments.

Star history