facebookresearch/flashlight

Features

• Distributed Training - A distributed training framework that synchronizes gradients and parameters across multiple devices to train large-scale models.
• General Deep Learning Frameworks - Provides the core primitives and implementation framework for building and training various neural network architectures.
• Activation Functions - Applies non-linear mathematical transformations to tensors to introduce complexity into the model's decision boundaries.
• Automatic Differentiation - Computes gradients through a computational graph to optimize model parameters via backpropagation.
• Automatic Differentiation Engines - Features a computation engine that tracks tensor operations to calculate gradients via backpropagation.
• C++ Machine Learning Libraries - A standalone C++ library for building and training neural networks using tensors and automatic differentiation.
• Computational Graph Tracking - Tracks tensor operations in a computational graph to enable backward traversal for gradient computation.
• Convolutional Layers - Implements 2D convolutional layers that slide filters over 4D inputs to extract spatial features.
• Deep Learning Infrastructure - Implements custom memory managers and data loading pipelines optimized for efficient machine learning workloads.
• Distributed Tensor Synchronization - Synchronizes tensor data across participating processes using all-reduce operations to maintain consistency.
• Distributed Training Frameworks - Implements a system for scaling machine learning model training across multiple compute nodes and hardware accelerators.
• Gradient Computation - Implements automatic differentiation tools for calculating function gradients to support model training and optimization.
• Distributed Gradient Synchronization - Implements mechanisms for coordinating gradient updates across multiple compute nodes during distributed training.
• Large-Scale Training Frameworks - Provides infrastructure for scaling neural network training across multiple devices and nodes using a reducer framework.
• Custom Neural Architectures - Allows the creation of tailored neural network structures by extending base modules and overriding the forward pass.
• Linear Transformations - Multiplies input tensors by learnable weight matrices and adds bias terms for vector space mapping.
• Normalization Layers - Provides normalization layers that standardize tensor values across batches to stabilize training and improve convergence.
• Recurrent Layers - Implements recurrent layers, including unidirectional and bidirectional sequences, to process temporal data or text.
• Neural Network Construction - Provides the process of designing and building deep learning architectures by stacking convolutional, linear, and pooling layers.
• Neural Network Modules - Ships encapsulated model components for managing parameters and assembling structural pipelines for data processing.
• Tensor Indexing - Implements advanced mechanisms for slicing and extracting subtensors using index literals and strided ranges.
• Tensor Initialization - Generates tensors with specific patterns including identity matrices, sequences, and constant-filled arrays.
• Tensor Libraries - Provides a library for creating and manipulating multi-dimensional arrays for mathematical and neural network computations.
• Tensor Management Utilities - Provides core utilities for creating and managing multi-dimensional tensors with configurable shapes and storage backends.
• Tensor Reshaping - Implements operations for modifying dimensional layout and reordering axes without altering the underlying data.
• Array and Tensor Manipulation - Provides a comprehensive suite of mathematical operations and indexing for multi-dimensional array manipulation.
• Parameter Synchronization - Aligns initial network weights across all distributed processes to ensure consistent starting states for every worker.
• Graph Construction Engines - Implements a mechanism for defining and building symbolic computational graphs by chaining neural network modules and layers.
• Element-wise Array Operations - Executes high-performance mathematical transformations, such as exponentials and logarithms, across every element of a tensor.
• Linear Algebra Routines - Provides multi-dimensional array management and linear algebra routines for data processing and model execution.
• Tensor Arithmetic - Implements element-wise addition, subtraction, multiplication, and division between tensors and scalars.
• Distributed Training Coordination - Sets up a communication backend to coordinate data and parameter exchange across distributed training processes.
• Tensor Comparison Operators - Provides logical and relational operators to evaluate equality and magnitude between tensor elements.
• Tensor Rearrangements - Provides operations to rearrange tensor axes and change data layout while preserving memory continuity.
• Dataset Batch Loading - Loads and batches datasets in fixed-size groups to optimize training stability and performance.
• Dropout Regularization - Implements dropout regularization by randomly zeroing tensor elements to prevent overfitting.
• Embedding Lookups - Retrieves high-dimensional vectors from a learnable dictionary to represent categorical data.
• Reduction Algorithms - Implements custom reduction algorithms to synchronize gradients across distributed devices.
• Loss Functions - Calculates the difference between predicted and target values using specific error metrics to guide optimization.
• Residual Block Composers - Provides utilities for creating residual blocks with customizable skip connections and scaling factors.
• Mixed-Precision Computing - Balances execution speed and numerical stability by supporting both full and mixed precision numerical formats.
• Tensor Memory Management - Uses RAII wrappers to lock and unlock device pointers, preventing premature memory freeing for tensors.
• Modular Layer Compositions - Assembles neural networks by nesting mutable parameter modules within sequential or residual structural pipelines.
• Model Performance Evaluators - Calculates loss and error metrics on validation sets to quantify model accuracy.
• Model Serializers - Implements utilities for saving and loading model weights and optimizer states to persistent storage.
• Sequential Compositions - Groups multiple modules into a strict linear order for sequential data flow during processing.
• Parameter Optimizers - Updates model weights using first-order optimization algorithms to minimize loss during training.
• Weight Normalizations - Reparameterizes weights in linear or convolutional modules to decouple magnitude from direction and accelerate training.
• Sample Batching - Packs individual samples into fixed or dynamic sizes for efficient processing.
• Tensor Serialization Utilities - Ships functions for saving and loading tensors and neural network modules to binary files.
• Training Memory Optimizers - Manages the clearing or retention of intermediate tensors to reduce peak memory consumption during training.
• Serialization Versioning - Persists model states and tensors to disk using version numbers to ensure backward compatibility.
• Asynchronous Task Execution - Distributes computational work across multiple CPU cores using a thread pool for simultaneous operation processing.
• Hardware-Agnostic Memory Interfaces - Provides a C++ interface that decouples tensor operations from physical hardware for flexible device memory management.
• Custom Memory Allocators - Provides a C++ interface to define custom memory allocation and management schemes.
• Automatic Differentiation - Executes mathematical functions on tensor wrappers that track dependencies for automatic differentiation.
• Custom Kernel Accelerators - Allows high-performance operations via direct memory pointer access for external accelerator libraries.
• RAII Resource Management - Utilizes RAII wrappers to lock and unlock device pointers to prevent premature memory freeing.
• Model Performance Tracking - Calculates error metrics and edit distance to track model convergence and weight quality.
• Machine Learning - Fast, flexible machine learning library.

Star history