# facebookresearch/flashlight

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/facebookresearch-flashlight).**

5,443 stars · 500 forks · C++ · MIT

## Links

- GitHub: https://github.com/facebookresearch/flashlight
- Homepage: https://fl.readthedocs.io/en/latest/
- awesome-repositories: https://awesome-repositories.com/repository/facebookresearch-flashlight.md

## Description

Flashlight is a C++ machine learning library and deep learning framework designed for building and training neural networks. It functions as a tensor manipulation library and an automatic differentiation engine that tracks operations to calculate gradients via backpropagation for model optimization.

The project is distinguished by its role as a distributed training framework, utilizing all-reduce gradient synchronization and distributed environments to scale machine learning workloads across multiple nodes and devices. It features a backend-agnostic memory interface and RAII-based management to decouple tensor operations from physical hardware.

The framework covers a broad capability surface including the construction of neural network architectures with convolutional, linear, and recurrent layers. It provides extensive utilities for tensor algebra, dataset management and batching, versioned binary serialization for model states, and monitoring tools for tracking training metrics and memory usage.

## Tags

### Artificial Intelligence & ML

- [Distributed Training](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-frameworks/distributed-training.md) — A distributed training framework that synchronizes gradients and parameters across multiple devices to train large-scale models.
- [General Deep Learning Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/general-deep-learning-frameworks.md) — Provides the core primitives and implementation framework for building and training various neural network architectures.
- [Activation Functions](https://awesome-repositories.com/f/artificial-intelligence-ml/activation-functions.md) — Applies non-linear mathematical transformations to tensors to introduce complexity into the model's decision boundaries. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Automatic Differentiation](https://awesome-repositories.com/f/artificial-intelligence-ml/automatic-differentiation.md) — Computes gradients through a computational graph to optimize model parameters via backpropagation.
- [Automatic Differentiation Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/automatic-differentiation-engines.md) — Features a computation engine that tracks tensor operations to calculate gradients via backpropagation.
- [C++ Machine Learning Libraries](https://awesome-repositories.com/f/artificial-intelligence-ml/c-machine-learning-libraries.md) — A standalone C++ library for building and training neural networks using tensors and automatic differentiation.
- [Computational Graph Tracking](https://awesome-repositories.com/f/artificial-intelligence-ml/computational-graph-tracking.md) — Tracks tensor operations in a computational graph to enable backward traversal for gradient computation.
- [Convolutional Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/convolutional-layers.md) — Implements 2D convolutional layers that slide filters over 4D inputs to extract spatial features. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Deep Learning Infrastructure](https://awesome-repositories.com/f/artificial-intelligence-ml/deep-learning-infrastructure.md) — Implements custom memory managers and data loading pipelines optimized for efficient machine learning workloads.
- [Distributed Tensor Synchronization](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-tensor-synchronization.md) — Synchronizes tensor data across participating processes using all-reduce operations to maintain consistency. ([source](https://fl.readthedocs.io/en/latest/distributed.html))
- [Distributed Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-frameworks.md) — Implements a system for scaling machine learning model training across multiple compute nodes and hardware accelerators.
- [Gradient Computation](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation.md) — Implements automatic differentiation tools for calculating function gradients to support model training and optimization. ([source](https://cdn.jsdelivr.net/gh/facebookresearch/flashlight@main/README.md))
- [Distributed Gradient Synchronization](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation/distributed-gradient-synchronization.md) — Implements mechanisms for coordinating gradient updates across multiple compute nodes during distributed training.
- [Large-Scale Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-training-frameworks.md) — Provides infrastructure for scaling neural network training across multiple devices and nodes using a reducer framework. ([source](https://fl.readthedocs.io/en/latest/))
- [Custom Neural Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning-implementations/custom-neural-architectures.md) — Allows the creation of tailored neural network structures by extending base modules and overriding the forward pass. ([source](https://fl.readthedocs.io/en/latest/extend.html))
- [Linear Transformations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/algorithms/linear-regression-implementations/linear-mixing-layers/linear-transformations.md) — Multiplies input tensors by learnable weight matrices and adds bias terms for vector space mapping. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Normalization Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/neural-network-layers/normalization-layers.md) — Provides normalization layers that standardize tensor values across batches to stabilize training and improve convergence. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Recurrent Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/neural-network-layers/recurrent-layers.md) — Implements recurrent layers, including unidirectional and bidirectional sequences, to process temporal data or text. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Neural Network Construction](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-network-construction.md) — Provides the process of designing and building deep learning architectures by stacking convolutional, linear, and pooling layers. ([source](https://fl.readthedocs.io/en/latest/mnist.html))
- [Neural Network Modules](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-network-modules.md) — Ships encapsulated model components for managing parameters and assembling structural pipelines for data processing. ([source](https://fl.readthedocs.io/en/latest/))
- [Tensor Indexing](https://awesome-repositories.com/f/artificial-intelligence-ml/tensor-indexing.md) — Implements advanced mechanisms for slicing and extracting subtensors using index literals and strided ranges. ([source](https://fl.readthedocs.io/en/latest/tensor.html))
- [Tensor Initialization](https://awesome-repositories.com/f/artificial-intelligence-ml/tensor-initialization.md) — Generates tensors with specific patterns including identity matrices, sequences, and constant-filled arrays. ([source](https://fl.readthedocs.io/en/latest/tensor.html))
- [Tensor Libraries](https://awesome-repositories.com/f/artificial-intelligence-ml/tensor-libraries.md) — Provides a library for creating and manipulating multi-dimensional arrays for mathematical and neural network computations.
- [Tensor Management Utilities](https://awesome-repositories.com/f/artificial-intelligence-ml/tensor-management-utilities.md) — Provides core utilities for creating and managing multi-dimensional tensors with configurable shapes and storage backends. ([source](https://fl.readthedocs.io/en/latest/tensor.html))
- [Tensor Reshaping](https://awesome-repositories.com/f/artificial-intelligence-ml/tensor-reshaping.md) — Implements operations for modifying dimensional layout and reordering axes without altering the underlying data. ([source](https://fl.readthedocs.io/en/latest/tensor.html))
- [Dataset Batch Loading](https://awesome-repositories.com/f/artificial-intelligence-ml/dataset-batch-loading.md) — Loads and batches datasets in fixed-size groups to optimize training stability and performance. ([source](https://fl.readthedocs.io/en/latest/))
- [Dropout Regularization](https://awesome-repositories.com/f/artificial-intelligence-ml/dropout-regularization.md) — Implements dropout regularization by randomly zeroing tensor elements to prevent overfitting. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Embedding Lookups](https://awesome-repositories.com/f/artificial-intelligence-ml/gpu-kernel-implementations/embedding-lookups.md) — Retrieves high-dimensional vectors from a learnable dictionary to represent categorical data. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Reduction Algorithms](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation/distributed-gradient-synchronization/reduction-algorithms.md) — Implements custom reduction algorithms to synchronize gradients across distributed devices. ([source](https://fl.readthedocs.io/en/latest/dist.html))
- [Loss Functions](https://awesome-repositories.com/f/artificial-intelligence-ml/loss-functions.md) — Calculates the difference between predicted and target values using specific error metrics to guide optimization. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Residual Block Composers](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/neural-network-layers/convolution-layers/convolutional-block-composers/residual-block-composers.md) — Provides utilities for creating residual blocks with customizable skip connections and scaling factors. ([source](https://fl.readthedocs.io/en/latest/contrib.html))
- [Mixed-Precision Computing](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/distributed-and-accelerated-compute/training-acceleration-tools/mixed-precision-training/mixed-precision-computing.md) — Balances execution speed and numerical stability by supporting both full and mixed precision numerical formats. ([source](https://fl.readthedocs.io/en/latest/common.html))
- [Tensor Memory Management](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-optimization-and-inference/hardware-and-acceleration/tensor-computing-libraries/tensor-memory-management.md) — Uses RAII wrappers to lock and unlock device pointers, preventing premature memory freeing for tensors. ([source](https://fl.readthedocs.io/en/latest/common.html))
- [Modular Layer Compositions](https://awesome-repositories.com/f/artificial-intelligence-ml/model-composition-architectures/hybrid-layer-compositions/modular-layer-compositions.md) — Assembles neural networks by nesting mutable parameter modules within sequential or residual structural pipelines.
- [Model Performance Evaluators](https://awesome-repositories.com/f/artificial-intelligence-ml/model-performance-evaluators.md) — Calculates loss and error metrics on validation sets to quantify model accuracy. ([source](https://fl.readthedocs.io/en/latest/mnist.html))
- [Model Serializers](https://awesome-repositories.com/f/artificial-intelligence-ml/model-serializers.md) — Implements utilities for saving and loading model weights and optimizer states to persistent storage. ([source](https://fl.readthedocs.io/en/latest/))
- [Sequential Compositions](https://awesome-repositories.com/f/artificial-intelligence-ml/module-composition/sequential-compositions.md) — Groups multiple modules into a strict linear order for sequential data flow during processing. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Parameter Optimizers](https://awesome-repositories.com/f/artificial-intelligence-ml/parameter-optimizers.md) — Updates model weights using first-order optimization algorithms to minimize loss during training. ([source](https://fl.readthedocs.io/en/latest/))
- [Weight Normalizations](https://awesome-repositories.com/f/artificial-intelligence-ml/weight-normalizations.md) — Reparameterizes weights in linear or convolutional modules to decouple magnitude from direction and accelerate training. ([source](https://fl.readthedocs.io/en/latest/modules.html))

### Data & Databases

- [Array and Tensor Manipulation](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation/array-tensor-manipulation.md) — Provides a comprehensive suite of mathematical operations and indexing for multi-dimensional array manipulation. ([source](https://fl.readthedocs.io/en/latest/))
- [Parameter Synchronization](https://awesome-repositories.com/f/data-databases/real-time-data-synchronization/parameter-synchronization.md) — Aligns initial network weights across all distributed processes to ensure consistent starting states for every worker. ([source](https://fl.readthedocs.io/en/latest/distributed.html))
- [Sample Batching](https://awesome-repositories.com/f/data-databases/data-aggregation-pipelines/batch-aggregators/sample-batching.md) — Packs individual samples into fixed or dynamic sizes for efficient processing. ([source](https://fl.readthedocs.io/en/latest/data_loading.html))
- [Tensor Serialization Utilities](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation/array-tensor-manipulation/tensor-transformations/tensor-serialization-utilities.md) — Ships functions for saving and loading tensors and neural network modules to binary files. ([source](https://fl.readthedocs.io/en/latest/serial.html))
- [Training Memory Optimizers](https://awesome-repositories.com/f/data-databases/memory-optimization-strategies/training-memory-optimizers.md) — Manages the clearing or retention of intermediate tensors to reduce peak memory consumption during training. ([source](https://fl.readthedocs.io/en/latest/autograd.html))

### Scientific & Mathematical Computing

- [Graph Construction Engines](https://awesome-repositories.com/f/scientific-mathematical-computing/data-modeling-processing/computational-graphs/graph-construction-engines.md) — Implements a mechanism for defining and building symbolic computational graphs by chaining neural network modules and layers. ([source](https://cdn.jsdelivr.net/gh/facebookresearch/flashlight@main/README.md))
- [Element-wise Array Operations](https://awesome-repositories.com/f/scientific-mathematical-computing/element-wise-array-operations.md) — Executes high-performance mathematical transformations, such as exponentials and logarithms, across every element of a tensor. ([source](https://fl.readthedocs.io/en/latest/functions.html))
- [Linear Algebra Routines](https://awesome-repositories.com/f/scientific-mathematical-computing/linear-algebra-routines.md) — Provides multi-dimensional array management and linear algebra routines for data processing and model execution.
- [Tensor Arithmetic](https://awesome-repositories.com/f/scientific-mathematical-computing/tensor-arithmetic.md) — Implements element-wise addition, subtraction, multiplication, and division between tensors and scalars. ([source](https://fl.readthedocs.io/en/latest/functions.html))
- [Automatic Differentiation](https://awesome-repositories.com/f/scientific-mathematical-computing/numerical-mathematical-foundations/automatic-differentiation.md) — Executes mathematical functions on tensor wrappers that track dependencies for automatic differentiation. ([source](https://fl.readthedocs.io/en/latest/autograd.html))

### Software Engineering & Architecture

- [Distributed Training Coordination](https://awesome-repositories.com/f/software-engineering-architecture/distributed-coordination-systems/distributed-training-coordination.md) — Sets up a communication backend to coordinate data and parameter exchange across distributed training processes. ([source](https://fl.readthedocs.io/en/latest/distributed.html))
- [Tensor Comparison Operators](https://awesome-repositories.com/f/software-engineering-architecture/logical-comparison-operators/tensor-comparison-operators.md) — Provides logical and relational operators to evaluate equality and magnitude between tensor elements. ([source](https://fl.readthedocs.io/en/latest/functions.html))
- [Tensor Rearrangements](https://awesome-repositories.com/f/software-engineering-architecture/sorting-algorithms/array-rearrangement-algorithms/tensor-rearrangements.md) — Provides operations to rearrange tensor axes and change data layout while preserving memory continuity. ([source](https://fl.readthedocs.io/en/latest/modules.html))
- [Custom Kernel Accelerators](https://awesome-repositories.com/f/software-engineering-architecture/performance-reliability/performance-optimization/computational-efficiency/custom-kernel-accelerators.md) — Allows high-performance operations via direct memory pointer access for external accelerator libraries.
- [RAII Resource Management](https://awesome-repositories.com/f/software-engineering-architecture/raii-resource-management.md) — Utilizes RAII wrappers to lock and unlock device pointers to prevent premature memory freeing.

### DevOps & Infrastructure

- [Serialization Versioning](https://awesome-repositories.com/f/devops-infrastructure/version-control-management/version-control/serialization-versioning.md) — Persists model states and tensors to disk using version numbers to ensure backward compatibility.

### Operating Systems & Systems Programming

- [Asynchronous Task Execution](https://awesome-repositories.com/f/operating-systems-systems-programming/asynchronous-task-execution.md) — Distributes computational work across multiple CPU cores using a thread pool for simultaneous operation processing. ([source](https://fl.readthedocs.io/en/latest/common.html))
- [Hardware-Agnostic Memory Interfaces](https://awesome-repositories.com/f/operating-systems-systems-programming/hardware-agnostic-memory-interfaces.md) — Provides a C++ interface that decouples tensor operations from physical hardware for flexible device memory management.
- [Custom Memory Allocators](https://awesome-repositories.com/f/operating-systems-systems-programming/kernel-core-internals/process-and-memory-management/memory-management/allocation-strategies/dynamic-memory-allocation/custom-memory-allocators.md) — Provides a C++ interface to define custom memory allocation and management schemes. ([source](https://fl.readthedocs.io/en/latest/memory.html))

### System Administration & Monitoring

- [Model Performance Tracking](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/observability-platforms/metric-performance-monitors/model-performance-tracking.md) — Calculates error metrics and edit distance to track model convergence and weight quality. ([source](https://fl.readthedocs.io/en/latest/meters.html))

### Part of an Awesome List

- [Machine Learning](https://awesome-repositories.com/f/awesome-lists/ai/machine-learning.md) — Fast, flexible machine learning library.