yahoo/TensorFlowOnSpark
TensorFlowOnSpark
TensorFlowOnSpark is a distributed framework for running TensorFlow machine learning workloads and model training across Apache Spark clusters. It functions as a cluster computing orchestrator that manages worker processes and resource allocation to scale deep learning tasks across multiple computing nodes.
The platform enables distributed deep learning training and large-scale model inference, allowing users to execute tasks across a cluster of servers to handle datasets that exceed the memory of a single machine. It integrates deep learning workloads with Spark data processing to create end-to-end pipelines for data transformation and model execution.
The system covers distributed data ingestion from remote storage, parallel model development to reduce training time, and the execution of parallel predictions across cluster nodes.
Features
- Spark Integrations - Bridges deep learning workloads with Apache Spark clusters to create end-to-end data processing and model execution pipelines.
- Distributed Deep Learning - Scales deep learning model training across a cluster of servers to process datasets that exceed single-machine memory.
- Resource Orchestration - Uses Apache Spark's resource management to handle worker process lifecycles and node reservations.
- Distributed Inference Engines - Implements a framework for splitting and executing machine learning inference workloads across multiple cluster nodes.
- Multi-Node Inference Scaling - Distributes large-scale model inference across multiple nodes to process massive data volumes in parallel.
- Distributed Training - Scales the training of deep learning models across multiple compute nodes to handle massive datasets.
- Distributed Learning - Implements frameworks for training deep learning models across multiple computing nodes in a parallel fashion.
- Parallel Inference Orchestrators - Provides orchestration logic to distribute model prediction tasks across a cluster to maximize inference throughput.
- Cluster Orchestrators - Coordinates distributed workloads and resource allocation across Spark clusters for machine learning pipelines.
- Parallel Training - Executes model training processes concurrently across a cluster to accelerate the development of large-scale models.
- Distributed Training Accelerators - Scales deep learning workloads across multiple hardware units to reduce total training time on large datasets.
- Model Weight Synchronization - Coordinates the synchronization of neural network weights and training states across distributed workers.
- Training Data Ingestion - Provides mechanisms for importing and preprocessing raw training data from HDFS or Spark for neural network consumption.
- Distributed Ingestion - Implements distributed orchestration for ingesting large-scale datasets from remote filesystems directly into worker nodes.
- Compute Graph Slicing - Implements techniques for partitioning the TensorFlow compute graph across multiple distributed nodes for parallel execution.
- Distributed Node Lifecycles - Manages the deployment and runtime supervision of worker processes across a distributed server environment.
- Frameworks and Libraries - Distributed training support for Apache Spark clusters.
