awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
Back to yahoo/tensorflowonspark

Projects sharing features with TensorFlowOnSpark

30 open-source projects similar to yahoo/tensorflowonspark, ranked by shared indexed features. Tags may describe platforms or build tools rather than the same primary purpose. Check each project’s use case, license, and deployment requirements before treating it as a replacement.

  • azure/mmlsparkAzure avatar

    Azure/mmlspark

    5,228View on GitHub↗

    Mmlspark is a distributed framework for executing machine learning models, data transformations, and AI service integrations across Apache Spark clusters. It functions as a distributed machine learning library and pipeline orchestrator, allowing users to integrate pre-trained cognitive services and custom models into large-scale batch and streaming workflows. The project is distinguished by its ability to incorporate external AI services and web APIs directly into big data pipelines for text and vision analysis. It provides a scalable model training framework that coordinates gradient boostin

    Scala
    View on GitHub↗5,228
  • exo-explore/exoexo-explore avatar

    exo-explore/exo

    48,237View on GitHub↗

    Exo is a distributed inference engine designed to run machine learning models across local hardware. It functions as a network orchestration layer that automatically discovers available devices to form a unified computing cluster, allowing users to scale artificial intelligence workloads by distributing computational tasks across multiple machines. The platform distinguishes itself through its ability to manage the entire lifecycle of local models while providing a standardized gateway for external applications. By translating local model outputs into industry-standard formats, it enables exi

    Python
    View on GitHub↗48,237
  • microsoft/synapsemlmicrosoft avatar

    microsoft/SynapseML

    5,230View on GitHub↗

    SynapseML is an Apache Spark machine learning library designed for building and scaling machine learning workflows and data pipelines across distributed clusters. It serves as a distributed machine learning pipeline framework and a distributed inference engine for executing hardware-accelerated predictions and deep learning tasks on large-scale datasets. The project functions as a cloud AI integration layer, allowing users to apply pretrained artificial intelligence services for text, vision, and speech within distributed pipelines. It also includes a dedicated suite of tools for distributed

    Scalaaiapache-sparkazure
    View on GitHub↗5,230

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Find more with AI search
  • h2oai/h2o-3h2oai avatar

    h2oai/h2o-3

    7,493View on GitHub↗

    h2o-3 is a distributed machine learning platform and automated machine learning framework designed for training and deploying predictive models using distributed in-memory computing. It functions as a deep learning framework and a distributed model scoring engine, capable of operating as a Kubernetes ML cluster to process large datasets in parallel. The platform distinguishes itself through automated machine learning capabilities that automatically select the best algorithms and hyperparameters to optimize model performance. It provides specialized deep learning toolkits for tasks including i

    Jupyter Notebookautomlbig-datadata-science
    View on GitHub↗7,493
  • angel-ml/angelAngel-ML avatar

    Angel-ML/angel

    6,795View on GitHub↗

    Angel is a distributed machine learning framework and graph computation engine designed to train predictive models and execute algorithms across a cluster of servers. It functions as a distributed parameter server that synchronizes model weights and gradients across multiple machines to handle massive datasets. The system provides a production environment for model inference deployment to provide real-time predictions for end users. It integrates with Spark to run machine learning workflows and data processing pipelines through a compatible interface. The framework covers distributed graph c

    Javahigh-dimensionalmachine-learningmodel
    View on GitHub↗6,795
  • horovod/horovodhorovod avatar

    horovod/horovod

    14,686View on GitHub↗

    Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

    Python
    View on GitHub↗14,686
  • nvidia/isaac-gr00tNVIDIA avatar

    NVIDIA/Isaac-GR00T

    6,222View on GitHub↗
    Jupyter Notebook
    View on GitHub↗6,222
  • chainer/chainerchainer avatar

    chainer/chainer

    5,918View on GitHub↗

    Chainer is an open-source deep learning framework built around define-by-run automatic differentiation, where computation graphs are constructed dynamically during forward execution. This imperative approach allows networks to be built using standard Python control flow, with gradients computed automatically through reverse-mode differentiation on the dynamically recorded graph. The framework supports GPU acceleration through a NumPy-compatible array backend with CUDA and cuDNN support, and provides a pluggable device abstraction that lets users switch between CPU and GPU computation without c

    Python
    View on GitHub↗5,918
  • facebookresearch/pytextfacebookresearch avatar

    facebookresearch/pytext

    6,298View on GitHub↗

    PyText is an extensible PyTorch-based framework for building, training, and deploying custom natural language processing models, including text classifiers, sequence taggers, and intent-slot predictors. It provides a modular toolkit that allows developers to assemble these models using pluggable registries for model architectures, data formats, and tensorizers, all configurable through YAML files without requiring code changes. The framework distinguishes itself through its comprehensive support for the full NLP model lifecycle, from training to production inference. It includes pre-built neu

    Python
    View on GitHub↗6,298
  • internlm/xtunerInternLM avatar

    InternLM/xtuner

    5,222View on GitHub↗

    xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised fine-tuning, and the optimization of vision-language multimodal models. It serves as a distributed training accelerator and a specialized framework for scaling Mixture-of-Experts models and aligning model behavior through reinforcement learning from human feedback. The project distinguishes itself through advanced memory and compute optimizations, such as sequence parallelism for ultra-long context windows and interleaved pipeline parallelism to reduce GPU idle time. It provide

    Pythonagentdeepseek-v3gpt-oss
    View on GitHub↗5,222
  • huggingface/acceleratehuggingface avatar

    huggingface/accelerate

    9,725View on GitHub↗

    Accelerate is a PyTorch distributed training library that abstracts the boilerplate required to run models across multiple GPUs, TPUs, and CPUs. It functions as a deep learning model scaler and distributed hardware orchestrator, allowing the same training script to run on different hardware backends without modifying the core logic. The project provides a distributed training command line interface for configuring compute environments and launching jobs across single or multi-node clusters. It includes a mixed precision training framework to implement FP16 and BF16 precision, reducing memory

    Python
    View on GitHub↗9,725
  • ai4finance-foundation/elegantrlAI4Finance-Foundation avatar

    AI4Finance-Foundation/ElegantRL

    4,342View on GitHub↗

    ElegantRL is a deep reinforcement learning framework and quantitative trading platform designed for automating financial decision making. It provides a system for designing and training agents using massively parallel GPU execution and includes a coordination layer for multi-agent reinforcement learning. Additionally, it features a GPU-based solver for NP-complete and nonconvex mathematical optimization problems. The platform distinguishes itself through GPU-accelerated environments that simulate thousands of parallel market interactions on a single device to accelerate data collection. It in

    Pythona2cbipedalwalkerhardcoreddpg
    View on GitHub↗4,342
  • flashlight/flashlightflashlight avatar

    flashlight/flashlight

    5,479View on GitHub↗

    Flashlight is a standalone C++ machine learning library and tensor library used for building and training neural networks. It functions as a comprehensive neural network framework and automatic differentiation engine, providing the tools to construct computation graphs and calculate gradients via backpropagation. The project serves as a distributed training framework, utilizing all-reduce operations to synchronize gradients and parameters across multiple compute nodes and devices. It distinguishes itself through deep integration of high-performance tensor manipulation, native device memory in

    C++
    View on GitHub↗5,479
  • wangshusen/deeplearningwangshusen avatar

    wangshusen/DeepLearning

    4,229View on GitHub↗

    This is an educational repository providing implementations and tutorials for deep learning, neural network architectures, and machine learning fundamentals. It serves as a reference for building multilayer perceptrons, convolutional networks, and recurrent networks using backpropagation and gradient descent. The project includes specialized frameworks for generative modeling via autoencoders and generative adversarial networks, as well as a toolkit for reinforcement learning that implements value-based, policy-based, and actor-critic methods. It also provides practical references for transfo

    TeX
    View on GitHub↗4,229
  • qwenlm/qwen-7bQwenLM avatar

    QwenLM/Qwen-7B

    21,343View on GitHub↗

    Qwen-7B is a pretrained causal language model designed for natural language generation, text processing, and complex reasoning tasks. It is available as an instruction-tuned model optimized for conversational interactions and a tool-use model capable of executing function calls and interacting with external APIs. The project provides a quantized version of the model to reduce GPU memory usage and supports the development of autonomous agents that can execute code and perform functions to complete complex goals. The system covers a wide range of capabilities including model fine-tuning throug

    Python
    View on GitHub↗21,343
  • tiiny-ai/powerinferTiiny-AI avatar

    Tiiny-AI/PowerInfer

    9,873View on GitHub↗

    PowerInfer is a high-performance local large language model inference engine and sparse inference framework. It provides a runtime for executing models on consumer-grade hardware, utilizing a GPU acceleration backend to optimize tensor operations for graphics processors. The system distinguishes itself through a sparse inference framework that increases generation speed by skipping computations based on activation sparsity in model weights. It includes a GGUF model converter for transforming weights and metadata into a unified binary format, as well as an OpenAI API compatible server for inte

    C++large-language-modelsllamallm
    View on GitHub↗9,873
  • uber/horovoduber avatar

    uber/horovod

    14,686View on GitHub↗

    Horovod is a distributed deep learning framework designed to scale machine learning training across multiple GPUs and nodes. It functions as an orchestrator for multi-GPU scaling and a tool for distributed gradient averaging, allowing users to increase compute capacity without rewriting core model logic. The project provides a consistent communication interface that supports multi-framework model distribution across TensorFlow, PyTorch, Keras, and MXNet. It leverages an MPI distributed training library to synchronize gradients across processes using collective communication operations. The s

    Python
    View on GitHub↗14,686
  • paddlepaddle/servingPaddlePaddle avatar

    PaddlePaddle/Serving

    921View on GitHub↗

    Serving is a high-performance framework designed for deploying and scaling machine learning models as production services. It functions as a distributed inference engine that enables the execution of complex data processing workflows by chaining multiple models into directed acyclic graphs. The platform distinguishes itself through its ability to manage the entire production model lifecycle, allowing for hot-swappable versioning that updates services without downtime. It supports horizontal scaling through distributed model sharding and optimizes high-dimensional data retrieval via specialize

    C++dagdeep-learningdocker
    View on GitHub↗921
  • gpustack/gpustackgpustack avatar

    gpustack/gpustack

    5,989View on GitHub↗

    gpustack is a GPU cluster management platform and LLM inference orchestrator. It functions as a centralized system for pooling and orchestrating graphics processing units across local servers and cloud environments, serving as a heterogeneous compute manager for diverse hardware and software configurations. The system provides a secure AI model deployment gateway that serves models as scalable services using key-based authentication. It includes a GPU resource scheduler that balances workloads across accelerators and coordinates multiple inference engines to map specific AI models to compatib

    Python
    View on GitHub↗5,989
  • rapidsai/cumlrapidsai avatar

    rapidsai/cuml

    5,209View on GitHub↗

    cuml is a GPU-accelerated machine learning library and framework that uses CUDA to accelerate tabular data preprocessing and model execution. It provides a suite of tools for training and deploying classification, regression, and clustering models on NVIDIA GPUs and GPU clusters. The library is designed for scalability, offering a distributed GPU machine learning environment that can spread computation and data across multiple hardware accelerators and nodes to handle datasets exceeding single-device memory. It mirrors standard estimator interfaces to allow the replacement of CPU-based models

    Python
    View on GitHub↗5,209
  • bigscience-workshop/petalsbigscience-workshop avatar

    bigscience-workshop/petals

    10,698View on GitHub↗

    Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted

    Python
    View on GitHub↗10,698
  • facebookresearch/pythiafacebookresearch avatar

    facebookresearch/pythia

    5,635View on GitHub↗

    Pythia is a multimodal research framework and distributed training system designed for building, training, and evaluating large models that combine visual and linguistic data. It provides a modular environment for developing vision-language models, focusing on the integration of image and text inputs into shared feature representations. The framework utilizes a modular architecture that decouples model building blocks into interchangeable components, allowing for flexible configuration of vision and language modules. It includes a benchmark suite for executing reference models against standar

    Python
    View on GitHub↗5,635
  • alibaba/x-deeplearningalibaba avatar

    alibaba/x-deeplearning

    4,299View on GitHub↗

    This project is a distributed machine learning platform and sparse deep learning framework designed for training and serving models with high-dimensional sparse data. It functions as an online model serving infrastructure and recommendation system engine, enabling real-time item retrieval and scoring using deep tree matching and neural networks. The system distinguishes itself through a multi-task learning framework that optimizes multiple objective functions within a shared representation space. It features a specialized online serving infrastructure that supports dynamic model hot-loading a

    PureBasic
    View on GitHub↗4,299
  • lightning-ai/lightninglightning-AI avatar

    lightning-AI/lightning

    31,189View on GitHub↗

    Lightning is a PyTorch training framework and distributed AI training orchestrator designed to decouple core research logic from the engineering boilerplate required for model training. It functions as a deep learning workflow manager that automates the process of pretraining and finetuning models across diverse compute environments. The project distinguishes itself by providing a hardware-agnostic training wrapper, allowing the same model code to execute on CPUs, GPUs, or TPUs without modification. It further manages the scaling of workloads from single devices to multi-node clusters and ser

    Python
    View on GitHub↗31,189
  • baidu/paddlebaidu avatar

    baidu/paddle

    23,959View on GitHub↗

    Paddle is a deep learning framework designed for building, training, and deploying large-scale machine learning models. It incorporates a distributed training engine for optimizing performance across multiple chips and a model inference engine for transforming trained models into production-ready formats for cross-platform execution. The platform features a heterogeneous hardware abstraction and a standardized software stack that allows models to run across diverse hardware architectures through a common interface. It also includes a scientific computing library capable of solving complex dif

    C++
    View on GitHub↗23,959
  • polyaxon/polyaxonpolyaxon avatar

    polyaxon/polyaxon

    3,746View on GitHub↗

    Polyaxon is a Kubernetes-native machine learning orchestration platform and MLOps pipeline orchestrator. It serves as a control plane for managing distributed deep learning workloads, automated machine learning pipelines, and experiment tracking. The platform distinguishes itself through specialized services for distributed training management, including MPI-based coordination for PyTorch and TensorFlow. It provides an automated hyperparameter optimization service utilizing Bayesian, random, and grid search algorithms, alongside managed interactive AI workspaces for launching Jupyter notebook

    MDX
    View on GitHub↗3,746
  • intel/ipex-llmintel avatar

    intel/ipex-llm

    8,836View on GitHub↗

    Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

    Python
    View on GitHub↗8,836
  • paddlepaddle/paddlePaddlePaddle avatar

    PaddlePaddle/Paddle

    24,138View on GitHub↗

    Paddle is a deep learning framework designed for building, training, and deploying neural networks. It provides a platform for constructing models using tensor-based computations and supports both dynamic and static execution graphs to facilitate research and production workflows. The platform functions as a distributed machine learning system, enabling the scaling of training workloads across multiple nodes and hardware clusters. It includes a comprehensive toolkit for model deployment and optimization, allowing users to convert external model formats, compress trained models for resource-co

    C++deep-learningdistributed-trainingefficiency
    View on GitHub↗24,138
  • dllxw/baby-llama2-chineseDLLXW avatar

    DLLXW/baby-llama2-chinese

    2,891View on GitHub↗

    This project is a training pipeline and framework for developing Chinese language models based on the Llama 2 architecture. It functions as a distributed GPU trainer and dataset preprocessing toolkit designed for both the initial pre-training of baseline models and subsequent supervised fine-tuning. The system distinguishes itself through a specialized workflow for Chinese text, incorporating a data curation pipeline that uses similarity hashing for deduplication and a tokenization process that converts raw text into memory-mapped binary files for efficient disk access. It implements a superv

    Python
    View on GitHub↗2,891
  • carperai/trlxcarperai avatar

    carperai/trlx

    4,749View on GitHub↗

    trlx is a reinforcement learning library and training framework designed to align large language models using human feedback. It serves as a distributed trainer and compute orchestrator for scaling high-parameter models across multiple GPUs and nodes. The project provides tools for reinforcement learning from human feedback and model alignment. It implements reward-model-based optimization and proximal policy optimization to refine model behavior based on goal-oriented rewards or human-labeled datasets. The framework covers distributed training strategies, including model parallelism, parame

    Python
    View on GitHub↗4,749