Azure/mmlspark

Features

• Spark Integrations - Wraps machine learning workflows within a Spark execution engine to distribute processing across a compute cluster.
• Apache Spark Pipelines - Integrates machine learning models and data featurization into distributed computing workflows using Apache Spark.
• AI Service Integrations - Provides connectors and interfaces for integrating external artificial intelligence services into distributed data pipelines.
• Distributed Inference Scaling - Executes trained models across clusters to generate predictions on large-scale data with hardware acceleration.
• Distributed ML Pipeline Managers - Orchestrates the integration of pre-trained AI services and custom models into large-scale batch and streaming workflows.
• Distributed Training - Coordinates the training and evaluation of machine learning models across elastically resizable compute clusters.
• Gradient Boosting - Trains gradient boosting models across a distributed cluster to handle large datasets.
• Hardware-Accelerated Inference - Uses GPU and CPU acceleration to execute trained deep learning models across distributed datasets for higher throughput.
• Machine Learning Pipelines - Incorporates machine learning models into batch and streaming workflows for consistent data processing.
• Machine Learning Workflow Libraries - Integrates machine learning models into distributed computing pipelines using a shared API.
• Distributed Training - Scales the training and evaluation of machine learning models across multiple compute nodes.
• Spark MLlib Integrations - Provides a distributed framework for executing machine learning models and transformations across Apache Spark clusters.
• Model Training Frameworks - Offers a toolkit for training and evaluating gradient boosting and classification models across resizable clusters.
• Multivariate Anomaly Detection - Identifies outliers across multiple variables by analyzing inter-correlations and dependencies between data dimensions.
• Scalable Anomaly Detection - Identifies outliers and multivariate anomalies in big data using distributed forest implementations and time series analysis.
• Data Access & Abstraction - Simplifies data experiments by abstracting access to various databases, file systems, and cloud data stores.
• Distributed Text Analytics - Processes massive amounts of unstructured text to extract sentiment, key phrases, and language identification.
• Unified Data Access Interfaces - Offers a standardized interface for reading and writing data across diverse databases, file systems, and cloud stores.
• Unified Data Connector Interfaces - Provides a common interface for reading and writing across diverse databases and cloud storage systems.
• ML Lifecycle Pipelines - Combines data featurization and model training into a single workflow to derive predictions at scale.
• AI Workflow Serving - Exposes distributed computations as low-latency web services for real-time inference.
• Face Analysis - Detects human faces and groups individuals based on facial similarity and identity verification.
• Distributed Computer Vision - Analyzes image content and detects facial features across large datasets using distributed computing clusters.
• Distributed Data Featurization - Provides utilities for normalizing numeric features and mapping categorical labels across distributed datasets.
• Distributed Training Optimizers - Optimizes distributed training through specialized synchronization to improve execution speed and memory usage.
• Forest-Based - Identifies anomalies in large datasets using distributed forest implementations.
• Image Content Analyzers - Identifies visual features, generates descriptions, and extracts text or tags from images.
• Inference Accelerators - Runs distributed model inference using hardware acceleration and standardized formats to reduce processing time.
• ONNX Runtime Inference - Executes trained deep learning and machine learning models across distributed data using the ONNX runtime.
• Model Persistence - Serializes machine learning models and metadata to a persistent store for retrieval and deployment.
• Model Auditing - Analyzes opaque models to understand decision-making and measure biases within datasets.
• Model Explainability - Calculates feature importance and contribution scores to explain why specific data points were flagged.
• Model Serializers - Persists model bytes and metadata to a storage layer for retrieval and deployment across environments.
• SHAP Value Computations - Calculates additive contribution values for each feature to quantify the final model prediction.
• Sentiment & Topic Analysis - Detects languages, extracts key phrases, and calculates sentiment scores from unstructured text.
• Time Series Anomaly Detection - Analyzes a series of data points to identify irregularities or anomalous latest points.
• Document and Unstructured Extraction - Transforms unstructured documents and receipts into structured data using OCR and machine learning.
• K-Nearest Neighbor Retrieval - Executes scalable k-nearest neighbor queries with conditional filtering on distributed data.
• Surrogate Model Explanations - Generates local interpretations for data by fitting a surrogate model around a specific observation.
• Sparse Data Processing - Implements fast and sparse data structures to handle high-dimensional text analytics at a distributed scale.
• External API Ingestion Pipelines - Ingests data directly into big data processing pipelines by calling arbitrary external HTTP web services.
• External API Integrations - Connects distributed data pipelines to external pre-trained AI services via HTTP requests for content analysis.
• General Machine Learning - Distributed machine learning framework for Apache Spark.
• Machine Learning - Distributed ML library with broad support.
• Machine Learning Frameworks - Distributed machine learning framework for Apache Spark.

Star history