mlflow/mlflow

Features

• Agent Evaluation Tools - Analyze agent performance by defining test datasets and custom scorers to assess both final outputs and intermediate tool usage.
• AI Gateways - Manage access to external AI providers by configuring API keys and defining model endpoints through a web interface.
• Experiment Tracking - Defines unique experiment names, specifies artifact storage locations, and attaches metadata tags to track development progress.
• Experiment Tracking Servers - Start a local tracking server with a single command to manage experiments, model artifacts, and metadata for machine learning workflows.
• Experiment Tracking Systems - Log experiment parameters, metrics, and models during training using explicit API calls or automatic logging for popular training libraries.
• LLM Execution Tracing - Captures detailed model call data including prompts, completions, and token counts.
• Model Gateways - Aggregates multiple AI service providers behind a single interface to manage authentication, rate limiting, and cost tracking for model requests.
• Model Lifecycle Management - Centralizes version control, tracks model lineage, and organizes deployment workflows through a structured registry system.
• Model Registries - Tracks versions, assigns aliases, and manages metadata for production-ready deployment workflows.
• AI Observability Suites - A diagnostic toolkit for tracing, evaluating, and monitoring the performance, quality, and operational costs of complex language model applications.
• Custom Evaluation Judges - Adapt base models to act as custom judges that align with specific business requirements and expert human judgment.
• Evaluation Frameworks - Configure the LLM used to power evaluation judges by specifying the provider and model name in the judge definition.
• LLM Provider Integrations - Connect various language model providers into a unified gateway to access chat and embedding capabilities through a consistent interface.
• Model Evaluation Frameworks - Runs systematic evaluations using built-in metrics to track quality and detect regressions in model performance.
• Model Inference - Sends input payloads to a specified endpoint or deployment target to generate predictions from a deployed model.
• Model Inference Servers - Deploy machine learning models to diverse environments by launching inference servers with REST endpoints for handling prediction requests.
• Model Packaging - Organizes model artifacts and configuration files into a standardized directory format that supports multiple interoperable model flavors.
• Model Serving Servers - Launch a local HTTP inference server using command-line tools to serve model predictions via standard REST endpoints.
• Prompt Engineering Environments - Offers a collaborative environment for versioning, testing, and optimizing prompt templates with integrated lineage tracking and automated evaluation workflows.
• Prompt Management Systems - Versions and deploys prompts with full lineage tracking while using automated algorithms to improve output quality.
• Prompt Registries - Creates versioned prompt templates with metadata and response format specifications.
• RAG Evaluation Frameworks - Evaluate RAG application performance using built-in judges that assess retrieval relevance, groundedness, and context sufficiency based on captured application traces.
• Prompt Repositories - Maintains prompt templates with version control and A/B testing capabilities.
• Agent Execution Tracing - Visualizes complex agent reasoning steps, tool calls, and retrieval processes for debugging.
• Automatic Tracing Instrumentation - Enables comprehensive observability by capturing execution details with minimal code changes.
• LLM Execution Tracing - Captures inputs, outputs, and execution details to provide full visibility into LLM behavior.
• LLM Evaluation - Provides a dedicated environment for evaluating LLM application performance through prediction functions and datasets.
• Model Serving - Launches a local webserver that accepts prediction requests in various data formats to serve saved models.
• Agent Deployment Servers - Provides a dedicated server environment for deploying and managing autonomous AI agents with integrated request validation.
• AI Gateway Management - Update existing endpoints by modifying model settings, connection configurations, or deleting them when they are no longer required.
• AI Observability Tools - Captures end-to-end execution traces and performance metrics to debug complex agent workflows and monitor production behavior in real-time.
• AI Observability Tracing - Captures complete execution traces of applications to monitor production quality, costs, and safety.
• Artifact Logging - Associates a local file with a specific run by logging it as an artifact.
• Automated Model Judges - Tracks quality metrics over time using automated judges to detect regressions and performance issues.
• Automatic Logging - Captures model parameters, metrics, and artifacts automatically by calling a single function before executing training code.
• Conversation Evaluation Tools - Evaluate existing conversation traces by grouping them via session IDs and applying multi-turn scorers to analyze production data.
• Experiment Visualization Dashboards - Visualize and compare logged experiments, runs, and performance metrics through a web-based interface or a hosted tracking server.
• Human Feedback Collection - Gather input from users and experts to validate judge accuracy, identify performance gaps, and improve overall evaluation quality.
• LLM Tracing Systems - Instruments LLM applications to capture and send execution traces to a tracking server for observability.
• Model Flavors - Ensures compatibility across various machine learning libraries and deployment tools by using a unified interface for loading and scoring models.
• Model Serialization Formats - Standardizes models into portable directory formats to ensure consistent loading across environments.
• Model Serving Endpoints - Deploys validated prompt configurations to live endpoints for real-time production inference.
• Model Versioning - Log models with input and output signatures to define expected data formats, enabling better model understanding and validation for downstream deployment.
• Observability Tools - Provides a visual interface to debug agent workflows and identify performance bottlenecks.
• Experiment Tracking Tools - Start a local tracking server using the command line interface to manage experiments, models, and metadata for development workflows.
• Distributed Trace Propagation - Maintains unified trace context across distributed services using W3C TraceContext headers.
• Model Access Governance - Provides centralized governance for model access, including rate limiting and cost tracking.
• AI Cost Monitoring - Tracks token usage and model efficiency to identify optimization opportunities.
• Model Evaluation - Allows developers to define custom functions to measure specific quality aspects of AI applications.
• Agent Deployment Tools - Launches agents using a server that provides automatic request validation and tracing for rapid production deployment.
• Agent Simulation Environments - Simulate user interactions with an agent by defining test cases with specific goals and personas to generate and evaluate diverse conversation scenarios.
• AI Application Monitoring - Captures complete traces of applications to monitor production quality, costs, and safety using standard observability tools.
• Automated Instrumentation Frameworks - Instruments application logic automatically by integrating with agent frameworks to capture execution traces without manual code changes.
• Conversational Evaluation Suites - Assess conversational agents by simulating multi-turn dialogues and applying scorers to evaluate interaction quality and safety at every step.
• Custom Model Definitions - Defines custom inference logic and artifact dependencies to deploy models as standard functions.
• Model Checkpoint Managers - Save multiple model checkpoints during a single training run and link performance metrics to specific versions for improved traceability.
• Model Comparison Tools - Analyze evaluation results across different agent versions side-by-side to identify regressions, debug issues, and inform future development.
• Model Containerization Tools - Package machine learning models into standardized containers with their dependencies and metadata to ensure consistent execution across various deployment environments.
• Model Deployment Pipelines - Provides a standardized framework for packaging, versioning, and deploying machine learning models across diverse production environments and serving infrastructures.
• Model Serialization - Save trained Keras models as artifacts and load them back for inference using dedicated functions that handle model serialization and retrieval.
• Prompt Lifecycle Management - Manages prompt evolution and deployment pipelines using versioned templates and aliases.
• Prompt Optimization Frameworks - Improves prompt performance automatically using genetic algorithms and metaprompting techniques.
• Training Instrumentation - Integrate tracking calls directly into training loops to manually log custom metrics, hyperparameters, and model states during development.
• LLM Development Frameworks - Platform for tracking experiments, evaluation, and model deployment.
• Machine Learning - Integrated platform for end-to-end machine learning workflows.
• Machine Learning Frameworks - Platform for managing the machine learning lifecycle.
• Machine Learning Operations - Open-source platform for the complete machine learning lifecycle.
• MLOps Platforms - Manages the end-to-end machine learning lifecycle.
• Model Management - Platform for ML lifecycle management.
• Observability and Evaluation - Comprehensive suite for LLM tracing and model evaluation.
• Observability And Monitoring - End-to-end platform for tracking and monitoring model applications.
• Perception and Machine Learning - Platform for managing the machine learning lifecycle.
• Experiment and Data Management - Lifecycle management platform for experimentation and deployment.
• Experimentation Tracking - Manages the end-to-end machine learning lifecycle.
• Artifact Management - Transfers artifact files or directories from a remote repository to a local filesystem destination for further analysis.
• AI Compliance Governance - Maintains audit trails and enforces content guardrails to meet organizational compliance requirements.
• Application Quality Monitoring - Tracks quality metrics over time to proactively identify and resolve issues.
• Automated Trace Evaluation - Configures automated judges to evaluate production traces for quality monitoring.
• Dynamic Autologging - Automatically captures training metrics and artifacts by injecting tracking logic into libraries at runtime.
• OpenTelemetry Exporters - Standardizes the transmission of execution traces by adhering to open observability protocols.
• AI Auto-Logging Tools - Instruments AI applications with minimal effort by using built-in auto-logging capabilities for supported libraries.
• AI Request Routers - Directs requests to various providers through a unified interface to manage rate limits and service fallbacks.
• Autologging Controls - Toggles tracking functions to enable or disable automatic data collection for specific libraries.
• Data Lineage - Track data sources automatically during model training by logging paths, formats, and versions of datasets read from distributed storage systems.
• Data Lineage Trackers - Record metadata about training datasets to maintain lineage and traceability between data sources and final model performance.
• Deep Learning Experiment Trackers - Capture training metrics, model parameters, and artifacts automatically during Keras model training by enabling a single-line configuration function.
• Inference Clients - Interact with model servers using standard HTTP requests to perform inference, check health status, and retrieve version information.
• Model Benchmarking Interfaces - Test registered models against defined datasets and scorers by wrapping them in a prediction function and passing them to the evaluation interface.
• Model Configuration Management - Stores inference parameters with prompt templates to ensure consistent model behavior.
• Model Signatures - Specifies input and output schemas with examples to enable automated validation and testing.
• Model Validation Schemas - Enforces data integrity by validating runtime payloads against predefined signatures during model serving.
• Transformer Model Management - Log transformer pipelines and individual model components to track model artifacts, metadata, and prompt templates for reproducible workflows.
• ML Orchestration Deployments - Deploy the tracking server using container orchestration tools or managed cloud services for production-scale environments.
• Authentication Clients - Supports multiple authentication methods for secure programmatic access to the tracking server.
• Application State Versioning - Snapshots code and configurations into versioned entities for reliable reproduction.
• Asynchronous Tracing - Captures execution context across asynchronous functions to ensure accurate tracing.
• Workspace Management Systems - Organizes resources into isolated workspaces to share infrastructure across teams while maintaining strict boundaries.
• Decorator-Based Instrumentation - Captures function inputs and outputs by wrapping code in decorators for performance analysis.
• Execution Path Visualization - Visualizes complex execution paths in multi-step agents to make every step debuggable.
• Performance Trend Analysis - Identifies performance patterns across large-scale deployments using summary interfaces.
• Trace Querying - Provides a query language to filter traces based on attributes and tags.
• Trace Sampling - Manages export volume using global sampling ratios and per-endpoint overrides.
• Usage Monitoring Tools - Provides aggregated visibility into token usage and financial costs for model calls.
• Execution Tracers - Enables deep inspection of agent execution traces to validate retrieval and tool usage.
• Performance Profiling - Analyze logged experiment traces to discover operational bottlenecks and quality issues, generating a comprehensive report with actionable recommendations for improvement.
• Autologging Customization - Passes arguments to initialization functions to customize model signature logging and refine data collection settings.
• Batch Inference Tools - Process input data in bulk using command-line tools or scripts to generate predictions and save results to an output file.
• Decorator-based Scorers - Create custom evaluation logic using a decorator to process application inputs, outputs, and execution traces for automated performance assessment.
• Dependency Management - Infers required packages automatically and defines custom environment configurations to ensure consistent model execution.
• Embedding Model Utilities - Log and load sentence transformer models with full metadata, model signatures, and support for native loading and generic inference interfaces.
• Experiment Identifiers - Fetches the unique identifier for a newly created experiment after successfully submitting a creation request.
• ML Infrastructure Configuration - Configure the tracking server architecture by plugging in custom backend stores for metadata and artifact stores for large files.
• Model Signature Generators - Generate model signatures automatically using language-specific type hints to enable runtime data validation and improve development environment support.
• OpenTelemetry Exporters - Exports OpenTelemetry traces from any language or framework by configuring the OTLP endpoint.
• Prompt Caching Strategies - Configures memory-based caching to improve retrieval performance for prompt requests.
• Semantic Search - Build semantic search systems by logging model parameters, saving corpus artifacts, and performing similarity searches using encoded document embeddings.
• Stateful Evaluation Scorers - Implement complex evaluation logic by extending a base class and overriding the call method to create stateful scoring behaviors.
• Access Control Policies - Controls access to resources by assigning granular permissions that define user interaction with assets.
• Authentication Middleware - Secure the tracking server using built-in authentication methods and network protection middleware to prevent unauthorized access.
• OIDC Authentication Plugins - Integrates with external identity providers to manage user sessions via OIDC.
• Declarative Tracing - Captures inputs, outputs, and execution time using decorators that maintain call relationships.
• Multi-tenant Isolation Policies - Enforces data and access boundaries by prefixing storage paths and applying granular permissions to separate assets across different organizational teams.
• Telemetry Systems - Offloads trace and metric data collection to background processes to maintain application performance during high-throughput operations.
• Tracing Instrumentation - Allows dynamic customization of span names and attributes during function execution.
• Wrapper-based Instrumentation - Wraps existing functions to capture execution context without modifying original definitions.
• Automated Trace Diagnostics - Automatically detects quality and operational issues within captured application traces.
• Observability Controls - Provides global control over trace collection to manage observability overhead and privacy.
• Programmatic Trace Analysis - Retrieves trace data via a Python API for analysis as structured data frames.
• Resource Monitoring - Monitor hardware resource utilization, including GPU, CPU, disk, and network metrics, to identify performance bottlenecks and optimize training efficiency.
• Session Tracking - Groups related execution traces into user sessions to analyze multi-turn conversation flows.
• Trace Context Management - Attaches request and user metadata to production traces for improved debugging.

Star history