replicate/cog

Features

• Machine Learning Model Portability - Bundles models and dependencies into standardized containers to ensure consistent execution across different hardware and software environments.
• Model Containerization Tools - Bundles machine learning models and their dependencies into standardized Docker containers.
• Declarative Image Synthesis - Builds standardized Docker containers by resolving GPU drivers and dependencies from a declarative configuration file.
• Containerized Model Wrappers - Wraps model code into a standardized image exposing a predictable interface for predictions and training.
• Framework Environment Configurations - Resolves compatible versions of hardware drivers and software libraries to eliminate environment conflicts.
• Inference API Servers - Provides a RESTful HTTP server and OpenAPI schema to expose model inference capabilities as standardized web APIs.
• Model Inference Servers - Implements a RESTful HTTP server that automatically generates OpenAPI schemas and handles request validation for containerized models.
• Model Inference - Provides the core logic for loading models and executing inference to generate predictions from input data.
• GPU Acceleration Configurations - Enables hardware acceleration and selects compatible driver versions based on the machine learning framework.
• Model Execution Environments - Defines a consistent interface and environment to ensure identical model operation across different hardware systems.
• Model Inference Servers - Runs a model as a live HTTP service that accepts prediction requests via JSON and returns results.
• Model Interfaces - Standardizes model interactions by defining input types, constraints, and default values through a unified interface.
• Input Parameter Specifications - Enables detailed specification of model input parameters with validation constraints and default values.
• Model Validation Schemas - Enforces data schemas for model predictions to ensure incoming requests match the expected format.
• Model Weight Management - Manages large model weights as external fixtures to optimize container image size and versioning.
• Model Inference APIs - Automatically generates RESTful HTTP servers and OpenAPI schemas to serve machine learning models as scalable web services.
• Environment Management - Provides a configuration system for resolving hardware drivers, system libraries, and Python dependencies to ensure consistent model execution.
• ML Image Builders - Bundles machine learning models and dependencies into standardized Docker containers with built-in inference servers.
• Build Dependency Management - Specifies runtime versions, package requirements, and system libraries needed to construct the execution environment.
• Data Type Validation - Validates incoming request data against specified ranges and regular expressions before the model is executed.
• API Request Handling - Handles model execution via HTTP POST requests and returns results along with performance metrics.
• Schema-Driven API Generators - Automatically generates RESTful HTTP servers and OpenAPI specifications from defined model input and output types.
• Local Model Execution - Allows executing packaged models on local hardware to verify behavior before production deployment.
• Incremental Inference Streaming - Implements server-sent events to stream model outputs incrementally, reducing perceived latency for the client.
• Model Training Pipelines - Defines standardized interfaces and environments for fine-tuning machine learning models and returning structured training results.
• Local Inference Verification - Runs packaged models locally or in notebooks to verify inference behavior before production deployment.
• Model Training Interfaces - Configures specific input and output requirements for model training workflows within a packaged environment.
• Model Training Packaging - Defines a function or class to handle model training that accepts specific inputs and produces weights.
• Structured Output Enforcements - Enforces custom schemas on model responses to ensure outputs are returned as complex, typed objects.
• Containerized Notebook Servers - Provides interactive notebook environments deployed within containers to isolate dependencies while maintaining consistency across development and production.
• Production Pipeline Integrations - Imports logic from exported notebook scripts into runner files to transition experimental code into deployable models.
• Concurrency Control Systems - BentoML limits the number of simultaneous prediction requests using a slot system to optimize hardware resource utilization.
• Container Image Registry Uploads - Builds container images and uploads them to remote registries for distribution and deployment.
• Process-Based Isolation - Executes predictions in separate subprocesses to prevent memory leaks or worker crashes from affecting the main process.
• Server-Sent Events - Uses server-sent events to deliver model outputs incrementally, improving perceived latency for long-running tasks.
• Subprocess-Based Isolation - Executes model inferences in separate subprocesses to isolate the main server from memory leaks or crashes.
• Inference Concurrency Control - Optimizes GPU and CPU utilization by limiting simultaneous prediction requests using a slot-based system.
• Request-Scoped Log Context - Tracks prediction IDs across asynchronous call stacks to attribute logs to specific requests.
• Concurrency Limiters - Sets the maximum number of simultaneous requests the model can process using an asynchronous execution method.
• Model Serving - Packages models into standard, production-ready containers.
• Cloud Infrastructure - Standardized containerization for deploying machine learning models.
• Infrastructure and Serving - Facilitates building Docker images.
• MLOps and Production - Tool for packaging machine learning models into reproducible containers.

Star history