# mercari/ml-system-design-pattern

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/mercari-ml-system-design-pattern).**

2,922 stars · 340 forks · MIT

## Links

- GitHub: https://github.com/mercari/ml-system-design-pattern
- awesome-repositories: https://awesome-repositories.com/repository/mercari-ml-system-design-pattern.md

## Description

This project provides a collection of architectural blueprints and design patterns for building, deploying, and scaling machine learning systems in production environments. It serves as a comprehensive reference for standardizing the end-to-end lifecycle of machine learning components, including training pipelines, model serving, and system observability.

The framework distinguishes itself by offering standardized strategies for managing complex operational requirements such as asynchronous inference, traffic routing, and service decoupling. It covers a wide range of patterns for model serving, including microservice-based architectures, containerized artifact packaging, and dynamic model loading, which allow for granular resource allocation and independent deployment cycles.

The system encompasses a broad capability surface for maintaining production-grade machine learning infrastructure. This includes automated workflows for batch processing and hyperparameter optimization, as well as robust traffic management techniques like circuit breaking, request throttling, and conditional routing. Furthermore, it provides methodologies for system observability, performance monitoring, and stress testing to ensure reliability and service stability under varying load conditions.

## Tags

### Part of an Awesome List

- [Machine Learning Patterns](https://awesome-repositories.com/f/awesome-lists/ai/machine-learning-patterns.md) — Provides a collection of architectural blueprints and design patterns for building and scaling machine learning systems.
- [MLOps and Lifecycle](https://awesome-repositories.com/f/awesome-lists/devops/mlops-and-lifecycle.md) — Provides a reference for standardizing the lifecycle of machine learning components including training and serving.

### Artificial Intelligence & ML

- [Machine Learning Operations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning-operations.md) — Provides tools and frameworks for deploying, monitoring, and maintaining machine learning models in production environments.
- [Machine Learning Systems](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning-systems.md) — Architects scalable and reliable production workflows for training, deploying, and maintaining machine learning models.
- [Automated Model Training](https://awesome-repositories.com/f/artificial-intelligence-ml/automated-model-training.md) — Automates data processing and hyperparameter optimization to produce optimized models.
- [Batch Prediction Processing](https://awesome-repositories.com/f/artificial-intelligence-ml/batch-prediction-processing.md) — Executes inference on bulk datasets at regular intervals to eliminate the need for continuous real-time availability. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Batch-pattern/design_en.md))
- [Hybrid Model Parallelism](https://awesome-repositories.com/f/artificial-intelligence-ml/hybrid-model-parallelism.md) — Runs multiple independent machine learning models simultaneously to aggregate results or generate diverse insights. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Microservice-horizontal-pattern/design_en.md))
- [Training Job Queueing](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/automated-ml-automl/training-job-queueing.md) — Implements error handling and retry policies to ensure training pipelines recover from failures. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Training-patterns/Batch-training-pattern/design_en.md))
- [Modular Pipeline Orchestrators](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/pipelines-and-orchestration/modular-pipeline-orchestrators.md) — Decomposes workflows into independent, sequential tasks to enable flexible execution and optimized resource allocation. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Training-patterns/Pipeline-training-pattern/design_en.md))
- [Machine Learning Evaluation](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-evaluation-analysis/machine-learning-evaluation.md) — Validates model performance and server stability using methodologies like shadow deployments and automated load testing. ([source](https://github.com/mercari/ml-system-design-pattern#readme))
- [Model Loading](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/data-and-checkpointing/model-loading.md) — Provides mechanisms for efficiently loading model weights and configurations into memory for execution. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Operation-patterns/README.md))
- [Model Deployment Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/model-deployment-pipelines.md) — Triggers the release of new model servers immediately upon successful completion of training pipelines. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Lifecycle-patterns/Training-to-serving-pattern/design_en.md))
- [Hyperparameter Optimization](https://awesome-repositories.com/f/artificial-intelligence-ml/model-optimization/training-efficiency/hyperparameter-optimization.md) — Runs iterative training and evaluation cycles to automatically discover effective settings and network structures. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Training-patterns/Parameter-and-architecture-search-pattern/design_en.md))
- [Environment Separation](https://awesome-repositories.com/f/artificial-intelligence-ml/model-serving-deployment/environment-separation.md) — Separates model development from production deployment to allow for quality assurance before release. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Lifecycle-patterns/Train-then-serve-pattern/design_en.md))
- [Asynchronous Inference](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-networks/asynchronous-inference.md) — Queues prediction requests for background processing and notifies the client once the result is ready. ([source](https://github.com/mercari/ml-system-design-pattern/tree/master/Serving-patterns))
- [Production Machine Learning Guides](https://awesome-repositories.com/f/artificial-intelligence-ml/production-machine-learning-guides.md) — Acts as a comprehensive guide to implementing reliable and scalable machine learning systems using industry patterns.

### DevOps & Infrastructure

- [Container Image Packaging](https://awesome-repositories.com/f/devops-infrastructure/container-image-packaging.md) — Bundles model files and serving environments into immutable images to ensure consistency and reproducibility.
- [Model Inference Microservices](https://awesome-repositories.com/f/devops-infrastructure/microservices-deployments/model-inference-microservices.md) — Decomposes machine learning models into independent, scalable service instances for granular resource allocation.
- [Model Serving](https://awesome-repositories.com/f/devops-infrastructure/model-serving.md) — Provides infrastructure and techniques for deploying and optimizing machine learning models for production inference.
- [Traffic Throttling](https://awesome-repositories.com/f/devops-infrastructure/traffic-management/traffic-throttling.md) — Prevents system-wide outages by rejecting excess incoming traffic before it reaches the model server. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Prediction-circuit-break-pattern/design_en.md))
- [Containerized Model Serving](https://awesome-repositories.com/f/devops-infrastructure/containerized-model-serving.md) — Provides architectural patterns for serving machine learning models via containers to ensure environment consistency. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Operation-patterns/Model-in-image-pattern/design_en.md))
- [Model Inference Deployment](https://awesome-repositories.com/f/devops-infrastructure/deployment-management/model-inference-deployment.md) — Provides systems for deploying models to production environments for real-time synchronous inference. ([source](https://github.com/mercari/ml-system-design-pattern/tree/master/Serving-patterns))
- [Event-Driven Pipelines](https://awesome-repositories.com/f/devops-infrastructure/event-driven-pipelines.md) — Links sequential data processing and inference stages into automated workflows triggered by events.
- [Prediction Service Scalability](https://awesome-repositories.com/f/devops-infrastructure/scalability-management/prediction-service-scalability.md) — Implements microservice architectures and caching strategies to handle varying traffic volumes for inference.
- [Health Tracking Sidecars](https://awesome-repositories.com/f/devops-infrastructure/sidecar-containers/agent-sidecars/health-tracking-sidecars.md) — Integrates logging and telemetry collection alongside prediction services to provide visibility into system performance.
- [Traffic Load Balancers](https://awesome-repositories.com/f/devops-infrastructure/traffic-load-balancers.md) — Balances incoming requests across parallel prediction services to gradually shift load between model versions. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/QA-patterns/Online-ab-test-pattern/design_en.md))
- [Preprocessing Decoupling](https://awesome-repositories.com/f/devops-infrastructure/worker-scaling/independent-model-component-scaling/preprocessing-decoupling.md) — Separates data transformation logic from model execution to allow for distinct resource allocation. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Prep-pred-pattern/design_en.md))

### Networking & Communication

- [Circuit Breakers](https://awesome-repositories.com/f/networking-communication/traffic-management-gateways/circuit-breakers.md) — Prevents cascading system instability by automatically stopping requests to failing model services. ([source](https://github.com/mercari/ml-system-design-pattern/tree/master/Serving-patterns))
- [Centralized Request Routing](https://awesome-repositories.com/f/networking-communication/centralized-request-routing.md) — Uses a centralized gateway to manage traffic distribution and conditional routing across multiple model versions.
- [Message-Based Process Orchestrations](https://awesome-repositories.com/f/networking-communication/communication-protocols-architectures/inter-process-communication/inter-process-communication-frameworks/message-based-process-orchestrations.md) — Coordinates decoupled prediction tasks and background workflows using message queues for fault tolerance.
- [Inference Parameter Overrides](https://awesome-repositories.com/f/networking-communication/grpc-server-reliability-toolkits/dynamic-parameter-adjustment/inference-parameter-overrides.md) — Enables dynamic modification of inference behavior through rule-based parameter injection. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Operation-patterns/Parameter-based-serving-pattern/design_en.md))
- [Conditional Routing](https://awesome-repositories.com/f/networking-communication/traffic-routing/conditional-routing.md) — Directs prediction requests to specific models based on contextual metadata to improve accuracy. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Operation-patterns/Condition-based-serving-pattern/design_en.md))

### System Administration & Monitoring

- [Monitoring and Observability](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability.md) — Provides comprehensive visibility into system performance by capturing application logs and traceable events. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Operation-patterns/Anti-patterns/No-logging-pattern/design_en.md))
- [Anomaly Detection](https://awesome-repositories.com/f/system-administration-monitoring/anomaly-detection.md) — Identifies silent failures and service degradation by triggering alerts on unexpected shifts in model output or input data. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Operation-patterns/Prediction-log-pattern/design_en.md))
- [Service Performance Monitors](https://awesome-repositories.com/f/system-administration-monitoring/service-performance-monitors.md) — Tracks response times for inference endpoints to ensure performance meets production requirements under varying traffic. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/QA-patterns/Loading-test-pattern/design_en.md))
- [System Metrics](https://awesome-repositories.com/f/system-administration-monitoring/system-metrics.md) — Integrates measurement units into pipelines to provide objective visibility into system performance. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Operation-patterns/Anti-patterns/Nobody-knows-pattern/design_en.md))

### Data & Databases

- [Batch Processing Schedulers](https://awesome-repositories.com/f/data-databases/batch-processing-schedulers.md) — Automates periodic training workflows by triggering data retrieval and preprocessing on a defined schedule. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Training-patterns/Batch-training-pattern/design_en.md))
- [Input Caches](https://awesome-repositories.com/f/data-databases/data-caching/input-caches.md) — Reduces latency by storing frequently accessed input data in high-speed memory layers during prediction requests. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Data-cache-pattern/design_en.md))
- [Inference Result Caches](https://awesome-repositories.com/f/data-databases/data-caching/persistent-binary-caches/inference-result-caches.md) — Accelerates inference by saving model outputs in fast-access layers to serve repeated requests instantly. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Prediction-cache-pattern/design_en.md))
- [Multi-Stage Pipeline Processing](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/document-llm-preparation/multi-stage-pipeline-processing.md) — Orchestrates complex inference tasks by chaining multiple sequential processing steps together. ([source](https://github.com/mercari/ml-system-design-pattern/tree/master/Serving-patterns))

### Development Tools & Productivity

- [Synchronous Prediction Interfaces](https://awesome-repositories.com/f/development-tools-productivity/rest-apis/model-prediction-interfaces/synchronous-prediction-interfaces.md) — Pauses calling workflows until inference requests finish to ensure predictable data flow. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Synchronous-pattern/design_en.md))

### Software Engineering & Architecture

- [Cache Aside Patterns](https://awesome-repositories.com/f/software-engineering-architecture/architectural-design-patterns/design-patterns/cache-aside-patterns.md) — Stores inference results in a high-speed memory layer to reduce computational load and improve response times.
- [Asynchronous Service Decoupling](https://awesome-repositories.com/f/software-engineering-architecture/asynchronous-service-decoupling.md) — Offloads inference tasks to background queues to prevent client applications from waiting for slow model processing. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Asynchronous-pattern/design_en.md))
- [Asynchronous Workflow Management](https://awesome-repositories.com/f/software-engineering-architecture/asynchronous-workflow-management.md) — Coordinates prediction triggers using message queues and automated retry logic for reliable processing. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Asynchronous-pattern/design_en.md))
- [Fallback Strategies](https://awesome-repositories.com/f/software-engineering-architecture/fallback-strategies.md) — Maintains service continuity by providing alternative responses or simplified workflows during traffic spikes. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Prediction-circuit-break-pattern/design_en.md))
- [Unified Model Interfaces](https://awesome-repositories.com/f/software-engineering-architecture/unified-model-interfaces.md) — Defines standardized APIs that provide a consistent execution interface across different model providers. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/Serving-template-pattern/design_en.md))

### Testing & Quality Assurance

- [Infrastructure Stress Testing](https://awesome-repositories.com/f/testing-quality-assurance/infrastructure-stress-testing.md) — Simulates high-volume concurrent requests against prediction services to determine resource capacity and bottlenecks. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/QA-patterns/Loading-test-pattern/design_en.md))

### Web Development

- [Caching and Delivery](https://awesome-repositories.com/f/web-development/static-site-performance-optimization/caching-and-delivery.md) — Improves response times and system resilience by applying caching and multi-stage processing to inference delivery. ([source](https://github.com/mercari/ml-system-design-pattern/blob/master/Serving-patterns/README.md))