zml/zml
Zml
zml is a machine learning model compiler and cross-platform inference engine that transforms model descriptions into optimized executable binaries for specific hardware accelerators. It functions as a model deployment toolkit and hardware-agnostic orchestrator, utilizing a tensor-based architecture definition to provide strong type checking during the compilation process.
The project distinguishes itself through the ability to shard tensors and distribute large-scale AI workloads across a logical mesh of multiple devices. It further supports the remote model lifecycle by authenticating and downloading gated model weights from cloud repositories to integrate them into a local inference engine.
The toolkit covers a broad range of capabilities, including model architecture validation against reference activations, cross-platform binary compilation for various operating systems, and the packaging of compiled models into container images and archives for server deployment. It also provides mechanisms for tensor buffer management and the porting of models from other formats.
Features
- Cross-Platform Inference Frameworks - Provides a runtime that executes pre-trained models across different operating systems and hardware architectures using a unified codebase.
- Model Deployment Toolkits - Provides a comprehensive suite for fetching remote model weights and packaging compiled binaries into container images for server deployment.
- Cross-Hardware Model Inference - Provides a unified codebase to execute pretrained models across heterogeneous hardware including CPUs, GPUs, and NPUs.
- Cross-Hardware Workload Distribution - Distributes large-scale AI workloads by partitioning tensors across a logical mesh of multiple hardware devices.
- Distributed Model Execution - Processes large-scale AI workloads by distributing model execution across a logical mesh of multiple devices.
- Distributed Model Orchestration - Orchestrates the distribution of large-scale model workloads by sharding tensors across a logical mesh of devices.
- Inference Execution - Executes compiled models by preparing input arguments and retrieving results from device memory.
- Model Compilers - Transforms high-level model definitions into optimized, hardware-specific executable machine code.
- Cross-Platform Deployments - Optimizes and packages models for deployment across diverse hardware from edge devices to cloud GPUs.
- Tensor-Parallel Inference Distributions - Splits large model weights across multiple hardware devices using tensor parallelism to handle massive workloads.
- Architectural Type Definitions - Uses structured types to define model layers and shapes for strong type checking during compilation.
- Pre-trained Model Application - Loads and executes pre-trained model implementations for tasks like image classification and text generation.
- Tensor-Based Architecture Definitions - Implements a structured tensor-based architecture definition to ensure strong type checking throughout the model compilation process.
- Architecture Definitions - Describes AI models using structs and functions that process tensors for strong type checking during compilation.
- Model Architecture Definitions - Uses structured types to describe network architectures ensuring strong type checking during the compilation process.
- Hardware-Specific Binaries - Translates model descriptions into optimized executable binaries tailored for specific CPU and GPU accelerators.
- Cross-Architecture Binary Compilation - Compiles model code into executable binaries for multiple target hardware architectures and operating systems.
- GPU-Accelerated Compilers - Compiles model definitions into optimized binaries targeting GPU and CPU accelerators for high-performance execution.
- Model Architecture Validation - Ensures network structure integrity through strong typing and verification of layer outputs against reference activations.
- Model Lifecycle Managers - Authenticates and downloads gated model weights from cloud repositories for local inference integration.
- Model Weight Management - Binds external weight files to model definitions for loading trained parameters into the inference engine.
- Model Format Converters - Translates model weights and architectures from PyTorch into a structured representation compatible with the inference engine.
- Cross-Platform Toolchains - Implements a toolchain to build model code across diverse hardware architectures and operating systems from a single source.
- Remote Model Loading - Downloads model weights and configurations from cloud buckets and HTTPS endpoints.
- Container Image Packaging - Bundles compiled model binaries and associated weights into container images for consistent server deployment.
- Model Container Execution - Wraps compiled models into Docker containers with a unified binary entrypoint for consistent inference execution.
- Model Memory Managers - Handles the allocation and transfer of weights and data between host memory and accelerator buffers.
- Device Buffer Managers - Provides a system for allocating and controlling hardware-resident memory buffers for tensor data.
- Dimension Tagging - Uses tagged tensors to simplify dimension handling and reduce errors during tensor operations.
- Tensor Buffer Offset Assignment - Manages the allocation of tensor data within fixed memory buffers to optimize inference loading times.
- Activation Validation - Verifies layer accuracy by comparing outputs against reference activations to identify mathematical or naming errors.
- Reference Implementation Validation - Verifies model conversion correctness by comparing layer outputs against trusted reference activations.
