mlc-ai/web-llm

Features

• Browser-based Inference Engines - Provides a browser-based inference engine for running large language models locally with hardware acceleration.
• Local Model Execution - Executes large language models directly on local hardware to ensure user data privacy and offline capability.
• AI Runtimes - Provides a high-performance AI runtime that leverages WebGPU for client-side machine learning.
• Hardware-Accelerated Inference - Leverages WebGPU to execute tensor operations on the graphics processor for high-performance local inference.
• Chat Completion Services - Provides standardized interfaces for generating conversational text responses using local language models.
• LLM Application Frameworks - Offers a framework for building conversational AI applications that run entirely within the browser.
• Background Processing Workers - Offloads intensive model computations to background threads to maintain main application responsiveness.
• OpenAI-Compatible APIs - Implements standard HTTP endpoints that mimic the OpenAI API specification for local model interoperability.
• Local Model Lifecycle Managers - Provides utilities for initializing, managing, and clearing large language models from browser memory.
• AI and Machine Learning - Inference engine for large language models on the web.
• Inference and Serving - High-performance in-browser LLM inference engine.
• Large Language Models - Running LLMs directly inside web browsers.
• More to explore - Runs large language models natively in the browser.
• WebAssembly Runtimes - Executes high-performance model logic and tensor operations using WebAssembly within the browser.
• Persistent Background Workers - Maintains model state and data availability across browser sessions using background service workers.
• Browser Extensions - Facilitates the development of browser extensions that provide persistent, local AI capabilities.
• Response Streaming Interfaces - Supports incremental text delivery to enable real-time response rendering in web interfaces.
• Conversation State Management - Manages conversational context and interaction history for multi-turn AI chat sessions.
• Custom Model Integrations - Supports the integration and execution of custom, externally compiled language model architectures within the browser.
• Local Model Integrations - Provides tools for integrating and testing local language model services within web projects.
• Model Weight Management - Handles the downloading, storage, and initialization of model weights for local inference.
• IndexedDB Stores - Utilizes IndexedDB for persistent storage of model weights to avoid redundant downloads across sessions.
• Background Thread Dispatchers - Offloads intensive model computations to background threads to ensure main UI responsiveness.

Star history