cactus-compute/cactus
Cactus
Cactus is an on-device AI inference engine designed for executing large language models, vision models, and speech-to-text systems on mobile and wearable hardware. It provides a programmable tensor computation graph for defining sequences of matrix operations and activation functions, alongside a local retrieval augmented generation framework that grounds model responses using local text files.
The project features a multiplatform SDK with language bindings for integrating AI capabilities into mobile applications and a model conversion system that transforms external model formats for optimized local execution. It utilizes a hybrid routing system to redirect workloads between on-device execution and cloud-based providers based on hardware capacity.
The engine covers a broad capability surface including on-device audio processing for voice activity detection and transcription, vector embedding generation for similarity search, and tool integration for parsing model outputs into external function calls. These processes are supported by optimized native kernels tuned for low-latency performance on mobile hardware.
Features
- Local AI Inference - Executes large language and vision models directly on mobile and wearable hardware using optimized kernels.
- On-Device Inference Engines - Serves as an on-device AI inference engine for executing large language, vision, and speech models on mobile and wearable hardware.
- Multimodal Input Processing - Performs inference on image and sound data to enable visual understanding and speech-to-text capabilities.
- Chat Completion Services - Produces natural language conversational responses based on chat history and configurable generation options.
- Retrieval-Augmented Generation - Grounds model responses using locally stored text documents and directories to provide context-aware generation.
- RAG Document Retrieval - Retrieves relevant snippets from local text files to provide grounded context for LLM responses.
- Inference Optimization Kernels - Utilizes native kernels tuned for low-latency, energy-efficient mathematical operations on mobile hardware.
- Local RAG Implementations - Provides a local retrieval augmented generation framework that grounds model responses using local text files without cloud access.
- Local Inference Engines - Provides an optimized runtime for executing large language models and vision models locally on consumer mobile hardware.
- On-Device Speech-to-Text SDKs - Provides on-device speech-to-text transcription using locally executed models on mobile and wearable hardware.
- RAG Frameworks - Provides a framework for building local retrieval augmented generation systems that ground responses in local directories.
- Speech Transcription - Provides local on-device speech-to-text transcription services with low-latency execution.
- Vector Embeddings - Generates numerical vector representations of text, visual, and speech inputs for similarity search and retrieval.
- AI Integration SDKs - Ships a multiplatform SDK with language bindings for integrating local AI capabilities into mobile applications.
- Mobile Framework Integrations - Offers native software kits to integrate AI capabilities into handheld and wearable operating systems.
- Tensor Computation Graphs - Allows defining sequences of tensor operations and activation functions as computational graphs for local execution.
- Graph-Based Execution Engines - Executes mathematical workflows as a sequence of tensor operations and activation functions via directed acyclic graphs.
- Language Bindings - Provides multiplatform software development kits and language bindings to connect the core engine to external applications.
- AI Integration Tools - Connects local AI models to external system functions and tools to perform actions based on model outputs.
- Model Request Routing - Redirects inference requests to cloud providers when local hardware capacity is insufficient.
- Cross-Framework Model Conversion - Transforms external model formats into representations optimized for mobile and wearable hardware.
- Function Calling Interfaces - Parses model outputs into structured function calls to interact with external system tools.
- Hybrid Local-Remote AI Routing - Routes AI workloads between local on-device execution and cloud-based providers based on hardware capacity.
- Local Speech-to-Text - Includes a low-latency on-device transcription system for converting audio input into text.
- On-Device Speech Recognizers - Performs local speech-to-text transcription and voice activity detection on handheld and wearable devices.
- Voice Activity Detection - Identifies periods of human speech within audio streams to trigger transcription and downstream processing.
- Mobile Model Format Converters - Transforms external model formats into optimized representations compatible with local mobile and wearable hardware execution.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
