cactus-compute/cactus

Features

• Local AI Inference - Executes large language and vision models directly on mobile and wearable hardware using optimized kernels.
• On-Device Inference Engines - Serves as an on-device AI inference engine for executing large language, vision, and speech models on mobile and wearable hardware.
• Multimodal Input Processing - Performs inference on image and sound data to enable visual understanding and speech-to-text capabilities.
• Chat Completion Services - Produces natural language conversational responses based on chat history and configurable generation options.
• Retrieval-Augmented Generation - Grounds model responses using locally stored text documents and directories to provide context-aware generation.
• RAG Document Retrieval - Retrieves relevant snippets from local text files to provide grounded context for LLM responses.
• Inference Optimization Kernels - Utilizes native kernels tuned for low-latency, energy-efficient mathematical operations on mobile hardware.
• Local RAG Implementations - Provides a local retrieval augmented generation framework that grounds model responses using local text files without cloud access.
• Local Inference Engines - Provides an optimized runtime for executing large language models and vision models locally on consumer mobile hardware.
• On-Device Speech-to-Text SDKs - Provides on-device speech-to-text transcription using locally executed models on mobile and wearable hardware.
• RAG Frameworks - Provides a framework for building local retrieval augmented generation systems that ground responses in local directories.
• Speech Transcription - Provides local on-device speech-to-text transcription services with low-latency execution.
• Vector Embeddings - Generates numerical vector representations of text, visual, and speech inputs for similarity search and retrieval.
• AI Integration SDKs - Ships a multiplatform SDK with language bindings for integrating local AI capabilities into mobile applications.
• Mobile Framework Integrations - Offers native software kits to integrate AI capabilities into handheld and wearable operating systems.
• Tensor Computation Graphs - Allows defining sequences of tensor operations and activation functions as computational graphs for local execution.
• Graph-Based Execution Engines - Executes mathematical workflows as a sequence of tensor operations and activation functions via directed acyclic graphs.
• Language Bindings - Provides multiplatform software development kits and language bindings to connect the core engine to external applications.
• AI Integration Tools - Connects local AI models to external system functions and tools to perform actions based on model outputs.
• Model Request Routing - Redirects inference requests to cloud providers when local hardware capacity is insufficient.
• Cross-Framework Model Conversion - Transforms external model formats into representations optimized for mobile and wearable hardware.
• Function Calling Interfaces - Parses model outputs into structured function calls to interact with external system tools.
• Hybrid Local-Remote AI Routing - Routes AI workloads between local on-device execution and cloud-based providers based on hardware capacity.
• Local Speech-to-Text - Includes a low-latency on-device transcription system for converting audio input into text.
• On-Device Speech Recognizers - Performs local speech-to-text transcription and voice activity detection on handheld and wearable devices.
• Voice Activity Detection - Identifies periods of human speech within audio streams to trigger transcription and downstream processing.
• Mobile Model Format Converters - Transforms external model formats into optimized representations compatible with local mobile and wearable hardware execution.

Star history