cactus-compute/cactus
Cactus
Cactus es un motor de inferencia de IA en el dispositivo diseñado para ejecutar modelos de lenguaje de gran tamaño (LLM), modelos de visión y sistemas de voz a texto en hardware móvil y wearable. Proporciona un grafo de computación tensorial programable para definir secuencias de operaciones matriciales y funciones de activación, junto con un framework de RAG (generación aumentada por recuperación) local que fundamenta las respuestas del modelo utilizando archivos de texto locales.
El proyecto cuenta con un SDK multiplataforma con bindings de lenguaje para integrar capacidades de IA en aplicaciones móviles y un sistema de conversión de modelos que transforma formatos externos para una ejecución local optimizada. Utiliza un sistema de enrutamiento híbrido para redirigir cargas de trabajo entre la ejecución en el dispositivo y proveedores en la nube según la capacidad del hardware.
El motor cubre una amplia superficie de capacidades, incluyendo procesamiento de audio en el dispositivo para detección de actividad de voz y transcripción, generación de embeddings vectoriales para búsqueda por similitud e integración de herramientas para analizar salidas del modelo en llamadas a funciones externas. Estos procesos están respaldados por kernels nativos optimizados para un rendimiento de baja latencia en hardware móvil.
Features
- Local AI Inference - Executes large language and vision models directly on mobile and wearable hardware using optimized kernels.
- On-Device Inference Engines - Serves as an on-device AI inference engine for executing large language, vision, and speech models on mobile and wearable hardware.
- Multimodal Input Processing - Performs inference on image and sound data to enable visual understanding and speech-to-text capabilities.
- Chat Completion Services - Produces natural language conversational responses based on chat history and configurable generation options.
- Retrieval-Augmented Generation - Grounds model responses using locally stored text documents and directories to provide context-aware generation.
- RAG Document Retrieval - Retrieves relevant snippets from local text files to provide grounded context for LLM responses.
- Inference Optimization Kernels - Utilizes native kernels tuned for low-latency, energy-efficient mathematical operations on mobile hardware.
- Local RAG Implementations - Provides a local retrieval augmented generation framework that grounds model responses using local text files without cloud access.
- Local Inference Engines - Provides an optimized runtime for executing large language models and vision models locally on consumer mobile hardware.
- On-Device Speech-to-Text SDKs - Provides on-device speech-to-text transcription using locally executed models on mobile and wearable hardware.
- RAG Frameworks - Provides a framework for building local retrieval augmented generation systems that ground responses in local directories.
- Speech Transcription - Provides local on-device speech-to-text transcription services with low-latency execution.
- Vector Embeddings - Generates numerical vector representations of text, visual, and speech inputs for similarity search and retrieval.
- AI Integration SDKs - Ships a multiplatform SDK with language bindings for integrating local AI capabilities into mobile applications.
- Mobile Framework Integrations - Offers native software kits to integrate AI capabilities into handheld and wearable operating systems.
- Tensor Computation Graphs - Allows defining sequences of tensor operations and activation functions as computational graphs for local execution.
- Graph-Based Execution Engines - Executes mathematical workflows as a sequence of tensor operations and activation functions via directed acyclic graphs.
- Language Bindings - Provides multiplatform software development kits and language bindings to connect the core engine to external applications.
- AI Integration Tools - Connects local AI models to external system functions and tools to perform actions based on model outputs.
- Model Request Routing - Redirects inference requests to cloud providers when local hardware capacity is insufficient.
- Cross-Framework Model Conversion - Transforms external model formats into representations optimized for mobile and wearable hardware.
- Function Calling Interfaces - Parses model outputs into structured function calls to interact with external system tools.
- Hybrid Local-Remote AI Routing - Routes AI workloads between local on-device execution and cloud-based providers based on hardware capacity.
- Local Speech-to-Text - Includes a low-latency on-device transcription system for converting audio input into text.
- On-Device Speech Recognizers - Performs local speech-to-text transcription and voice activity detection on handheld and wearable devices.
- Voice Activity Detection - Identifies periods of human speech within audio streams to trigger transcription and downstream processing.
- Mobile Model Format Converters - Transforms external model formats into optimized representations compatible with local mobile and wearable hardware execution.
