RunanywhereAI/runanywhere-sdks

Features

• Local Model Execution - Enables the execution of large language and vision models locally on hardware for privacy and offline availability.
• On-Device Inference Engines - Implements a high-performance runtime for executing large language and vision models locally using hardware acceleration.
• Voice Pipelines - Implements a coordinated offline pipeline for voice activity detection, speech-to-text, and text-to-speech synthesis.
• Voice Interaction Management - Maintains a continuous voice session with automatic activity detection and event-based callbacks.
• AI Integration Tools - Connects local models to external functions and custom code via structured tool calling and argument parsing.
• Speech-to-Text Translation - Converts mono PCM audio into text using specialized speech-to-text models.
• Conversational Voice AI - Provides a coordinated pipeline for building hands-free assistants using voice activity detection, speech-to-text, and text-to-speech.
• Conversational Voice Pipelines - Coordinates voice activity detection, speech-to-text, and text-to-speech into a continuous conversational loop.
• GPU Acceleration - Utilizes hardware-specific GPU drivers to increase the processing speed of local AI models.
• Hardware-Accelerated Inference - Provides mechanisms to verify if the inference engine is utilizing hardware-accelerated GPU execution.
• Local Model Orchestrators - Orchestrates the lifecycle, hardware acceleration, and streaming execution of local machine learning models.
• Local RAG Implementations - Implements retrieval-augmented generation using private local datasets and offline LLMs for grounded answers.
• Local Model Lifecycle Managers - Provides tools for downloading, caching, and loading model assets to optimize local device storage and RAM.
• Multimodal Vision Inputs - Provides tools to process and interpret combined image and text inputs for visual analysis and descriptions.
• Natural Language Generation - Produces human-like natural language responses from prompts using configurable temperature and token limits.
• On-Device Models - Ships a comprehensive SDK for running large language, vision, and speech models locally on edge hardware.
• Real-Time Audio Transcribers - Transcribes live audio input and emits partial results for immediate user feedback.
• Incremental Synthesis - Produces audio chunks incrementally so playback begins before the entire synthesis task completes.
• Local Speech Synthesis - Converts text into natural-sounding audio using neural models running locally on-device.
• Tool Calling - Connects on-device models to external functions via typed tool definitions and automated execution.
• Vector Embeddings - Provides vector embeddings of text to enable semantic search and memory operations directly on-device.
• Vector Retrieval Systems - Implements local document chunking and similarity search via embedding models to provide grounded AI responses.
• Vision-Language Inference - Executes multimodal models that process combined image and text inputs to generate analytical descriptions.
• Voice Activity Detection - Identifies when a user starts and stops speaking using configurable sensitivity thresholds.
• Voice Conversational Loops - Coordinates a single cycle of audio-to-text, AI reasoning, and text-to-audio synthesis.
• Local Model Loading - Handles the downloading of model files from remote URLs and loading them into device memory.
• Multimodal Analysis Engines - Implements a multimodal analysis engine to process visual content and text prompts together for image description.
• Word-Level Timestamps - Tracks start and end times for individual words to synchronize text and audio playback.
• Retrieval-Augmented Generation - Implements a retrieval-augmented generation pipeline that grounds AI responses using locally indexed documents.
• Argument Parsing - Extracts tool names and arguments from raw model text for manual execution logic.
• Diffusion Model Managers - Manages the registration and downloading of diffusion model packages for local image generation.
• Inference Telemetry - Tracks tokens used and generation speed to monitor local model performance and efficiency.
• Image Generation - Creates images from text prompts using hardware-accelerated diffusion models on-device.
• Model Preloading Endpoints - Loads specific models into memory during idle time to reduce latency when starting a user task.
• Incremental Inference Streaming - Streams model outputs token-by-token using asynchronous iterators to reduce perceived user latency.
• Model Adapters - Injects lightweight LoRA adapters into base models at runtime to change behavior without reloading full weights.
• Behavioral Constraints - Defines personas, operational constraints, and response formats for models through system prompts.
• Model Persistence - Caches downloaded models in persistent storage to prevent redundant network requests.
• LoRA Adapter Loaders - Applies lightweight LoRA adapters to base models at runtime to modify behavior without full reloading.
• Structured Output Generators - Constrains model responses to specific JSON schemas to ensure machine-readable and type-safe outputs.
• Speaker Diarization - Detects and labels different speakers within a single recording to distinguish between voices.
• Concurrent Model Execution - Supports the simultaneous execution of multiple models in memory to facilitate complex processing pipelines.
• Text Generation Controls - Provides controls for adjusting creativity, token limits, and streaming behavior during text generation.
• Model Memory Reclamation - Removes unused large language or speech models from memory to reclaim system resources.
• Document Ingestion Pipelines - Provides a pipeline for chunking, embedding, and indexing raw documents to facilitate local vector search.
• Schema-Constrained Sampling - Enforces structured JSON or XML output formats by constraining token sampling during the model generation process.
• Raw Audio Captures - Records raw audio at specified sample rates, providing audio chunks and volume levels for AI processing.
• Camera Feed Capture - Accesses the device camera to provide live RGB frames for analysis by on-device vision models.
• Token Streaming - Delivers AI model generated tokens to the user interface in real-time via asynchronous streams.
• Event Monitoring Streams - Provides an event stream to track AI-specific lifecycle events, including generation status and model loading.
• Inference Performance Monitoring - Tracks inference-specific metrics such as tokens per second, latency, and time to first token.
• Model Serving & Deployment - Runs AI models on-device for mobile platforms.

Star history