TEN-framework/ten-framework
Ten Framework
Ten Framework is a multimodal large language model agent framework designed for building low-latency conversational agents. It integrates voice, text, and visual inputs in real time to facilitate human interaction.
The project includes a real-time speech processing pipeline for streaming transcription, voice activity detection, and speaker diarization. It also features an avatar synchronization engine that coordinates character lip animations and visual outputs with synthesized speech.
The framework covers edge AI deployment through containerized packaging and direct integration with embedded hardware boards. Additional capabilities include a telephony gateway for connecting agents to phone networks via the Session Initiation Protocol and tools for real-time visual generation of sketches and doodles.
Features
- Real-Time Conversational AI Frameworks - Provides a low-latency framework for building multimodal conversational agents that integrate STT, LLMs, and TTS.
- Voice Activity Detection - Identifies speech segments within audio streams to manage natural turn-taking and conversational flow.
- Real-Time Transcription - Converts live audio input into text representations in real time for downstream language model processing.
- Multimodal AI Orchestrators - Coordinates multiple AI model types including vision, speech, and language into a unified context for agentic workflows.
- Multimodal Conversational Interfaces - Provides a framework for building conversational agents that integrate text, voice, and audio processing for real-time interaction.
- Full-Duplex Multimodal Interaction - Processes simultaneous audio and data flows to enable fluid, real-time conversational turns and natural interruptions.
- Real-Time Speech Processing - Implements a real-time speech processing pipeline for streaming transcription, voice activity detection, and speaker diarization.
- Unified Speech Pipelines - Ships a unified speech pipeline integrating transcription and activity detection for bidirectional voice interaction.
- Edge AI Model Deployment - Runs conversational logic and agent services on embedded hardware and edge devices for low-latency interaction.
- Speaker Diarization - Analyzes real-time audio streams to detect, separate, and label multiple distinct speakers.
- Containerized Deployments - Packages customized agent services into images for consistent execution across cloud platforms and virtual machines.
- Containerized Packaging - Bundles customized agent configurations into portable images for consistent deployment across cloud and edge environments.
- Embedded Hardware Deployment - Integrates agent logic directly onto embedded development boards for communication-driven physical device interaction.
- Edge AI Deployment Pipelines - Provides tools for packaging conversational agent logic into containers and deploying them onto embedded hardware boards.
- Hardware Integration - Allows agent logic to run directly on embedded development boards for physical device interactions.
- Audio-Visual Signal Alignment - Coordinates synthesized audio output with avatar lip movements to ensure precise temporal alignment.
- Modular Provider Interfaces - Implements a pluggable provider model to decouple multimodal capabilities like speech-to-text and avatar animation from core logic.
- Lip Synchronization Engines - Coordinates character avatar mouth movements with synthesized audio output for realistic speaking digital humans.
- Conversational Avatar Animators - Synchronizes character movements and facial expressions with LLM-driven conversational audio.
