xinnan-tech/xiaozhi-esp32-server
Xiaozhi Esp32 Server
This project is an AI voice assistant backend and gateway server designed to connect ESP32 hardware to large language models. It enables real-time conversational AI by processing streaming speech-to-text and text-to-speech interactions, allowing hardware devices to engage in natural language dialogue.
The system is distinguished by a modular plugin framework that loads custom feature extensions at runtime and a retrieval-augmented generation engine that queries external knowledge bases for factual accuracy. It further personalizes interactions by using voiceprint mapping to identify individual speakers and maintain long-term contextual memory.
The platform covers a broad capability surface including IoT hardware control via function calling, secure device authentication using bearer tokens, and full-duplex communication through WebSockets. It also provides a web interface for system and device management, overseeing configuration and gateway traffic routing.
Features
- Conversational Voice AI - Implements an end-to-end conversational AI system integrating STT, LLMs, and TTS for hardware devices.
- Knowledge Base Retrieval - Integrates a retrieval-augmented generation engine that queries external knowledge bases to ensure factual accuracy.
- LLM Orchestrators - Orchestrates the connection between large language models, external plugins, and IoT hardware for multi-modal interaction.
- LLM Tool Calling - Maps natural language intents to executable functions for hardware control and data retrieval.
- Full-Duplex Multimodal Interaction - Maintains persistent bidirectional connections for real-time streaming of audio and control data.
- Conversational Dialogue Systems - Implements a system for natural, real-time conversational dialogue between users and AI-powered hardware.
- Personalized Assistants - Implements voiceprint mapping to identify individual speakers and deliver personalized responses and memory.
- LLM Gateways - Acts as a dedicated gateway server connecting ESP32 hardware to large language models for voice interactions.
- IoT and Hardware Control - Sends programmatic commands from a central server to control physical components on ESP32 devices.
- Speech-to-Text Pipelines - Processes spoken audio in real-time chunks to minimize latency during voice-to-text conversion.
- Streaming Pipelines - Provides real-time streaming speech-to-text and text-to-speech processing for fluid voice interactions.
- Device Management - Manages security keys and gateway routing to maintain secure connections for a fleet of IoT devices.
- Speaker Verification - Matches incoming audio signatures against stored voiceprints to verify and identify the speaker.
- Speaker Identification Frameworks - Identifies individual users by their voiceprints to deliver personalized AI responses.
- Remote Hardware Controllers - Enables control of physical ESP32 hardware states through function calling and communication protocols.
- Communication Gateways - Routes device communication through MQTT and UDP gateways using dynamically delivered endpoints.
- Bearer Token Authentication - Uses unique bearer tokens and identifiers to secure the communication between ESP32 devices and the server.
- Device and Connection Authorization - Authenticates hardware devices via tokens and unique IDs to ensure secure persistent connections.
- Extensible Plugin Architectures - Provides a modular framework for extending system functionality via runtime plugins.
- Intent-to-Skill Mappings - Translates natural language user intents into specific programmatic actions and tool executions.
- Runtime Capability Extensions - Allows the dynamic loading of modular feature sets at runtime without restarting the server.
- Modular Plugin Frameworks - Features a modular framework that loads custom feature plugins at runtime to extend device capabilities.
