Building blocks for voice-to-voice AI

This talk demonstrates building fast, reliable voice-to-voice AI bots using open source tools, covering key components and showing 50 lines of core code.

Overview

Building fast, reliable conversational voice bots on top of today’s generative AI models and tooling requires combining a number of technologies. Major components include network transport, audio compression and processing, text-to-speech, multi-turn LLM inference, speech-to-text, tool use, and interruption handling. We’ll start with a demo of a voice bot built with Open Source libraries running at 500ms voice-to-voice latency. Then we will do a lightning tour of the demo bot’s components and 50 core lines of code.

Links

https://github.com/rtvi-ai
RTVI-AI defines an open standard for real-time voice/video AI inference.
https://x.com/kwindla/status/1815892810584723609
Groq accelerates Llama 3.1 voice bots to 500ms voice-to-voice latency.

Tech stack