What you’ll build
A voice assistant you talk to naturally: it notices when you start and stop speaking, answers out loud within a second or so, can be interrupted, remembers what you allow it to, and can take actions through tools (calendar, tasks, deploys) with confirmation for anything risky.
Drew’s own desktop voice assistant uses this exact cascade: Deepgram streaming speech-to-text, Claude as the brain, and ElevenLabs streaming text-to-speech, with barge-in, a self-echo guard and a name check before actions. It aims for first audio in under 800 ms, stops speaking the moment you interrupt, and asks for a read-back confirmation before high-stakes actions. Those design choices shape this guide.
Architecture and data flow
- Wake: a local wake-word engine or a push-to-talk key opens the microphone. Keep wake detection on the device.
- Listen: audio streams to Deepgram Flux over
/v2/listen. Flux emits turn events:StartOfTurn,EagerEndOfTurn,TurnResumedandEndOfTurn. - Think: on
EndOfTurn(or early, onEagerEndOfTurn), send the transcript plus context to your LLM or agent. The agent can call tools, including MCP servers for your business apps. - Speak: stream the LLM’s tokens into ElevenLabs’ text-to-speech WebSocket as they arrive; send
flushat the end of the turn; play audio chunks as they land. - Barge-in: when Flux reports
StartOfTurnwhile the assistant is talking, stop playback, cancel the LLM request and drop the unspoken tail. - Remember: write short summaries and explicit preferences to a local store you can inspect and delete.
Managed alternative: Deepgram’s Voice Agent API runs listen, think and speak over one WebSocket (Settings, then streamed audio, with events such as UserStartedSpeaking for barge-in). It’s less work and less control.
Components and why
| Component | Choice | Why |
|---|---|---|
| Speech-to-text | Deepgram Flux | Model-integrated end-of-turn detection, configurable thresholds (eot_threshold, eager_eot_threshold, eot_timeout_ms) and a recommended 80 ms chunk size |
| Brain | An LLM or agent platform with tools | Claude, Codex-family models or your agent of choice; tools through MCP |
| Text-to-speech | ElevenLabs Flash over WebSocket | Flash models are documented at about 75 ms model latency; the WebSocket accepts streaming text from an LLM |
| Memory | Local SQLite | Inspectable, deletable, private by default |
| Secrets | OS keychain | Keys never sit in code or config files |
| Code home | GitHub | Reviewable changes; Actions runs your tests |
Step outline
- Spec and repo: define three voice tasks you actually want, the tools each needs, and which actions need confirmation.
- Text-first loop: build the agent with tools as a chat app first. Voice comes after the brain works.
- Add speech-to-text with Flux; log turn events and tune thresholds on your own voice and room.
- Add text-to-speech streaming with flush at end of turn; measure time to first audio.
- Barge-in and echo: stop audio on
StartOfTurn; mute or gate the microphone while the assistant speaks (a half-duplex guard) unless you have solid echo cancellation. - Latency budget: measure each hop (speech end → transcript → first token → first audio). ElevenLabs notes network round trips add 50–200 ms on top of model latency.
- Safety rails: read back and confirm before sending, paying, deleting or deploying; fall back to text when audio quality drops.
- Memory with consent: store only what the user approves; give a “forget that” command.
- Wake word last, once push-to-talk works well.
AI platforms that help
Any of the four coding agents can build this from a clear spec. Keep the code in a GitHub repository from day one so every agent change arrives as a reviewable diff.
| Platform | Where it helps most |
|---|---|
| Cursor | Building the UI with the agent in the editor, then checking it in the built-in browser tool |
| OpenAI Codex | Handing off well-specified pieces (parser tests, packaging) to run in the background |
| Claude Code | Terminal-first scaffolding, refactors and writing the test suite |
| Google Antigravity | Running parallel agents on separate parts (UI, storage, packaging) |
| Grok Bot | Not for building the app itself, but as an example of an agent that works in the background with approvals for sensitive steps |
Costs
- Deepgram bills streaming speech-to-text by audio duration; ElevenLabs bills text-to-speech by characters or credits. To verify: both pricing pages for your models and volume.
- LLM costs scale with conversation length and tool calls. To verify with your provider.
- A local wake-word engine and SQLite add no usage cost; check the wake-word engine’s license.
Risks
- Consent and recording: tell people when the microphone is live; don’t record others without consent where the law requires it.
- Voice cloning: only clone voices you have the right to use; ElevenLabs’ Prohibited Use Policy forbids cloning or impersonating a living person’s voice without their explicit and informed consent.
- Wrong actions: speech recognition errors can trigger the wrong tool call. Confirm high-stakes actions with a read-back.
- Always-on privacy: keep wake-word detection local and show a clear listening indicator.
- Latency creep: every tool call adds delay; say a short acknowledgment while a slow tool runs.
Sources
Deepgram and ElevenLabs official docs are the primary sources for the speech pipeline. Design targets come from Drew’s own build notes, shared without private details.