What you’ll build
A system-wide dictation tool in the style of Wispr Flow: hold a hotkey, talk, let go, and polished text appears in whatever app has focus (email, Slack, a doc, a form). Under the hood: streaming speech-to-text, a quick cleanup pass, and a paste that puts your old clipboard back.
Drew’s own voice-assistant build includes this exact flow (hotkey → speech recognition → cleanup → editable preview → clipboard), with a local speech model as the default. This guide uses Deepgram for the cloud speech step and keeps a local fallback.
Architecture and data flow
- Hotkey down (a global shortcut, e.g. Ctrl+Shift+Space) starts microphone capture.
- Audio streams in small PCM chunks over a WebSocket to Deepgram’s live transcription endpoint; interim and final transcripts stream back.
- Hotkey up closes the stream and collects the final transcript.
- Cleanup: a short LLM prompt fixes punctuation, removes filler words and applies your style (“keep my wording; don’t add content”). If the model is unavailable, a deterministic fallback (capitalization and punctuation rules) still returns usable text.
- Paste: save the current clipboard, write the cleaned text, simulate paste into the focused app, then restore the saved clipboard.
- Optional: show an editable preview first, or append the text to a daily note.
Components and why
| Component | Choice | Why |
|---|---|---|
| App shell | Tauri (Rust + web UI) | Small native app; plugins for global shortcuts and the clipboard |
| Speech-to-text | Deepgram streaming (Nova family for dictation) | Low-latency streaming; smart formatting adds punctuation and numerals; keyterm prompting helps with names and jargon |
| Local fallback | whisper.cpp | Works offline and keeps audio on the device |
| Cleanup | A small, fast LLM, or a local model | Fixes filler words and punctuation without rewriting meaning |
| Paste | Clipboard plugin + OS paste keystroke | Works in any app without per-app integrations |
| Code home | GitHub | Versioned, reviewable changes from your coding agent |
Deepgram’s Flux model is built for voice agents and turn detection; for push-to-talk dictation the hotkey already marks the turn, so a Nova model with smart formatting is the simpler fit. Test both on your own voice.
Step outline
- Repo and spec: create a GitHub repo; list the hotkey, the apps you’ll test in, and a latency target.
- Hotkey + mic: register the global shortcut and capture audio while it’s held.
- Stream to Deepgram: open the WebSocket on key-down, send audio chunks, read transcripts; keep the API key in the OS keychain, never in code.
- Add keyterms for your company, product and people names.
- Cleanup pass with a strict prompt and a deterministic fallback.
- Clipboard dance: save → write → paste → restore. Test with images and rich text on the clipboard.
- Offline mode: route to whisper.cpp when there’s no network or when you choose “local only”.
- Package and test in your top five apps.
AI platforms that help
Any of the four coding agents can build this from a clear spec. Keep the code in a GitHub repository from day one so every agent change arrives as a reviewable diff.
| Platform | Where it helps most |
|---|---|
| Cursor | Building the UI with the agent in the editor, then checking it in the built-in browser tool |
| OpenAI Codex | Handing off well-specified pieces (parser tests, packaging) to run in the background |
| Claude Code | Terminal-first scaffolding, refactors and writing the test suite |
| Google Antigravity | Running parallel agents on separate parts (UI, storage, packaging) |
Costs
- Deepgram bills streaming transcription by audio duration. To verify: the current rate for your model on Deepgram’s pricing page.
- The cleanup pass costs a fraction of a cent per use with most small models. To verify with your provider.
- The local fallback has no usage cost.
Risks
- Privacy: audio leaves the device when you use cloud speech-to-text. Make the mode visible and offer local-only.
- Clipboard clobbering: always restore the previous clipboard, including non-text content.
- Over-editing: the cleanup prompt must not add facts or change meaning; show a diff or preview for long dictation.
- OS permissions: global hotkeys, microphone and simulated keystrokes need explicit user permission on each OS.
Sources
Deepgram, Tauri and MDN docs are the primary sources. Wispr Flow is cited only as the reference product category.