floure
← All posts
The Hidden Engineering Behind Real-Time Dictation

The Hidden Engineering Behind Real-Time Dictation

·3 min read·Akshat Kotpalliwar

Real-time dictation has a hard constraint: text must appear before the user notices the delay.

Research puts that threshold around 300 milliseconds. Below that, things feel responsive. Above it, users start waiting. Above 500ms, they get annoyed. Above a second, they try again or give up.

floure's pipeline stays between 113 and 233 milliseconds end to end:

Stage Time
Audio capture (16kHz, 16-bit) 10ms
Ring buffer processing 5ms
VAD decision 3ms
Whisper inference (small.en) 80–200ms
Text post-processing 5ms
Output insertion 10ms
Total 113–233ms

Keeping it in that range requires optimization at every stage.

The audio buffer

Microphones produce a continuous stream of audio. Whisper processes fixed-size chunks. These two things do not match naturally.

floure uses a ring buffer - a fixed block of shared memory that bridges the gap:

  • The capture thread writes audio to the buffer continuously
  • The VAD thread reads the buffer independently to detect speech
  • The transcription thread picks up completed speech segments when the VAD signals an utterance
  • Old audio is overwritten by new audio in a circular pattern

The buffer holds 30 seconds of audio. All access uses atomic operations. No locks. No threads waiting on each other.

Whisper optimizations

Whisper is accurate, but the large model is slow on consumer hardware. Four techniques bring it into real-time range:

Quantization. Model weights are reduced from 32-bit floats to 16-bit or 8-bit integers. Memory bandwidth drops 50–75%. Inference runs nearly 2x faster on GPUs. Accuracy loss is under 0.5%.

Batching. Multiple audio chunks are sent to the GPU together. GPUs are designed for parallel work - processing 4 chunks at once is faster than processing them sequentially.

SIMD. On CPUs without discrete GPUs, SIMD vector instructions (AVX2 on x86, NEON on ARM) accelerate matrix multiplication by 3–4x. Whisper's attention mechanism is heavily matrix-math dependent.

Memory mapping. Model weights are memory-mapped from disk instead of loaded entirely into RAM. The OS pages in only the weights currently needed. Peak memory usage drops 40%.

Combined, these optimizations achieve 3x the speed of vanilla Whisper with identical word error rate.

Text output

Inserting text into the target application is more complex than it sounds. Different applications accept input differently:

  • Clipboard API - Fast, but some applications block clipboard access
  • Keyboard simulation - Works everywhere, but slower and can conflict with shortcuts
  • Accessibility APIs - Most reliable, but platform-specific

floure tries clipboard first, falls back to keyboard simulation, then accessibility APIs if needed. The fallback chain is automatic and completes in milliseconds.

Consistency over speed

A system that delivers text in 100ms but spikes to 800ms feels worse than a system that delivers text in 200ms consistently. Users adapt to rhythm. When the rhythm breaks, the experience breaks.

floure achieves consistent latency through CPU core pinning (the audio thread stays on one core), pre-allocated memory pools (no allocations during transcription), and Rust's memory model (no garbage collection pauses).

Every frame produces a result within the same time window. No spikes. No surprises.