Live Brain-to-Text Decoding
Streaming attempted speech from intracortical signals into text
Overview
Live Brain-to-Text Decoding
Streaming attempted speech from intracortical signals into text
I built a real-time decoder for the Brain-to-Text '25 dataset. Four implanted microelectrode arrays supply 512 neural features every 20 ms; a GRU acoustic model turns those signals into phoneme probabilities, and an n-gram WFST search produces a revisable text transcript every 80 ms. An optional GPT-2 pass re-ranks candidates after the sentence ends.
This is research on one participant's attempted speech. The accuracy figures below are exploratory results on a development split with calibrated recording-day layers and dataset-provided sentence endpoints, not independent test or clinical performance.
Watching the neural input stream
The live harness plots threshold crossings and spike-band power from selected electrodes while a trial plays at its 20 ms data cadence. The model uses 512 such features across 256 electrodes, followed by causal smoothing, recording-day calibration, and 280 ms patches. This is the actual harness electrode inspector, using the released processed features rather than raw spike times.
From phoneme probabilities to live words
A five-layer GRU predicts a distribution over 39 phonemes, a word boundary, and CTC blank every 80 ms. A 4-gram WFST search turns those probabilities into partial text that can revise earlier words. The screenshot shows one in-progress copy-task sentence, its greedy phoneme stream, and the frame-processing lag; that lag is not word latency.
Accuracy across recording sessions
The harness reports pooled word error rate, per-session variation, transcript revisions, and diagnostic phoneme error rate. The reference causal pipeline reached 8.51% WER with 4-gram search on 1,253 val-dev sentences. Checkpoints and decode settings were selected using validation data, and sentence endpoints come from the dataset, so this is exploratory development-split accuracy.
Measuring the delay instead of assuming it
A paced replay feeds the decoder every 20 ms and measures frame-window-to-output lag. In this recorded reference run the median was 9.48 ms and the 95th percentile 13.45 ms; first possible output still waits for the 280 ms input patch. These are processing measurements on a replay, not end-to-end communication or word latency.
Re-ranking after a sentence ends
The optional GPT-2 pass re-ranks the WFST's sentence-final candidates, reaching 6.87% WER on the same val-dev scope. That improvement arrives after the dataset's trial endpoint. The harness also examines when partial words can be committed without later retraction, exposing the accuracy cost of showing a word as final too early.
Additional context
Trading future context for delay
I compared causal, bidirectional, bounded-look-ahead, and sliding-window GRUs under a shared 20,000-batch schedule and decode grid. A look-ahead model using 18 future frames (1.44 s of future context) reached 7.50% val-dev WER, versus 8.71% for its zero-look-ahead capacity control at the same acoustic scale. This is an exploratory accuracy-versus-delay experiment, not a measured live deployment result; shorter look-ahead gains were smaller and less certain.
A working path from intracortical features to continuously updated text, with live decoding, sentence-end re-ranking, and future-context experiments evaluated separately.