How to read a decoding paper

Every week a paper or a press release announces that brain signals have been decoded into text. Most are true and few mean what the headline implies. The gap between the two lives in five pieces of small print, and once you know them you can read any result in this field in about a minute, including ours.

First: which number is being reported

Character error rate counts the edits, substitutions, deletions and insertions, needed to turn the guess into the true sentence, divided by the sentence's length. Word accuracy is the same arithmetic at word level and reads higher because whole words either survive or do not. They are not comparable to each other and neither is comparable to accuracy on a two-choice task, where coin-flip performance is already fifty percent. Translate every claim to one question: what would chance score here? A seventy percent result on ten candidates is modest; a seventy percent result on open vocabulary is extraordinary.

Ask what chance would score. Then ask who else saw the test set.

Second: whose brain, which session

The most common trap is the within-session test. Train on the morning's recording, test on a held-out ten minutes of the same session, same person, same electrode positions, same electrode gel drying at the same rate, and you will report a lovely number that says nothing about Tuesday. The honest ladder has three rungs: within-session, cross-session with repositioned electrodes, and cross-subject with no personal data at all. Every rung down costs accuracy, and the drop is the actual research problem. A paper that does not state which rung it stands on is telling you something by omission.

Third: what helped across the finish line

Modern pipelines score their output with a pretrained language model, and Meta's Brain2Qwerty is admirably explicit that the language model in the loop is load-bearing. That is legitimate; a decoder that knows the statistics of English is decoding brain signal plus prior, and users would want it that way. But a reader should know when rescoring is included, because the neural decode alone is a different and worse number, and the difference measures how much of the sentence the brain signal actually carried.

Fourth: the size of the stage

Count participants. Two or three highly trained subjects is a demonstration; dozens of naive subjects is a result. Count the vocabulary: a closed set of twelve commands is a classification problem, genuinely useful and much easier than open sentences. Count the machine: the Brain2Qwerty headline numbers, sixty-one percent average word accuracy with a best participant at seventy-eight, come from magnetoencephalography, a room-sized, shielded, million-euro instrument. The wearable-EEG version of the same experiment sits near sixty-five percent character error. Both numbers are honest; only one of them ships.

Fifth: what the model was actually pretrained on

MANAS-1 deserves its reputation: sixty thousand hours, a four-hundred-million-parameter EEG foundation model that transfers across recording setups. But its pretraining is general EEG, sleep, pathology, rhythms, not language-paired speech. Its existence proves the foundation-model recipe works on this signal; it does not put words on the screen. That final step needs what no one has collected, which is the whole reason our Phase 1 exists. We will read every paper in this field with the five questions above, and we invite you to read ours the same way.