Systematic Spike Sorting Errors Revealed by Juxtacellular Ground Truth in the Awake Brain

Existing ground-truth data for spike sorting come almost entirely from anesthetized or in vitro preparations. Yet phenomena expected to challenge spike sorting most (spike overlap and bursting) depend on brain state and differ in the awake brain. Therefore, error estimates from existing benchmarks may not transfer to awake recordings, and the types of errors in the awake brain remain largely uncharacterized. To address this, we generated a ground-truth dataset (16 neurons) using paired juxtacellular-extracellular electrophysiology in awake mice, and found spike sorting performance, particularly recall, to be lower than conventional benchmarks might suggest. Generally, false negatives sharply increased with intrinsic physiological factors like bursting, whereas false positives were more strongly associated with extrinsic factors, such as putative spike overlaps. To quantify contributions to errors from intrinsic and extrinsic factors, for each observed juxtacellular waveform we predicted the corresponding extracellular waveform. In the best-performing sorter, we estimate approximately half of false positives were attributable to credulous attempts to resolve putative overlaps. Bursting and putative overlaps were each associated with a roughly doubled false negative rate when they occurred separately, and with a sevenfold increase when they occurred together. These co-occurrences accounted for approximately 40% of a typical neuron's false negatives, despite comprising only approximately a fifth of its spikes. Thus, the moments when spike sorters must rely on the assumption of waveform invariance the most to resolve overlaps are exactly the moments when bursting makes that assumption least valid. We also estimate that three-quarters of false negatives had been detected but assigned to other units. These spikes were not cleanly oversplit: post-hoc merging was not productive because the loss in precision was three times the gain in recall. Because bursting and spike overlaps often co-occur, sorting spikes by waveform similarity and repairing the result by post hoc merging is unlikely to adequately balance precision and recall in the situations of greatest scientific interest. That is, while bursting and spike overlaps may involve only a minority of spikes, the resulting sorting errors may disproportionately distort the interpretation of neuronal population dynamics. We thus conclude that uncertainty quantification in sorters' output is essential.