FastReseg: using transcript locations to refine image-based cell segmentation results in spatial transcriptomics

Abstract

Spatial transcriptomics (ST) is a rapidly advancing field, yet it is challenged by persistent issues with cell segmentation accuracy, which can bias biological interpretations by making cells appear more similar to their neighbors than they truly are. FastReseg introduces a novel class of algorithm that employs transcriptomic data not to redefine cell boundaries but to rectify inaccuracies within existing image-based segmentation outputs. By combining the rich information from image-based methods with the 3D precision of transcriptomic analysis, FastReseg enhances cell segmentation accuracy. A key innovation of FastReseg approach is its transcript scoring system, which scores each transcript for its goodness-of-fit within host cell using log-likelihood ratio. This scoring system facilitates the quick identification and correction of spatial doublets, i.e. cells erroneously segmented due to close proximity or spatial overlap in 2D. FastReseg approach offers several advantages: it reduces the risks of circularity in deriving cell boundaries from expression data and minimizes spatial-dependent biases arising from erroneous segmentation. It also addresses computational challenges often associated with existing transcript-based methods by introducing a heuristic, modular workflow that efficiently processes large datasets, a critical feature given the increasing size of spatial transcriptomics datasets. Its modular workflow allows for individual components to be optimized and seamlessly integrated back into the overall pipeline, accommodating ongoing advancements in segmentation technology. By enabling efficient management of large datasets and providing a scalable solution for refining cell segmentation, FastReseg is poised to enhance the quality and interpretability of spatial transcriptomics data even as underlying image-based cell segmentation techniques evolve.