Evaluating FastReseg for Transcript-Based Segmentation Refinement in Spatial Transcriptomics

Spatial transcriptomics enables high-resolution mapping of gene expression within intact tissue sections while preserving spatial context. A critical step in spatial transcriptomics analysis is cell segmentation, as segmentation accuracy directly determines transcript assignment and downstream cell-level interpretation. However, image-based segmentation methods commonly used in these platforms remain sensitive to tissue morphology, staining quality, and cellular density. This sensitivity leads to segmentation errors that are not always apparent from morphology alone. In this thesis, the FastReseg post-segmentation refinement framework was applied to spatial transcriptomics data generated using the NanoString CosMx Spatial Molecular Imager. FastReseg operates on segmented cells and leverages transcript-level spatial information to detect and correct segmentation errors without performing de novo segmentation. Transcript assignment consistency was quantified using log-likelihood ratio scoring based on reference expression profiles. Subsequently, spatial dependency modeling was employed to identify cells exhibiting structured patterns of poorly fitting transcripts indicative of segmentation errors. Cells flagged by this procedure were further analyzed at the transcript level using support vector machine-based spatial modeling to identify and group misassigned transcripts. These transcript groups were subsequently refined through conservative, rule-based operations. These operations included trimming, merging with neighboring cells, or the creation of new cell identities when supported by strong spatial and transcriptional evidence. The results demonstrate that segmentation errors are relatively rare but spatially structured, occurring preferentially in regions of high cellular density and at interfaces between transcriptionally distinct cell populations. Importantly, visually plausible segmentations were shown to harbor biologically inconsistent transcript assignments that could only be detected through transcript-level spatial analysis. Refinement actions were highly targeted, affecting only a small fraction of cells and transcripts while preserving the overall segmentation structure. Overall, this work highlights the value of transcript-informed post-segmentation refinement for improving transcript assignment accuracy in spatial transcriptomics data. The FastReseg framework provides a scalable and conservative approach to segmentation quality control that complements image-based methods and enhances the reliability of downstream spatial analyses.