Augmentation and Ensembles: Improving Medical Image Segmentation with SAM and Deep Networks

This study explores novel ensemble strategies to improve image segmentation performance, particularly on medical image data. We investigate how the Segment Anything Model (SAM), despite not being explicitly trained for medical image segmentation, can still produce relevant information for model training. Building on these insights, we propose augmentation techniques that integrate SAM information directly into the images, enhancing the learning process of segmentation models. Each proposed augmentation method comes with its unique advantages, thereby to leverage the strengths of each approach, we introduce AuxMix, a model trained with three distinct SAM-based augmentation techniques. We conduct experiments on the state-of-the-art models, evaluating the effects of each technique independently and within an ensemble framework. The results show that our ensemble strategy, combining complementary information from each augmentation, leads to a robust and improved segmentation performance.