Neuron Recognition and Characterization in Cultures: An Instance Segmentation-based Approach

In this thesis, an instance segmentation-based framework is developed for the recognition and characterization of neurons in microscopy images. The methodology consisted of two main components: an instance segmentation model based on the YOLOv7 architecture, and a generative adversarial network (Wasserstein GAN with Gradient Penalty, WGAN-GP) for generating synthetic binary masks. The dataset, consisting of microscopy images with annotated samples, was preprocessed and partitioned for both segmentation and generation pipelines. YOLOv7 was trained on a custom-labeled dataset, and its performance was quantitatively evaluated using standard metrics, including precision, recall, F1-score, mean Average Precision (mAP), and Intersection over Union (IoU). Qualitative evaluations were also conducted to assess prediction behaviour under varying visual conditions. These evaluations confirmed the validity of the YOLOv7-based segmentation model, which demonstrated strong detection and localization capabilities even in visually complex neuronal cultures. To address the limited availability of annotated data and enhance model generalization, a WGAN-GP model is implemented to generate realistic binary masks simulating neuron morphology. The generative model was trained, and its output was analyzed across training stages to ensure structural diversity and convergence stability. Post-processing steps are applied to binarize and validate the generated outputs. The results indicates GAN-generated masks displayed structurally coherent and diverse neuron-like patterns. Overall, the thesis contributes two validated modules: one for real-data segmentation and another for synthetic mask generation, each supporting future augmentation strategies.