Grounding and Robustness Evaluation of Image-Based Surgical Visual Question Answering Models

Multimodal Large Language Models (MLLMs) are applied to Visual Question Answering (VQA) in endoscopic and surgical imaging. However, standard benchmarks assess whether a generated response matches a reference answer, without establishing whether the prediction depends on visual evidence or on linguistic priors, candidate-option structure and output-processing conventions. This thesis investigates how accuracy on EndoBench relates to visual dependence, decision stability and reliance on the answer interface. Following an initial analysis of Surgical VQA datasets, ColonGPT, LLaVA-Med-7B and HuatuoGPT-Vision-7B are evaluated through a protocol-aligned reproduction of EndoBench. Their baseline predictions are examined using cross-task random image replacement, random and cyclic option-order perturbations, multiple -choice versus free-form generation and target-region occlusion, complemented by qualitative attribution analysis. Results reveal heterogeneous behavioral profiles that aggregate accuracy alone does not capture. Image replacement substantially affects ColonGPT and HuatuoGPT\allowbreak-Vision\allowbreak-7B, whereas the limited net change observed for LLaVA-Med-7B hides opposing question-level transitions. Option permutations produce prediction variability despite small changes in average accuracy, while free-form generation substantially reduces performance for LLaVA-Med-7B and HuatuoGPT-Vision-7B and exposes output-format failures. Target-region occlusion may indicate region specific reliance for HuatuoGPT-Vision-7B in coordinate-based region recognition, but yields limited effects for the other models and less conclusive results in the Visual Grounding task. The thesis also identifies inconsistencies affecting the interpretation of EndoBench results and develops a behavioral evaluation pipeline based on controlled interventions. The findings indicate that benchmark accuracy is a composite measure that should be complemented with evidence of visual dependence and robustness to non-semantic perturbations. The source code and evaluation pipeline are available at https://github.com/Carraro-Riccardo/Surgical-VQA-Robustness.