Reinforcement Learning from Human Feedback to Fine-Tune Vision Models

Recent advances in Reinforcement Learning from Human Feedback (RLHF) have demonstrated the effectiveness of human-guided fine-tuning in aligning large language and vision-language models with human intent. This inspires the idea that this method can be useful also to train or fine-tune smaller vision models for more specialized tasks that may have a larger applicability in industrial settings. This thesis investigates how sparse human qualitative evaluations and preference signals can be seamlessly integrated into the training and adaptation of vision models, thereby reducing dependence on costly pixel-level annotations. We propose a unified framework that embeds human interactions directly into learning loops via reinforcement learning techniques and reward modelling. Through a series of experiments, we show that models trained with human feedback adapt more rapidly and robustly to novel visual scenarios, achieving significant improvements in label efficiency without sacrificing performance. The final goal is to use RLHF methodologies to adapt vision models for novel domains, enhancing their practical deployment potential.