An Empirical Study of Decentralized Fine-tuning for Machine Learning Models

In a world where data collected by edge devices is of high interest, gathering it at a central point has become more challenging due to current data protection regulations. Decentralized paradigms, such as federated and gossip learning, have gained popularity as a solution to train models while avoiding to share raw data. In decentralized machine learning, edge devices train a local version of the model on their own private data. Moreover, local models are then iteratively shared and merged to produce an aggregated model. Federated learning has been proposed as an initial solution, which relies on a central server for the aggregation. In gossip learning, on the other hand, the model sharing and aggregation is performed in a peer-to-peer fashion, giving peers more control over which peers to collaborate with. Previous studies tested feasibility and limits of decentralized machine learning training models from scratch, which requires access to a large pool of data. In this thesis, we analyze a different aspect of decentralized training, exploring the case of decentralized fine-tuning of machine learning models with a limited amount of available data. Our research involves extensive experimentation with a number of model architectures and datasets, considering different data distributions and imbalances to simulate a real-life setup, demonstrating its applicability across different scenarios.