Scalable Cloud-Agnostic architecture for IoT data analysis and machine learning

The rapid expansion of the Internet of Things (IoT) has introduced significant challenges in managing and analyzing the vast amounts of data generated by interconnected devices. This thesis focuses on developing a scalable, cloud-agnostic architecture to efficiently handle data from IoT devices. The core objective is to create an architecture that seamlessly integrates with various cloud platforms—such as Aruba Cloud, AWS, and Microsoft Azure—while supporting both horizontal and vertical scaling. The system must be capable of ingesting data via the MQTT protocol, managing data from online devices and archive data sources, and performing complex data preprocessing and analysis, including machine learning tasks. Security is a critical consideration, with the architecture designed to protect data both in transit and at rest. To achieve these goals, the thesis involves a detailed examination of existing technologies, an analysis of different cloud providers, and the definition of a flexible base architecture. Practical testing on a real case study is used to validate the architecture’s performance and scalability. The proposed solution addresses the growing demands of IoT data management and provides a robust framework for future applications in diverse cloud environments.