AI-Enhanced Monitoring for Data Warehouses

Modern Data Warehouses (DWH) and Data Lakes rely on intricate ETL workflows and interconnected data platforms, making them susceptible to infrastructure failures, application anomalies, and performance bottlenecks. Traditional monitoring approaches often struggle to provide proactive insights, leading to operational inefficiencies and increased maintenance costs. This thesis focuses on the development of AI-driven monitoring models, leveraging both custom-built and pre-existing architectures to enhance anomaly detection, failure prediction, and automated response strategies. A key component of this research is optimizing prompt engineering to improve AI interpretability and decision-making. Additionally, vector databases are explored for efficient knowledge base management, enabling fast and context-aware retrieval of historical insights. By integrating these techniques, the proposed solution aims to enhance monitoring accuracy, reduce downtime, and provide intelligent, real-time recommendations for issue resolution. The effectiveness of the approach is validated through case studies in large-scale data environments, assessing improvements in response times, error mitigation, and system stability.