Reengineering the Galois System: Toward Reproducible and Generalizable LLM‑Based Structured Retrieval

In this thesis, we present the GaloisPy system, a software tool written in Python that re-implements the Galois system, a Java application developed by a team of researchers from the University of Basilicata in collaboration with the French research institute EURECOM for retrieving structured data from large language models, or LLMs, using SQL syntax. GaloisPy introduces several improvements and optimizations designed specifically to enhance its performance, such as including chat history management, dynamic Pydantic models for structured outputs, and prompt rewriting from SQL-like language to natural language, with the last being crucial for the system's accuracy. In this work, the results of various experiments conducted will be presented, comparing the performance of GaloisPy against the original Galois system using the GPT-4.1 nano model by OpenAI. These tests analyze both systems’ performance in different scenarios, including one where data is generated solely using LLM internal knowledge and another employing the Retrieval-Augmented Generation, or RAG, strategy. From the results obtained, it is evident that GaloisPy achieves significantly better performance compared to the original, as indicated by the metrics used. However, it also has some limitations, such as the ineffectiveness of the Key-Scan strategy compared to the Table-Scan, and challenges in managing complex queries, especially those involving JOIN clauses. Finally, in the RAG scenario, it was observed that using a reranking model, coupled with the small size of the LLM used, can reduce overall performance.