LLM-driven automation for insurance operations: agentic graph RAG for ticket resolution, document pre-labeling and extraction, and semantic FAQ retrieval

This thesis collects the work carried out during an internship at Leitha, the technology company of the Unipol group, and explores how Large Language Models and Generative AI techniques can be applied to three concrete problems in the insurance domain: the handling of support tickets, the extraction of data from documents, and semantic information retrieval. Across all three, the guiding thread is a systematic comparison between proprietary and open source models, aimed at assessing their quality, their cost, and their feasibility in a real production setting. The first project is a library for the analysis, summarization, classification, and documental grounding of support tickets for the BeRebel product. Here the entire reference documentation is kept available to the model through context caching, while a Graph-RAG workflow orchestrated in LangGraph handles technical diagnosis by navigating an internal knowledge base across multiple hops. A LiteLLM abstraction layer keeps the whole system independent from any specific model provider. The second project concerns the automatic extraction of information from Italian identity cards. A ground truth of about two hundred documents was annotated by hand, starting from an automatic pre-labeling produced with Gemini and then refined in Label Studio. On top of this dataset an internal Python library was developed, exposing an extraction API driven by Pydantic schemas and dedicated prompts, together with a dashboard for visual comparison. The evaluation is performed field by field as an exact match and is tracked with MLflow, comparing Gemini models against open source ones such as Qwen and Gemma on accuracy, recall, and F1. The third project is a comparative evaluation of semantic retrieval architectures over the BeRebel corporate FAQs. It contrasts bi-encoder and late interaction approaches, with and without cross-encoder re-ranking, weighing retrieval quality against latency. The results show that the proprietary gemini-embedding-001 model leads the ranking, and that a modern multilingual re-ranker can meaningfully improve the open source retrievers. Taken together, the work provides empirical evidence to support design choices for adopting Generative AI in the insurance sector, and shows how open source models often stand as a competitive and sustainable alternative to proprietary ones.