Enterprises increasingly rely on customer support systems to retain users and gain insight into product issues, yet scaling human support operations remains costly. To address this challenge, companies often use Retrieval-Augmented Generation (RAG) to automate responses to customer tickets based on existing documentation. However, RAG is limited when the information needed to resolve a ticket changes frequently, is highly specific to an individual customer, or is poorly documented. To address these limitations, this thesis proposes and evaluates two agentic extensions to an existing RAG-based customer support system. The first extension uses a tool-augmented language model to query additional resources, such as source code and customer account information, and injects the results into the response-generation process as additional context. The second uses an end-to-end agent guided by several playbooks to investigate tickets and generate responses independently of the RAG system. To evaluate whether these additional capabilities improve support performance, the three approaches were tested through controlled replay experiments on one hundred support tickets and live monitoring in a production environment. In the replay experiments, an LLM-based evaluator strongly preferred both agentic configurations over the baseline RAG system. These improvements, however, came at the cost of higher latency and computational requirements. The observations from the live environment were consistent with the controlled results, with the end-to-end agent achieving more favorable customer-response signals than the other systems. Taken together, these results indicate that tool-assisted investigation can improve automated customer support when the required information is dynamic, customer-specific, or absent from the knowledge base. The results also suggest that extending an existing RAG pipeline with additional context can capture much of the benefit, while an end-to-end agent is often less expensive and can provide additional value for more complex inquiries. Building on these findings, future work should explore cost-aware routing policies, experiments with human validation of the evaluator, and mechanisms for isolating and controlling individual agent capabilities to determine which tools affect performance in practice.
Enterprises increasingly rely on customer support systems to retain users and gain insight into product issues, yet scaling human support operations remains costly. To address this challenge, companies often use Retrieval-Augmented Generation (RAG) to automate responses to customer tickets based on existing documentation. However, RAG is limited when the information needed to resolve a ticket changes frequently, is highly specific to an individual customer, or is poorly documented. To address these limitations, this thesis proposes and evaluates two agentic extensions to an existing RAG-based customer support system. The first extension uses a tool-augmented language model to query additional resources, such as source code and customer account information, and injects the results into the response-generation process as additional context. The second uses an end-to-end agent guided by several playbooks to investigate tickets and generate responses independently of the RAG system. To evaluate whether these additional capabilities improve support performance, the three approaches were tested through controlled replay experiments on one hundred support tickets and live monitoring in a production environment. In the replay experiments, an LLM-based evaluator strongly preferred both agentic configurations over the baseline RAG system. These improvements, however, came at the cost of higher latency and computational requirements. The observations from the live environment were consistent with the controlled results, with the end-to-end agent achieving more favorable customer-response signals than the other systems. Taken together, these results indicate that tool-assisted investigation can improve automated customer support when the required information is dynamic, customer-specific, or absent from the knowledge base. The results also suggest that extending an existing RAG pipeline with additional context can capture much of the benefit, while an end-to-end agent is often less expensive and can provide additional value for more complex inquiries. Building on these findings, future work should explore cost-aware routing policies, experiments with human validation of the evaluator, and mechanisms for isolating and controlling individual agent capabilities to determine which tools affect performance in practice.
From RAG to Tool-Augmented Agents: Design and Evaluation of a Two-Tier LLM Customer Support System
VISONÀ, FRANCESCO
2025/2026
Abstract
Enterprises increasingly rely on customer support systems to retain users and gain insight into product issues, yet scaling human support operations remains costly. To address this challenge, companies often use Retrieval-Augmented Generation (RAG) to automate responses to customer tickets based on existing documentation. However, RAG is limited when the information needed to resolve a ticket changes frequently, is highly specific to an individual customer, or is poorly documented. To address these limitations, this thesis proposes and evaluates two agentic extensions to an existing RAG-based customer support system. The first extension uses a tool-augmented language model to query additional resources, such as source code and customer account information, and injects the results into the response-generation process as additional context. The second uses an end-to-end agent guided by several playbooks to investigate tickets and generate responses independently of the RAG system. To evaluate whether these additional capabilities improve support performance, the three approaches were tested through controlled replay experiments on one hundred support tickets and live monitoring in a production environment. In the replay experiments, an LLM-based evaluator strongly preferred both agentic configurations over the baseline RAG system. These improvements, however, came at the cost of higher latency and computational requirements. The observations from the live environment were consistent with the controlled results, with the end-to-end agent achieving more favorable customer-response signals than the other systems. Taken together, these results indicate that tool-assisted investigation can improve automated customer support when the required information is dynamic, customer-specific, or absent from the knowledge base. The results also suggest that extending an existing RAG pipeline with additional context can capture much of the benefit, while an end-to-end agent is often less expensive and can provide additional value for more complex inquiries. Building on these findings, future work should explore cost-aware routing policies, experiments with human validation of the evaluator, and mechanisms for isolating and controlling individual agent capabilities to determine which tools affect performance in practice.| File | Dimensione | Formato | |
|---|---|---|---|
|
Visona_Francesco.pdf
Accesso riservato
Dimensione
4.12 MB
Formato
Adobe PDF
|
4.12 MB | Adobe PDF |
The text of this website © Università degli studi di Padova. Full Text are published under a non-exclusive license. Metadata are under a CC0 License
https://hdl.handle.net/20.500.12608/116130