Key Information Extraction from Visually-Rich Documents

Key Information Extraction (KIE) from Visually-Rich Documents is a critical challenge due to the diverse formats, intricate layouts, and the integration of both textual and non-textual elements. This thesis explores recent open source methods for extracting key information. The study presents the progression of Information Extraction (IE) approaches, from traditional rule-based systems to the latest deep learning models. GLiNER, LayoutLM, LiLT, and Donut, are evaluated for their performance for a specific KIE task on two private dataset containing complex documents in the Italian language, and then compared to third party services (GPT-4o) and human performance in order to gather insights for performing KIE in an industrial setting. The research compares the accuracy, inference time, and cost of these models. It reveals that while all models exhibit a certain degree of accuracy for some fields, other are much more difficult to extract. Donut particularly excel in the task, showing robust performance extracting all fields and beating results obtained by human. Other than the most precise model, it also is the cheapest model, due to his end-to-end structure that doesn't require to run an OCR, it requires a smaller amount of data preprocessing, thanks to its flexibility and being a generative model, however it requires extensive training to reach peak accuracy and can't be run as a zero-shot or few-shot model. The research also lays the groundwork for further exploration into more sophisticated models and techniques that can handle even more complex document structures in the future.