A large share of eukaryotic protein sequence never folds into a fixed three-dimensional struc- ture. These intrinsically disordered regions bind, regulate and organise the processes of the cell, and almost everything known about them is recorded in the literature: a region is estab- lished as disordered because an experiment reported in some publication showed it to be so. DisProt is the reference database collecting those findings, assembled by curators who read the publications and enter what they assert, so its growth is limited not by the supply of published evidence, which accumulates with the literature, but by the time of the people who read it. The purpose of this project is to measure how much of that reading a large language model can do. Three instruction-tuned models were asked to return, from the full text of one publica- tion and nothing else, the record a curator would have entered from it: which protein, which residues, on the strength of which experiment, and the sentence in the paper that carries the claim. A corpus was assembled from the DisProt entries behind the CAID3 benchmark targets, retrieved, converted to text and submitted under a prompt developed iteratively against ob- served failures. The output was scored against the curated annotations and, separately, residue by residue on the CAID3 benchmark, with paired confidence intervals throughout so that a difference between two models can be told from sampling noise. The findings show that the two stronger models identify the annotated proteins with com- parable accuracy, while the lighter model, run under an earlier version of the prompt, trails on nearly every measure. Where a region is recovered, its boundaries usually match the reference coordinates exactly, and recall is the main limitation. Compared with two leading CAID3 pre- dictors on the proteins each model covers, the extractions trail the structure-based method on the reference that includes disorder inferred from X-ray structures, but outperform both meth- ods on the reference that excludes it. This indicates that reading the literature and reading the sequence do not recover the same evidence and that the two approaches can be complementary.

A quantitative assessment of LLM positional information extraction in structural biology literature

VENANZI, MARCO
2025/2026

Abstract

A large share of eukaryotic protein sequence never folds into a fixed three-dimensional struc- ture. These intrinsically disordered regions bind, regulate and organise the processes of the cell, and almost everything known about them is recorded in the literature: a region is estab- lished as disordered because an experiment reported in some publication showed it to be so. DisProt is the reference database collecting those findings, assembled by curators who read the publications and enter what they assert, so its growth is limited not by the supply of published evidence, which accumulates with the literature, but by the time of the people who read it. The purpose of this project is to measure how much of that reading a large language model can do. Three instruction-tuned models were asked to return, from the full text of one publica- tion and nothing else, the record a curator would have entered from it: which protein, which residues, on the strength of which experiment, and the sentence in the paper that carries the claim. A corpus was assembled from the DisProt entries behind the CAID3 benchmark targets, retrieved, converted to text and submitted under a prompt developed iteratively against ob- served failures. The output was scored against the curated annotations and, separately, residue by residue on the CAID3 benchmark, with paired confidence intervals throughout so that a difference between two models can be told from sampling noise. The findings show that the two stronger models identify the annotated proteins with com- parable accuracy, while the lighter model, run under an earlier version of the prompt, trails on nearly every measure. Where a region is recovered, its boundaries usually match the reference coordinates exactly, and recall is the main limitation. Compared with two leading CAID3 pre- dictors on the proteins each model covers, the extractions trail the structure-based method on the reference that includes disorder inferred from X-ray structures, but outperform both meth- ods on the reference that excludes it. This indicates that reading the literature and reading the sequence do not recover the same evidence and that the two approaches can be complementary.
2025
A quantitative assessment of LLM positional information extraction in structural biology literature
Proteins
Disorder
LLMs
Disprot
File in questo prodotto:
File Dimensione Formato  
Venanzi_Marco.pdf

Accesso riservato

Dimensione 9.3 MB
Formato Adobe PDF
9.3 MB Adobe PDF

The text of this website © Università degli studi di Padova. Full Text are published under a non-exclusive license. Metadata are under a CC0 License

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.12608/115853