Large Language Models (LLMs) can produce different answers to semantically equivalent questions, making output coherence an important consideration for reliable open-domain question answering. Question Prompting is an inference-time intervention that provides multiple semantically equivalent formulations of a target question and has been shown to improve answer accuracy and question-answer coherence. The study introducing the intervention evaluated it under near-greedy decoding, without examining its effects on abstention or confidence calibration. Similarly, the Question Prompting study and the work introducing a dataset used in its analysis evaluate coherence under deterministic or near-greedy decoding. This thesis evaluates Question Prompting across 13 open-weight model configurations from the Qwen 3.5, Ministral 3, and Gemma 4 families on the PopQA-TP dataset. The evaluation separates output variance associated with question phrasing from stochastic decoding noise, and examines calibration and selective prediction with explicit abstention accounting. The analysis further considers whether these effects vary with entity popularity and assesses the sensitivity of evaluation results to a character encoding corruption affecting a subset of the dataset. The results show that stochastic decoding contributes substantially to output variation, while Question Prompting reduces variation associated with question phrasing to near zero across the evaluated models. Question Prompting also changes model abstention behavior in addition to accuracy, and its measured calibration effects depend strongly on how abstentions are treated. Selective prediction improves when abstentions are excluded, while larger shifts are generally observed for less-popular entities. Repairing the encoding corruption also changes measured accuracy, abstention, and calibration on the affected subset. The findings suggest that the impact of coherence improving interventions should be assessed through accuracy alongside calibration and an explicit abstention policy rather than accuracy alone, as such interventions can alter model uncertainty.
Large Language Models (LLMs) can produce different answers to semantically equivalent questions, making output coherence an important consideration for reliable open-domain question answering. Question Prompting is an inference-time intervention that provides multiple semantically equivalent formulations of a target question and has been shown to improve answer accuracy and question-answer coherence. The study introducing the intervention evaluated it under near-greedy decoding, without examining its effects on abstention or confidence calibration. Similarly, the Question Prompting study and the work introducing a dataset used in its analysis evaluate coherence under deterministic or near-greedy decoding. This thesis evaluates Question Prompting across 13 open-weight model configurations from the Qwen 3.5, Ministral 3, and Gemma 4 families on the PopQA-TP dataset. The evaluation separates output variance associated with question phrasing from stochastic decoding noise, and examines calibration and selective prediction with explicit abstention accounting. The analysis further considers whether these effects vary with entity popularity and assesses the sensitivity of evaluation results to a character encoding corruption affecting a subset of the dataset. The results show that stochastic decoding contributes substantially to output variation, while Question Prompting reduces variation associated with question phrasing to near zero across the evaluated models. Question Prompting also changes model abstention behavior in addition to accuracy, and its measured calibration effects depend strongly on how abstentions are treated. Selective prediction improves when abstentions are excluded, while larger shifts are generally observed for less-popular entities. Repairing the encoding corruption also changes measured accuracy, abstention, and calibration on the affected subset. The findings suggest that the impact of coherence improving interventions should be assessed through accuracy alongside calibration and an explicit abstention policy rather than accuracy alone, as such interventions can alter model uncertainty.
The Effect of Question-Prompting on Calibration and Uncertainty Structure in LLMs
MARRELLI, SEBASTIANO
2025/2026
Abstract
Large Language Models (LLMs) can produce different answers to semantically equivalent questions, making output coherence an important consideration for reliable open-domain question answering. Question Prompting is an inference-time intervention that provides multiple semantically equivalent formulations of a target question and has been shown to improve answer accuracy and question-answer coherence. The study introducing the intervention evaluated it under near-greedy decoding, without examining its effects on abstention or confidence calibration. Similarly, the Question Prompting study and the work introducing a dataset used in its analysis evaluate coherence under deterministic or near-greedy decoding. This thesis evaluates Question Prompting across 13 open-weight model configurations from the Qwen 3.5, Ministral 3, and Gemma 4 families on the PopQA-TP dataset. The evaluation separates output variance associated with question phrasing from stochastic decoding noise, and examines calibration and selective prediction with explicit abstention accounting. The analysis further considers whether these effects vary with entity popularity and assesses the sensitivity of evaluation results to a character encoding corruption affecting a subset of the dataset. The results show that stochastic decoding contributes substantially to output variation, while Question Prompting reduces variation associated with question phrasing to near zero across the evaluated models. Question Prompting also changes model abstention behavior in addition to accuracy, and its measured calibration effects depend strongly on how abstentions are treated. Selective prediction improves when abstentions are excluded, while larger shifts are generally observed for less-popular entities. Repairing the encoding corruption also changes measured accuracy, abstention, and calibration on the affected subset. The findings suggest that the impact of coherence improving interventions should be assessed through accuracy alongside calibration and an explicit abstention policy rather than accuracy alone, as such interventions can alter model uncertainty.| File | Dimensione | Formato | |
|---|---|---|---|
|
Marrelli_Sebastiano.pdf
accesso aperto
Dimensione
1.34 MB
Formato
Adobe PDF
|
1.34 MB | Adobe PDF | Visualizza/Apri |
The text of this website © Università degli studi di Padova. Full Text are published under a non-exclusive license. Metadata are under a CC0 License
https://hdl.handle.net/20.500.12608/116676