Large language models have become routine tools for writing scientific Python, yet they exhibit a failure that ordinary code generation does not catch: a function passes its tests and runs without error while returning numerically wrong values, for instance when an accumulation silently stays in single precision. Such errors are invisible to the generate-and-test loop that drives standard agentic refinement. This thesis asks whether a locally deployable hybrid of specialized models in the 7 to 32B range can generate scientific Python more reliably than a single small coder, without sending code to a cloud model. The proposed system combines parallel generation of four candidates, hierarchical static triage by linters, type-checkers, and numerical invariants, a specialized math reviewer, and a heavy reviewer. The two reviewers are specialized through preference optimization, and the coder through reinforcement learning on an execution-based reward. Code-generation quality is measured by sub-step pass@1 on the SciCode benchmark and on a custom contamination-free benchmark. On the SciCode test split the assembled system raises sub-step pass@1 from a single-shot baseline of 19.8% to 30.2%, a gain of 10.4 points (p < 0.001), within its latency budget. Parallel generation with triage is the largest single contributor, adding 6.2 points (p = 0.001) at 1.26 times the median latency. The specialized reviewer outperforms a homogeneous control by 6.6 points (p = 0.002). Training each component improves its own metric but does not carry into the assembled system. The 30.2% result places the system among the strongest early-2025 frontier models on the public leaderboard, and the custom benchmark reproduces the gain. Assembling and filtering small local models yields a measurable improvement on numerical code, driven mostly by cheap parallelism rather than the specialized parts. Reliable verification of generated numerical code remains open: 61% of the failures that survive the full system are silent wrong values rather than crashes.

Agentic Code Generation with Parallel Self-Refinement and Math-Aware Verification on Local LLMs

CAZZADOR, LORENZO
2025/2026

Abstract

Large language models have become routine tools for writing scientific Python, yet they exhibit a failure that ordinary code generation does not catch: a function passes its tests and runs without error while returning numerically wrong values, for instance when an accumulation silently stays in single precision. Such errors are invisible to the generate-and-test loop that drives standard agentic refinement. This thesis asks whether a locally deployable hybrid of specialized models in the 7 to 32B range can generate scientific Python more reliably than a single small coder, without sending code to a cloud model. The proposed system combines parallel generation of four candidates, hierarchical static triage by linters, type-checkers, and numerical invariants, a specialized math reviewer, and a heavy reviewer. The two reviewers are specialized through preference optimization, and the coder through reinforcement learning on an execution-based reward. Code-generation quality is measured by sub-step pass@1 on the SciCode benchmark and on a custom contamination-free benchmark. On the SciCode test split the assembled system raises sub-step pass@1 from a single-shot baseline of 19.8% to 30.2%, a gain of 10.4 points (p < 0.001), within its latency budget. Parallel generation with triage is the largest single contributor, adding 6.2 points (p = 0.001) at 1.26 times the median latency. The specialized reviewer outperforms a homogeneous control by 6.6 points (p = 0.002). Training each component improves its own metric but does not carry into the assembled system. The 30.2% result places the system among the strongest early-2025 frontier models on the public leaderboard, and the custom benchmark reproduces the gain. Assembling and filtering small local models yields a measurable improvement on numerical code, driven mostly by cheap parallelism rather than the specialized parts. Reliable verification of generated numerical code remains open: 61% of the failures that survive the full system are silent wrong values rather than crashes.
2025
Agentic Code Generation with Parallel Self-Refinement and Math-Aware Verification on Local LLMs
Language models
Scientific code
Multi-agent systems
Preference tuning
Code verification
File in questo prodotto:
File Dimensione Formato  
Cazzador_Lorenzo.pdf

accesso aperto

Dimensione 806.41 kB
Formato Adobe PDF
806.41 kB Adobe PDF Visualizza/Apri

The text of this website © Università degli studi di Padova. Full Text are published under a non-exclusive license. Metadata are under a CC0 License

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.12608/119132