LLMs Evaluation for Android Vulnerability Repair

Automated Program Repair (APR) aims to automate the process of fixing software defects, including syntactic errors, semantic bugs, and vulnerabilities. Emergent studies have explored the use of Large Language Models (LLMs) for APR related tasks, such as code refinement and fault localisation, but they are mainly involved in the repair of syntax and semantic bugs. The specific focus on vulnerability repair, especially in the context of Android applications, remains limited. This project wants to take a step further in this direction, by contributing a new dataset of real-world Android vulnerabilities and corresponding fixes and by evaluating three prominent LLMs (ChatGPT-3.5, Google Bard, and Android Studio Bot) on such a dataset in a zero-shot scenario, to understand their actual abilities in repair Android vulnerabilities. Notably, although Android Studio Bot is strictly related to the Android Studio platform and should facilitate Android code generation, its performance for the specific vulnerability repair task is to be considered unsatisfactory, letting ChatGPT-3.5 and Google Bard stand out.