Assisted Pre-training for Decoder-Only Language Models

Large Language Models (LLMs) are becoming increasingly popular and have demon strated strong capabilities across a wide range of natural language processing tasks. Nonetheless, pre-training large language models remains a complex and resource-intensive process. Several techniques have been developed to simplify this procedure, including methods that reduce model size while aiming to preserve performance. One of the most widely used approaches is knowledge distillation, which transfers knowledge from a larger pre-trained model (the teacher) to a smaller more efficient model (the student), often resulting in minimal performance degradation. This technique typically involves training the student model to replicate the output distribution produced by the teacher, rather than learning directly from the training corpus. This thesis proposes and analyzes a novel approach that inverts the traditional di rection of knowledge transfer. Instead of distilling from a larger to a smaller model, we explore the potential of transferring a combination of corpus-based knowledge and information from a smaller model to improve the training of a larger one.