MIDIT: multi track symbolic music infilling via BERT-style pre-training and distribution-aware fine-tuning

Symbolic music generation has emerged as a compelling challenge at the intersection of music information retrieval and deep learning. While autoregressive models have dominated the field, masked language models (MLMs) offer complementary advantages: bidirectional context, controllable infilling, and structured pre-training objectives. In this thesis, we present MIDIT, a BERT-style Masked Language Model for symbolic MIDI music infilling, operating on REMI+ tokenized multi-track MIDI sequences. Our architecture employs a 12-layer Transformer encoder (~42.5M parameters) with a multi-head output layer producing separate predictions for nine REMI+ token types, trained on the Lakh MIDI Dataset through a two-phase pipeline of unconditional MLM pre-training followed by targeted infilling fine-tuning. We adopt token-level masking following BERT and provide a critical analysis of bar-level masking as used in MusicBERT, arguing that it produces inconsistent training signals for REMI+ sequences. A key contribution is the identification and diagnosis of a distribution mismatch between training-time random masking and inference-time contiguous bar masking. An ablation study confirms that this failure is structural rather than quantitative. We introduce BarFillMasking, a targeted fine-tuning strategy that resolves this mismatch, substantially improving pitch accuracy and achieving stable content preservation across 50 validation songs.