A system-theoretic perspective on Transformers

This thesis begins with an in-depth analysis of the Transformer architecture, which has become the foundation for a wide range of sequence modeling tasks due to its powerful attention mechanism. Despite its success, recent studies have highlighted certain limitations of attention, particularly in terms of computational efficiency and scalability for long sequences. To address these issues, this work explores an alternative class of models based on State Space Models (SSMs). In particular, the S6 model—a recently proposed SSM—was studied within the context of the MAMBA architecture, which leverages the strengths of state space formulations while aiming to match or exceed the efficiency of Transformers. Following a thorough analysis of S6 and its integration into MAMBA, a novel SSM-based model is introduced. This new model, developed as part of this thesis, demonstrates improved performance over S6 in specific application scenarios. The thesis also features an extensive experimental section, where S6, the proposed model, and other baseline architectures are compared across different benchmarks, with a focus on accuracy, efficiency, and generalization capabilities.