Analysis of Iterated Self-Attention in Transformer Architectures

In a Transformer, each self-attention layer replaces every token with a weighted average of all the tokens in the sequence, with weights computed from the tokens themselves. Repeated over many layers, this averaging tends to make the token representations indistinguishable, a phenomenon known as rank collapse, which limits both the expressive power and the trainability of deep attention networks. Dong, Cordonnier and Loukas (2021) showed that, without skip connections, the collapse is doubly exponential in depth once the tokens are close enough to one another. Later work proved exponential collapse from every initial configuration, through lower bounds on the individual attention weights; these bounds depend on the size of the tokens and do not improve as the tokens approach each other, so the global and the local rate come from separate arguments. The effect of skip connections is only partly understood. This thesis studies a single attention head as a discrete-time dynamical system, with depth playing the role of time, and analyses it through the diameter of the token configuration and the Dobrushin coefficient of the attention matrix. For pure self-attention, we prove that the diameter $D_t $ after $t$ layers satisfies $D_{t+1} \leq (1 − exp(−a D_t^2))D_t$ for every configuration, where a depends on the norm of the query-key matrix. The contraction factor depends only on the differences between tokens and decreases as they approach each other, so that a single inequality describes the whole trajectory: far from consensus it gives exponential decay, and near consensus it reduces to the cubic estimate behind the doubly exponential collapse of Dong et al. We then add the other components of a Transformer layer one at a time. With a value matrix whose powers are bounded, the tokens still collapse from every configuration, and they approach a trajectory of the linear map defined by the value matrix; their common representation therefore converges for every initial configuration exactly when the powers of this matrix converge, as they do in pure self-attention. For expansive value matrices, collapse is guaranteed only near consensus. A residual connection passes the disagreement between tokens through unchanged at first order, so that attention acts on it only at third order. Near consensus, a single layer can then no longer reduce the disagreement by a fixed factor, whatever the parameters, which addresses, for one layer, the question of a lower bound left open by Dong et al.; far from consensus, no such bound holds uniformly over all parameters. A Lipschitz feed-forward network changes the constants of these estimates but not their cubic exponent. Finally, we show that under standard initialization the exponent governing the global bound grows with the model dimension, so that the contraction certified in the first layers is numerically indistinguishable from none. The results identify the effect of each component of a layer on token collapse; which regime a given network actually occupies depends on properties of its parameters, at initialization and after training, that are left for future work.