CoSy-seminar with Claudio Altafino

Everyone is welcome and the first 40 people to register will be treated to a free lunch sandwich. If you do not want lunch, you are still welcome to join. Registration formula link: https://doit.medfarm.uu.se/bin/kurt3/kurt/8905685

Abstract: Transformer architectures have rapidly become the standard algorithm for many problems in deep learning (e.g. large language models or image classification). At the core of a transformer lies a so-called self-attention mechanism. In this presentation I will show how to model such self-attention mechanism as a continuous-time multiagent-like dynamical system model living on a sphere. Such dynamics is related to a multiagent version of the Oja flow, a dynamical system that computes the principal eigenvector of a matrix, corresponding for transformers to the value matrix. In the ``single-head'' time-invariant case, the equilibria of a self-attention dynamics can be classified into four classes: consensus, bipartite consensus, clustering and polygonal equilibria. For this simplified dynamics, multiple asymptotically stable equilibria from the first three classes often coexist. Interestingly, equilibria from the first two classes are always aligned with the eigenvectors of the value matrix, often but not exclusively with the principal eigenvector.