RC RANDOM CHAOS

Beyond Attention: Grassmann Layers Rival Transformers

· via Hacker News

Original source

Attention Is Not What You Need (2025)

Hacker News →

Researchers challenge the assumption that self-attention is essential for sequence modeling performance. They introduce a competing architecture based on Grassmann manifolds that performs comparably to standard transformers while offering clearer mathematical interpretation.

The proposed Causal Grassmann layer replaces the typical attention matrix with a geometric approach using Plucker coordinates. Instead of computing pairwise interactions in an unstructured tensor space, the system encodes local token relationships as subspaces on a Grassmann manifold, then fuses these features back into hidden states through gated mixing. This keeps computation on a mathematically structured manifold rather than in an opaque tensor space, potentially enabling better understanding of model behavior.

On language modeling benchmarks, attention-free Grassmann models with 13-18M parameters achieve perplexities within 10-15% of transformer baselines. On the SNLI inference task, a Grassmann-based variant actually outperforms the transformer baseline. Beyond performance parity, the authors argue this manifold-based approach creates a more structured path toward geometric interpretability and invariant-based reasoning in neural networks.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.