Attention, without backpropagation

The respectable story says attention needs backpropagation.

That is not an unreasonable story. Attention is the part of a transformer that decides what to look at, and modern systems usually teach it with one global error signal threaded backwards through the whole network. Every piece gets told how it helped or hurt.

We took that teacher away.

Each part had to update from what it could see locally. No global backward pass. No post-hoc explanation from the loss. The interesting question was not whether a local rule could work once, in a tiny case where almost anything can look clever. The question was whether it would still work after the problem stopped being polite.

The ingredient that mattered

The soft version did the sensible thing: a weighted average. Look at every option a little, in proportion to how promising it seems.

This is emotionally appealing and mechanically leaky.

As the number of options grew, the soft version kept donating attention to distractors. Not much to any one distractor, but enough in aggregate to wash out the signal. It was attention by committee, and the committee wanted everyone to feel included.

A sharp winner-take-all competition held up where the soft average collapsed. Going from 4 options to 16, the soft gate's quality dropped off a cliff while the competition kept about 85% of the achievable performance, with no leakage onto distractors.

The annoying detail is that a plain softmax is not enough either. It has the right formal costume, but not the right behavior. The competition has to be genuinely sharp and conserve its total attention.

That was the useful part. Not a global correction signal. Not a tiny trainer hiding in the mechanism. Just a local contest that refuses to spend attention twice.

The honest size of it

This is a small, controlled result: toy vocabularies, a handful of seeds, and a narrow question. It holds substantially, not perfectly. It does not say transformers are solved without backprop, and it does not turn a small mechanism result into a theory of intelligence.

The win is cleaner and more useful:

You don't need backprop to learn where to look. You need the competition to be sharp, local, and conserved.