Knowledge lives in the overlap, not in the parts

Every time we wanted to know what the system had actually learned, we did the obvious thing: we opened it up and read the pieces. This weight, that connection, this one element. It kept telling us almost nothing.

The signal isn't in the parts. It's in the overlap — which stored memories light up together. Two things are related not because of a number on the edge between them, but because they share a population of traces that co-activate. The shape of that shared activation is where the structure actually shows up.

We didn't decide this on taste. Reading a system out through overlap beat reading it through individual weights on a straight classification test (0.749 vs 0.641). And it quietly dodged a problem the weight-level view couldn't escape: a whack-a-mole where every fix that improved one pattern degraded another. Three different weight-level mechanisms got stuck in it. The overlap readout simply didn't have the problem — it was looking in the right place.

A separate project of ours landed on the same idea from the other direction: it treats overlapping memory traces as its unit of meaning, and that overlap is what carries its sense of grammatical category. Same conclusion, reached from the far end.

The catch

Overlap tells you where to look, not that the thing is there. If the system never wrote the structure in the first place, no readout will conjure it out of the overlap — that's a different wall entirely. But whenever you're tempted to judge a system by squinting at its individual parts: don't. Look at what they share.