The model learns the shortcut, not the rule

"The keys to the cabinet are on the table." The verb agrees with keys, not with cabinet — and the word that decides it can sit a variable distance back. That's a real, non-local rule, and exactly the kind of structure we'd love a system to discover on its own.

Here's the trap. On ordinary text, a dumb shortcut — just agree with the nearest noun — is right about 92% of the time. The cases where it fails are rare (~8%), and worse, you can't even point at them without already knowing the answer. Spotting "where the shortcut breaks" requires knowing which noun is the real subject — which is the whole thing you were trying to learn.

So we asked: will any unsupervised, predict-the-next-thing objective prefer the real rule? We tried four structurally different ones, on 1.5 million real sentences. All four picked the shortcut. The real rule isn't broken — hand it over and it tracks the subject correctly through the hard cases. It just loses every contest where it has to be discovered, because being lazy pays off too well.

Why this is a wall, not a bad day

Four different objectives, real data at scale, all failing for the same reason, and we located that reason precisely: it's in the discovery objective, not the mechanism and not the representation. When a failure is that consistent and that well-pinned, it's a property of the approach.

The way through

The escape isn't a cleverer objective — it's a different diet. Stop weighting every example by how often it shows up (which drowns the hard cases) and feed the system a curriculum heavy on the ambiguous ones. We later confirmed that lever works: concentrate the budget on distinct hard cases and the difficult tail lifts sharply, no architecture change required.

The rule was never the problem. The problem was that the shortcut kept winning the audition.