04 / How It Goes Wrong
Attention relationship
Every pair of items in the context can influence each other. Double the amount of context and you roughly quadruple the number of pairs competing for notice.
How you’ll hear it
“Why does adding more material make it worse?”
“More material, more competing connections for it to weigh.”
Avoid
Assuming more context always helps.
The longer version
Every piece of context can relate to every other piece, and the model has to weigh those relationships. As context grows, the number of relationships grows much faster, which is one reason long, cluttered inputs give weaker answers.