Discussion about this post

User's avatar
Mira Kamdar's avatar

It is a testament to my heat-addled brain that I read the headline as "The Exhaustion of AI".

Expand full comment
Derek Neal's avatar

There's something else quite interesting here, unless I'm being fooled myself by a bit of Hinternet trickery. Did the LLM really use the phrase "attention exhaust?" If so, that's a fascinating construction I don't think I've ever seen before, but it recalls "data exhaust," which Shoshana Zuboff wrote about (I believe she popularized the term a bit). If I'm right that this term hasn't been used before in this way, it's evidence of the LLM "seeing" a relationship between the way "attention" and "data" are talked about in its training data and intuiting that "attention exhaust" is a possible term based on statistical relationships. This is one interesting way to use LLMs--when they "hallucinate" or do something novel, it's not simply a mistake, it's telling us that there's some sort of relationship among different words and ideas that we (humans) might not have noticed but is present in the vast amounts of text that we've produced.

Expand full comment
3 more comments...

No posts