I completely agree, but I also think that an industry disrupting architecture tweak akin to the “Attention is all you need” is VERY possible to emerge at any moment.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.
It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.
I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.