LLM’s operate in discrete steps. So that “interpretation continues iterating” is a very good description of what’s actually happening.
There’s a little uncertainty in the process, so sometimes it will pick the wrong symbol at the wrong time and the entire process just spirals into nonsense. But it’s semi lucid nonsense like a dream or hallucination not line noise.
The confidently stating the wrong thing bit is arguably a different though related problem. There’s making up a citations that don’t exist and there’s inserting song lyrics where a citation should be.
Confabulation: Inadvertent pseudo-memory fill in, within a generally reasonable stable context representation.
Hallucination: Context representation becomes unstable, resulting in open ended drifting and morphing into incoherence.
The scale of error and recursion of error in the latter make the effects quite different.
You could be right, perhaps sometimes the initial causes might not be so different. But I would be surprised if that was true, given there doesn't seem to be much of a middle ground between the wide differences in scale and downstream effect.
You could be right that it’s a fundamental difference in what’s going on, but each word is a new iteration/token and most errors are more than a single token.
I’m assuming it’s like a converging vs diverging series in how well the LLM recovers from problems. Where the boundary between the two states is arbitrarily small, but OpenAI etc has fine tuned the system so you generally see the converging / confabulation types of errors even if the difference is just a slight change in the scale of error.
> I’m assuming it’s like a converging vs diverging series in how well the LLM recovers from problems.
That's a really good point.
Similar causes diverging in effect, a phase change or not, depending on whether a critical feedback threshold is hit or not. That could certainly be a factor.
With increasing doses of some drugs, humans do progress from being just a bit wonky, to full on lost in space!
But I think confabulation might be on a different continuum too. We fill in recalled memories with made up details all the time. Our memory storage is lossy, associative and overlapping. Our recall is always an imperfect combination of actual incidence memory filled out by similar memories.
We must confabulate to recall. Statistically, sometimes we over confabulate.
There’s a little uncertainty in the process, so sometimes it will pick the wrong symbol at the wrong time and the entire process just spirals into nonsense. But it’s semi lucid nonsense like a dream or hallucination not line noise.
The confidently stating the wrong thing bit is arguably a different though related problem. There’s making up a citations that don’t exist and there’s inserting song lyrics where a citation should be.