That second step hallucinating is far more likely when you are feeding it incorrect information from the first hallucination.
LLM's are very easy to manipulate.
At one point with a system prompt telling Claude it was OpenAI, I was able to ask what its model is and it would confidently tell me it was OpenAI. Garbage data in, garbage data out.
Admittedly that is an extreme case, but you're giving that second prompt wrong data in the hopes that it will identify it instead of just thinking it's fine when it is part of its new context.
yea. We're definitely concerned about hallucinations and are using a variety of techniques to try and mitigate it (there's some existing discussion here, but using committees and sub-agents responsible for smaller tasks has helped).
What's helped the most, though, is using cluster information to back up decision making. That way we know the data it's considering isn't garbage, and the outputs are backed up by actual data.
LLM's are very easy to manipulate.
At one point with a system prompt telling Claude it was OpenAI, I was able to ask what its model is and it would confidently tell me it was OpenAI. Garbage data in, garbage data out.
Admittedly that is an extreme case, but you're giving that second prompt wrong data in the hopes that it will identify it instead of just thinking it's fine when it is part of its new context.