Well... since o1 (Sept 2024), most models generate a "hidden" thinking trace before the visible answer... surely you have seen this by now: "Wait, that's wrong...", "Ah, now I have the full picture...". The model prunes dead ends when it composes the final answer.
Demanding sources for this is odd. It's literally been a headline feature of every frontier model for two years.
I guess you are "technically correct" that no model can "un-emit" tokens... but I don't think that is what anyone was saying or an interesting point to make.
Edit: see also this recent post, which details another place where revisions can happen, upstream of the reasoning token emission:
You're talking about "thinking" models which are mostly regular LLMs trained to output a "<thinking></thinking>" token delimiter before the answer that will be shown to the user, and that's a clever use of the Chain of Thought idea. All of which I'm completely familiar with.
GP was saying
> That would be true of non-iterative models that just emit an output from beginning to end.
The disagreement here seems to be a confusion between the pure model behavior and the behavior that users experience, which is the model wrapped in a harness. It’s normal and natural now to refer to the whole system as ‘the model’ informally, even though that is technically not accurate. That ship sailed a while ago.
Since the LLM is now designed to run in a harness, it’s really not even wrong any more.
Genuine question: How is that a value proposition when Cloudflare offers a CDN for free with unlimited bandwidth, that you could just put in front of the sweaty VM in Helsinki?
Not trying to be obtuse, I really don't get how other providers can compete with that, I can't imagine Vercel's CDN is so significantly superior to make it worth it.
For that matter, the entire site could be in a Cloudflare worker with all the content in R2 (no egress fees, just storage). Likely to barely exceed the $5 baseline price for a paid account. (not sure on the storage costs, but likely well under $100/mo)
Yes, and I didn't mean to imply that a single VPS is all you needed. But I wanted to put things into perspective for the other posters who claimed that you couldn't possibly serve a site like this from a single machine, purely in terms of performance.
The current SOTA closed model providers are also all rolling out access to their latest models with better pricing (e.g. GPT-5 this week), which seems like a confounding factor unique to this moment in the cycle. An API consumer would need to have a very specific reason to choose GPT-4o over GPT-5, given the latter costs less, benchmarks better and is roughly the same speed.
To be a little more fair... that example is tidily slotted into the EXAMPLES section, under the heading "You can extract images from a video, or create a video from many images".
I don't think most people read the man pages top to bottom. And even if they did, then for as much grief as you're giving ffmpeg, llm has an even larger burden... no man page and the docs weigh in at over 8k lines.
I get the general point that ffmpeg is a powerful, complex tool... but this is a weird fight to pick.
Demanding sources for this is odd. It's literally been a headline feature of every frontier model for two years.
I guess you are "technically correct" that no model can "un-emit" tokens... but I don't think that is what anyone was saying or an interesting point to make.
Edit: see also this recent post, which details another place where revisions can happen, upstream of the reasoning token emission:
https://www.anthropic.com/research/global-workspace
reply