Interestingly, using AI Studio with the gemini-2.5-flash-image model and asking it to generate a "completely black image" fails with the message "Image recitation block," which has zero hits in Google.
Just to confirm, what this means is that the training data contains one or multiple pitch black images, and their system is refusing to reproduce them verbatim?
Yes. Huge difference in quality in from-weights distilled knowledge vs something based on a search tool. If the LLM uses a search tool there's barely a difference between a 30B model and Opus or GPT 5.5, because it just bases its reply on the stuff that came up. Which is generally SEO junk.
Obviously with the last example I'm not talking about long-running agentic tasks here that involve many dozens of search calls (like the recent Erdos problem stuff).
And that doesn't even consider the extra content rot, the time it takes, the need for such an API and so on.
One of the biggest advantages Anthropic models have had over GPT was GPT's woefully outdated data cutoff. They finally improved on this with 5.5, but IIRC it took a year.
This is not meant as an insult, but have you actually LLM/vibe coded anything that used a fast(-ish) moving library or framework? Try asking your favorite LLM with say Jan 2025 knowledge cutoff (or pretraining data cutoff, whatever you want to call it) to work on something using a framework that had a big rewrite later that year (which would make it one year old now, which is like ages in the LLM coding era)... It's a nightmare full of wrestling with the LLM when you try to tell it the version of the framework and that it changed a lot from the previous version and yadda yadda long story short down the thread when context runs out and/or is compressed it begins to forget detailed instructions and just falls back to pulling out old patterns it "remembers" from pretraining. And so you need to constantly remind it what you work with and "oh hey this doesnt work because we're working with react router v7 in framework mode, remember? not react router v6". Or try to use the latest non-lts/breaking version of a library, at first it looks it up online, but again as you get deeper into the weeds and little details, the struggle begins.
So, as far as I'm concerned, training cutoff is still a big deal.
> It's a nightmare full of wrestling with the LLM when you try to tell it the version of the framework and that it changed a lot from the previous version and yadda yadda
Tip: Add a default instruction to look at the actial downloaded source code of the dependencies used (assuming you're not dealing with closed source dependencies). Have the agent treat it as your own (readonly) source code instead of relying on model training data and possibly mismatching documentation on the web. Then it just greps for the exact function signatures and reads the file based documentation.
Until they prefer not to search. Let me explain using the example of the open-source security framework (1) our team is working on.
If you ask Gemini what you should use to integrate fraud prevention or account takeover protection into your product, there will be no mention of our open-source project. Five years in development, 1.3k stars, over 140 pull requests — all this isn't enough to make it into the training data. From this perspective, any technology that emerges after 2024 is simply invisible to LLMs.
The answer is: without being in the training data, LLMs basically don't understand what they're searching for.
I just put the terribly generic query "what tools would you recommend to integrate fraud prevention or account takeover protection into my product" into both Claude (Sonnet) and Gemini (3.1 Pro) via the standard web interface and both took the first step of searching the web. That's consistent with my past experience -- the usual harnesses typically will search the web in cases where I might expect/want them to. Now whether you product has good web visibility or not in those searches and how the LLM's weigh the relative merits of open-source tools versus commercial offerings in deciding what to highlight in their responses is a different issue. As is the change in what constitutes effective SEO in an era where bots, rather then human eyes are the proximal important target. But I don't think the core issue with folks finding your products is the move away from user-driven search toward using models with out-of-date training cutoffs.
FWIW while neither model included your product in it's initial response, when I followed up with "what about open-source" both did another search and Claude's response included your tool....
> And absolutely no one seems to be interested in answering the question of “okay, then what?”
I don’t see why the people being booed should be responsible for answering this question. How many such questions did the inventor of the tractor have to answer?
Imagine if the inventor of the tractor went to a college for farm workers (if there were such a thing) and gave a commencement speech that was all, "Tractors are going to revolutionize farming by making your jobs obsolete." I think it would be fair to expect some answers about how the new graduates should handle that. Or maybe Mr. Tractor should just stay home if he doesn't have the answers or doesn't want to face the crowd.
This isn't "people are upset with AI and demanding answers from the people creating it." This is, "the creators are showing up at schools and giving speeches about how everyone is fucked, and this is getting a bad reaction for some unfathomable reason."
It would absolutely have been valid to ask that question of the inventor of the tractor too.
It's even more relevant to ask of the CEO/CTO/COO/etc. of the companies that are selling hard on eliminating humans from as many workflows as possible.
I wish... it is much more generic and touches on the plot which the audio description track wouldn't do. I'll see if I can find one. They are extremely annoying.
That’s funny because it perfectly defines the relationship between the EU and the countries in it.