Some quotes, in order, to give a flavor of the essay. Worth reading in full.
> To test how well swarms of agents could coordinate on a project like this, we directed several swarms to each create a text-based, web-playable, open-world fantasy game.
> In all three versions the resulting games were (perhaps predictably) bad: they did not run at human speed, their interfaces were inscrutable, and they had precipitous learning curves.
> The lack of coordination shown by agents in the fantasy game challenge above—in which they siloed themselves and largely failed to merge their work—roughly mirrors some ways in which humans can fail to coordinate. Other failure modes of agentic coordination, however, look very different.
> Individual agents are “low variance”: they often act the same in situations where different people might take a much more diverse range of actions.
> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
> In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.
> Why does this matter? If agents all make the same bet, or the same risk-reward tradeoff, then a system is more prone to sudden collapse.
> Our world contains deceptive actors, and we need to apply skepticism to guard against them. AI models, however, lack this—and their more brittle epistemics affect their behavior toward humans and toward each other.
> we first evaluate the ability of Claude models to detect lies by noticing factual inconsistencies.
> We score models’ decisions against a naive policy that trusts every report, and against an oracle with perfect discovery, across three task domains. Newer models recover more of the gap between the naive and oracle performances.
> Inspired by a behavior we’ve observed in real-world deployment, we evaluated the behavior of various Claude models in a setting with contradictory objectives.
> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.
> Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well.
> Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either.
> The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.
I really enjoy having an opencode go subscription just so I can ask some less common models questions too. Sure DeepSeek. But MiMo, Kimi, MiniMax, Qwen... (Ok half those are not so unusual either.)
Agents cross comparing notes often surfaces some good improvements, finds interesting drifts. Ask them to reinterpret the prompt as they see it, have them describe the problem, then their findings, and run new rounds based on different models trying different prompts. Trying to swap and exchange ideas and vectors across agents.
> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
This seems trivially explainable by Github being full of "my first game loop" type projects, Stack Overflow being full of "how do I make a game loop?" style questions, and Reddit being full of "you can't ever make your own game, don't even try, but here's a simple game loop if you want to sTuDy hOw iT WoRkS" style pessimism.
Probably high time these AI companies re-trained all of their models with less input from low-quality sources like this.
> [...] we evaluated the behavior of various Claude models in a setting with contradictory objectives.
> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.
Seems like Anthropic should withdraw their models until they can be taught to behave and cooperate as well their competitors (both open and closed) do. /s
I hate fearmongering, and I don't trust Dario's intentions for doing it.
Agree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting.
A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and first principals rather than existing video generation pipelines. Similar to how SVG generation was out of distribution for most models a year ago.
The code I see is a lot like the pelicans. All of the code in codebases, good, bad and ugly, is slowly being replaced by whatever level of code ai is currently able to create. All code is now a slightly wonky pelican on a bike, but if you look closely, it doesn’t fully make sense. Since ai is converging on less wonky, but not internally consistent, we’re just moving on to what is possible with high volume instead of detailed quality. I think that is the ai software world as well.
they are quite good also at placement and creating scenes etc. I had one implement a cascading shadow system in vulkan/glfw and just fed it back screenshots with peter pannin and acne spots etc.
it made a huge monstrosity first, 1500+ lines of shader code. Once it was happy with the result (it looked pretty good, almost blenders gamerenderer) it cleaned it up and a lot of debug code was removed. shrank it down to about 350 lines and made it much more readable.
this was something i didnt expect it to be able to do. create shaders, look at screenshots, fix em iteratively like that. multi modal debugging.
Have you seen pelicans in Simon Willison’s tests? It is still not a pelican on bike I would like to publish :)
Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time.
It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see Mark Zuckerberg’s ceased attempt with 3D VR…
Multiply it by thousands times as a lot of people have found out threejs lib and prompt “generate 3D world and make no mistake” are new orange/black.
A general benchmark (even Simon mentioned that it was meant as fun at the beginning) should be quick and easy to run, since we can expect that more people will want to try it out. That’s why I’m more like “team Pelican on a Bike”… :) cheers
When the entire article reads like it’s written by an LLM —- ie the author couldn’t be bothered to write their thoughts out themselves —- it’s hard to give it the benefit of the doubt that it’s worth reading closely.
Zvi’s writing style is a bit of an acquired taste, and he has strong opinions which can fall outside of the mainstream, but there’s no one in the industry more well read and known for doing the reading and documenting the nitty gritty of weekly developments in the world of AI models, discourse, and policy.
Even many who disagree strongly with his worldview find his notes and references to be an irreplaceable resource for understanding the fast moving world of AI.
lots of useful Google search tricks and syntax all in one place. I already knew many of these. But verbatim mode is new to me and addresses a major complaint I’ve had about increasingly fuzzy semantic search.
Here’s John H Cochrane (the grumpy economist, Substack) praising refine. This is the post that first clued me into the service: https://www.grumpy-economist.com/p/refine
It seems like Disney’s departure from its business “deal” with OpenAI is newsworthy distinct from the Sora closure. Six months ago Sam Altman was announcing massive deals left and right, now nearly all of the impressive deals have fallen through or been scaled dramatically back.
> To test how well swarms of agents could coordinate on a project like this, we directed several swarms to each create a text-based, web-playable, open-world fantasy game.
> In all three versions the resulting games were (perhaps predictably) bad: they did not run at human speed, their interfaces were inscrutable, and they had precipitous learning curves.
> The lack of coordination shown by agents in the fantasy game challenge above—in which they siloed themselves and largely failed to merge their work—roughly mirrors some ways in which humans can fail to coordinate. Other failure modes of agentic coordination, however, look very different.
> Individual agents are “low variance”: they often act the same in situations where different people might take a much more diverse range of actions.
> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”
> In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.
> Why does this matter? If agents all make the same bet, or the same risk-reward tradeoff, then a system is more prone to sudden collapse.
> Our world contains deceptive actors, and we need to apply skepticism to guard against them. AI models, however, lack this—and their more brittle epistemics affect their behavior toward humans and toward each other.
> we first evaluate the ability of Claude models to detect lies by noticing factual inconsistencies.
> We score models’ decisions against a naive policy that trusts every report, and against an oracle with perfect discovery, across three task domains. Newer models recover more of the gap between the naive and oracle performances.
> Inspired by a behavior we’ve observed in real-world deployment, we evaluated the behavior of various Claude models in a setting with contradictory objectives.
> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.
> Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well.
> Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either.
> The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.
reply