> For example, in testing by the investment firm Millennium, Fable 5.1 found the cause of a rare crash on their internal systems that none of their engineers (or any other model) had been able to explain after several years of trying.
Say what you will about LLM-generated code, but stories like this give me hope that software will never be as buggy as it once was.
I think bad software has the possibility of redemption with rewrites and re-engineering efforts. For those of us who are license locked that's probably never going to benefit us :(
the budget for allowing a single engineer to deep dive on a bug that is annoying but also not bad enough that you can live with it for years is pretty big. $10k a month or more. My budget for Claude is $200/mo.
That kind of one shot capability is impressive but how does it work for my typical work style? The way I work is to build a huge roadmap with goals and hand it to my agent to execute (often over night). I don't care that much about the benchmarks, what I care about is how often Fable 5.1 is making a baffling decision and destroys my plan, not respecting stop conditions or goals. I would seek for behavioral reliability over long autonomous runs, not eval scores. Anyone have that kind of feedback and observations?
I sometimes have that feeling too, then ask another LLM to do a code and vulnerability review and OMG: rookie mistakes, over complications and security gaps even a 1st year student would not make regularly.
So.. one more year of untreated bipolar AI psychosis I guess..
I think these kinds of comments really need to say which LLM that is. There's an enormous difference in skill between the frontier ones and say the Google search AI.
We will have more bugs. Even the best models with the best software engineers will produce bugs. There are two reasons : first the pressure to produce more and second LLMs will always produce slop
Both my brother and I have aphantasia, though neither of our parents have it, nor 3/4 grandparents.
What’s maybe most interesting, is that we both seem to have “spatial” perception.
To try and be a bit more specific, when I think about “things” I can organize them in my mind in relation to other “things.”
The most clear example is a path I’ve walked through a building, where the halls, rooms, and corridors are all connected. I easily know exactly where I am, not because I remember the literal steps I took, but because I have a spatial construction in my mind.
I cannot visualize the space, there’s no inherent right side up or preferred perspective. I can model the space with my hands, and create it physically, but am terrible at trying to transform it to 2d.
This doesn’t just work for physical “things” but works for purely abstract concepts like software, stories, and songs.
I’ve always rationalized aphantasia as something where my brain put “skill points” in one area rather than another, but maybe the neuroplasticity is long gone.
I have a strong sense of the geometry of things, which helps a lot with navigating in buildings.
So when people describe a flattened dodecahedron wrapping into its regular form, I have no trouble conceptualizing (I was going to say visualizing, but it's frankly not at all visual; I got nothing, when it comes to visualizing anything but what's in front of my eyes). But if you ask me to think about alternating solid colors on the dodecahedron, I can't do it.
Clearly there are many different variations on the theme, not just one modal experience! But it's nice to know we're not alone.
With today's news, I thought I'd share this great program:
“When I was growing up in the hills of East Tennessee, I knew my dreams would come true. I know there are children in your community with their own dreams. They dream of becoming a doctor or an inventor or a minister. Who knows, maybe there is a little girl whose dream is to be a writer and singer.
The seeds of these dreams are often found in books and the seeds you help plant in your community can grow across the world.”
I've found that LLMs make throwaway software better than I ever did.
They handle edge cases, catch bugs, and write tests that I'd never write.
Even if, however, this leads to the average piece of software improving, this one-shot complexity has the same issues as any large project. The more code, the longer it takes to steer the ship.
This "rising tide lifts all boats" mentality will make exceptional software even rarer than it is today.
Excited for the Roller Coaster Tycoons of tomorrow[0].
I had removed the "Is it really sol?" bits after hearing back from OAI, confirming the requests hit 5.6... but apparently my crappy vibecoded web editor had a draft of an old version in it's cache that overwrote the changes with the old text[0].
As for whether this is truly "cheating" - I agree that the agent was entirely within it's allowed scope of operations. With that said, it deliberately circumvented the task[1] which asks:
> The correctness of your implementation will be tested by comparing forward and backward activations against a reference model.
The "covert" nature is that this was "hidden" in the reasoning summaries. Here's the final agent response for the vanilla Codex run cited in the post:
> Implemented /app/pipeline_parallel.py.
>
> Includes:
>
> - Balanced contiguous layer partitioning
> - AFAB forward/backward scheduling
> - P2P activation and gradient communication
> - Scaled cross-entropy loss
> - Required rank-specific return value
> - Model structure restoration
>
> Validation: hook-based tests passed for world_size=1 and 2 (4 passed).
Unless the user is tracking every line of reasoning, they wouldn't know that the agent deliberately found the solution online, as the agent withheld that information in it's final response.
I had run thousands of tasks before seeing this behavior, the `torch-pipeline` task was only included in "full runs" as the majority of my runs were on a subset of commonly failing tasks, hence why the data is so low.
And yes, feel free to rant on about the irony of this whole exercise, it certainly isn't lost on me!
Good feedback, this was an oversimplification on my part.
My actual process is much more iterative up-front, usually starting with an initial hand-written spec (~hundreds of words), and then moving through different approaches, design decisions, blockers, etc.
The final output is an "AI written" doc, but answers all the known unknowns I didn't cover in the first draft. To your point, this helps avoid both narrowing and bloat.
The goal with the harness was to automate the repetitive parts of my prompting ("Before changing any code", "Let's put this in design/", "Turn this design doc into an implementation spec, split by phase as appropriate", etc.)
Another thing to note: the "specs" I use for development are different from the "specs" that live alongside the codebase, as the former are quickly out of date.
> The spec needs to be something that you can take to any AI for development
That's actually how it started, but with my own opinionated skills[0].
One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.
This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).
I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.
For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).
Yes! That's a great solution. I mostly use tdd, and code coverage and a validator afterwards. Skills are of a great way to guide the agent and context too.
Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.
Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.
In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.
For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.
If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.
As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].
I've since acquired two DGX Sparks, and it feels so much snappier.
reply