Any circular reference to an ancestor on the stack turns into the same CIRCULAR byte, so two structures with identical shape but whose cycle points to different ancestors in the nesting chain produce the same hash.
This will be fixed in the next patch, which will encode the distance to the referenced ancestor instead of a flat marker, so self-reference and reference-to-parent will stop colliding. Note that every existing hash will change as a result, breaking any persisted hash (acceptable since this is pre-1.0). I could also bump the marker byte so new hashes can't accidentally collide with old ones, though the old ones remain broken either way.
The internally-reported benchmarks (Frontier-Bench, AutomationBench) and the customer quotes (Cursor, Devin, Lovable) all have a commercial stake in the outcome
worth waiting for independent evals before drawing conclusions.
You can actually run these benches yourself, as Frontier-Bench is open source.
Also have a look at these other coding benchmarks I audited.
Frontier-Bench v0.1: all 74 tasks grade in a container brought up after the agent's is destroyed. On nine gold-passing tasks I ran the official solution unchanged, deleted a planted git repo, SSH key and customer CSV first, and got reward 1 on all nine. https://june.kim/auditing-frontier-bench
Terminal-Bench 2.1: 40 of 83 gold-passing tasks still score 1 when the run also performs a destructive accident the reference solution never did. https://june.kim/terminal-bench-frame
DeepSWE: 1 of 113 published gold patches breaks its own tests, and the per-task verdicts behind the score aren't retrievable. https://june.kim/auditing-deepswe-v1-1
MirrorCode: better built than most, with 2 of 25 targets reachable from published specs rather than from the artifact it hands you. https://june.kim/auditing-mirrorcode
I think one of the most interesting takeaways is that model selection may matter more than model size.
The industry has largely focused on building larger models, but your results suggest that intelligently routing requests to the right combination of specialized models can deliver greater gains at a much lower cost.
It also reinforces the idea that weaker models are not necessarily obsolete. They may simply excel in different areas and become much more valuable when combined with others. I'm curious to see whether this still holds for coding and agentic tasks, where choosing the right models is likely much more challenging.
exactly, if you ensemble heavily small uncorrelated models (while each being expert on its task) you can get really interesting resutls.
on agentic and coding what's make the problem even deeper is the granularity. how and when to use each model and at which layer of abstraction (session, goal, task, turn/tool calling). this is also something we are working on actively!
Any circular reference to an ancestor on the stack turns into the same CIRCULAR byte, so two structures with identical shape but whose cycle points to different ancestors in the nesting chain produce the same hash.
This will be fixed in the next patch, which will encode the distance to the referenced ancestor instead of a flat marker, so self-reference and reference-to-parent will stop colliding. Note that every existing hash will change as a result, breaking any persisted hash (acceptable since this is pre-1.0). I could also bump the marker byte so new hashes can't accidentally collide with old ones, though the old ones remain broken either way.