I think it's also important and heavily overlooked to develop and maintain open source "pro" level models. Those that are able to think for 80 minutes and yield heavy solutions.
I'm not an expert in LLMs so it's hard to understand how much are we lacking, is it just the compute and thinking strategies / parallel chains, or something specific architecturally. But I feel there's value there and I haven't seen anything like it available so far.
We have autograding for code through tests written by hand, and additionally do manual code audits if we see suspicious behavior. We also do grading the old-fashioned way for writeups.
We do indeed catch students who don't follow the honor code. It's very obvious from how the code looks, as well as the rate of progress. Since we use Modal for class submissions, we have code deltas for every time they run something on B200s. The diffs often contain something like 300 lines in 5 minutes, in which case we review and report based on how egregious/provable it looks.
My guess is that with the new UI they've either mistakenly or deliberately reduced the thinking effort or the budget, even if they write "extended thinking" in the app. And it seems that they're (mistakenly or deliberately) now routing some queries to the Flash model despite the selection of Pro/Thinking.
It was super bad right after launch, yet the AI Studio or API calls were on par to the earlier experience. Despite the selected "extended thinking", for the first time in a very long time the LLM outputed fully incorrect and a non-working code for a super simple problem (schema matching), which it can solve easily (and did through the API). So it's definitely not a coincidence.
I'm hoping that it's temporary, because otherwise we're back to GPT-5.0 levels of bad and it will kill the coding usage.
I would guess it's because ChatGPT Pro allows for 80min "think". I've never had even remotely similar think times with Gemini Deep Think. It's generally around 10-15min for math problems, and get increasingly shorter for continued interactions.
Good question! I might add them, but there were multiple reasons:
1. Most variants on HIGH/XHIGH provide only marginal improvements in accuracy, but at drastically increased latency and cost. One special example is Gemini 3.1 Flash Lite, which on High used 1.5M reasoning tokens, and it's cost was 5x the one of running 5.3-Codex: https://aibenchy.com/compare/google-gemini-3-1-flash-lite-pr...
2. On medium it seems like most models use a similar amount of reasoning tokens, this should be a more fair comparison.
3. Most models in the wild are used on medium (chat apps, default coding apps, tools, etc.).
4. Running on models on HIGH/XHIGH can lead to huge costs for me maintaining the test suite. I might add more models on high, if I can do it in a sustainable way.
5. Running models on HIGH would make running tests suites take much longer, so the results won't be published as fast.
6. Some models even show degradation when used on HIGH, as they tend to overthink/doubt themselves more. This seems to be a trend especially for new models, which wore trained to actually say "wait, but" quite a lot...
Overall, I am happy with how the current leaderboard/comparisons work. I might test some models on high, but for me, a better indication of true intelligence of a model/AGI is how well it does with "none"/no reasoning, than how well it does with high.
It's not "13 parameters to reason", they just rotated the full 8B parameter space in 13 dimensions and found a rotation that was still able to reason.
Depending on the latent structure, it's possible a nice rotation that would be perfect for some one specific problem, but you still got to search for it, and it's not a guarantee to exist.
But it's a nice step towards LLM parameter-space interpretability.
reply