I also believe this. Post-training LLMs with vague metrics can only be achieved with RLHF, which is not impossible, but extremely costly and difficult. Instead, companies will opt for RLVR, focusing on math and programming tasks. This pushes objectives away from writing quality; often far away. That is why older models, in my view, actually read better than newer ones. It's by design.

You can brute force it by making it try random stuff then judge itself on it. You don't have to always use an LLM's output. Sometimes you can use that plus other things to add flavor. An LLM is actually really good at judging if something is good or bad. It just has a really hard time coming up with new things. But if you had unlimited compute you can throw in some rng and whimsy and get something resembling what humans do.

I agree that this should be something that researchers reflect on. GPT-2 is one of the primary models to research on nowadays, and many recent developments have come from studying it as a test bench.

Imagine if CRISPR was considered "too dangerous" to publish because of the potential ethical ramifications, and that only a special few should be aware. It is utter self-righteousness, and it is shameful behaviour. The world cannot adjust itself to what it cannot see, so you risk greater catastrophe by keeping it secret.

The open dissemination of knowledge at every increment is the only way for society to truly deal with what is to come.

It's good to see validated numerical proofs seeing a resurgence now that they are substantially easier to achieve.

Others might be able to chime in, but my experience is that AI is effectively taking proofs that were once iterative (pushing current ideas further, tightening arguments) but considered time-consuming, and turning it into very low-hanging fruit. So stuff like this, as well as the new lower bound on the asymptotic ratio of zeros on the critical line, really aren't that impressive anymore. But it is likely necessary to do nonetheless, just as low-hanging fruit has always been.

The real frontier remains (even if many were not operating there) the development of new definitions and ideas that push right past the walls that were previously there. AI still seems to be shockingly poor at doing this, and you can feel that when you use it for proper hard problems.

In a frontier scientific research environment, funding is often limited, so personal subscriptions are more common. Fable can hit a 5-hour usage limit on the Max subscription tier before it finishes a single complex math prompt. Most of the time you get one or two if you reduce context and question complexity. Most of my colleagues are cancelling their subscriptions because of that, and just using Sol instead, which is virtually unlimited on the Pro tier.

Hmm. Would you mind sharing an example of a complex math prompt that you would use? Because I found that even Sonnet can solve fairly complex math problems fairly easily if you give it the right tools, so I'd like to give it a shot myself if you'd like.

No it really is about the test suite, and provably so. As another poster pointed out, speed is a superoptimization problem and the test suite provides the constraints. If the constraints are appropriately set, even a naive genetic algorithm will eventually improve the outcome over time, provided suitable mixing of the proposal scheme. LLMs provide measurably better proposals than naive approaches, so the entire chain is sound. The issue really is an inability to set appropriate constraints on what the user is looking for, and poor quantification of the multiple objectives one should try to balance in practice. What's great is that's a human problem. Diverting that to the models is obviously a disaster.

I agree that there has been a glut of subpar developers for years, and that has lowered the bar significantly. This is mostly because core values shifted. So let's keep our eyes on what really matters rather than acting elitist.

Tests do not prove the absence of bugs.

I hope you understand the context in which that was said. The point of that statement is that the only way to rigorously verify correctness of a program is by using formal methods. Those are often too difficult to achieve by humans, which is why there is an entire program of developing autoformalization using LLMs. You are supporting what I have said.

In practice, no one rigorously "proves" that their program works at present, and well-written tests do suffice. The definition of "well-written" here is circular, granted, but there isn't really an alternative. Even strong programmers should be writing high-quality testing suites.

Llms are great at converting natural language to a spec because that's a translation from one language to another.

Only a moron would then use them to code up that spec.

The Chinese labs are picking up on the low hanging fruits on efficiency, and no, you do not need to abandon transformers, you just need to push them closer to the more computationally efficient architectures of the past. OpenAI and Google seem to be trying a few things too.

Anthropic clearly are not though, and to call their operations wasteful is an understatement.

Even if you get all of your games via Steam, provided you have them downloaded, you can still run them without Steam if you know where to look. Obviously GOG is far better in this regard, but preservation is not a concern on PC, outside of games that are reliant on a server.

Starting next generation when prices are going to be obscenely high, I'm really struggling to see why anyone should invest in a console as opposed to a (mini-)PC.

Depends. Steam is not DRM itself, you can open the game folder and copy the game to another computer and steam doesn’t care. The game often has its own DRM though which will stop you.

GOG is different in that they ban DRM outright.

Some games won't start if Steam isn't running but you can work around that with a fake steamapi.dll, it's not a deep DRM check.

> The game often has its own DRM though which will stop you

I think you missed the "know where to look" part. It's called a Steam emulator, for starters. Note that I speak about this strictly for preservation purposes, as people who raise Steam DRM as an obstacle to preservation have not done their homework.

My expertise lies in deep learning theory, and yes, the "intelligence" is coming primarily from scaling up, among other things. There are good reasons for this, but essentially it comes down to taking advantage of a narrow statistical trick, where a very well-crafted model/optimizer pair that has a strong implicit bias toward simplicity can exhibit progressively increasing performance with respect to model size. Marcus Hutter's lab has shown that you can phrase this in terms of Solomonoff induction, so this bias is truly universally effective. An effective bias can continue to improve performance with larger model sizes by taking advantage of the curse of dimensionality in a way not dissimilar to how more data generally gives you a better answer (indeed, there is a duality taking place here, but I digress).

To be clear, it is an extremely narrow model class that can do this; we just got "lucky" and worked our way to it. That's why we still teach general statistical principles which often forbid this sort of behavior as a rule of thumb.

Neither Claude nor GPT are acceptable for writing English text. Personally I have found Gemini to be far better, and that is really all I use it for.