Another thing that sort of puzzles me about benchmarks is that LLMs are not deterministic and do not always complete a problem. So what are the results actually representing? The best run? The average? It is all in some ways a falsehood

That's a good point and conventionally if benchmarks aren't run as "one shot", it is denoted as "benchmark@K". Inference time scaling has historically shown improvement.

Generally though, many of these fairness complaints do go away if there is "3rd party testing". Right now, companies reporting their own benchmarks has all the problems that 3rd party testing resolves in many other industries.

I don't think it is vague in the slightest. Take the most simple examples, how many LLM's have you tested making them? There are stylistic choices pertaining to games that is well beyond a 0/1 reward. Even something as basic as breakout or flappy bird can have wildly different quality between models. Yeah, you could call this animal on a bike benchmarking, but I don't think it is. IMO the problem space occupies an interesting area where you can ignore the pass/fail and focus on the actual level of the model to do something beyond that.

I doubt the OP meant something like creating the whole tech stack for WOW.

You seem to think I was disagreeing somehow.

I was just asking what kinds of games and with which technology.

Neither is stated in the original comment, and the answer obviously isn’t “every kind with every technology”.

Your first failure was trying to get claude to do anything :)

It ain't bearing enough load, any load on this one.-

There is, in my personal opinion, a reason that reasoning is not front and center. Partially related to distillation.

From what I understand pre-training is totally irrelevant to this and as far as post training goes there will be multiple steps, for claude and codex and the like that ship with a harness, the harness is definitely included in evaluation. However, they will definitely include evaluation from a variety or even none, and settle on something that works the "best" for a release.

Disclaimer: No first hand knowledge

Tmux is good because: 1) sessions stay running if you disconnect 2) window management - you can split your screen, have an octobox a-la redzone style, and focus/unfocus etc.

This is based on developing on a remote server - but even locally, I find it invaluable. Multiple terminal windows are fine, but some times you want multiple windows. Even in a pre-ai world, you might want to run a process, see the code, edit, and maybe have htop or something like that. If you ever NEED multiple terminal windows for the same thing, tmux is really the answer.

One reason I find it hard to grasp is that when I work on Windows, it's almost always through RDP or VNC. There is never any question of a terminal (or a process launched from it) staying open between login sessions, because it never depended on any particular connection to begin with. It's just there, on the desktop, which might stay up for months before I need to connect to it again. Any terminals or other processes I launch on that system will be there until I close them intentionally, or until Microsoft decides to reboot the system without my permission.

So when I dip my toes into the Linux world, the first thing I do (or rather ask an LLM harness to do, being much less familiar with Linux than I am with Windows) is set up an RDP host. That way I can follow the same basic workflow on both OSes. I don't need to do anything special with terminal windows.

I guess if I were running truly headless systems that don't even have a VGA frame buffer, I'd be more interested in multiplexing. But so far I haven't had to do that, and would probably resist it because a persistent desktop strikes me as better in general than letting applications including terminals worry about their own lifetimes.

I have been in RDP land but I would never go back unless required (windows). While it could just be personal experience/failures, latency was not good and reliability was not good. Things are probably better as of late though I hesitate to assume software has improved... It is just much simpler to live in the terminal for dev work, especially with agents. That being said, I use MacOS and Windows daily for non-dev work. Either way, your points are valid, and whatever works for you works for you friend :)

I just wanted to echo your comment. Most people, in my experience, pursue an undergrad program with the intention of being employable. This is my personal experience, but also a learned opinion from being a teaching assistant for some years. Places like waterloo stress co-op, it is built in to the program. IMO the average person seeking an education are served under a model that places them in or near industry related work at some point. While I understand the original post is relating to PHDs, I think most programs (or rather, participants in programs) would love to have this sort of integration despite it being a hard problem.

Interestingly enough, Kitchener, adjacent to Waterloo has a large population of German-descent. I wonder if the OP's spouse's experience in Germany has any relation.

It is RLVR, Not a puzzle, Not leetcode

> Only systems which required less than $10,000 to run are shown. (Notes[1])

Am I lost or are their many models on this ranking (Opus 5 included) that clear this?

Many models are much cheaper through their subscriptions' included usage. That could be what's happening here.

Claude gives you something like $5000 of tokens on a $200 plan.

Isn't it ~$3000 per week? Extrapolating from the current limit on Pro plans.

They don't exactly say. I was extrapolating from some sessions usage (my figure was per month, so weeks times ~4).

I honestly just use GPT models nowadays, Claude models are too restrictive and more of a quitter and fable/whatever is just too expensive to be worth it.