> It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.
And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.
What about multi-token prediction and speculative diffusion? That’s a different mechanism of prediction, even if it serves only to accelerate decoding.
As you say, that's just an efficiency play and, as I understand it, doesn't change the behavior of the models beyond perhaps a small amount of sampling noise.
Anthropic's API has two nines availability and Claude Code is a TUI made with React that can regularly consume more than 1GB of RAM, and the codebase is utter slop. They couldn't fix the flickering bug for over a year!
And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol).
It's not about the people there being smart or not, it's about their and the company's priorities, resources and what they choose to focus on.
You cannot just "try all possible optimisations". It takes time, effort, and money that could otherwise be spent elsewhere (especially for training, where each training run is especially costly, and optimisations might be promising early on, but cause the final performance of the model to be worse). You need smart people interested in unglamorous work, and if you're swimming in VC money, it's far more straightforward to just throw more GPUs at the problem.
> Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
I'm not sure what you mean? You can see the intelligence of LLMs progress predictably and stably according to scaling laws. LLMs have to encode language in addition to intelligence so there's a minimum bound for them to output sensible text (you can train specialised tiny models to solve basic puzzles without language). Start at around 127M and compare models of increasing parameters and you'll see a clear progression in intelligence.
> It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them
How is that a basic tenet? Simple, easier to parallelise algorithms that have lower memory requirements, or can take better advantage of hardware, or don't hit a plateau the more compute you throw at them, can absolutely beat cleverer algorithms. E.g. brute forcing rendering with Monte Carlo path tracing will give you more physically accurate results than ray tracing or rasterisation algorithms that rely on a bundle of hacks to approximate global illumination, transparency smooth shading, etc.
I stopped using Cursor because of how terribly optimised it is (worse than VSCode despite being a fork). It would routinely take up 50% of the CPU resources on my MacBook M4 and gigabytes of RAM for absolutely no reason.
I switched to Zed, and I'm never going back to Electron/non-native IDEs.
Why is it unethical? I'm both a freelance engineer and a business owner that sells software, and I've both sold my labour for equity/revenue share, and for a flat hourly rate.
If I charge a client $50k for some software and they made $1 million profit from it, good for them? As long as they pay our mutually agreed upon rate on time and there was no hostile negotiation, why should I feel suddenly entitled to more money if that wasn't in our contract? How do I know how much of the value is from my work and not their marketing or idea?
What you're saying seems as crazy as me saying that someone who bought my software for $99 and used it on a multi-million dollar project is being unethical unless they give me more money. How on Earth does that make sense? Should I be forced to switch to a royalty model? What if I make more selling copies at a flat rate, what if I don't want to have to investigate the finances of thousands of customers and have to deal with that whole trouble?
For me it's the same thing regardless of whether I'm selling my labour or a product. I can choose whether to accept a flat hourly rate, equity, or a mix of both, and usually the better deal is the hourly rate.
If I find a way to hire a software engineer for market rates (say, $200k/year in the US) and get $2M revenue from their work, good? They can ask for a raise or a bonus, we can renegotiate, they can leave if they're unhappy, but I'm not obligated to give them more money than was in our agreement anymore than they're obligated to give me their salary back in the project fails.
There's an argument that if someone agrees to a bad deal, that's their own fault. Where I think it becomes unethical is where there's a significant power imbalance that disadvantages one side.
Suppose I buy a painting from a flea market for $100, get it evaluated by a specialist, and then discover it's actually worth $100,000. In this example I have no inherent advantage over the seller; neither of us knew the value of the painting at the time it was sold.
Now suppose a famous TV antique dealer stumbled across that painting instead, and immediately realizes its true value. The seller recognizes the dealer, and the antique dealer offers to buy the painting for $25. The seller, trusting the antique dealer's judgement, agrees to the discount.
Would you say in both examples everyone acted ethically? This is a genuine question, as I can certainly see the argument that using the assets you possess to secure yourself the best deal possible is just business, and yet I would personally see the antique dealer in the second example as being exploitative.
When it comes to companies there's a similar disparity in power. An employee requires money to live, while someone founding or investing in a company often has enough of a financial safety net that they won't starve if the venture fails. Equally, any would-be billionaire is explicitly looking for employees who generate vastly more value than their cost. You don't get rich by paying people what they're worth; you get rich by underpaying them and pocketing the difference.
The other problem, and one you've touched on, is how do we assess the value of an individual employee? This is obviously not easy, and businesses also have no incentive to work it out or reveal that information to their employees even if they knew. On the contrary it benefits employers to keep their employees as much in the dark as possible.
Aside from the ethical problems there's a practical one. The very existence of billionaires implies that a significant number of people are undervaluing their work. It's a pricing problem that the market isn't solving, and is only getting worse.
A cursory search says 74-90% (in the US), but also that’s just tech companies and usually you need to be early. It’s also often in the form of options that take years to exercise and companies have gotten very creative lately in how they screw people out of them.
Demand is so high and supply so low customers will go to anyone that has any gear, period. Anthropic is paying xAI for GPUs from 2022, not the latest Nvidia release.
And yet Anthropic is paying xAI over a billion dollars a month for those out of date GPUs in their first datacentre (H100s being nearly 4 years old at this point).
Even A100s are still barely available on the major clouds despite being 6 years old.
Yeah most of the performance increases have mostly been from architectural improvements like reduced precision tensor cores. AFAIK FP4 is basically the limit for floating point matmuls, after which you need to switch to integer addition if you want to reduce bits, and I don’t think we’ve figured out 1-bit LLMs just yet.
Do you think the work will still apply to speculative/alternative decoding methods like MTP and block diffusion, which are making batch=1 decoding less memory bound? Kernel launch overhead and memory transfer become less and less significant as a % of time when computing multiple tokens at once.
Why not, it's one way to look at it!
Although I have yet to see other work with speculative decoding higher than ~1,000 tokens/s., because the other bottlenecks start to matter at that point, and they need to be solved to go further.
Our view is that MTP / speculative decoding could help getting a X multiplier (X = 2 to 6) on the tokens per second speed we currently achieve.
We are a bit greedy, we want to stack optimizations on top of each other to get the maximum speed possible.
It involves additional compute to verify the predicted tokens during the forward pass (it's like a small batch), which should be totally doable for dense models, and will be more tricky for MoEs because it could mean activating more experts and thus more active parameters.
Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.
And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.
reply