It is pretty clear at this point that current models are good at maths and problems with verifiable rewards. And puzzles are essentially math problems. Still a long way before we can say their "fluid intelligence" is effectively applicable to the real world.
I keep wondering why there aren't more real world tests.
Maybe hook up a bunch of the AIs to a stereo camera and a couple of microphones and give them control over actuators to so they can drive cars. Then lets race them around a somewhat complex course.
When they are good enough at driving on tracks, put them on the road. Maybe see which can drive a truck with 400 cases of Coors from Texarkana, TX to Atlanta, GA and back within 28 hours.
It’s so weird to think how computers have mastered stuff that we used to think took intelligence (like chess, go, mathematics problems) but are doing so poorly at things any idiot can do (like drive a car).
The stock market is not a reflection of reality and revenue and users are up. Software stocks are generally down because all the money is going to the current bubble (hardware) before we come back down to earth.
The article actually cites the opposite - reported users going down at many media sites, including Facebook for the first quarter ever, and Netflix now refusing to report user counts until 2027 when they used to proudly report user count increases. With that change of behavior, Netflix users are definitely NOT going up, and are likely declining
And until this week, they would release numbers every six months. Now they are NOT releasing numbers.
If the numbers were up, they would post them.
This sudden change of Netflix' reporting shows they decided it is less bad to raise the flag of halting reporting than to expose the size of the subscriber loss. That is not good. Maybe they are just fearful of showing any loss at all, or maybe it is a really large loss. Eventually, we'll know something.
No, this is just pseudo-science and resembles educational research where they try to find a causation to things that are probably just attributed to genetics. Think about genetics determining the size of your engine and motivation being the amount of energy supplied to it.
Yeah I don't think any of the labs have some secret sauce for intelligence either. It seems most of the advancements are still coming from hardware, making LLMs more efficient and throwing more compute and data at problems. And even those problems still require a lot of prompt engineering: https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98...
The secret sauce is training data. They’re not just taking advantage of more compute (which obviously is necessary but as mentions basically a commodity). They are paying billions to data labelers and making judgements about the nature of the training data they best need to make the product they want. This seems to get pushed aside as a minor point but it’s the primary differentiator of the big labs.
I'm pretty sure at this point that Anthropic is training mixture models (at least in the heavy pre-train) and deploying them dense with explicit loss on thinking trace coherence.
Having a thinking trace that is legible, coherent, and immediately implies the explicit turn output and/or tool use seems difficult if not impossible to reliably get from mixture models.
I predict MoE is a transitional technology, it's got too many problems and the benefits are...kinda grandfathered into the dogma at this point.
While scaling laws hold (more weights = better), and time / financial costs are not trivial the incentives are in place to have MoE. MoE means you can have more weights without increasing the critical path of evaluating it.
I am curious what you believe the problems with it that would cause people to prefer using less weights. I'm not following what you mean by MoE can't have legible thinkings trace or tool use when existing models with MoE can.
Weights are not created equal: while interpretability is a young field the prevailing view at the moment is that MLP (hence experts) in a mixture model are substantially where dense encoding of factual information resides, attention is even less easily interpreted but it should be uncontroversial that temporal/sequential modeling occurs here.
So it's more consistent with available empirics to say that an architecture can be characterized along a spectrum from fully dense to mixture (a sub spectrum) to Engram-style lookup, and the amount of model power allocated at this point or that will recover different performance profiles.
By far the most stark example of how much performance in reasoning is left on the table is Qwen3.6-27B, which depending on the task, comparison model, and whose benchmarks you believe outperforms mixture models 15-60x larger in total parameter count.
It's badly under-studied (in public) because of the paucity of modern dense models at the near frontier, but even that one data point pretty much rules out the cocktail party version of the Chinchilla-adjacent scaling thesis (which wasn't about modern MoE to begin with).
The "Mixture of Parrots" work is a good jumping off point if you want to get a modern literature review.
>the prevailing view at the moment is that MLP (hence experts) in a mixture model are substantially where dense encoding of factual information resides
Yes, because that's where all the parameters are. For reference in GLM 5.2 98% of the weights are for the experts.
>The "Mixture of Parrots" work is a good jumping off point
The paper shows increased performance on knowledge dependent task while having similar reasoning capabilities. This backs up what I was saying about how the weights unlock extra performance without increasing inference costs as much as a dense model would.
>model power allocated at this point or that will recover different performance profiles
While increasing the number of weights makes the model better, where those weights are does matter in how much better the model gets and also matter in regards to the cost of training / inference. Model design is a big set of trade offs and I see MoE as a useful tool that will survive in the trade off space.
>reasoning is left on the table
Even so, if there was 2 models with an equivalent amount of reasoning ability and priced the same would you rather pay for the one with narrow knowledge or wider knowledge.
>because of the paucity of modern dense models at the near frontier,
You don't need to be at the frontier to benefit from MoE. Even open source models that are behind the frontier, benefit from being able to host experts on different machines, and scale individual, commonly used experts separately from each other. On the other end with small models you are probably resource constrained so you want to maximize the tokens generated per second. This makes going for purely dense models niche like you are saying.
I think we're fundamentally reading from the same sheet of music but drawing different conclusions.
Mixture models have compute advantages at training time, everyone agrees about that, that was the original rationale (popularized at the time with `mixtral-8x7B` among others). This seems to be likely to remain an economically relevant strategy for (especially) pretrain: in a training setting you have already paid for fast interconnect at scale, a dense architecture doesn't buy you anything in a big pretrain and it costs you a lot of traffic and to a lesser degree batch size under the roofline. The heavy, FLOPs intense, interconnect intense parts of training run benefit enormously from MoE: I don't dispute that though I suspect we are well into convergence on the target precision (4) and the target format (NVFP4 or similar). At some point the whole Internet is in the pretrain at the terminal generalizing precision, the pretrains of the various labs start to look a lot alike, and the sauce remains in the later parts of training along with the proprietary data sets and what not. Frankly all the labs would benefit from standardizing the Common Crawl recoverable pretrain to greater or lesser degree, it would lower everyone's costs without changing the competitive landscape much. But that's my prediction/opinion, that's why I said "suspect".
The evidence is suggestive if not fully conclusive that mixture models are strictly losing in most regimes during inference: the exemplar of Qwen3.6-27B (which outperforms Alibab's own mixture model at ~ ten times the size) is very suggestive, and it's not the only argument for this. Because most/all modern MoE requires the activations of the previous layer before routing the subsequent experts, you are pretty much paying for the HBMe3 or GDDR7 to hold the whole thing even though some small fraction of it is under your roofline on any given token or draft verification.
This is grossly wasteful under all trajectories (even parity at N parameters between dense and MoE, which we have evidence is off by 1-2 orders of magnitude): in a "local LLM" setting (from bedroom to regional office, anything other than an NVL72 or Ironwood rack) you are sharply constrained by both total available accelerator DRAM and accelerator memory bandwidth: you are probably not getting under your roofline even with pretty slow tensor units (e.g. GB10). This use case matters and looks like it's going to matter more and more over time (the GB10 in particular is going into about a gigaton of RTX Spark laptops next year). In a datacenter setting, you're paying for extreme interconnect (training class hardware setups basically) for pure forward pass that wouldn't otherwise need training optimized gear. Even multi-trillion parameter dense models can fit in 8x or 16x RTX 6000 Pro style setups (hell, there's a DGX branded one) and with all of the geometry, tiling, scheduling, and interconnect needs mapped out up front, the design space is really forgiving on all manner of tensor parallelism, pipelining, KV cache sharding, it's a very friendly constraint space up to like, 3-5T parameters. MoE at inference time works for two groups of people: people who are willing to page experts in per token/draft batch in local LLM settings (not probably ever going to be mainstream, people really dislike that level of slow), and the vendors of extreme performance interconnect i.e. vendors selling training-class equipment as necessary for inference. And you pay in so many other ways: grouped GEMM is no one's idea of a good time, the kernels are fiendishly difficult, therefore they are not abundant, it's no fun.
The path forward here is non-obvious, and I don't claim to have it all figured out. But since you seem interested enough to carry the conversation past the pleasantries, a more substantial version of my thoughts on the matter can be found at: https://cdn.s4.gl/preprints/pairwise-interlock.pdf
MoE is just activating fewer weights per token than the whole model. It will continue to make sense for as long as compute is more expensive than memory (at scale).
even meta that sucks at doing anything is releasing frontier models. making an top ai is easier than making twitter clone( threads) if you have enough money.
It is not really about graphics as it is about the scale of a game. Many studios don't even create their own custom engines like they used to to push the most out of hardware and use Unreal Engine. Games take long now because they are much larger in scope and are typically open world. Even the sequel to Breath of the Wild took 6 years and the graphics aren't exactly staggering but the scope of these games have improved a lot which isn't a bad thing. You can get lost in some for dozens of hours. I would imagine studios like From Software even enjoy the ambition of creating games like Elden Ring.
Technology is what you make of it. If you want to make it your best friend and your only friend, that's your choice. But I would guess more people would use it as a personal tutor. If it works as well as the demo, this already crushes most language learning apps. Actually the only apps for learning language that it would fail to replace would still be Tandem and HelloTalk because it still can't replicate the nuance of real human interaction.
"Blame the individual" doesn't really make sense when we're talking about things that have a society-wide impact. As an analogue, personally I'm not on social media, but I am affected by social media because everyone around me is on social media and their happiness and wellbeing also impacts me. AI companies need to stop pretending they can just outsource responsibility for their products.
I think assuming people will use it as a tutor/learning tool is.. way too optimistic. A small fraction will, but the majority will just view things like a second language as something not worth learning.
Go read "Careless People". The social media people knew exactly how damaging their product was early on, as evidenced by the fact that they wouldn't even let their own children use it. I think AI people understand how damaging their product is too; and so I will never respect them unless they take responsibility for the things they create. Going "not my problem bruh, I got my bag!" is a bad ethical position.
