A decade ago, I worked on a simulation program that involved a C++ core with a Python wrapper and DB interface. End users cared a lot about throughout with a rather limited size, weight and power budget. We spent a lot of time optimizing the core -- but basically hit a bottleneck once we got to about 15% of the time that was spent in malloc-related functions. It turned out that was all in the Python layer. Probably there was some level of bad code in our Python code, but it was impractical to figure out where that was. I was shocked because I assumed the simulation core (which ended up almost allocation-free by the end) would always dominate CPU usage.
Both allocators and Python have probably gotten better since then, but it was a fascinatingly large and stubborn fraction of CPU time.
My guess is that it's not complete or stable or qualified enough for them to publish it as a supported feature. Maybe the power draw is too high and it triggers thermal throttling too quickly, similar to early Intel AVX-512? Maybe they are still sorting out how to have each process choose SVE vs NEON mode, so libraries using the other one don't cause a fault? (Note that this seems limited to M4 and newer CPUs.)
I agree this is an important feature. I have very often wished that my Mac had wider vectors.
No, they use the same definition of memory bandwidth as others, but Apple Silicon has a lot of memory channels. In previous generations, prefill has been compute-limited and decode is fast.
https://blog.exolabs.net/nvidia-dgx-spark/ outlines a combination of a DGX Spark and an M3 Ultra that took advantage of fast prefill on the Nvidia hardware and fast decode on Apple Silicon.
It takes almost 8 years for $3.25/mo of electricity savings to break even with a $300 difference in up front cost. Assuming the time value of money is zero, that is.
I can't speak to your specific setup, but if the two SIMs use different carriers then they will often need to use different frequencies. A phone could in theory use twice as many radios, but the emitted power limits would be the same, so you'd have reduced range or throughout for each (plus additional size, weight and power for the second radio). Access points have different SWaP budgets than phones.
When I travel, I try to use an eSIM that uses a network my home provider partners with. This seems to improve the chance that voice and data over LTE or 5G NR work simultaneously.
Dumb question: If the limit is L, why do regulators care if I have one device with two radios each outputting L or if I have two devices each with one radio outputting L. Either way, total output is 2L.
What is the objective?
* To limit range? Then regulators shouldn't care about 2 radios in one device outputting L, as long as one radio doesn't output 2L.
* To avoid cooking nearby electronics and people? How do they account for the unpredictable, theoretically limitless number of devices in an area? What about someone working in a cell phone store or passengers on a Tokyo train at rush hour? Do regulators just set L low enough that any realstic number of devices together is safe?
Usually, the exposure limit is based on an active phone next to an ear. RF field strength falls off with the square of distance, so a phone on each side of your head is most dangerous to the respective ear and bits of your brain close to your skull; each phone is considered acceptably safe to the middle of your brain and the opposite ear.
Is it that impressive? It is a model generating short strings from scratch, so I do not think there is significant token parsing or memory allocation going on.
In compute performance, 10M tok/s * 4096 parameter/tok * 4 byte/param = ~160 GiB/sec implied memory bandwidth, vs nominal ~300 GB/sec for the M5 Pro's RAM. That seems even less notable once we consider that 4096 parameter * 4 byte/param = 16 KiB of parameters fit easily within L1 data cache (the M5's efficiency cores each have 64 KiB of L1 data cache). 40 GFLOPs means ~10 FLOP/cycle, about 2.5 NEON instructions per cycle given that the core instructions are fused multiply-adds (FMAs). A random web page I found says the M5 family cores have a 4-wide SIMD block, meaning this gets about 63% utilization: respectable but not super high.
Yes, but their "run rate" assumes the current level continues unchanged, right? If revenue per month goes down then the projected run rate is optimistic. However, revenue has been increasing, so the reported run rates have underestimated revenue.
I think the better question is how much more than $65B/year revenue they need to cover what they are spending on capex and model development. I would bet money their revenue in the next year is over $75B (vs $65B), but also that their amortized costs exceed their revenue.
"Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated.
Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.
It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc.
Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.
> The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.
Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.
Yes, it's notable that they mean "production" in the sense of 40 vehicles rather than 40 models or model variants. It's close to prototype quantities rather than production; US law allows the military to build up to 10% as part of "low rate initial production".
Yes, this. At work, I have side-by-side 32" 4K monitors. At home, I have a 59" curved 7680x2160 monitor that is the same size as the two work monitors combined. My home setup is much more ergonomic than work. Beyond your point about angles being bad with two monitors, pretty often I have three things open that are wider than 1920 pixels worth of content. At home, each can be 2560 pixels wide. At work, I can either have bezels in the middle of one or shrink two of them. And the fairly aggressive curve at home means pixels are close to the same angular size.
Both allocators and Python have probably gotten better since then, but it was a fascinatingly large and stubborn fraction of CPU time.
reply