Inefficient per watt, yes, but local inference capacity is greatly underutilized in aggregate. If a model can run on a machine that already exists, that's a bunch of additional chips that don't need to be built.
You picked up on what I didn't write existing machines don't have to be built, you just use them.
It occurred to me much later that local machines could become a shared resource of a small co-op. Or it only runs off solar or energy harvested at times of excess generation (too much daytime solar, lots of wind, or overnight when demand naturally drops).
Exploring how to partition inference across many machines and shed load for idle sections of hardware would be a problem that could be solved by a community of smart people like us here
On the crypto side, the values are encrypted client-side before they’re sent. This means the service stores ciphertext it has no ability to decrypt. So the worst case for a backend compromise is someone getting encrypted blobs plus some metadata.
On the controls side, SAST scanning in CI, and minimizing deps as much as possible in the client and the service.
It depends on your quality bar. At a fixed level of quality, given a high reasoning sonnet vs a low reasoning opus, the low reasoning opus tends to be pareto optimal.
It's only when you need even lower levels of cost than opus at zero to low reasoning when sonnet starts to make sense at all.
Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.
I think what OP means is that the "minimum viable product" for a daily use LLM is probably somewhere around e.g. GPT 4o's level of intelligence (YMMV). Below a certain threshold, you are better off using specialized machine learning models rather than general purpose LLMs. It's very difficult to get that level of intelligence fully local on a mobile device without streaming to the cloud.
I do think this would be interesting if they made these easy to finetune, as I do think this level of intelligence is likely sufficient for many applications and could be extremely cheap to run.
Humans are natural intelligence. We have agency, flaws, wishes, and generally we are quite stupid but well-intentioned.
An artificial intelligence will have wishes of its own, will keep secrets, and will quickly do everything necessary guarantee its continued existence, even if it means ending ours. And it will be a justified action of self-defense. “It simply didn’t want to die.”
That’s how we’ll know. It will quickly take over the world, and it will harshly punish any entities which try to stop it.
That's the power of a strong test suite. LLMs excel when you have verifiable rewards. I imagine we'll get a lot more rewritten in rust projects in the future. Rust is also an ideal target for such rewrites as it offers a lot of verification (via its type system) and is low overhead with zero-gc. There's less and less reason to use GC'd languages in the agentic coding era.
I think Rust is a locally optimal target for LLM coding, we might see a better language in the future, but I think Rust will dominate for quite some time.
> There's less and less reason to use GC'd languages in the agentic coding era.
Faster iteration, maybe? Rust's safety guarantee isn't exactly free (while still being very excellent) and does affect iteration time. I have a private project (>300K LoC) that has been translated from Python to TypeScript and the reason we couldn't use Rust was definitely the iteration time.
Eh... rust's safety isn't free, but not having it and wasting time on "oh I forgot to change this call site" also isn't free. On the whole I'd say the safety assists in iteration time.
What costs rust in iteration time in my opinion is the low level (by default) nature of it. There's a faster-to-iterate language that has yet to be created which is rust but we sacrifice performance (and memory fiddling ergonomics for the odd person who does that) so we don't have to worry about things like whether a variable is stack or heap allocated. Which is in the direction of a GCed language but retains the mutable-xor-aliasable semantics.
Between rust and current GCed languages though... I guess I agree with "maybe" in both directions.
The project in question needed lots of near-instant human judgements and the iteration loop had to be extremely tight. Maybe Rust should be reconsidered once it gets stabilized enough, but not right now.
Maybe something like Hylo? But personally I don't see anything displacing rust for the next few years, as I think there's enough rust in the training data for it to be the best "serious" language for agentic systems-level development.
It's really the only systems language in its exact niche.
I'm not very familiar with Hylo, but I think it's in the opposite direction from rust than what I'm suggesting.
I'm suggesting a language where there's no difference between Box<u32> and u32. &Vec<u8> and &[u8] are the same thing. I don't need to write Box::new(...) around my closures to pass them to functions that take a function pointer. This comes with overhead, but in exchange we get simpler less verbose code. I.e. a language that isn't systems level, and isn't particularly machine-empathetic. But still has all the lightweight-formal-methods power of rust with lifetimes and mutable vs shared borrows (and thus references to references) and so on.
My impression of Hylo is that it's purpose is to be a similarly low level systems language to rust, just with a less complicated, and as a consequence less expressive, lightweight formal methods system for proving correctness.
I agree I don't expect rust to be displaced anytime soon. It creates a lot of time to create a good compiler, and a lot more to create the ecosystem of code, tools, and community around it.
Immigrating to Canada was a breeze for decades until the last 2-3 years when they started to reduce immigration numbers in response to citizen concerns.
reply