I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device.

"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.

Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.

Scaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters.

I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.

Scaling is definetly hard - but there is no absolute requirement to put huge flagship models into this technology (although it might be possible over time).

A fairly dumb but FAST model has it's own totally distinct use-cases even if it can't be scaled in size. Think about a LLM-infused-Alexa where the response time is instant. Where you can request it looks at hotel options in Montreal, and it starts answering in half a second rather than a few minutes.

Plus some sort of slow smart + fast dumb combo architectures might also work really well for different classes of problems.

> Where you can request it looks at hotel options in Montreal, and it starts answering in half a second

Yet the answers will get outdated quickly whilst the silicon is fixed.

>Yet the answers will get outdated quickly whilst the silicon is fixed.

Bro is living in 2020 before rag was widely introduced.

I was told updates require replacing at least two layers of metal, though not whole thing. Was that not accurate? Can you say more?

You can change the prompt, and give the agent tools, and that keeps the agent up to date.

The agent doesn't know the date, or know what hotels there are in Montreal, it sees:

> System: You are an AI agent. The date is 11th August 2026. Your knowledge cut-off is March 2023. User is based in <date>. If you need to search for something to support the user say {search:<term>} and a list of options will be provided along with instructions on how to access. Or say {help} for a full list of commands.

> User: Can you help me find hotels in Montreal for next weekend?

The AI then interacts with the tools given in the base prompt, which can obviously be updated. So it then goes:

> AI: Of course, let me search for that. {search: hotels in montreal for 16th August}

> System: [Provides list of websites]. Say {read[n]} to read option or say {start subagent:<goal>) to register subagent.

> AI: {start subagent: List hotels on booking.com available on 16th August}.

[etc etc, then eventually]

> AI: Yes I have searched for you and I found a few options!

While you can't change embedded knowledge, a good model knowing that the date is 5th January 2040 can infer certain things (e.g. while it might not have been trained on certain deaths, it can probably guess that it should search before answering if it means a person would be 102 and their last information is from 2024)

You are talking about two different things.

Yes, to update the blueprint for new models two layers will be updated. That is the NN.

To instead update the data on which to operate you could use a RAG to query.

(As in "the Pathfinder 2.0 NN is on the chip; the geodata is in the OpenGeoMaps dump-DB-nightly" - not really overlapping with LLM+RAG but may give an idea in a different scenario.)

> before rag was widely introduced

And when did RAG start to work properly as a mature, reliable technology?

I mean, it's pretty damn reliable now. Has been for simple high contrast Q&A for a while now.

I've just been doing research and experiments for work related stuff.

Typically we've used plain embeddings for a lot of high contrast documents aka discrete facts.

However I've been working with a >1000 page document of complex procedures with incredibly low contrast where embedding falls flat.

There's top down/graph searching, bottom up/embedded; alts like colbert, reranking, reasoning, search agents and now (though seemingly quite new) specific search agent models.

Ultimately I found that a reasoning enabled search agent doing a hybrid of bottom up (with reranking) followed by top down, gave the absolute best results. Paired with Luna for cheaper and faster tokens it benchmarks pretty well even for vague references to procedures.

I would imagine that search specific models just coming out are even better and I'll have to evaluate using these but for now the above works well for us.

Having an agent get vector search results to use as anchors and then being able to explore the sections and subsections above that, then eventually digesting as much as is relevant (big context, cheap tokens) is amazing.

Very interesting and very good (though, let us say, much more recent that ancient): only, it paradoxically does suggest that the computational cost (when compared either to a simple "search by vectors distance" or to the task failures that we can have witnessed by chief implementations*) strongly suggest the benefit of 1000x speed boosting, energy conservative Taalas board.

*(Of course it has "always" worked well for «simple high contrast Q&A», ever since the base embeddings technology worked properly: that is almost by definition; it is on real world use cases, where the nuances of reality are present, that it failed miserably.)

And outside of idiotic demos, who exactly is going to ask an LLM to look at hotels in Montreal for them? This usecase has never made sense to me in the slightest

In the slightest? The user goes to their computer and types in "Montreal hotels" and Google comes up with no shortage of results, including a bit from their LLM. So that's already happening, but how do you narrow down the results from that initial search? Click around on Expedia for an hour? You probably know what you care about, just tell the LLM that you have dogs or are a vegan or whatever instead of wasting a bunch of time doing it by hand yourself.

> who exactly is going to ask an LLM to look at hotels in Montreal for them

Anybody who has a specific informal query ("SELECT ... FROM ... WHERE has_carpark AND ... ORDER BY score(has_jacuzzi , walk_distance(...) ...) DESC") but does not want to research and cross the different scattered info himself (does not want to build the virtual DB himself).

I mean it was just an example, but I get AI to do things like this all the time... ChatGPT planned my latest work trip by looking through flights, hotels and pulled together 3 options of itineraries for me, and wrote an email I could send to my client with estimated costs etc.

It's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about one transistor per bit, but can probably be packed more densely than general logic. Something like, say, 150 megabit/mm² is not a lot. N6 has a 850 mm² reticle limit. This roughly tracks, the article says the chip has 8B parameters and apparently spends about half the area on ROM. There's a reason AI accelerators just use a ton of silicon area (each HBM3 die is >1000mm² of silicon). I imagine this is not terribly viable unless they make it a lot more space efficient e.g. using MLC ROM if they don't already, or use stacked dies with a ROM-optimized process. And then we're back to not cheap, though reticle chips were never in the cheap area to begin with.

Taalas exploits the low cardinality to store one 4-bit weight with one transistor. (They are using metal layer traces for the ROM, and connecting an access transistor to light up one of 16 options.)

Their system is honestly very efficient for the weights, the problem is the KV-cache. That's why HC1 only supports such short context, they use SRAM for that and spend most of what's left of the die for it. The recent advancements that made attention more efficient are probably going to be very useful for them.

Do you really need a KV cache if inference is that fast though?

... Yes. Quadratic is really bad for large enough n, and you need that big context for useful work.

A full wafer like Cerebras is about 60x that, and N2P has about 3x the transistor density. So right now it's technically feasible to etch a 1.4 trillion parameter model. So roughly DeepSeek-V4-Pro class. Imagine that running a factory, for example.

Cerebras have special techniques to work around etching errors / bad cores on their wafers. This is possible since their wafers are effectively hundreds of identical copies of redundant cores. Can't do that for a globally unique model.

Etching failure in that situation would be like brain-damage in a human, all sorts of weird effects would start appearing.

There's several ways to engineer around that as the errors are detectable. There's a big literature on how to trade off speed or transistors for error correction. [1]

(Is Cerebras doing something novel? CPUs and memory blocks have been doing those things for a long time too, since the error rate is otherwise too high for normal size chips as well)

[1] see eg https://www.vlsimentor.com/dft/redundancy-bisr to get some basic concepts

A few hundred bad bits/transistors in a trillion+ parameter model would compromise its abilities not one iota...the models are inherently lossy and resistant to "brain damage"...

> (each HBM3 die is >1000mm² of silicon)

Did you mean that each HBM3 stack is that large? Because it only takes one glance to see that the memory chips are much smaller than reticle-sized GPUs they sit next to.

I have found that for some personal prose-related projects, QWEN 3.6 35B A3B is an amazing model even quantized down to 4 bits. I actually find it's "writing" style as a GM for an LLM-powered solo text adventure game, better than even some of the faster/dummer frontier models like GPT-5.6-Luna or Haiku 4.5, and it runs (slowly) on a 3090 with a 80k context.

So I have faith in these embedded LLM chips when it comes to fun projects like that. I have not personally found my quantized QWEN good at agentic tasks, though, and it LOVES to make shit up when asking questions about documents in the prompt.

Consider a model like https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0 which just came out. 0.9B parameters and very accurate for doing a very specific task: document features classification.

Now imagine you have a chip which is just that model, but can do it at absolutely insane speed. Like tens of thousands of documents a second.

Same for things like text-to-speech or speech-to-text. Think of the accessibility wins if subtitling becomes insanely accurate and fast and omnipresent.

There are all sorts of domains like that, and the trend has been such that smaller models are getting smarter and smarter. If you can stick them in parking meters, traffic lights / street crossings, mobility aids, etc etc I just see so much potential win.

350x is only about 10-20 years of improvement, using CPU FLOPS as the benchmark.

Haven't CPUs largely plateaued? They're just getting bigger, more power hungry, and multiplying cores.

Physics has hard limits and Moore's law is long dead.

AFAIK these don't have to be CPUs. ASICS are well suited for this purpose. This will be the most cost effective way to run LLMs.

wouldnt you want gpu, fpga, or dsp as the benchmark?

its lots of parallel calculations, rather than one blazing fast one

There's certainly incentive to do so. And its only an engineering problem haha.

You could imagine different layers on different chips, though, i think...

Yes, it also opens up a faster recurring revenue model for hardware companies, faster model obsolescence than how often you change a computer, a server, or a GPU card. I hope they can figure out trillion-parameter models rapidly. Nvidia happened to be the best option for AI after building machines for graphics, so it makes sense they weren't the best idea from scratch for this specific use case. Especially given the scale of the demand and the possibility of recurring revenue, I hope a lot of smart people will try to solve it, compete with each other, and deliver us extremely fast and cheap intelligence.

And one can say LLMs are not as smart as a human, but a lot of the reasoning humans do for product and service generation isn't smart at all—it's just a bit of fuzzy input/output plus some reasoning rules. And then if you hook a robot up to the LLM, you can get results in atoms instead of bits.

I'm very excited about the future. I also hope it will stop money from flowing to bureaucrats who are incentived to keep the problems open to keep the money flowing, and instead facilitate sharing directly with the people (for example, no money to the state to solve homelessness—instead, spay instead with intelligence output to build a house and provide food as part of taxes.)

Now the fun part, how will having an LLM in my washing machine help anything

Think bigger: A washing machine and dryer with an arm and a camera.

It can sort your dirty and clean clothes before washing and after drying, based on your preferences.

It defaults to common sense things like washing similar materials and weights, keeping delicates in their own load, grouping sensitive colors like red.

It can insult the plumbing with "you're a dumb pipe".

same way that having wifi does. by providing no actionable value, but boosting marketing materials

WiFi is fine for notification although Bluetooth would be enough to not get forgotten cloths stuck there for days.

But why it calls home and why we have to create accounts to just get a notification.

Never mind washing machine, how about a missile or a drone?

Or a bomb.. "let there be light"

https://www.youtube.com/watch?v=h73PsFKtIck

You load it, tell it what's in and it sets the program, tells you what it set and why. You approve and off it goes

I do that today by turning a dial and pressing the start button.

Not saying it's the superior way, but to be honest I wouldn't know how to start some washing machines without consulting manual first, having very dumb simple appliance myself.

I'm sure the manufacturer will figure out a way to:

a) Have it display advertising on a screen on its front size (like some fridges now)

b) Only work with approved laundry pods

c) Other new revenue generating opportunities for its manufacturer

Tell the washing machine. "Washlexa I'm putting my gym clothes in. It was a hard workout I sweated a lot!"

and the washer now knows what to do. No more temp, length, 2nd rinse proxy controls.

And result is tiny little clothes, because 80C seemed right for the job.

Washlexa:”Sorry, you were right I was supposed to use regular programme, but I used wrong one, do you want me to wash them again?”

make the "washing machine" a bucket with a robotic arm that you can put in any bathtub

I now picture people talking to their washing machine like the intro of american psycho.

I have seen smiling people talking to their connected car, replying with a dumb condescending bad-actor voice.

I had also seen Frankie Boyle in his parody of Knight Rider: "Michael, I am Kitt, your car, stop taking the medications, they want you to take them so you will be unable to talk to me"...

Washing machine was so inefficient whole time, it just heat water and spins. When you had to do by hand the hard decision is when to stop.

I believe there is a lot we can improve in washing machines

Assistive technology.

Imagine an energy efficient IC for a small multimodal model that can do voice to text, text to speech, question/answer, tool calling, and structured output.

Wire that up to a microcontroller that parses the structured output to constrain the model (rather than giving the model direct hardware access).

Now you have an assistive tech mode for supporting vision impaired users without requiring them to configure an app on their phone, pair devices, etc. And so now the user can just speak to the washing machine to tell it what to do.

And because models are getting better and better at multi-language support, you can rely on a single model to cover a wide range of spoken languages. And therefore you don't need a bunch of variants of this chip for a single product line.

TLDR this gives a path to replace "always online" and "wifi enabled" devices with fully on-device capabilities without being forced to abandon assistive technology support.

I don't think that the ability to record and transcribe people's words will encourage manufacturers to remove Wi-Fi functionality. If anything, that would encourage them to keep it. Imagine the marketing possibilities.

sure but once the capability is there it's not unlikely we'll see regulatory pressure from regions like the EU or certain US states to ban or heavily limit sale of "collect all your data" wifi enabled devices.

IMHO the only reason there's not been pushback from the EU already is because so much assistive tech is only possible with wireless connectivity currently. Once privacy conscious assistive tech is viable those crackdowns will probably be on the horizon even if only for the security risk associated with millions of internet connected devices with poor security posture.

that would be so cool, but it'll be a long time to make the cost of an AI chip cheaper than a wifi enabled one if that's even possible.

It can ponder the meaning of its existence.

"My Job Is To Open and Close Doors" [1]

[1] - https://www.youtube.com/watch?v=49t-WWTx0RQ

It's possible in the future we will have Rick and Morty style AI in literally everything just because it's so easy to add it.

    Sentient Switchblade: "Hi Beth! You've gotten taller! Shall we resume stabbing?"

Commoditize your complements - still a winning strategy.

If you make chips, you want models to be free.

I think this combined with a bit of memory and something like the “high-bandwidth flash” they just announced (if it works out), could be an interesting thing for some resident (burned) experts + active moe streamed from HBF.

I expect one day having small lower power demand drive sized devices with proprietary burned-in models that are quite fast running on-device in robotics and such. Commoditizing LLMs via burned and locked hardware seems likely when LLMs have stabilized (when we reach a year between releases again) and the hardware is capable and “disposable” enough. “Buy a robot and upgrade it forever* (5 years) with newer models (sold separately)”, at least until the planned hardware obsolescence that the “interface has changed to support newer hardware, so you’ll need to upgrade (again) to use the latest features”.

The plans basically write themselves.

The Taalas chip already had a memory region to keep fine-tune weights.

I like "good enough" LLMs for search and quick trivia. But "cars, appliances, etc" is exactly where LLMs are between noxious and dangerous.

My food processor could use a self-cleaning feature. It could only be made worse by some system that, IDK, changes the setting based on off-hand comments about "I don't know what his beef is.."

I think you’re making another good point as well:

Specific traces for specific inferencing will mean that some generations get deprecated. Look at H265.

According to aljazeera's article, "the core investor would be a New York venture capital investment firm created by Joshua Kushner, the brother of US President Donald Trump’s son-in-law, Jared Kushner."

Welcome. Not a dupe if some time has passed between submissions without much discussion or upvotes. And this is the source.

Welcome. Not a dupe if some time has passed between submissions without much discussion or upvotes. And this is the source.

Tried it today. Works fine for navigation. I am missing live traffic info. This is the one thing keeping me in Google Maps the past years.

Same. Apparently there's ongoing work to integrate live traffic: https://codeberg.org/comaps/comaps/projects/21877

> RDW said the EU sets stricter requirements for safety during vehicle approvals than the U.S. "This means that the FSD Supervised version in the U.S. is NOT comparable to the FSD Supervised version in the EU," it said, without providing further details.

The Netherlands has one of the highest road safety standards in the world. More info on differences between USA and NL on the site of RDW (Dutch)

https://www.rdw.nl/nieuws/2026/toelichting-rdw-op-europese-t...

Interesting to put this into perspective to Tesla's master plans from 20 years (https://www.tesla.com/secret-master-plan) and 10 years ago (https://www.tesla.com/master-plan-part-deux)

My 75" Samsung The Frame (2024) uses 70w in 'art mode'. It has a motion sensor and you can configure to fully switch off after some timeout.

I see a lot of blocked requests in my OPNsense firewall (not sure what exactly) but I see that with almost all 'smart' devices (which I like to keep local).

Wow, that really puts the 10W my laptop uses into perspective.

Not sure if it's the case now, but early Frame models had fans which could be quite audible.

The system I worked on never had fans in, but was rated to operate at 75C instead.

Android source code will be released twice per year instead of every quarter.

Great. So now nobody gets bugfixes until after the main vendors get priority access to it. for six months.

There's no way this isn't intentional hostility towards forks.

> There's no way this isn't intentional hostility towards forks.

Of course it is. But it isn't new. This was declared in March last year. We discussed it a lot here. It's only now that it's going into effect.

You guys are getting bugfixes?

Ofc to the root kit installed by "them", better battery life and WiFi stability.

/s

The security fixes will be published normally.

Security fixes are hardly the only software problems Android has.

The writing was on the wall the day they shipped Doze with GPS as an OS feature.

All those years back I started calling it, since I built software for (long-lived) HMI devices that ran on Android

We were doomed when they stopped shipping a dialer or SMS app in AOSP.

“Phone by Google” is disgusting.

And then Signal removed SMS support so we're stuck with Gogole's crap.

There are several SMS clients on F-Droid.

Yea, SMS and phone apps are quite numerous. I don't think it's a problem, the subsystems all the apps use is open enough and not hard to build against.

Except for RCS, that's completely locked down and is pretty solidly becoming literally just Google. Fuck RCS.

Not just Google is the problem, the entire industry is the problem. Almost all of the cell-based standards are locked away and purely depend on the operators, major infrastructure companies like Motorola, Ericsson and Huawei and modem implementors like Qualcomm, Apple or Broadcom.

Implementing them independently is extremely difficult and even if you manage to do it you cannot have them commercially available due to radio regulation and patents. Even academic research can only be done with collaboration of those huge companies.

It is impossible to make a phone that is LTE capable completely independently (or even without nation state support). You cannot implement VoLTE or RCS without support from the carriers. They all have their own proprietary protocol on top of the standards.

Google has basically infinite money and their own patents and industry relationships and government support so they can figure out RCS. An indie company, even with infinitely motivated engineers and good funding do not have any of it.

It is not completely impossible, if you do it in a clever way, e.g., make the modem removable, https://puri.sm/posts/breaking-ground/

For purely data, sure, that works. I have actually done it for multiple projects but for industrial purposes. It is slower than what a phone can achieve with an integrated baseband and SoC but would be good enough

For VoLTE, it is possible to get very basics with external modules but it is also very time and operator dependent. You need to have the profiles of each and every single operator you may support. If the phone would be globally available, this means thousands of profiles. Your module needs to be configurable after deployment. You still carry the risk of operator changing their profile or switching to a different encoding that your module doesn't support.

RCS is completely proprietary to the specific operator. There are currently no external modules that supports it, nor I beleive will be due to the complete proprietary nature. Google and Apple internally handle their pairing with the ones that support it.

On top of these two, you have actually a significantly bigger problem. The reason companies like Qualcomm integrate baseband chips with the main SoC die is the power efficiency. With external modules you will never reach an integrated circuit efficiency. Moreover you are sacrificing valuable battery space to the extenal module.

This is all true, and my phone (from the link) is not very energy efficient and doesn't support all mobile bands. But it exists and is usable. Calls work for me, too.

> You still carry the risk of operator changing their profile or switching to a different encoding that your module doesn't support.

Do you change your module accordingly.

At this rate, given a few more years Google will announce that they're moving to a "once a century" release model for AOSP.

They'll decrease the frequency by a year every year so they never actually release it again

Would it be so bad? The recent Androids have been going downhill. E.g. the mandatory edge-to-edge nonsense.

Got to say, I like the current Android versions. In the early days I flashed my Motorola Defy every second month with some cool new ROM. Always rooted and Xposed, always enabling something new.

Now I run a S23 Ultra and after two years it still does everything I need. OneUI 8.0 and Android 16. For work (app de) I also have a Pixel 7a, always with the newest Android Beta. Also works well.

Even the entry level phones work OK to pretty good now. My Samsung A16 5G (also for work) functions surprisingly well for 150€.

> Now I run a S23 Ultra and after two years it still does everything I need.

Maybe, but it is fully under Google and Samsung's control, and is choke full of spyware. You couldn't pay me to use a stock (Googled) Android phone for this reason alone.

Back when I used Android phones, tweaking was pretty important to me too. I still remember when I installed CyanogenMod on a Motorola XT1565, those were the days... Eventually, LineageOS, and then some new phones happened, not all of which were rootable, though I eventually ended up with a OnePlus 7 Pro which was pretty tweakable and even opened the possibility of bootloader re-locking, until a TWRP bug wiped my device and I pretty much stopped tweaking. Was never quite able to get EdXposed working right again...

How well is rooting supported on these newer Android versions/devices? If I install LineageOS on my device, for example, I can be reasonably sure that Magisk will work fine. But how well does it work on a stock, locked-down ROM?

Most devices doesn't have unlockable bootloaders now thus you can't even root them unless it was a popular device and a temporary /finicky hack was found.

This is them trying to strangle Graphene and LineageOs. We desperately need an ecosystem where manufacturers are legally compelled to publish the source code for their drivers and similar so as to make it easier for alternative Oses to exist.

Android will soon become fully closed source. The writing is on the wall.

The founder of GrapheneOS commented on this a few days ago here on HN, and basically said it doesn't impact GrapheneOS.

https://news.ycombinator.com/item?id=46550366

No, neither of them register as a blip on the usage stats and both have, and will continue having, priority access through OEM partnerships.

Plenty of drivers are proprietary. There are many ways of doing so, like much of it can exist in userspace, or in firmware, or using a shim in the kernel.

Just because it's legally required doesn't mean everyone obeys the law I agree.

I assume this means the reviews under "Trusted by Listing Agents" are AI generated as well?

No, I had a couple of agents give it a spin. I am letting them use it for free in exchange for feedback.

I've been able to successfully update a windows 10 machine which was not enrolled in the ESU progam.

The model is openly distributed, enabling others to improve it. Currently there are numerous uncensored and improved versions of Deepseek r1 out there already.

This community effect of improving models is why in my opinion the best LLMs will soon all be open.