The license you receive when you download Gemma off of Google's website is not necessarily the same license that AT&T gets when they deploy Gemma as a customer service bot. That's the whole point. AT&T can work directly with Google for a licensing and legal framework that provides certainty.
From what I’ve been seeing, the Mac studios do look like they have potential. I was looking to drop $10k-$15k on one until recently. After comparing a Radeon 7900 XTX vs Ryzen Halos 128GB vs M1 MacBook Pro 64Gb, I landed on just getting an external closure setup with Nvidia RTX 5090.
The model I’m specifically targeting to use at high speeds is Qwen 3.8 27b @q4ks. This model actually proved to be good at coding (it sits somewhere between Sonnet 5 and Opus 5 capability). M1 got 10 tok/s, Ryzen Halo 20tok/s, and Radeon 7900 XTX 50tok/s (can only do 128k context window in Radeon card).
The prefill gets extremely slow around 50k tokens in context window (whatever prompt processing stage entails could be wrong about phases here). It takes about 2 hours to fill the context.
Even with a drafter model intended for speed instead of mtp, I can’t get past 70tok/s, still is extremely slow to process prompts as context grows, and drops down to 40-50tok/s anyway making this config still moot for improvement on my Radeon card.
The only thing I can point to slowing me down is bandwidth of the card itself.
I am waiting to actually get my 5090 right now and I am betting that the 1700 Gbps of capacity will fix my prompt processing speeds. I don’t need full PCIe lane bandwidth to serve my house I just need to load the full model into vRAM and let the GPU do its thing.
Additional benefit to the external enclosure route is being able to migrate the inference between devices more easily. I can develop out the infrastructure then migrate the card to be hooked up to a shared node in the house with all the tools necessary for my family to take advantage of the privacy enhancement that comes with local inference.
How are you actually using the local model? I've played with Qwen 3.8 27b on ollama and the coding harnesses (Claude Code and OpenCode) seem to fail way more often then using the cloud models. And by fail, I mean the edits don't apply cleanly, it goes to add python code, but doesn't indent it properly, or the edit doesn't apply and so it tries again and again and eventually wipes out a different function then it "intended". It just gets really frustrating compared to the relative stability of Claude Cloud.
My use-case is only coding, every model sucks at writing good literature and there is no way around that (have had people try to debate me on this, but it's a taste thing, I have extensive English writing skills from my school years).
Prior to two weeks ago, I was just using Pi and Ollama.
I have tried my hand at putting together a few harnesses and I finally landed on what I like. Been working on this small app to handle running llama-server for me from any device that has the llama-cpp stack setup: https://github.com/SamInTheShell/loom
try using pi harness, hae not encountered these sort of problem myself
also yuou can ask codex to look at the transcript and figure out the solutions to tool call failures that way
Just wanted to share, I had a SaaS AI drop me a script to bench ninfer against llama-cpp and it is impressive. The place that it's doing better at than llama-cpp seems to really be late in the context window.
Thanks for sharing, I'm definitely trying it out after I get through my project milestones for the 5090. The README claims 700tok/s for Qwen 3.8 27b, that would be amazing, I'm only expecting an increase from my ~50tok/s on my Radeon to 200tok/s on the 5090.
Junk. Just another company that doesn't understand why I pick up a phone or use a support chat or send an email. Before I even started in corporate, I stopped checking email. Now when I call in to get support for things, everytime I get a clanker, I'm guaranteed to talk to a human customer service person to chew them out and cancel the service.
Just use a harness that discards all except the most recent thoughts. This model is probably the first real small model that does well on long horizon tasks.
I did. Distributed too and supports erasure coding. The reason is because nothing quite had the architecture or features I wanted in a deployment pattern that matched my use case. All major databases that are validated are complicated to operate. I built something that operates one way for a non business use. It’s great, doesn’t need to serve a million people, just about 5 or so. Doesn’t need to be the most optimized thing ever.
I’m opting out of any corporate targeted code for personal use, anything not designed for user experience gets scrapped when I’m the user and I have Software on Demand as a Service.
It's worth running, even quantized. I liked Meta Muse Glimmer's outputs, but qwen3.8-27b@q4_k_s kinda seems way better. Haiku/Sonnet kinda pairing in workflows?
It's easy to follow, it's a knee jerk tribalism reaction and not a logical or informed one. Go attracts backend or anti-frontend people so anything intermingling the two will draw out the commenters who bemoan anything that realistically builds a modern website. Just ignore it, your choice for dev tooling is practical and makes sense to me.
As someone who has had to clean up systems that only did a _npm install_ to get compromised, despite being good Renovate citizens, it doesn't matter if it is not "on the production path".
I don't care to hear arguments about how npm was _configured wrong_.
A tool with insecure defaults, with a long and storied history of being the vector by which data has been exfiltrated and companies hacked, needs to do an awful lot to justify still being used for any infrastructure our team is responsible for.
The proxy caches and package solution for Go has had its own drama.
My big contention with this project is that if I wanted to use node (or npm; or vite) in any capacity, I would have chosen node for that. I don’t want to use those tools in this context because it serves me worse than a well thought out solution.
The thought out solution is to isolate your frontend into its own directory. You can embed and serve that as part of your Go server. That frontend could have been done with react, vite, jsx, whatever typescript nonsense you want; without the Go parts necessarily needing npm or node in the build process.
reply