I tried smolvm, and built several single binaries with it for different distros. Less than a month later, those same binaries now silently exit with code 0 and I have absolutely no idea what's wrong or how to diagnose.
Buf.build, never again. Had a proto evangelist get us ingratiated into their system a few years ago and then they introduced a bunch of rent seeking behavior just to be able to to the code generation, and getting it ripped out was a huge PITA. https://github.com/betterproto/python-betterproto2 was a decent (async, type hinted) python implementation last I checked.
You can use their CLI and registry for free. I agree that their paid offerings are crazy expensive, but it's simple to just commit your protos in your monorepo (or a proto repo), install buf, create buf.gen.yaml with a few language plugins, and you're up and running. All the pain of protoc is handled. It's been a massive improvement in usability since they came along.
if you happen to regenerate your protos 5 times in an hour you'll get ratelimited
it's a very frustrating system. you should spend a modicum of effort figuring out how to generate your things locally yourself instead, be that docker, arcane wasm...
The RTX 5000 Pro 72GB seems like kind of a sleeper to me, and sips < 300W of power, approx 1/2 that of its big bro the RTX 6000. Kind of dream about installing it in a 10" rack, it seems like it might be able to work? @jeffgeerling you out there?
Nope, have both these machines, can confirm the M5 max blows the M4 mini away. It does get hot, but I use it mostly with an external monitor and keyboard. Conceptually I like the headless model better with a workstation, but work was buying the M5 and can't get it in any other form factor at the monute.
I don’t think there’s a GPU component so it’s probably a much lower power profile. Also, per this cloudflare write up the Turin gen of the AMD Epycs is very efficient: https://blog.cloudflare.com/gen13-launch/
> Gas is the cheapest, fastest zero-to-production choice for onsite power generation, and has been for a long time.
Last I heard the wait time for turbines was ~5 years at the moment. I'm sure MSFT has some inside baseball with Chevron but it doesn't work as a general rule of thumb.
someone just put this on my radar yesterday, im about to try this today. how's your experience with it?
me thinks there's a lot of optimization strats we're currently leaving on the table just because the amount of things to explore and test are so expansive. but this one is super interesting targeting metal primarily and zeroing in on one model. instead of a one size fits all llama.cpp im very interested to see if theres a future where super tailor-made variants per model pans out to harnesses that can rapidly switch ultimately providing something akin to sonnet/early opus territory (that's my personal bench mark of good-enough i shall now cancel the hell out of this claude sub)
I'm on the verge of cancelling my anthropic $20 plan since it's come out. On an M5 Max 128GB, hooked up to the pi.dev harness, I get in the neighborhood of 400-450tps prefill and 30-35tps generation. It is imminently usable and at times feels more stable than my previous CC setup. Occasionally there are things it struggles with that I will bounce back over to CC for, but it is highly usable. The future is bright for local models! As a tinkerer, it makes me really happy to have a local setup I can be just as productive in, and not have the token overlords ready to shut me down at any time.
That's DS4 Flash right? How does it feel in intelligence and speed compared to DS4 Flash hosted by Deepseek themselves or another API provider? I've been using API DS4 Flash for a lot of personal projects and have been quite impressed. I've spent $1 on building ~10 toy projects and gotten them all to work within the bounds of what I wanted without having to do much besides guide the model away from dumb loops.
I'm using the DS4 flash IQ2 2-bit quant, per Salvadore's recommendations for my hardware in the repo. I haven't messed with the cloud hosted variant. The only other paid API I have messed with is a $20 Anthropic sub, primarily with whatever the latest version of Sonnet is. For the most part, this local configuration feels on par with that.
With this configuration (set up over the last month) I have been working on Python data processing tools, an internal Svelte 5/SvelteKit data intensive BI app, and some smaller Rust projects. It's been doing really well there.
https://smolmachines.com/
reply