Let’s not forget to count all the people who use a MS office product at work, and are therefore opted into counting for copilot numbers, which almost _certainly_ counts towards ChatGPT numbers by default!
Talaas have done this with a llama 3 model.
Runs at like, 16k/tokens a second oror something obscene. Very little power draw too.
Doesn’t need hbm or lots of memory, because the hardware can just forward the data straight to the next layer and you don’t need to round trip through memory.
They claim to be working on an approach to make the underlying hardware a bit more reusable between models.
Yeah, if you have a fixed llm topology, you can just effectively burns 2 top layers of the chip as Rom (model weights) - which has a per area density even better than dram - so it’s just attention and kv streaming that is hbm to sram transfer.
Most big model weights will not fit a single reticle sized chip - so you’d have prob 30 different chips to split the model .
And you’d need super fast chip to chip comms for the all-reduce and similar.
So scaling to 1T models is hard - and a long lead time - but can be very power efficient.
This reads an awful lot like post-how justification of poor business practices.
It’s got a bit of”uhhmm actually, poor management and high turnover is good actually” vibes, which is then (over)extended to a kind of carte-Blanche justification of “why using kms and having no idea of what’s going on” is good-and-desireable.
Which is like, certainly a take, and I can think of at least one “technical skills hating” exec from a past life who’d read this and foam at the mouth to feel justified in their decisions to try and throw all engineering practices out with the proverbial bath water.
I read it more as pointing out the inevitable and accepting the implications. You can not call it the result of poor practice when every business beyond a certain size has this problem.
Perhaps the problem could be mitigated by keeping the company itself small, but that has nothing to do with programming anymore.
> Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory
Careful, if you say that too loudly, the "get rid of async just spawn more threads!!!!!" people will come out of the woodwork to yell at you about how _all_ async is a lie and we should instead pretend none of it exists and just spawn more threads.
I am now trembling at the thought of warriors who will skin me alive :).
Jokes aside, there are use cases for rayon, use cases for tokio async, use cases for "may" coroutines, use cases for a custom scheduling policy, or use cases for a combination of these.
We went with "may" coroutines (with its thread pinning) + custom numa work pinning (to a may thread) due to "may's" lightweight nature and not having to have our functions colorized. We use rayon where cache locality penalty is minor compared to the millisecond latency gains due to work stealing - but this is only for an edge case.
Our log broker Monolog (akin to kafka) which sits atop Dip uses tokio async because it plays nicely with zmq while "may" doesn't.
So I am a big fan of using the right tools for the job. Each technique has its own pros and cons. Informed decisions based on use cases matter above any dogma.
Our stack consists of two consumer facing mobile apps, Slyp and SlypBusiness, which necessitated developing the underlying infrastructure over time for scaling and efficiency.
That stack includes Dip (our database substrate), Monolog (our log broker), Singularity (our KV engine), Craft (a mobile app for app development using flow charts, snippets, and a WYSIWYG designer), and Portal (a mobile app for remote container management and a HITL AI workflow).
Craft and Portal are MVP-style functional, but aren't in production yet. Everything else is being rolled out gradually.
You're more than welcome. Happy to discuss any of it in more detail. Rather than digress further in this thread, feel free to reach out privately if you'd like to continue the conversation.
> but if it falls prey to these pitfalls, it would make me pause before adopting it.
This issue isn't really a Tokio concern, it's pretty straightforward to write Rust code that has back pressure mechanisms. The "it's not reasonable" in my mind implies that if someone goes and _does that_ then there's not much a library can do to restrain the developer.
This reads an awful lot like the prompter had a semi-confusing time with some async, and had their favourite model write an upset blog post about it.
I don't think these systems are perfect, nor are they fit for every use-case, but some of the complaints ring a bit hollow.
> fetching a database record over the network, then immediately crunching the data. But what happens when that data crunching involves parsing a 10MB JSON payload
Mixing IO-sensitive code and blocking code causes issues, who knew? I'm not quite sure how these libraries are supposed to magically _save_ you from this?
> When these latency spikes occur, the answer is always the same: separate your runtimes.
Well yeah. "Why doesn't my daily-driver cut sick lap times around the Nurburgring?" If you want to run blocking, non-interleaved code, don't do it in an executor expecting small, interleaved, non-blocking tasks. The docs for Tokio even mention this, and provide a number of worked examples of integrating/bridging sync and async code.
> If a developer must manually partition I/O and compute, strictly police the boundaries to prevent deadlocks, and ferry data between two different runtimes with two different mental models, the async abstraction has failed.
Not necessarily. If I'm chasing a performance target, and the tool gets me 80-90% of the way there, to the point where my next task is optimising layout and caching, I call that a win. That's performance ground we'd have to address at some point if we want to go faster, so getting there easily means:
- those people who don't need to go faster because their perf requirements are met are happy
- those people who need to go further get to skip the intervening work, and go straight to these optimisations.
Edit: also wanted to make a point about the "memory blowup" the author mentions. It's entirely possible to do the _same_ load-shedding the author wants in Rust/Tokio/Monoio/etc, this is again, just a matter of building your application in a way that uses, and communicates back-pressure. I gather that they appear to want something which is a bit more opinionated and "batteries included" as far as functionality goes, which is totally fine-and-cool, but calling Tokio + Rayon _bad_ because they explicitly don't do this is a bit of a miss.
This entire post was written/prompted as a pitch for the author's own concurrency framework. These types of "current state of the world - bad, my solution - good" posts have been annoying for far longer than LLMs have been around, but the LLMified over-embellishment coupled with the vagueness makes it so much worse.
> don’t understand why anyone thinks management of any kind was involved in this at all
Because it’s marketing gold? Any marketer worth their salt would be super keen to get their claws into a project like this.
Anthropic and friends are in then business of selling LLM’s. This is a demonstration of LLM capability, they’d be foolish to not even try to cash in on it.
> I just don't care anymore because it's not my responsibility or my code at all at that point
Yep, 100%.
Business has made it clear they don’t care, so there’s no point in burning one’s energy. Throw the whole thing on auto, check out, and go do something else during the day.
I agree, that work in these places is likely short lived, if for no other reason than working in them is awful and demoralising. My point is, that these places exist, and have such a hard-on for “oh my god, AI!!!1!1!” that putting in extra effort there, is a waste of your own energy.
I'm just thinking: The result is net-net the same. Either they terminate your contract as a salaried employee or they do it as a consultant. The fee is higher as a consultant and you never have to say you were fired. You're making a really good argument for getting into consulting.
You can just answer the same rote questions to different companies for max salary, then when the going gets tough you can essentially 'fire' the company and say they've moved past your core consulting paradigm. Maybe suggest a new consultant and move one. Rinse; repeat.
Parent comment said they end up "...pass[ing] it off to QA."
Edit: suppose that very well could have been tongue-in-cheek.
Side note: as someone who has been interested in programming for a while, but didn't end up in a software dev track in life, it's been pretty wild watching the ride you all have been going on lately. I used to be pretty bummed I didn't get to do that kind of work for a living, but lately I've been feeling more and more like I dodged a bullet.
Not that I don't have my own AI-related junk I have to deal with where I did end up, of course. I think most have.
reply