i also used nixos as the base, but i went a little overboard with it, this is 7.1.2 supporting all devices and a custom driver that fixes the UMA page leak issues, and the wallpapers are all kernels at or above parity with what you can get out of the box.
i'm not quite ready to OSS the whole thing, it's got a few rough edges, but if anyone wants to alpha test, caveat emptor and it's yours.
There are a ton of DeepGEMM kernels in all of the flashinfer adjacent stacks including trtllm. Bunch of the headers still have the CLion pragmas and shit.
No one serious about the idea can afford either vanity metrics nor ignorance of the code. The bar is higher, not lower, to operate with this much automation in the water supply. It's mostly a lot more math and a lot more work.
It's an extreme form of any startup: you trade off capital for years of your life.
I have to do like, low paying web dev to fund it, so I can't promise timelines, but it's coming and it will be free to anyone and fast as fuck.
I estimate about 20 bucks an hour at current spot rates in the hundreds if not thousands of tokens per second.