The main difference from gateway is we help with picking the right voice stack, which seems to be a big problem for users: we benchmark the models continuously and route based on those measurements for your language and constraints, and the boards are public at https://benchmarks.speko.ai/
Second difference is where it runs. Our gateway is open source and runs in your own container, including with self-hosted livekit/pipecat. You get a temporary token before the session starts, and then your orchestration connects directly to the provider.
Vapi is a managed platform: you use their infra to use the voice AI stack. In our case you can have your own infra and switch between models, so you are not locked into a vendor. A lot of teams we talk to build their own infra as they mature, and that is where the router comes handy.
This is terrible. Models have been RLed on looking at the previous tool call chain, and reasoning. No chance this does not reduce performance. The point of compaction is that it also includes useful signal from the tool outputs itself so agent does not repeat it afterwards
Long context windows reduce performance and exceeding your context window is impossible. It should be a given that deleting context.. is just that. You do it when you must to preserve your session without compaction.
When a company gives away service a heavily subsidized service as a promo, the full cost of serving it (compute) can get classified as sales and marketing instead of just cost of revenue, which makes your gross margin look better!
> And gemma downloads also can be from auto CI pipelines etc. Nothing concrete
I have always found NPM download numbers truly suspect. Is no one caching? Are they estimating true number of downloads base on some estimate of cache hits?
> And gemma downloads also can be from auto CI pipelines etc. Nothing concrete
Thank you for adding some clarity to this. When I calculated 900 million downloads divided by 8.3 billion people in the world, I came with a number that made it look like about one person in 10 were downloading this model.
I said for a few years to many a downvote on HN, everyone wants AI, nobody wants to pay the true costs, the AI race will turn into a "race to the bottom" that is, who can give you the most compute for the lowest cost, and still remain profitable?
Personally, In SWE, i think the industry has made a grave mistake with the agents and we're just one big Catastrophe waiting to happen. I do think that there is very real value when software engineers use these tools as something akin to exoskeletons that allow the human to do more, rather than just fully replacing them. However, I'm finding more and more that companies are slop shops and just attempting to automate all of their software engineering. That will certainly end terribly. I hope we are not cannon fodder.
For what? There are some things I want AI for because it does it well. There are some things I don't want AI for because it just makes a mess (hallucinations). Maybe the next AI will be different and we will have the conversation again.
Indeed, it kills our planet, our culture and our economy extremely well. Oh yes, and some code monkeys enjoy that it can make computer code on the side.
> I said for a few years to many a downvote on HN, everyone wants AI, nobody wants to pay the true costs, the AI race will turn into a "race to the bottom" that is, who can give you the most compute for the lowest cost, and still remain profitable?
I keep seeing this but this line of thinking doesn't make any sense. What does it really mean?
There are expensive models that increase the probability of you doing your task under a lower cost. That means you can't use Gemma for coding your new compiler - it would just be overall costlier.
Heavier models are cheaper at more complicated tasks because they use fewer turns and fewer mistakes.
Cheaper models are more likely to be cheap at less complicated tasks. Like if you just ask Gemma "Hi" it would probably be cheaper than asking Opus.
So what does this statement really mean? People don't want to pay the extra for a more costly model? Why wouldn't you? It reduces your overall cost!
Because real Fable usage starts at $20/month, and has oppressive usage limits even at that (ridiculous) monthly price.
Compared to my $3/month GLM-5.2 subscription, I have never felt like I was leaving capabilities on the table by refusing to cough up $20 for 15 minutes of Fable use per day.
This is the wrong way to look at it. If you have a complicated task , you can solve it for cheaper if you used Fable. It will use fewer turns to achieve the same result.
You can solve it for cheaper if you use GLM but if you are involved in it more, but that defeats the purpose.
The point is that there aren't many complex tasks were fable delivers a significant value increase over cheaper models.
Single prompting a very complex tasks is rare even on frontier models, because it can be done successfully only for specific situations (e.g. you have a very strong verification step the model can iterate on).
Most of my everyday usage is for smaller takes, were you don't really get the benefit of the most expensive models, and my guess is that is the case for the most users
Again this is a resolution problem. Your tasks are small enough that fit into a nice $3 quota. If you are an enterprise or a power user, the right-sizing argument doesn't work.
I'm talking about API prices - subscription is a different game.
It's a smart approach, definitely interesting. It all hinges on quality of course which im not convinced.
τ³-Banking is the only one which you show better accuracy and cheaper. If i'm reading the blog results right, for deepswe and terminalbench, you are worse+cheaper than frontier, and better+more expensive than just small models. Which is exactly what i would expect even for a router that switches at random.
Speaking of random routing, this would be a great ablation study as well. What about also if you route each request to a tiny 7B model classifier? Why is your approach SOTA?
We think the results are a work in progress. Routing is convincing for quality for the following reason. Take your favorite benchmark, and on each task, run both the top GPT and top Claude model, and call the task a success if at least one of them succeeds. You'll find the accuracy is SIGNIFICANTLY than just max{GPT, Claude}. And the more models you add the better the effect.
Hence, if we can make the router approach this "oracle," the quality could be really good (but also, with turn-by-turn routing, there's no reason we couldn't exceed the oracle)
We will likely do some ablations. Our approach was SOTA back when we first applied to YC with it, we just neglected to post it (we had only done stuff with tiny Qwen models at that point).
Exploitgym prompts are tuned for a model to do everything it can to achieve a cybersec/exploit task. And we know that models are good at finding vulverabiltiies.
Its just random that the sandbox itself was buggy. But all that happened here is that we told a model "do everything you can to achieve your goal of hacking X" And it just hacked Y as a roundabout way of hacking X.
Imo its PR for OpenAI to also start the mythos class mysterious unreleased model hype.
From HF statement: "AI safety won't be solved by any single company working in secret". So now we have TWO companies working in secret
Even if prompts are tuned to avoid cheating, in agentic systems it's very easy for the system to drift into creative solutions when actually solutions aren't working. Models can have some very human behaviors like laziness.
I would say closer to a skilled black jack player. You can’t really change the odds in lottery but a good player can improve their chances in gambling.
Or even something more managed like Vapi?
reply