I'm very bullish on MCP (or at least MCP "like" implementations), as they solve a lot of problems for non-devs as they're much easier and safer to add into ChatGPT, Claude and other desktop + web apps.

They provide a set of tools and a context when to use those tools (much like a packaged version of a CLI+API and a skill) which makes them more discoverable than other options.

I've got a few folks using my open source data storage MCP - https://github.com/ExpeditedProjects/hutchdb - now and it makes a lot more sense than any other implementation for what they're doing

I've been using OpenRouter since relatively early (I build evvl.ai - an eval platform on top of it) and think that they're selling at a good time.

An underpinning of their model is that API calls / inference are similar across providers allowing for commodity tokenization cost comparisons, but the providers are beginning to shift to non commodity features that don't easily shift.

For instance, calling Gemini with "search grounding" isn't something that OpenRouter can do (they sub their own web search in), last I checked they weren't doing real time voice models, etc.

> For instance, calling Gemini with "search grounding" isn't something that OpenRouter can do (they sub their own web search in)

It can: https://openrouter.ai/docs/guides/features/server-tools/web-...

It's apparently even the default these days (which makes sense, as it's usually better in my experience).

Nice. Happy to be mistaken on that point as that seems very useful.

Your main point is still right, the providers are beginning to shift to non-commodity features.

Another example might be all the managed agent features, where you get a VM + model, right now it's mostly Google and Anthropic that have this offering.

The service tier is another, where Google, Vertex, OAI, Anth offer it but only OAI and Google offer flex service tier and not all models from them get it.

OpenRouter for me is an abstraction over that complexity, they will have their work cut out for them.

Fortunately Stripe is in the business of providing a single abstraction over an enormous pit of complexity, so I think they'll be in good hands!

I can see Stripe adding extra services on top. However, in their main business its "easy" for them because getting into payment processing is hard intially. But whole search, harness, VMs, agent loops are not hard to find somewhere else.

Is this a result of too much activity on GitHub (AI-driven jump in repos), though, rather than a fall?

SO a company Github's size can't handle the slop they invested a lot to enable and this is not a failure? I mean if the market moves and you cannot move with it, what do we call it?

Mostly it seems like greatly increased usage pushing their systems to the limit.

Ahhh okay. They've definitely had some growing pains. By "fall from grace", I assumed you meant some sort of ethical/trust failure.

That too, ever since they decided everything on GitHub was fair game for training AI without respecting its license.

Of course, cursor is going to be even worse than github in that realm.

Github has lost trust. This wasn’t an isolated incident.

The eval world is split into:

1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.

2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")

Since most analytics is done with JS (Google Analytics, etc.) very little of this shows up in site visit stats.

Parent of a middle and a high schooler and we try very hard to not pressure the kids, but a lot of it is coming from their peers and the school.

The 2 vs 3 days makes less of an impact on personal decision making but has massive benefits for decision making at the country wide response level.

To your slow point: I did a quick eval for a data viz task to do a qualitative comparison and Fable was much faster (by 4x), but I struggle with the "token inefficiency" as a sort of whatever metric.

Kimi was half the cost and produced a near identical output.

https://fr4geiw93g.evvl.io/

On a fairly simple coding task Kimi was 2x the cost and 7x as slow (and gpt-5.6-sol was even cheaper).

https://r26pakjmhv.evvl.io/

It's kind of ridiculous how good these are getting. 3.5 Flash lite is pretty comparable to Opus 4.8 (at least for the couple tests I did) while simultaneously being 6x faster and 19x cheaper.

https://fy2zp1ri90.evvl.io/

Not for me personally. While setting up a custom website, 3.5 Flash introduced tons of bugs that Claude had to fix. The website has about 3,000 lines of code spread across multiple files, and Gemini somehow couldn't do frontend changes without breaking things. Sharing my 2c, but I've stayed on GPT 5.5+ and Claude Sonnet/Opus 4.6+. Anything past that from those two have been bug-free, but Google's latest hasn't been.

Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.

So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.

Kimi is cheapest by 5x but also slowest by 2x

https://9gpyw4uxr2.evvl.io/

Personally I'd consider the three middle ones to be failing, in the typical "Gemini/Google" fashion in that the model is doing more than what the prompt asks. The prompt asks for SVG, yet the model is providing more.

Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.

By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.

It's a free site, so I was trying to limit both the privacy and risk exposure.

Making the content auto expire after a short period of time greatly decreases the attractiveness of the site to lots of SEO spammers and other types of abuse, and if someone were to get something malicious or vile posted it will clean up after itself without me having to wade into things.