If your maximum addressable market is “the whole economy,” as seen in SpaceX filings, then a city-sized call centre (distributed, of course) really is ‘t that much of an ask.
> But it’s just as likely to make an output better.
No, for any particular output token the model's true logits are definitionally the 'best' that the model can achieve.
This is inherently probabilistic. The model's top-1 guess is not guaranteed to be optimal, but it should be so a proportionate fraction of the time. Same with the top-2, top-3, etc.
Watermarking necessarily alters the output distribution away from the model-set distribution, and that alteration is inherently 'worse' in expectation.
You can liken this to a weather forecast. If there's a 25% chance of rain, the forecast should say so (or a 'sampled' deterministic forecast should predict rain 25% of the time). If the forecast is 'watermarked' and predicts rain 27% of the time under identical circumstances, it's a worse forecast.
That being said, this is a case of hiding a message in a noisy channel. Watermarking only needs to communicate one bit ('yes watermark'), so the effects can be arbitrarily small provided one is willing to tolerate an increase to the text size needed for reliable detection.
I think you can also just use the random number generator (seeded with a secret key) as the watermark. Then the probability distribution is exactly the same.
> Hyperscalers don't run computing at some multiple more efficient than on prem.
I'd disagree here. I see two avenues for an efficiency multiple, albeit a single-digit multiple:
* Client aggregation allows a hyperscaler to average out demand spikes from uncorrelated clients, reducing the peak:average demand ratio and allowing better budgeting of compute.
* Dynamic batching allows typical requests to run in batches of more-than-1 and/or overlap, offering better internal compute utilization ratios (e.g. interleaving output and input streams). The small limit of on-device LLMs will run with batch sizes of one with strong memory bandwidth bottlenecks.
For an example of these factors in action, see the API cost differential between batch, standard, and 'fast' processing. OpenAI prices these tiers at a 1:2:4 ratio.
A hyperscaler can pack compute and that is it. It doesn't get you some multiple higher in efficiency. And it is a net negative when oversubscribed, which due to MBAs ruining all the clouds is how they operate. Customers pay more for less when using the cloud.
Go get a job a hyperscaler, they want to smoke what you are smoking.
I am not talking about diurnal cloud workloads, I am talking about the native efficiencies of hyperscalers vs on-prem. They have no magic and they all think they are going to make up their business overheads in exorbitant saas pricing.
Yes, they can do all of these things. But then the margins are eaten up in the process, hyperscalers could be half the price of on-prem, but they are more. They have no magic, or the magic they do have they are wasting (which is good for the people). You can beat a hyperscaler on price at a single rack.
> As things settle down and commoditize, the value of switching on a dime diminishes as people lock into their favorite models
I can imagine just the opposite outcome from the same scenario: as people settle into their favorite but commoditized models, competition for marginal inference cost will take over. A company like OpenRouter that promises the cheapest tokens by the minute becomes essential on the low-cost margin.
I think that OpenRouter and equivalents get pushed out of the market only if the froth calms down (as you posit) and winning models stay proprietary, perhaps with their own unique API surfaces.
> A company like OpenRouter that promises the cheapest tokens by the minute becomes essential on the low-cost margin.
Okay, I can see that, but if their value is just cost optimization, their ability to mark up the tokens becomes increasingly difficult as well. Or, people will build a router themselves to avoid paying the markup, possibly with reduced features, but someone will open source it. Heck, Claude or GPT can probably one-shot it these days. Either way, I think the whole OpenRouter model is going to struggle unless the market stays frothy.
First, cost optimization directly contributes to how much markup they can charge. Say on average they deliver savings of 20%, that is excess value OR/Stripe can mostly capture.
Also, you don't need to mark up tokens much if you're a commodity volume business. Think of Costco and their margins & membership fees. Not everything has to be high margin, not everything has to be a SaaS subscription.
Well, it depends on how they deliver that savings. If it’s through something unique and defensible, something that only they (with some moat) can do, then yes, they can argue for a fraction of the savings. But if prices just go lower because things are commodities, then they are going to be viewed as middleman markup on top of already low prices. In that case, their value add is considerably lower.
You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs.
All other things being equal, increasing the speed of some part of the development process will increase the overall pace of development. However, By Amdahl's law that increase will be sublinear, and that is why we should take "pull requests" as an imperfect metric.
We also don't get to pick the form that 'better technology products' take. While we'd probably like to keep cost(/effort) and complexity constant and increase robustness and performance, the market equilibrium might be 'worse is better' and reward whiz-bang features and lower effort.
Also, large companies are still experimenting on how to integrate LLMs into their workflows. Due to the fast cadence of releases, people forget that LLMs became robust (regardless of the capability level/parameter count/data size) enough to use semi-reliably in company-specific ways only 1 year ago. And bigger the org, the slower the process. I don't expect it to settle and get productive used across the majority of very large companies for another year atleast.
High prices are mostly a result of DC capacity. As more and more DCs get built out, prices will drop. At a unit level the inference business is extremely sound regardless.
> You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs.
Right, but we didn't invest a trillion dollars of capital into any of those things in the span of a couple years, thus forcing them to capture value and show a return on such a massive investment.
> Is the ROI there to pay for the trillions in commitments that have been bet on that ROI? That looks like a clear no at this point.
That's only a potential crisis for those who have made concrete investments.
On the use side, the 'cost' of AI spans more than two orders of magnitude. Looking at recent models (<6mo) with reasonable performance (intelligence index >= 45) on OpenRouter, the output cost ranges from $50/MTok (Fable) to $0.153/MTok (DeepSeek Flash 0731).
From the perspective of a user of LLM/agent assistance, there's very likely a range where the benefits outweigh the costs.
If the ROI for the model developers isn't there, then that just impairs the future trajectory of the field. Current models are just bits that aren't going anywhere, and as long as they can be served (in inference) above their marginal cost they will continue to be so-delivered.
A reasonable guess about the algorithm is 'A Watermark for Large Language Models' (https://arxiv.org/abs/2301.10226). The idea is that each generated token (or bigram) seeds a strong PRNG that splits the vocabulary into a 'green' and 'red' set. The sampler then tries to select a 'green' next-token for generation.
After-the-fact checking only needs the vocabulary splitter, which is independent of the LLM. Over a sufficiently large text non-watermarked text would expect to use green and red tokens with the baseline probability, and that difference can easily become statistically significant over sufficiently long texts.
The basic algorithm has obvious knobs to tune, among them the initial ratio of red to green tokens and how hard the sampler tries to pick a green token. These would balance fidelity to the original distribution against watermark detectability (minimum required content length for statistical power).
Anthropic actually tells you the approach they use, and it's not that. From their Claude Text Watermark page[0]: "Claude’s text watermark is a version of the SynthID-Text approach published by Google DeepMind in a Nature paper in 2024."
The Nature paper is "Scalable watermarking for identifying large language model outputs"[1]. This method does not separate out tokens into separate classes, but merely uses a seed for the PRNG that selects which among the most likely tokens generated by the LLM will actually be output. This has the advantage that there's no green and red token sets, so no token is systematically favored or disfavored. If a particular token is overwhelmingly predicted to be the most likely candidate, it will almost certainly be selected, so the watermark doesn't affect that. Even if there are several choices of output token at a point that have similar probability of selection, the watermark doesn't systematically bias in favor of one token or the other.
This is actually a quite elegant method of watermarking that, contrary to people's fears, won't adversely affect the model output. The main concern I have with it is that it appears that you can't actually test the watermark locally, without uploading it to Anthropic. I'm not sure why that's the case, since there's no particular reason the watermarking key has to be private, except if you want to prevent others from generating text with their own LLMs that is watermarked to look like it's generated by Anthropic - but everybody wants their text to not have the watermark.
> But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty.
When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked' even if I only succeed in this dictum (e.g.) 10% more often than the baseline.
most people have at least two, but generally around four to six (or more, thanks internet) interactional styles (not "selves", just things which vary depending on context, subject, and people they are interacting with). probably more. some of it is due to simple physical comfort levels (right now I am in a physical position where capitalizing is more difficult); some might be due to talking to a peer group instead of a group of kids or a priest or boss at a job, etc. unsure how that will shake out with AI but it unnerves me.
There's more to it than encoding one bit. You also want to avoid false positives. You can encode a single bit by XORing all the bits in the UTF-8 encoding. But then you get a lot of incorrect hits. The lower your tolerance for false positives, the more it acts like you're actually requiring more bits in your payload.
> Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?
That's precisely the author's point: deliberate backdoors will be more adversary-exploitable than ever before, but the demand for such from law enforcement agencies is likely to ratchet upwards.
But that's one step after, I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases? Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it? How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
I'm not sure you understand what is being described here?
If you're a US company building an app/device/etc. such that an intelligence agency like the CIA or FBI would want access to the data in that product which is normally secured, then they're not going to try to sneak it in there without your development team knowing. They're going to have a meeting with the owners of the company and say, "hey, we'd really like you to implement this backdoor for us, and in return we won't cause you in problems."
Note that this does certainly already happen a lot, but it's also not something that can happen across the board (like the author points out). Apple, being one of the largest companies in the world and who has one of their biggest selling points being their security, has explicitly refused to do this to the point that the intelligence agencies couldn't break into an iPhone until another company found a way to do so.
>I'm trying to understand how those backdoors even end up in software if everything gets automatically reviewed by the people working on these codebases?
The people working on these codebases are "in on it." Of course, we're talking backdoors which are very subtle, target very minimal infrastructure, and are known about by very few people. But, like I said, companies are currently shipping products with backdoors in them knowing they exist already. AI doesn't change that dynamic.
>Wouldn't things like these be flagged by systems other than the developers tasked by the agency to implement it?
The only people who would have access to the systems that can even be flagged by this stuff would be people who would know about it. Keep in mind that the scale we're talking here is massive. Think about how software development works at companies like Apple, Microsoft, Google, etc. There are devs working in offices all around the world where they only ever have access to a fraction of the code that company owns. These companies are very capable of keeping their stuff locked down. It's a necessary component of their work.
>How can law enforcement actually get these things implemented without big parts of the engineering team not seeing warnings about it happening?
Hopefully my explanation at this point is clear, but just to be concrete: backdoors are, by design, very hard to detect. That doesn't mean they're just sneakily written code that humans don't notice as they read over it, but, instead, they're very subtle implementations in very specific parts of huge systems that are already locked down to the point that the number of people who even have access to those portions of the systems are very limited. These agencies don't slip in backdoors without anybody noticing; they convince the minimal number of people needed to know about it to implement it. Again, we're talking about a meeting between the directory of a three letter agency and a CEO, where the CEO then directs the CTO to implement the backdoor who then instructs the handful of very high-ranking engineers to do so.
AI systems in these companies may very well flag these backdoors to the people who already know they exist, then these people can tell the AI "hey, those are their on purpose, so just move on," and the other 99.99% of the company will never know they exist.
I suppose it's important to emphasize, again, that these systems are incredibly massive and complex and most people at these orgs don't have any access to most of these systems, so it's not like you can expect an intern running BugBot across a repo and expecting it to find a backdoor.
I think people take the FBI or CIA too literally. I imagine they don't need to talk to owners and it's probably not even ideal. It might just be easier to get plants in the organization.
I would imagine most big companies, like Microsoft, have dozens of CIA and FBI plants in their organizations. Agents who are legitimate software engineers, tasked with acquiring intelligence and undermining security.
> If you're a US company building an app/device/etc. such that an intelligence agency like the CIA or FBI would want access to the data in that product which is normally secured, then they're not going to try to sneak it in there without your development team knowing. They're going to have a meeting with the owners of the company and say, "hey, we'd really like you to implement this backdoor for us, and in return we won't cause you in problems."
And then you say, loudly and publicly, "all the source code of our software is public, and our binaries use binary transparency so it's not possible for us to build a binary that doesn't match the source, and people will rapidly find this in our source code at which point we go out of business and you stop having a product to backdoor in the first place".
(And you move out of the US.)
And since this is a foreseeable future, you should start acting now to prepare for that future.
With laws like Chatcontrol and all, other jurisdictions are not necessarily any better.
In fact, we're increasingly seeing a desire to build the "backdoor" directly into the software, e.g. mandatory age verification, client-side scanning, etc.
> And then you say, loudly and publicly, "all the source code of our software is public, and our binaries use binary transparency so it's not possible for us to build a binary that doesn't match the source, and people will rapidly find this in our source code at which point we go out of business and you stop having a product to backdoor in the first place".
> (And you move out of the US.)
And then everybody claps.
Name me a software or hardware company—one big enough that the US government would actually care to force them to add a backdoor—that would be willing to give up the US market? The only one that's shown the least bit of backbone is Apple, and while I like them and appreciate what they've done in that vein so far, they're never going to move to open source software running their stuff, and they're pretty well embedded in the US, and very, very unlikely to try to move regardless of the headwinds there.
I'm fully with you that this would be a wise and moral thing to do, but frankly, our tech companies are neither wise nor moral. They are self-serving, greedy, and many of them have wanted to become the neofeudal overlords of a new order since before Trump started smashing the old one.
I'm not suggesting giving up the US market. I'm suggesting moving out of the US and continuing to serve the US market from elsewhere, because the US does not have a nation-wide firewall. And working with organizations mounting legal challenges to "please destroy your company in order to put in a backdoor for us".
Certificate Transparency has essentially eliminated the problem of backdoored CAs, because attempting to do so would destroy an entire CA. Binary Transparency can do the same for software.
I'm not sure exactly what you think that will accomplish...?
Companies have to follow the laws of the countries they operate in, not just the countries their physical headquarters are in.
That's why, for instance, Apple has to follow the DMA in Europe.
Furthermore, especially for many of the tech companies, where they are located is an integral part of their culture. They are Silicon Valley. You're going to have a very, very hard time convincing any of them to up stakes and move.
And further-furthermore, move where? Europe has, unfortunately, made similar authoritarian noises (eg, Chat Control). China is already more of an authoritarian state than even Trump's USA. Ditto for Russia, and, AIUI, India, though both in somewhat different ways.
The degree to which you have to follow the laws of a place you have no legal nexus in (e.g. no employees) is limited. Non-US companies can and do refuse overreaching requests from the US sometimes.
Apple is subject to the DMA in europe because 1) they have employees in Europe and 2) they want to import and sell physical phones into Europe. It's easy to block imports or fine employees. It is not especially easy to prevent people from spending money on a service provided entirely via the Internet from another country.
It is not at all impossible to impose sanctions on an entire company. But if the US tried to do so over a refusal to put in a backdoor, and the company very loudly made that clear, that would to some extent be exceptional free marketing.
Why can't the backdoors themselves be more sophisticated? I understand that expecting such sophistication from legacy companies is a joke at this point.
> In the notebook, I compute a MiniLM embedding of every real Wayfair classification. I compute the embedding of the fake, hypothetical embedding from the LLM. I then dot product the fake embedding into the real ones to find the most similar. Producing: [the right answer]
Isn't this begging the question that the hallucinated classification will be more selective with respect to the real schema than the query itself? What would the dot product of <E(search query), E(schema)> have given?
Even if that is too vague, smaller LLMs are capable rerankers; return the top N matching true categories and ask for a contextual ordering.
Yes what you're describing is a classic way of doing query understanding.
I've found, though, getting it in the language of the vocabulary has generally improved performance.
Further, when searching for "blue shoes" you want to separate the color from the item type. So its useful to have a dumb LLM do this for you. And with the LLM in the loop, its further useful to get it into the language of the taxonomy to improve embedding retrieval accuracy.
There are of course many ways to skin the cat here :)
reply