Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to generate $3-$4 worth of tokens per hour and still maintain a usable per-user throughput.
There’s also input tokens. For many agentic use cases input/output ratio can be 2:1 or even 4:1. Non-quantised DeepSeek V4 Flash costs $0.14 / $0.28 on most inference providers with ZDR.
When self-hosting the model, cache hits are basically free. RAM cache also helps with hit rate (prefixes may be around for hours instead of minutes).
To be clear this project doesn’t aim to achieve the best inference economics per token. MI300X doesn’t have native MXFP4 so it’s not even the right platform for the model. That’s why very few deployment recipes are available.
It’s interesting to me because MI300X is quite accessible to a small team with budget for just 1-2 GPUs. DeepSeek V4 Flash otherwise wouldn’t even fit on 2x H100s.
We can run several coding agents during the day and batch inference jobs overnight and serve the entire team with guaranteed privacy, without compromising precision or speed.
In fact we found that many inference providers are quantising the weights or even KV cache, and due to the low prices they serve at massive batches, resulting in unstable throughput. I ran GSM8K as a quick validation test and this deployment is “better” than the OpenRouter endpoint in a statistically significant way (I wouldn’t name the provider here). I will run some follow up benchmarks and update the repo when I find some time.
830t/s is burst aggregate. ~500 is sustained and it's for 8 concurrent users. Meaning for $1.99/hour if you serve 8 users it's 8*$0.54, not just $0.54.
You shouldn't rent one out if you're just serving it for yourself, but from a financial standpoint if you sell to users you can take a 100% margin.
Bro 500 Aggregate. so that's 500 * 60 * 60 = 1.8M output which is .5$ at best... Not including pre-fill and stuff.
This is not the real margins, even if you are selling to 8 users it's 90 tps per median stream. So assuming that .6-.7$
This is not even remotely worth it.
You need to 3x this tps(~1500 tps) to be worth it, and that's what most providers are doing, at 20-30 users at 50-60 tps with better optimized batch processing and kernels you can make some profit.
This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.
> trying to run it locally or with your own hardware is pointless.
Serving local models has advantages other than price. If you work in restricted industries, or have a strong need to protect your IP, or if you just value privacy more than cost, you now have options.
If you don't do any attention steering, custom decoding or meddle with the weights maybe. Services are worthless unless all you do is write positive prompts.
As others have mentioned, there's the privacy factor as well.
Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher.
Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.0028 = $0.8358 with DeepSeek API pricing.
For comparison, it would take 2M / 8000 + 1M / 800 = 1500 seconds to process this amount of tokens with the linked framework, which is about $0.83 when we assume $2/hr for one MI300X.
However, other inference providers have 10 times higher prices for cached tokens, which results in a comfortable margin.
And we should not discount that DeepSeek also gets paid in data, which is probably more valuable to them.
And I believe that this framework still has some room for optimization for generation with high batch sizes.
Your math is a bit funny if you're assuming the 1/0.5/98.5 ratios: you doubled input and output tokens but not cached. If you double cached tokens to match your original ratio it works out to around $1.11, and if you 10x the cached token cost it's around $6.08.
Based on your $0.83 estimate, the margin isn't great. This is within shooting distance of "at cost" which is probably pretty close to what DeepSeek is operating with, ignoring the value of the data they're collecting of course.
> And I believe that this framework still has some room for optimization for generation with high batch sizes.
If that optimization can bring this scenario closer to $0.50 then it gets pretty compelling, otherwise I'm not confident.
Oh, I messed up. Half-way through, I thought it would be a good idea to double the numbers so I don't have to deal with half millions, but forgot to also double the 98.5. Unfortunately, I can not edit it anymore.
I think the margins of DeepSeek may be a bit better than with this vibe-coded framework here, since they had the liberty of optimizing their models for their own hardware.
At the time, open frameworks were not anywhere close to achieving that number. Not sure whether they caught up. The software wizards at DeepSeek are quite skilled.
> should not discount that DeepSeek also gets paid in data, which is probably more valuable to them
That's agentic feedback loops for training, right? Any more detail on this, such as how they actually tell whether that data is good or not? That seems like a very hard problem, and like the value of that data is low compared to just building their own, controlled RL gyms.
Agents usually start with ingesting the existing code base, and DeepSeek can use those code bases for pretraining. And they will have filters on top of that to throw out garbage.
I am not sure how they are using the data for post-training, but there probably are ways to get signal out of it, e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with the generated code, or started a new session with the same starting point as before, i.e. they git-stashed.
Generally, you can train on data that is quite bad (e.g. the entire internet). It will still work, but take much longer compared to clean data.
> e.g. sentiment analysis when the user begins cursing at the agent, or checking whether the user continued another session with the generated code, or started a new session with the same starting point as before, i.e. they git-stashed.
Thank you for elaborating, that's already useful. Anywhere I can learn more about this? I'm very interested in it!
Definitely not. Inference is not as expensive to operate as many people seem to assume. The frontier labs are probably making a lot of money from selling tokens. It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.
They claimed that OpenAI and Anthropic have positive gross margins. I don't think there are many credible claims saying that's not the case (at least for API usage)?
Use nvidia hardware instead and use a larger cluster serving many more users concurrently. Easily 10x–20x higher token rate per GPU with public solutions like dynamo and sglang.
I think Deepseek is selling roughly at cost (perhaps a slight premium). They don’t guarantee that they don’t train on the submitted prompts, so I suspect they are mining the data. Mining for what? Well, who knows. Best case, mining to make Deepseek better. That said, I use Deepseek all the time. It has done a whole lot of ‘ls’ commands on my system, though.
If you have 2x DGX Spark it will run quite nicely. They cost only $8000 or so and use less power so you may be able to rent them cheaper than the MI300X.
As other calculated, even with a Mi300 you could not saturae it enough with one stream to break even with the DS API, so I think renting sparks would make it even harder cause they are considerably slower.
The MI350p exists and should run a decent quant (say, the ~96GB antirez mix) well, but you can get two rtx pro 6000s for one of these, or 8x (actually more) r9700 + probably the gear to run them, etc.
Otherwise, you can probably buy one of these second hand from somewhere (SXM A100s are available that way) and run it in an adapter board.
You can get one on ebay for like 20k, but it comes without the backplane and i dont think there is a pcie card adaptor from china like the ones for h200.
To be fair the development of GPUs have stalled over the years. If they kept up with the progress instead of focusing on enterprise market, likely 256GB consumer GPU would be a norm today.
Basically right from Lisa Su's speech: "AMD is essentially taking one of its MI350X accelerators and cutting it in half, resulting in a card with half as many compute resources, half as much memory, and perhaps most importantly, a bit over half of the power consumption"
The big question is whether the demand will stay if the subsidized pricing ends. That's what the bubble talk is about. Right now all the players compete for market share and don't care about the losses (hence the debt). But what happens if no one wants to lend them anymore?
There is no evidence they are losing money on inference, though?
Also if they are keeping the price low because they want to gain market share and reduce the competitiveness of Chinese models they won't be able to raise prices without providers serving open models (at cost + low margin) severely undercutting them.
There's no evidence they're making money, and we already know from the projected datacenter capacity in a few years that there will be for more supply than demand, so the major providers will have to repay that debt. Even if they are making money on inference, it's nowhere near enough to cover the bill. It's a losing proposition either way, especially with Chinese models now in play.
There is. Specifically the pricing for open models from third party providers on OpenRouter since inference is a "commodity" at this point. Unless we think that Opus/GPT-5.6 are many times less efficient than GLM 5.2 or Kimi Openai and Anthropic are making money from inference.
> it's nowhere near enough to cover the bill
Obviously it does not cover R&D, marketing and other spending but nobody has ever claimed that here.
I see your point, but Anthropic and OpenAI are not only less efficient, they have much higher operating expenses because of their massive AWS/GCP spending (due to not owning it) and have more capital expenditures than everybody else. It's not comparable to third-party providers making some profit on the margin.
They will be instantly bought out by companies, not individuals. The consumer bubble won’t pop for quite a while yet. Production also won’t ramp up while lack of real competition keeps the demand high.
Thing is, GPUs will always be on demand, look at their history, initially for gaming, then for hash cracking, then 3D rendering, then for crypto mining, and now AI training and fine tuning. When AI bubble bursts, there will be another bubble taking over.
The only solution is more companies making high end units, only competition will make it better for consumers.
Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued?
I can tell you it will pop in 10 years and when it pops, it will still be 20x bigger than in 2026. Does that even make any sense?
People said AI bubble will pop soon in 2024 and that it was overvalued. Turns out, many AI stocks 10x, 20x since 2024. Actual usage has gone exponential as well. Anthropic revenue went from $100m ARR at start of 2024 to $80b ARR today.
> Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued?
Well, given the literal trillions being spent, the only ways this pays off are:
1. AI replaces a non-trivial fraction of human employees.
2. Someone builds a Culture Mind, and humans become (hopefully) pampered pets of AIs we don't understand. Seems unlikely, but it would arguably count as a payoff even if it made money meaningless.
Or maybe the AIs don't want pets, and you get SkyNet. Which definitely doesn't care about paying off anyone's investments.
When you look at various news articles about investors, yeah, there are definitely a lot of rich people who think that they're going to automate all human labor or just bring about the Singularity. Possibly with them in charge of the rest of us. If you don't make these kinds of wild assumptions, then yeah, this is looking like one of the biggest bubbles ever.
Did you read your own article? It's well less than $3 trillion.
Now let's build the model out more. What is the projected revenue, backlog, improvements in existing big tech businesses such as AI helping Meta's ad business?
1. AI replaces a non-trivial fraction of human employees.
I mean, it's pretty clear that's going to happen. How could it not?
The only question is whether resources are optimally allocated at the moment to prepare for this. That seems unlikely at best. So yes, there is probably a bubble, and if so, then yes, it will pop, and then life will go on, with resources better allocated. Just like when the dot-com bubble popped.
> Across all tested generations, divergent paths serialize linearly with the number of paths k, following T(k)≈sk with no super-linear reconvergence penalty. Warp execution efficiency falls as 32/k, the penalty is independent of occupancy, and predication removes the serialization cost
There is more! I started to figure out how ADC/DAC chips actually work. I believe our CM6533 and its competitor the CM108 both use delta-sigma. The output isn't a staircase of those samples but a rapidly switching (MHz rates) 1-bit modulator (think: PCM) output, and a final analog filter smooths it into a continuous waveform.
Many countries now assume you have a phone. For example getting UK visa requires a smartphone. I don't think going without a phone is feasible nowadays.
Another question is if going with a burner phone that has just sim card and bank card, sufficient. But then you need appleid/google account on the device, and this again links back to your phone number, and it's not easy in practice to have proper clean device.
You can simply use an Apple account for your primary device and a Google account for your burner (or vice versa). Or set up a secondary Apple or Google account. Or use a device/OS that doesn’t require one of these accounts!
I hinted there how the NS chain of lookups works from . to your domain. The point is that we wanted to be able to move name servers around the ip addresss, but that wouldn't work for many domains. So - in some contexts moving IP's rapidly is possible, in some it's not. Fun.
(I'm ex CF) This is backwards. Nobody "allowed" anyting. CF serves a customers need. You can argue with the solution but you can't argue with the core problem. It's more healthy to start the conversation of _why_ CF services are valuable.
CF serves something it convinced customers they need.
Static blogs hiding behind bot protection (in some cases blocking legit users from GrapheneOS because it's difficult to fingerprint them) because someone convinced them they'll be DDoSed by bots otherwise is a loss to the Internet.
A lot of self-hosters running CF tunnels because they don't know better also contributes.
> It's more healthy to start the conversation of _why_ CF services are valuable.
Begging the question. It's what TFA is about - telling people they need CF.
> A lot of self-hosters running CF tunnels because they don't know better also contributes.
Are you saying CF documentation is better than Computer Science / Networking education resources? Why don't people know better? I thought the tunnels are mostly used to bypass NAT's.
> Static blogs hiding behind bot protection
I'm not sure what is the proportion of the static vs dynamic sites, but I would argue that for wordpress CF is adding real value.
> I thought the tunnels are mostly used to bypass NAT's.
While not free, you can do with with TCP HAProxy streams on a cheap VPS. A lot of people using them to bypass NAT don't realise that Cloudflare decrypt the traffic on the way - that's what I meant about them not knowing better.
It's not about being incompetent. Quite often in selfhosted subreddits and forums you will see people surprised that Cloudflare can see their traffic in plaintext.
Of course, they probably don't, but the fact that they can and that their policies now influence XX% of internet traffic is bad for the open internet.
I think it counts as "allowed" regardless of utility. CF is so massively over-weight on the internet that it's impossible to trust them with anything because if they can be forced to do something by a hostile government (hint: they can be!) then they can get away with it invisibly and affect billions of people.
That is something that should not be allowed to exist. It's one of the reasons monopolies (or even majority-opolies) are bad. It's a weapon hanging on the wall, waiting to be used.
Then I think the real question is why haven't any serious competitors emerged that can handle the essential services that Cloudflare provides?
Are there network effects like what happens with Microsoft in the business computing space? With Microsoft, I'm also aware of a great amount of anti-competitive behavior, and though I haven't seen that from Cloudflare personally and haven't heard accusations of it, I also haven't paid attention.
When I learned econ 101 in high school there was a concept of a "natural monopoly" like an electricity utility, a concept that was probably mostly post-hoc rationalization of the regulatory structures that were chosen a century ago, but it at least was a coherent narrative. I can't see any coherent narrative about Cloudflare's services being a natural monopoly. So I'm left wondering if they are just way better at what they do than anybody else, and perhaps the space isn't big enough to drive a competitor to enter it?
I hope somebody on HN has a much better explanation of this than I do.
I suspect a big part of it is that CF is running other businesses on the side, and offering basic features at a loss - they've artificially depressed the price of the service so it's hard to compete with them on only that service.
Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do. It's heavily used as a way to steal small markets simply by being successful in a different large one.
> Everyone using the free service likes that, of course, but honestly I wish we'd make it illegal to do.
I can see the benefit, but if you made free services illegal companies like google would just offer gmail for a penny.
You'd could go farther and say that no one should be allowed to provide a good or service for less than it costs them to provide it. That still has problems though. For example, Netflix sends servers to ISPs to set up on their network so that segments of that ISP's customers are pulling data from one of those boxes distributing the load. ISPs are happy to do it because it means fewer customers call them to complain about slow speeds/buffering issues with netflix. If netflix wanted to start selling CDN services to others, having all those boxes at every major ISP in the nation means the cost to deliver that service will be much lower for netflix than it would for some new start up. You can't really make it illegal to offer a service at a price lower than it would cost the poorest and least efficient/capable would-be competitor though.
We'd also risk losing a lot. Everyone would be forced to pay youtube some amount proportionate to what it costs for youtube to exist. There are also extremely expensive to run services like the internet archive which would be bad to price people out of
There are plenty of ways to mitigate that "losing a lot" (like government subsidies: run X, get paid $Y per Z -> easier access for both users and providers. we already do this for a lot of research and tons of industry, it's not some utopian dream), and since a lot of the free internet stuff we have now is ad-supported and addiction-oriented... I'm not convinced that'd be bad to lose. The adtech business causes horrific damage, and is consistently used as a weapon against people (ICE buying info commercially to get around laws that would normally block their access, for example). Not everything has to be allowed to exist.
For Internet Archive: IA is a non-profit that currently runs fine on donations, not ads, and in a healthier environment they'd get subsidies because they serve a clear public benefit, one which the government also relies on fairly frequently. They're a wildly different category of business than Cloudflare, both obviously and legally, and I think they'd be fine. And we'd be significantly better off if they had competition, because that'd serve as a distributed, duplicated backup.
I'm not claiming it'd be trivial, or perfect. I'm claiming it'd be worth the effort.
Maybe you could avoid the thornier issues entirely by outlawing the adtech industry instead of free services. With the backbone of surveillance capitalism out of the picture most of the free services would stop existing or move to pay models anyway. Worthwhile free services like the internet archive and wikipedia that aren't spying on users could continue to exist without worry. We'd still lose youtube I think, but maybe video hosting isn't best centralized either.
yes, I think that's a good idea too. and possibly easier to achieve.
I don't think advertising is inherently bad, and at some point you get ambiguous about "is this advertising or is it just describing something so I can find it in a search engine" and it's literally impossible to eliminate it all. and being able to tell people with problem X that solution Y exists can be a good thing, in moderation. it's more that the current maximally-invasive insanity and e.g. tons of highly distracting billboards are probably a net loss for humanity. São Paulo shows that it's quite nice without it, for example.
Seems more of a moral hazard of government intervention than of ventures whose economies of scale demand large market cap to most economically serve customer needs.
"Most economically" can be bent to nearly any purpose, there's almost always a cost-benefit to owning more of what you do, or externalizing costs onto others. It's not enough of a reason, and it's why laws restricting private behavior to serve the greater public exist.
Which is very nearly the only reason to have governments: forcing restrictions that lead to better results in aggregate.
(malicious control is another reason, but you kinda can't argue public benefit there)
"Most economically" means to satisfy the desires of the buyers without impairing the satisfaction considered more important. It has public benefit built in.
Regulation could conceivably disallow a single company to control such a significant portion of internet traffic. The parent can be interpreted as lamenting the absence of such regulation.
It's certainly a huge risk when one company's outage can take down so much of internet. Decentralization and the ability to route around damage was a core feature of the internet. The more we depend on the internet, the less acceptable that loss becomes.
cloudflare (and CDNs more generally) certainly started out as an own goal. Warnings were ignored and most people did the easy thing instead of the right thing.
Lysk is a European defence startup, based between Paris and Warsaw. Our mission is to create modern software tailored for hostile environments. Our customers include European military and police units. We are in stealth but have already been recognised as one of France’s top 20 seed companies. Backed by Project A, Europe’s leading defence VC, and angels including the founders of Cloudflare, Quantum Systems, and ElevenLabs. We’re assembling a world-class team at the intersection of modern software and defence. We do not work on weapons systems or hardware products.
reply