Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
Its a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part of system design interviews.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
“changing data once its running in production is hard, changing the data structure is even harder”
100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
The fact that it took them 4 months to roll out does not mean this is the hard part: it's simply a coordinated rollout with incremental, staggered deployments and rate-limited migrations.
Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.
However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.
So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.
I do not think Cloudflare was a less-than-peers optimized product when they launched. This is one of their blog posts which describes taking one aspect even further.
I think Cloudflare became big only because they were so much more optimized than others that they offered some services for free that others were not offering. If running costs are high, you only burn (VC) cash and then you exit.
There is the entirely plausible option of the NSA indirectly bankrolling them to counteract the growing number of TLS connections. We terminate your TLS for free in our server and you don't have to change anything was a sweet deal for website operators _and_ those that want to look at unencrypted traffic. Given enough sustained funding they could undercut competitors and grow to what they are today.
Quite. I was a VMware fanboi (25+ years, man and boy)
I still look after a few VMware estates and a lot of Proxmox ones (that used to run VMware).
Hilariously, VMware is described as "enterprise class", which I can only conclude means MVP and a bit wanky.
Today I repaired a Proxmox HA + Ceph node using boring old normal Linux skills and as it turns out I have 30 years of those. Part way through a remote v8 to 9 upgrade I think I lost comms due to using OpenvSwitch for networking and despite using tmux for the upgrade session. Anyway, the Proxmox ISO was useless for rescue but the classic systemrescuecd worked nicely and I could run dpkg in a chroot.
VMware "used" Linux and never really gave back. I don't miss fixing vCentres and all the other nonsense that "Enterprise" wankery has foisted on me over the years.
Sorry, the very simple thing you’re trying to do is too complex and non-standard for our support team to handle. We’ll gladly sell you a consultant for $400/hr. He’ll work on modifying our system, and then we’ll sell those modifications to everyone else.
To me it always meant needlessly complex and overspecced for what's needed. I think probably due to Java's enterprise years.
Why solve the problem directly when you can abstract everything away into FactoryFactoryImplementationInterfaceFactorys, and have something that is both a memory-hog and completely unassailable to any normal programmer seeking to understand it or make changes?
Those old school systems are often much more stable than any newer systems. Autozone looks to use something like that and I’ve never seen them have issues as a customer.
In recent history I've done some work on Pick/Multivalue systems first deployed in the 1980s that have been rolled forward today and I love every minute of it. Dick Pick had some opinions and most of them were pretty good.
The art is in knowing how to write software that doesn't perform like shit without doing all the work of measuring and refining. If you can save $100k in hardware costs in a couple days by just knowing what you're doing, that optimization is not premature.
And yet when Prof. Donald Knuth wrote that in 1974 paper[1] it was in this context:
> "The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal"
also:
> "In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by
virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable"
In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals.
In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".
> And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".
Oof that hit hard. The last project I worked on suffered from a very similar disease, and it has really taken a toll on me psychologically. To work day in and day out on something that you can prove cannot work is unbelievably demoralizing. From an organizational standpoint, it makes a lot of sense to have an "internal consultant" who can deliver bad news like this. I tried to do it from the "inside" which was a huge mistake--got a negative performance review saying I had a "communication problem" because nobody wants to hear "negativity". I can come off online as kind of an asshole, so this may not seem credible, but I did actually deliver this news in a professional, measured manner. It's just that organizations are allergic to it, and their antibody response kicks in. You need someone who is not affected by the organizational hierarchy (or at least not that branch of the tree) to step in and deliver the bad news without fear of retaliation.
You're never going to get promoted with that attitude!
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
Remember that everything has an opportunity cost. Running a lot of servers might cost $10 million annually, but if the product team had to choose between a project that would recoup $5 million of that vs. an opportunity to earn $50 million ARR for the same amount of work, the logical answer would be obvious.
Depends on how you measure your ROI and how it could be different from how your company measure their ROI.
The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.
In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
You’re assuming performance has zero impact on customer retention, spending, etc which is demonstrably false. Further future costs aren’t bound by the current customer base or fiscal quarter.
Insufficiently optimized code kills companies in highly competitive markets.
“Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.”
> The problem with optimizations is that you are competing in prioritization with other features.
I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users.
I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost.
> In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
> People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration.
I can easily picture both tech and non-tech individuals acting this way; in the end egocentric people and employees with a herd mentality exist in both groups. I'm old enough to have participated in several major version rewrites aimed at resolving performance and technical problems. More often than I care to admit, these projects failed, often after several delays, and didn't show any real improvement over the existing system, apart from hopeful conversations about the future and tech presentations at local conferences.
I agree rewrites are generally not productive unless the original system is so far gone that it makes sense to start over.. but that's actually very rare.
The thing I think is absolutely ridiculous is when organizations can't figure out how to get work done which improves the performance (cost, efficiency, speed, etc) of their existing features. "Prove to me it has {financial, customer, etc} impact" is just how managers who care about nothing other than rolling out new features prevent the work from being done. It makes sense, they get promoted when their underlings deliver some splashy new thing. But it's an incredibly stupid way to build software. If the thing was worth building in the first place, surely it's worth making it good, right?
We give perverse incentives to the marketing department to bring us the most customers they can, instead of the most appropriate customers.
Someone told me at my second lead position that they had landed a top-tier customer for our demo-ware and the first words out of my mouth were, "FUCK ME". Not the response they were expecting, but then that guy never did end up understanding me the entire time we worked together. My bosses did though.
When your system is new you're a loss leader for all of your customers. Every dollar they bring in costs you two, possibly more. You have to get to positive MRR before the venture capital runs out, and over a long enough time horizon, you will go for a new round and find out that a recession is about to start and the VC guys getting cagey is the first clue it's coming.
Meanwhile if you spend all of your time and energy on reducing costs instead of increasing revenue, then your competitors catch up with you. Particularly if their funding rounds are half a cycle off from yours - one of you will be the last one to have gotten a cash infusion before the money dried up.
Most recently I worked at a place that didn't understand my calls for sobriety until it was too late. So I got put in charge of a rear-guard action that was too little too late, and our customers all fled to much cheaper competitors who could do 80% of what we could for half the price. I learned some good stuff, but the company got bought by a competitor who scrapped those systems.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
<strange high pitched equivocal sounds>
I do a lot of tuning work that others either can't be arsed to do, or more often are scared shitless of touching (learned helplessness). I tend to work on things that have to go right the first time in general, but I firmly believe that I learned that skill doing optimization work early in my career, not the other way around. Through a series of unfortunate events, my first specialization was performance, before I got good enough at anything else for people to listen to me.
Ultimately, the single thing that fucks up performance work, far and above any other concerns or sources, is that the vast majority of developers want to skip from Make it Work to Make it Fast without going through Make it Right. Then they have to use dodgy 'optimization' tricks that embrittle the entire thing, destabilize production and make every new feature feel like walking a minefield.
So what happens is that when someone who actually knows what the fuck they are doing comes along and wants to build a superhighway, they find the route littered with abandoned 'vehicles' that have sunk into the mud up to their door handles and left to cement in place. Clearing the proverbial route becomes a bigger technical challenge than building the road. And so you'll find that your peers have a bunch of ideas about what needs to be done but none of the stomach to do any of it.
Which is why I say "stubbornness" is the most important tool in my performance toolbox, far above cunning.
This assumes that you have plenty of cash to burn in the process, which is approximately correct for VC-backed ventures, and for offshoots of large corporations that play a lomg game.
I have a mountain of bad things to say about Microsoft, but they definitely had loss leader products that never made a dime but pulled you into their ecosystem. So they funneled money from a couple of cash cows back into the other projects to keep it enticing.
> start generating profit, and then you can start optimizing your costs.
this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit.
Or, in the current era of ai, the cost of the capital investment is far exceeding the ability to generate profit off it. The optimization in how the resources gets used will be needed to cut the costs down, and allow increase in the scale of usage for the same hardware. That's where profits would lay.
Of course, in order to achieve any of this, you'd need the runway to survive until such times. A small scale operator won't have this runway, and so die before they can accomplish anything profitable (or get big by begging for investor money to grow large - as we've seen in the past 20 years of tech).
I've seen a lot of people relearn the same couple of lessons over an over again.
One, most of the things you believe scale linearly scale at nlogn because of physics. Storage is much worse, scaling at n^1.5. What fit in a single 4U case is orders of magnitude slower when it no longer fits in a single data center. Inserting records into a table with a billion rows is much slower than a table with ten thousand. Wipe the surprised Pikachu look off of your face.
Second, every architectural choice needs to be reviewed every time traffic increases by an order of magnitude, and preferably before so that you have time to stew on possible solutions and allow serendipity the time it needs to find you a better answer. If you have made too many architectural decisions, you will find there aren't enough hours in a month to keep up, even if you never sleep. That's the real reason we practice Principle of Least Power. Not security, but job security. Keep the system small enough that you have time to keep the plates spinning while also implementing new features to attract or retain customers.
What I find surpring here is this being about DNS. Simple optimization should have been done maybe 1997. Letting it build up to 100 TB is noteworthy, but on the other hand for IT that's common. It's surprising but also totally expected...
My house is a ~700 sqft. condominium, gov. subsidized for lower income individuals, and even my mortgage is more than 300k… maybe I’m just basing my info off of coastal city prices, but is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
Can you ground the discussion by mentioning what you think these cities are? Taking Columbus, OH as the most average of American cities and a 20m isochrone map from city center, there are currently 0 parcels for sale with 3+ ("several") acres under $300k. There are a few within 30m drive, one of which even has a possibly habitable structure. The rest are bare agricultural land you'd need additional investment to actually live on.
I'm just looking at Kansas City as an example right now.
Tons in the $200s btw, and this is in the city less than 20 minutes from the Nelson-Atkins Museum, WW I museum, the river, farmers market, KC Chiefs / Royals / Sporting KC / KC Current teams, parks, schools, newly rebuilt international airport, etc.
I picked the convention center as city center (union station would also be a reasonable choice) and I was only able to find 2 improved lots under $300k. Notably, neither is quite 3+ acres or within 20 minutes of the things you listed, and both are on septic. And this is in a metro where home prices are 36% below national median.
First if you us the convention center, you place the housing center off of where most of the housing actually is. Maybe 40% is in Kansas, with another 60% in the metro being on the Missouri side, roughly. The population density of the metro drags further south though if you look at it on a heatmap.
First if you us the convention center, you place the housing center off of where most of the housing actually is.
Convention centers are usually downtown. Mean population centers reflect the surrounding geography and can drift pretty far from an "idealized" circular urban area. My area's is out on an island for example, nowhere near downtown. Jackson county's is east of Raytown. I'm not sure why you'd specifically want to live within 20 minutes of the mean population center, so it wasn't the obvious interpretation of your original statement.
Between $175k and $350k center on KCMO alone there are over 760 results.
No, there are 17 results once you filter for 2+ acres. Most of them are mislabeled or unimproved property. If you filter those out too (because this is a housing discussion), what remains within 20-30ish minutes are the two I linked.
I realize multiple acres of land is a bit of a silly requirement, but you're the one who suggested there are tons of houses with "several acres within 20 minute drive to a lot of US city downtown areas". That's why I was surprised.
> That would buy you thousands of square feet and often several acres within 20 minute drive to a lot of US city downtown areas.
This is my direct quote.
Let me break this down:
1. That would buy you thousands of square feet
True, clearly seen in the search results.
2. and often several acres
True, you just filtered down to several results currently on the market.
> I realize multiple acres of land is a bit of a silly requirement
Why? I live slightly further than 20min outside of town. There are massive, beautiful homes with tons of land out here, and tons have gigabit fiber from GFiber or several of the other providers in the area.
But that's not a requirement if you don't want it to be! There are clearly tons of homes available in the city with 3-4+ bedrooms and 2+ bathrooms in the $200 and $300ks is the point.
Desirable neighborhoods are by definition expensive. The trick is to find a neighborhood you like where your home can just be a home and not a top-heavy investment.
Acknowledging this isn’t always easy or possible, but just pointing out that this is a self reinforcing problem.
> is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
What does "city" mean to you? For some, it's 500 people, or 5,000. For some, it's 5 million. Define that first. The US is a big place, and I know people that don't live within 50 miles of another human.
If the extra $200/month means you can’t afford it, you probably shouldn’t be buying a house…
Yeah that sounds cruel and messed up, because it is. I don’t like how expensive real estate is and I wish we could have another 2008 without a bailout. But the point stands. If you hafta pick between eating and paying your mortgage, you made a bad financial decision. Blame whoever you want, I am of the opinion that people still have agency and are responsible for their decisions, as unpopular as that seems to be these days.
This reasoning is largely centered around the runway being finite. You obviously can't have costs so high you are making a huge loss, but also there's little value in improving margins past profitability until you actually have a stable segment of the market.
This is true if you can scale out (ie you can add resources to your system). But for a robot for example, just adding a GPU can just flop your product completely: you need more battery, more weight, suddenly your unit economics is out of the window... Your next hardware iteration will be very slow to come and very expensive.
So here, you better not have a system wasting too much resources pretty early on after the prototype phase.
You underestimate the cost of "optimization", sometimes it means actually rebuilding large parts of the system. I would not say it is the "easiest" part, but it is usually not what will kill your business though.
from a business perspective, this might be considered the only way, but it is not. at large volume scale it becomes more, but often large scale is lacking optimisations in the first place.
its not wrong in my eyes, but definitely not the only path to take.
>optimization is by far the easiest part of the process
Not if the whole thing is architected poorly but was a requirement of the hour so it became big. Then optimisation becomes an art, but definitely not the ‘easiest part of the process’
> start generating profit, and then you can start optimizing your costs
Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...
I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.
"The boss" is not who says what is good enough. Ultimately it's the customer. In many industries it seems good enough is not very good.
Then there are industries where the customer complains if code is slow. They will actually hire expensive consultants to analyze and benchmark the code. And while the consultants likely are not more talented than inhouse staff, now you have both sides very interested at looking at the problem from engineering perspective.
It is a comment on the absurdity of orbital data centers. Mountaintop data centers sound absurd, but are more feasible and efficient than orbital ones in nearly every aspect.
Cooling is not the crux of the real problem, it's the fact that we have no way to replace single failed units in a running space-based data center without another launch - and if youre stressing your total launch cadence with 'new' datacenters, at what % do you repair or replace the whole slab.
The launch tempo, following the invention of a functioning approach to in-space single node replacement for even a modest portion of the planned workload capacity is something that strains credulity, even at the normal earth-level maintenance rate.
Addressing the increased failure rates due to the hard rads and geomagnetic effects, while demanding that orbital systems remain above nm% load - that's n% of the hardware still operating - at 100% power and thermal, or 100% of hardware at m% of power and thermal, or the intersection of those two slopes at any given time - in order to meet shareholders profit expectations pushes that launch cadence and cost - to maintain the baseline of workload... and well, the math of that for even a minimal % of earthbound current deployed demand is just staggeringly many launches per year.
Maybe i'm missing something, but bigger vehicles for putting larger payloads doesnt make it better, it makes it worse.
That's fine, if the argument for DC in space is just "Let's put them in the hardest place possible". Then less hard -> absurd, implies more harder -> more absurder.
But space based dc accomplish something that mountaintop dc do not. The different list of benefits/tradeoffs are why space DC are proposed and mountaintop ones are not. It's a difference of kind, not degree. It's not a meaningful experiment to just try to build DC in hard places and then we can finally validate space.
Stated benefits in particular:
- Power available 24/7 for "free"
- coms w/o interruption using existing infra
- Rideshare (SPX can build out capacity while other lifts pay some of the bill for lift)
- Nonregulation
- Very low latency to "places of interest far from USA mountains"
And no, I do not believe that mountaintop automatically satisfies these benefits in a smooth way such that mountaintop is a meaningful stepping stone towards space.
The Sun is visible from Earth as well, the last time I checked.
In LEO you don't get power 24/7 because you are only 500km above the Earth. Yes the Sun is more attenuated on Earth but what we care about is $/W not raw wattage, and Earth certainly has cheaper $/W than space.
> - coms w/o interruption using existing infra
I'm perplexed how comms might be easier in space than on Earth where you can just run a cable.
> - Rideshare (SPX can build out capacity while other lifts pay some of the bill for lift)
On Earth you don't need to rideshare because you don't have to ride a rocket.
> - Nonregulation
Space is more regulated than Earth. The only way to get to space is via a rocket which is the same as an ICBM. Governments regulate the process of building ICBMs and what payloads can ride on them.
If you want non-regulation then go to international waters or find a bribable government.
> - Very low latency to "places of interest far from USA mountains"
The latency is not terrible in LEO but it's nowhere near as good as on Earth.
We're losing the direct chain of thought here. My assertion is that "Nonexistence of Mountaintop DC is not a counter-example to space DC". That's it. The reasons were spelled out.
Your points: "Mountaintop" is how comms is easier in space vs on earth. Starlink already serves many rural areas simply b/c it is easier to go to/from space in some places than "running a cable". "Latency is nowhere near as good as on earth" is just false. "Mountaintop" is why. But more broadly, my most recent vacation cabin has higher latency than starlink offers. Case closed I guess?
And one more on latency: I was referring to latency in areas of interest far from USA mountaintops / USA in general. You might want to peruse the DARPA programs on low latency in-situ, closed loop comms for in theater (sometimes space based) compute. Something close to the action.
Power: "Mountaintop" is how space has a better power case than earth. Not all of earth. Mountaintop earth.
top level comment was talking about a wind turbine on a mountaintop. That's an attempt at 24h power which is very likely strictly worse.
You can step back and make larger arguments, but this thread is narrower.
"Space is more regulated than Earth". Yes, again, you're talking about wider counts of regulation. Just go look around at the pushback to data centers and you'll see some of the case for DC in space. The path to getting equipment into space is clean - just get permits and launch same as SPX does for starlink. The path to building a data center on a mountaintop probably encounters at least some non-paperwork pushback that's likely to trip big political fights. That's it. Are there a lot of mountaintops that are sufficiently cold to warrant "cooling" arguments that are not part of large state/federal parks?
So going back to the thread - if you believe that a mountaintop datacenter is a counter example to the feasibility of a space-based data center, then I think you're making a category error on some of the above criteria. Your comments don't dissuade me at all about that because they don't address either side of that argument.
Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
reply