Frankly yes. Do it in a way that doesn’t violate outright or cover your tracks. Customers deserve to know the truth of the type of company they are dealing with.
So someone should potentially get fired and face legal action to satisfy your curiosity which you'll get sated anyway when you read the postmortem in a few days?
It’s not like OP is putting a gun to anyone’s head!! If they want to share we’d appreciate but there is no obligation.
Are you also shocked that you can stand in the middle of the highway and get hit by a car and die? You can do it if someone asks you but you’re not forced to.
It's not the postmortem du jour that we want, the whole pattern is a mystery. Do they push releases every day and bring down the site every day by botched rollouts? What really is it why they can't GH keep the core service running (e.g. pushing commits and pulling changes)?
I would take what C-level execs say with a grain of salt. Remember these folks hire people with plenty of system design experience. They likely also have the money for hardware resources.
If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.
Something more serious is at play. And I want to hear from the actual underlings.
More load does not just mean needing to increase capacity, which would be simple for Microsoft to do.
Additional load reveals limits and breaklines that were never known before - your temporary integer IDs are suddenly too small, you reach the maximum file descriptors in a Linux distro, your hash function suddenly has repeating conflicts, your DB index is suddenly oversaturated.
Having massive exponential growth of your traffic would probably require constant refactoring to meet the ever changing system limits.
A good rule of thumb I once heard (from Jeff Dean, I think) was that (in a complex system) every 10X you go up in scale you need to re-consider basic design elements, trade-offs, etc, and probably change some of them.
It's hard to design something (large and complex) for much more than 10X scale with only HW capacity increases.
Infra is a cost center right up until its absence begins to undercut your value proposition to customers, and then it becomes an investment. GitHub seems well on the wrong side of that equation right now.
In general true. But most of paying customers, including enterprise have made a bet that they cut down expense and resources on operational experience, system management knowledge to minimum because cloud .
So even if these cloud/SAAS solutions perform poorly it is not like customers have wherewithal to create alternate solutions.
>They likely also have the money for hardware resources.
Hey man, in case you hadn't heard, the hardware industry is having a lot of trouble keeping up. "Expand our capacity" isn't as easy as it was 2 years ago.
My pet theory is they backed themselves into a corner with Actions. That was a step change in computational scaling effort. It also fell immediately on the heels of the Microsoft acquisition.
Something about these two factors seems to have conspired to wreak absolute havoc on their uptime in the ~7yr hence. The AI slopocalypse hasn't helped matters, but the foundations were laid far earlier.
Would love to hear from an insider, either for or against this conspiracy theory.
First, that source is unreliable, particularly for before the acquisition. (We don’t know whether the data is accurate, it’s based on GitHub’s own self reporting.)
Secondly, the scale is WAY off and makes it look far worse than it is. It makes it look like 99.5% availability is practically zero availability.
Finally, GitHub’s availability is not a binary all-or-nothing proposition. They report incidents on a granular level, for example, webhook firing can be impaired while Git hosting may be working fine.
0.5% downtime is an hour a week. If that happens for 10 minutes a day at 11pm PT it doesn’t matter, if it happens for half a working day once a month, it’s a disaster.
reply