I tell you what... If it weren't for Route53 being as good as it is and getting all of our domains sucked into it's gaping maw, we would be much more likely to hop to different cloud vendors. It is very nice having the entire enterprise tech stack managed through a single vendor when you are a tiny company like we are.

That said, I am losing patience with how slow the EC2 instances are considering what we pay. I've got management asking increasingly-probing questions about our monotonically-incrementing AWS bill. All of this would be fine if perceived/actual performance weren't also dropping for us over time (I.e. intel spectre mitigations). I can almost feel how the AMZN profit margins are squeezing us at this point... It's almost a weekly conversation now with Azure or even a return to on-prem being brought up. "Do we move now or later? Is the frying pan hot enough yet?"

Our organization is small enough to comfortably fit onto a single 2S 128 core AMD Rome system. Why shouldn't we just lease out a half-rack somewhere local (I.e. near the developers who can care for it) and then stick a few of their systems, a switch, router and management hardware in there? This all began on-prem with us moving to AWS, and after 6 years in the cloud circus it's starting to feel like a safe place to return to. Perhaps we will just move our compute to on-prem and continue to use AWS for backups and DNS. All I know is my TR 2950X workstation can compile our solution ~10x faster than our Jenkins server which is running on a T2.Large. Imagine giving Jenkins 32 Rome cores. This is something we could actually afford if we owned the hardware.

There are also some other compelling factors for us to consider moving back on-prem. Emerging technologies like Blazor create a strong argument for keeping your workers near the datacenter. Very few businesses truly require more than 1 physical datacenter. Yes, you might also have a DR site, but you can arguably run all of the functions for 95%+ of businesses out of a single physical location. Also, having a physical location where you can hook up any arbitrary hardware means we could also pull our iOS build machine in-house and use proper high-end Apple hardware on the same local network as the rest of our infrastructure.

For what it's worth, I think that was the premise of hybrid on-prem and multicloud. If you can trade-off availability in your stateless compute and worker layer and you're fine mostly managing it on prem, the cost savings could be very compelling. However, you'll still pay for bandwidth, which is the real killer and recurring expenditure. At the end of the day, there's no getting around that unless you move fully to your own metal, but then you'll likely realize costs related to maintaining your own metal that made you want to switch to the cloud in the first place. One way you might try to test this out is by using bare metal AWS instances -- if it's going on the right direction but you want more savings, you could move to using AWS Outposts with your on prem metal.

Just to make some conjecture here, I think that there's a difference between not liking the market's prices and thinking that the market is mispriced. Another commentator's point that the difference in bandwidth between AWS/Azure/GCP/OCI and DO/etc being a factor of 10x should give you an idea of some of the price discrimination going on.

Ya. S3 is awesome but everything else on AWS sucks. I have AWS and only use S3 and nothing else.