My gut feeling agrees with "I don't trust the Dev's explanation tbh." Companies and developers often try to get away with doing Bad Things, and cry wolf publicly without offering up specifics about what they were trying to get away with. "Spin up 10 VMs for 500k rows of data" offers no explanation to just what those 10 VMs are doing. There is a big difference between "using memory and cpu" and "saturating the network in abusive ways".
Random speculation of one possibility: each of those 10 instances were suddenly doing something unexpected and spammy with the network. Maybe sending 500k+ emails (one per row of data claimed by the developer) over SMTP in a very short period, or jumping to massive spikes in torrent traffic, or crawling sites to scrape data (maybe each row of 500k is just a top-level domain name, and they crawl every URL on those domains, possibly turning 500k rows into hundreds of million of http requests).
The postmortem will be interesting. If DO is truly at fault here, that email after the second lockout saying the account is locked after review, no further details required... bad.
> to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.
Tildes preceding numbers means "approximately". Why doesn't he know exactly how many VMs he spun up?
Was he actually spinning up 1 VM per record and only allowing 10 VMs to be running concurrently?
I'm not a Pythong dev but why can't you execute 10 instances of Python on a single VM?
If you need to dedicate an entire VM to processing 1 row, what the hell is it doing?