I received one of the phishing messages as well as the follow up / apology. An interesting wrinkle is that both were handled by sendgrid and used the same dkim selector. I would guess that a set of sendgrid api credentials shared with some 3rd party service was compromised.
- One of our domains had a DKIM trust with Mailgun (3rd party)
- Mailgun was integrated with a planning service (4th party?)
- Planning service was integrated with a CRM (5th party?)
- CRM was integrated with a website (6th party?)
Website got pwned, spam ensues using the entire chain all the way back to our domain. This was a while ago but I think the website was pwned, leaked API credentials for the CRM, those were locked to only read the address book for sources (not even destinations! but '*' was allowed...) but because the software was crap the planning/calendaring service was registered as a 'source', which included API creds. The planning service itself was pretty good, no further grab-keys-via-API, but using what was already allowed you could send raw MIME messages and it would just use the Mailgun API it had access to.
Luckily for me, I was on a prometheus spree and had an exporter grab the Mailgun metrics every few minutes (Ironically to support the CRM team because they didn't have any good metrics of their own and did like to blame everyone else), so while it was configured to look for dips, it also triggered on spikes because those tend to end with dips too.
I think in the end nobody learned from it because every team/vendor covered their ass with "well we only run it in datacenters with firewalls so this is the cloud at fault" and I don't think anyone got flak for it (but some definitely deserved a fair bit).
