Great question! We address this in detail on our Security page [0], but I'm happy to give a high-level overview here:
1. Don't go down! We run two independent compute clusters on different managed infrastructure products (GKE and GAE) and route between them at the DNS layer to help avoid downtime
2. We store local encrypted fallback files on your infra via our CLI [1]. These local fallback files are fully managed and allow the CLI to continue to serve your secrets, even if our API or your internet connection is down. I use this heavily whenever I'm on a flight
Is there a scenario where the CLI fallback gets out of sync and you may as well not have the fallback? How is the fallback file secured and what's stopping someone else from accessing those secrets of they can get at the encrypted file itself? Finally, what if a disgruntled ex-employee still has that fallback - can access be revoked reliably even if your API is down or they just turn off the internet connection?
The local fallback file is updated each time new secrets are fetched. It is possible for your Doppler secrets to change between your last successful fetch and your next unsuccessful fetch, and in that scenario you wouldn't be operating off the latest secrets. We don't see the fallback file as an excuse for availability; our service needs to be up.
The fallback file is encrypted using AES-GCM with a key derived from the auth token and other metadata using PBKDF2. It is theoretically possible for an ex-employee to store the fallback file and retain a copy of the auth token and other metadata, even after the token has been revoked. In this case, they could construct the key using their privileged information. However, this attack would require active malfeasance by an authorized party during the period in which they're still authorized. It would be easier for the bad actor to store the raw secrets (by logging process.env, for example) than storing the encrypted file. To really solve this issue, the secrets would need to be time-bound and dynamic. Dynamic secrets are on our roadmap, but the issue you point out is a really hard problem. (And if you'd like to help, we're hiring![0])
I think if you substitute all your questions with env vars, all the same issues present themselves. You have to roll secrets whenever someone with access no longer has access, and the env vars can be stale just the same as the encrypted file.
What happens if Doppler is down or if there is a SNAFU when syncing the env in production?