Interesting that they wish to discourage key pinning. This is pretty commonly used by apps to make them harder to reverse engineer. It prevents you from installing your own cert in the system's trust store and then using that to man-in-the-middle the app's communication with its backend.
It's important to note that this is preventing pinning /intermediates/ which are re-issued about annually. It is usually a mistake to do that. It is possible that due to a large-scale incident, we'd have to revoke an intermediate immediately and switch to a backup.
While we don't recommend key pinning, there's nothing to prevent pinning Let's Encrypt's root CAs.
Do clients generally check for revocation for every cert in the chain? You would think you would definitely want to as a compromised root cert would be highly damaging!
There's unfortunately a lot of nuance and sadness here.
Generally, a root is trusted by virtue of being in your root store and can't be revoked via OCSP/CRLs. An operating system security update should remove distrusted roots.
The mainstream browsers all have a push-based mechanism that they can use to rapidly revoke a root or intermediate faster than that. If something truly bad happened, Let's Encrypt would ensure that happened ASAP.
However, many clients do no revocation checks at all. That's a big part of why the ecosystem is pushing to shorter lived certificates, intermediates, and roots.
It doesn't really mean anything to have "revocation" of a Root. The way forward is to explicitly cease trusting this Root if you believe it has been compromised. For example Microsoft can ship a Windows Update which distrusts roots they by default trust in Windows.
Exactly what clients do about revocation for Intermediates and End Entity certificates varies, the revocation information will be available publicly in one (or both) of two forms:
The Online Certificate Status Protocol. OCSP results have a lifetime so you can check back periodically, and they're created via a distinct key from the Root, which has its own certificate (provided in the response) showing that it's trusted to do this specific work.
Certificate Revocation Lists are signed documents with a list of revoked certificates. CRLs are again signed using a distinct key with its own certificate showing that it was authorised to sign these lists.
Some client vendors may (semi)automatically aggregate information to produce their own summaries so that their clients can rely on the aggregate rather than fetching revocation information during use.
OCSP in particular was designed so that it could be stapled which is a technique where your server fetches and then includes the OCSP results for your end entity cert (and possibly the intermediate) with the cert itself during a connection handshake, the client thus now has a valid (albeit perhaps not entirely fresh) OCSP answer without doing its own checks, since OCSP is signed you can't forge this although of course if you have a 48 hour OCSP good answer you could continue playing it for up to 48 hours even if subsequent OCSP results said revoked.
Unfortunately implementation for OCSP stapling has been fairly poor, it's usually wrongly or at least badly implemented by people who apparently have no idea what the goal was and so it can make your service needlessly less reliable if your server is defective in this way, so the main browsers are not (last I knew) much interested in pursuing this further.
It's specifically to discourage intermediate key pinning. If folks want to pin their own end-entity public key (and always re-use the same key when renewing their cert), go for it -- dealing with compromise of their own key is their own problem to solve. Or if they want to pin a root public key to ensure some other CA doesn't issue a MITM certificate, go for it (although that doesn't prevent a bad actor from getting the same CA to issue a MITM certificate; there are other mechanisms to prevent that).
Just please don't pin intermediate CA keys, which should be opaque to the end-user and need to be able to change quickly without breaking a bunch of apps.
Monitor CT for your domain name and if you find an "obscure CA" misissuing for your domain, report it! This may result in the obscure CA getting explicitly distrusted.
At some level this is not that great a solution, but it's really so much better than what we had 13-14 years ago.
CT logs only mean that if after the horse has bolted, you can start the process to close the door within hours.
That's adequate for low-value targets, but it's hardly sufficient for high-value targets. If a CA mis-issued a cert for *.aws.amazon.com should we be waiting around for a manual process on some mailing list to invalidate it?
I agree conceptually, and I think it's sad that we don't have anything more absolute. (I was a fan of HPKP, which has been deprecated as a "footgun" because apparently users often didn't understand what it was doing or weren't cautious enough when using it.)
But the CT system seems to work very well in practice. While the detection and remedy part is awkwardly manual, there are people working hard on them. There are also (following a "you have to disclose all intermediates ahead of time" rule introduced by Mozilla) fewer intermediates and we actually have a list of them.
Detected incidents involving intentional misissuance are very rare. When unintentional misissuance happens, the responsible CA has to publicly explain how it happened and what it will do to prevent the situation from happening again.
And then what? What are you supposed to do in the meantime until browsers have decided to distrust the CA?
Also, how do you detect "obscure" CAs in the log? If your CT monitoring daemon has some logic to distinguish "good" from "bad" CAs, why not use that logic directly in the app and implement certificate pinning?
A CT-honoring client should reject an end-entity certificate that isn't accompanied by a SCT. In other words: a dodgy CA that skips CT to avoid disclosure of their mis-issued certificate should be unable to convince any CT-honoring client to accept that certificate.
> This is pretty commonly used by apps to make them harder to reverse engineer. It prevents you from installing your own cert in the system's trust store and then using that to man-in-the-middle the app's communication with its backend.
As an aside - this isn't generally very effective or worthwhile nowadays.
Anybody doing reverse engineering at a level where they're installing their own system certificates & intercepting traffic can easily run many off the shelf scripts (Frida scripts, Objection, apk-mitm, etc) which will disable certificate pinning like this automatically. This takes _seconds_ to disable and is standard practice.
Even doing so manually is not especially difficult - the pin is fairly predictable value, and a reverse engineer can access all content of the mobile app, so can easily search for and just replace it before installation.
From a reverse engineering perspective, there's little-to-no value in purely client-side protections like this. You're not increasing the reverse engineering difficulty significantly and you do create many practical problems with e.g. certificate rotation in future.
OTOH there is an interesting case to be made for certificate pinning as a protection for users being unknowingly MitM'd. That's a different scenario that may well be worth defending against, but due to the many downsides of cert pinning, using certificate transparency to mitigate this is strongly preferable nowadays.
Mostly agreed - I think the one difference is that to disable the pin, you have to modify and sideload the APK, which a non-rooted phone may not permit you to do. (Or your modified apk might not have access to the app's local data, security keys, etc).
In contrast, with a non-pinned app, you can monitor traffic of that app as it is installed right now.
You should be able to fully inspect all traffic originating from your device and reverse-engineer everything running on it, so any change that makes it harder for others to stop you from doing so is a good one.
> Interesting that they wish to discourage key pinning. This is pretty commonly used by apps to make them harder to reverse engineer
I don't think you'd typically pin intermediates if that was your goal: you'd pin the site cert you control. It could be as simple as a whitelist of certificate SHAs.
Pinning the site cert (or its key) is not great because it makes it very hard to rotate your certificate or key.
And you've got to be able to do that, in case your key is disclosed or possibly disclosed.
Pinning intermediates doesn't make a lot of sense; unless you have a deep relationship with the CA, they can and do change intermediates, and may not be able to issue from the old intermediate after the change. But, I've been involved in cert pinning where we had to pin intermediates, because that's the only viable information the client API returned we could use for pinning (and really, it was pinning based on the name of the intermediate --- any system trusted Root CA could have made an intermediate specifically to bypass our pinning, but it was the best we could do on that platform; of course, this was also the platform where we ended up with a wrong pin in our final build (platform owner shut down their app signing CA), which meant some ugly workarounds).
Pinning roots makes sense, but you need to pin more than one. And roots from more than one organization. And you need a succession plan. You can't use a new root on an existing hostname until all of the deployed clients with pins that could use that hostname have become irrelevant. If your all your trusted CAs stop issuing, you're pretty screwed.
You can sometimes use an in-house root, which makes continuity easier, but not if you need to share hostnames with browsers. You can't get an entity cert signed by two issuers, so there's no way to present a cert that is trusted by browsers and also signed by your root. You can present alternate paths for the issuers of your cert (and any intermediates) and if the x.509 libraries do the right thing, any valid path is accepted; but not all libraries do the right thing, and again, that doesn't apply to the entity cert.
Pinning isn't really for anti-revere-engineering. It does discourage some reverse engineers, which is a bonus. But it's main purpose is to protect data integrity between the client and the server --- there's plenty of cases of CAs in system root stores that issued certs that could be used to intercept and manipulate traffic. There's also plenty of cases where something installed on the user's device adds itself to the system root store and intercepts traffic with the intent of passing it through but does not do so correctly and causes hard to diagnose errors.
A pinned cert makes it easy to detect these MITMs and refuse to work if they're in place. A dedicated reverse engineer is "just" going to patch out the pin, and go on with their day, no big deal. But ISP MITM or corporate MITM or on device MITM is going to be stopped, and that's worthwhile.
That's sort of outside the threat model though, isn't it? If an attacker can install a root cert, the entire thing is compromised.
If you're concerned about that and want to pin, why have a chain of trust at all? Couldn't you just embed self-signed certs in the binary at that point?
If you mean preventing users from clicking through the SSL warnings in a browser if they're being MITM'd, that makes sense to me.
> If you're concerned about that and want to pin, why have a chain of trust at all? Couldn't you just embed self-signed certs in the binary at that point?
Not if you need to serve https traffic to clients and browsers from the same servers / same hostnames.
> If you mean preventing users from clicking through the SSL warnings in a browser if they're being MITM'd, that makes sense to me.
That comes from HSTS, not from cert pinning. Cert pinning in browsers is only available if you can convince the browser to include it in their static lists, which means you've got to be big... so that is what it is.
> Not if you need to serve https traffic to clients and browsers from the same servers / same hostnames
That's what I was getting at, that makes sense to me.
> That comes from HSTS, not from cert pinning.
HSTS is a terrible half assed solution that relies on adding domains to magical predistributed lists. I wouldn't trust it if I cared enough about MITM to go to all this trouble.
> HSTS is a terrible half assed solution that relies on adding domains to magical predistributed lists. I wouldn't trust it if I cared enough about MITM to go to all this trouble.
Sure, but cert pinning in browsers is done with the same magical redistributed lists; it's just even harder to get them changed.
I'd recommend pinning (your) keys because knowing cert SHAs ahead of time means you need the certificates themselves and in a nasty situation the CAs you relied on to issue those certs may be out of the picture, if you're pinning a key that's fine, CA #5 will cheerfully issue you a brand new certificate against the same key you were using with CA #1 before something went badly wrong - but with cert SHAs you have to bootstrap all of that fresh if it happens.
If you pin keys you can even pin a key you haven't and never plan to use, keeping the corresponding private key in the company safe as a hedge against something going badly wrong.
> reusing private keys is an enormous vulnerability.
As long as the private key is stored/handled safely and RSA/ECC is not broken, it is not vulnerable.
I do agree that key rotation is better/recommended practice.
> a single leaked private key means your entire site is compromised
The leak is the actual vulnerability. As long as the leak is still there and you are not aware of the compromised private key, a fresh new private key will probably leak again.
However, the chances of leaking may be greater if a private key has to be used in multiple locations.
They're mainly trying to issue certs for public facing websites. If your service is in a position where someone can install an alternative root CA on your webserver, you're already pwned!
The threat model isn't "someone will install stuff on our servers". The threat model is "some user will look at the network traffic and see all the personal data we're hoovering from their phone".
Sometimes also "some user will look at the network traffic and notice that we have 0 server-side security".
Also, some companies[1] have been caught being naughty and installing CA certs which allowed third parties to MiTM local connections. This was a major issue and, at least in my case, what drived many to pin certs at the time.
> The threat model is "some user will look at the network traffic and see all the personal data we're hoovering from their phone".
If you're talking about someone wanting to examine an app they have installed, there's nothing a developer can do about it.
There are frameworks like Frida that allow you to disable pinning or disable certificate verification entirely in an app. I did this myself once while reverse-engineer the API that an app was using. Configured my phone's system proxy to point to Burp Suite, then used Frida to disable certificate verification in the app so it'd let Burp do its MitM magic, and I could see all the app's network traffic.
Does this not simply mean they want people to pin "ISRG Root X1" instead of pinning "Let’s Encrypt R10" because they want the ability to rotate the latter?
Although IMHO saying "don't pin this cert which expires in 2027, instead pin this cert which expires in 2035" is kinda a weird distinction to make as those are both too short.
Pinning your CA's certificate protects against two things:
1. All-powerful NSA types who've managed to steal a CA's private key or get a certificate mis-issued. It's a nigh-irreplaceable attack opportunity, but the moment they use it it'll be revoked within hours, so naturally it's reserved for the absolute highest value target. For example, [1]
2. Users trying to reverse-engineer your app.
You can imagine which of these is the most common - perhaps you don't want the user to be able to bypass the pin, even intentionally.
I believe they are using "certbot" to mean "Let's Encrypt", in which case their advice is sound -- if you truly have to pin a key, pin your own end-entity cert's key, not the CA's key.
By certbot key I mean letsencrypt key, sorry for confusion.
That means that you need to hard-code the public key of the certificate that you're using in the client application. That certificate might be signed by letsencrypt or any other CA, it doesn't matter. If you're pinning CA key, you put trust into entity which you don't control. They can replace any of their keys. They can stop signing your certificate. It makes little sense to pin CA certificates.
Just make sure to have a plan for key rotation in case you need to revoke your key.
These three "levels" are the minimum that would work in the Web PKI and thus usual (for public services).
The reasoning goes like this. Roots represent entities you trust (or in most cases, which are trusted on your behalf by some entity that's thought about what constitutes trustworthiness e.g. your browser vendor, OS vendor, or maybe your workplace). Accordingly the root "certificates" aren't really certifying anything, the X.509 certificate format is a convenient structure to put this data in, but unlike most certificates they don't represent a certification of anything by anybody, in human terms these documents tend to say e.g. "We, ISRG claim that we are ISRG and we're a Certificate Authority and we're signing this document to say so". OK, but anybody could make such a self-signed document, there's no way to "authenticate" them, you must have decided to trust some of these claims (or as I said, your vendor probably did for you)
The private key corresponding to a root is therefore very precious, billions of people trust signatures made with this key so it's a huge deal if it's stolen/ compromised. As a result we require that these keys aren't online, they're typically manifested as a small physical object (a "hardware security module") locked in a safe, the idea is that the key itself cannot easily leave the object, so by locking it in a safe we've protected the key.
But since these roots can't be online, we can't make certificate with them online, so a simple system of the sort you might build at work, with a CA root that directly issues certificates, isn't possible. This is where the Intermediates come in.
A ceremony is conducted, with third party auditors and senior personnel from the CA to watch as the Root is used to sign one or (as here for ISRG) a handful of Intermediate certificates. These documents can usefully be verified because they were signed with the Root's keys, if we trust a Root we can check that indeed it has signed this Intermediate. The CA is also required to have a technology so that it can (in a reasonable amount of time) revoke an intermediate if for example it was compromised or stolen.
Unlike a root, the Intermediates can then be kept online (at ISRG only some are used this way, others are just backups against the unexpected) and used to sign End Entity (what you've called Server) certificates in real time shortly after they are ordered, once the appropriate checks have been carried out. HSMs are used again, but in this case they're attached to a server where the issuance software is running rather than locked in a safe.
Some of these anti-tiger rocks even come with an anti-tiger warranty. If you can confirm that you were attacked by a tiger and the rock failed to protect you, you can make a claim for the price of a new rock.
Let's Encrypt, and WebPKI in general is meant for public consumption of certificates.
I think what this means is, if you're needing some stricter limitations on these, use private PKI within your own devices. E.g. for Machine to Machine comms.
You pin public key, not certificate. You can keep your private key for whatever period you want and issue multiple certificates with the same public key.
