FOXTROT/ALFA Special Issue: Let’s Encrypt Fail

Hello and welcome to a quick special issue of FOXTROT/ALFA for Wednesday, 4 March 2020! I’m back from my holiday but I immediately dove up to my neck into work again and hadn’t planned on re-starting this newsletter for a few days yet. But then, this huge Let’s Encrypt story broke and I thought I’d better give you a quick heads-up in case you’re managing a webserver or two and are using Let’s Encrypt.

Let’s Encrypt Revoking 3 Million Certificates

There’s a serious issue with Let’s Encrypt’s backend software, that is causing the CA to revoke 3,048,289 currently-valid certificates by 03:00 tonight (CET). As ZDNet reports:

The Let’s Encrypt project will revoke more than 3 million TLS certificates on Wednesday, March 4, 2020, due to a bug it discovered in its backend’s code. More specifically, the bug impacted Boulder, the server software the Let’s Encrypt project uses to verify users and their domains before issuing a TLS certificate.

The bug impacted the implementation of the CAA (Certificate Authority Authorization) specification inside Boulder. CAA is a security standard that was approved in 2017 and which allows domain owners to prevent Certificate Authorities (CAs; organizations that issue TLS certificates) to issue certificates for their domains. Domain owners can add a “CAA field” to their domain’s DNS records, and only the CA listed in the CAA field can issue a TLS certificate for that domain.

Let’s Encrypt found and patched a bug in Boulder over the weekend that was introduced in July and would have allowed attackers under certain circumstances to obtain valid certificates for domains that specifically do not allow this because no CAA field is set. This was because Boulder wouldn’t check this field correctly for all applicable domains.

It is very unlikely that someone exploited this bug, the project said.

Nonetheless, today, the Let’s Encrypt project announced it was revoking all the certificates that have been issued without proper CAA checks, following industry rules, as dictated by the CA/B Forum.

If you are admin for a webserver that uses HTTPS and you are managing its TLS certificates from Let’s Encrypt, you might want to check if your certs are part of the problem. If they are, there’s probably a certbot renew --force-renewal in order.

I’ve checked fab.industries already and you shouldn’t experience any TLS errors on my site tomorrow. Fingers crossed.

I’ll be back with the regular newsletter in a few days. Until then: Hang in there and check your certificates!


This is an archived issue of my daily newsletter FOXTROT/ALFA. You can find more information about it, including how to subscribe via email, on this page.