How Let’s Encrypt certificate changes affect Live Patching Customers

The expiration of a root certificate in the Let’s Encrypt certification chain causes multiple issues, especially when coupled with older versions of OpenSSL like those in CentOS 7.

OpenSSL behaviour in that version would fail validation if it found a “bad” (read: expired) certificate anywhere along the certification path. This has a ripple effect, making the connections to KernelCare’s servers fail. Users of live patching services like KernelCare (any version) on CentOS 7 are encouraged to update the ca-certificates package, which removes the affected certificate and thus allows the live patching client to resume working as normal.

The root certificate that expired in the Let’s Encrypt certification chain (DST Root CA X3) will remain in the chain until 2024. Recent versions of OpenSSL correctly ignore the expired certificate and validate using the alternate certificates present in the chain, but older versions of OpenSSL will fail the verification. This causes serious issues; TLS connections will fail when they should not. In an unfortunate twist of fate, the “certbot” utility itself will fail to update the chain and renew the Let’s Encrypt certificates (which would resolve the issue).

In CentOS 7, there is already an updated ca-certificates package that addresses the issue by removing the expired certificate, which then causes OpenSSL to no longer fail the validation of KernelCare servers’ certificates. If you have systems running CentOS 7, you should update this package as soon as possible to fix any issues related to failed connections. Note that this affects many other software packages and is not just a KernelCare specific issue, so updating ca-certificates is highly recommended in any scenario.

Updating the ca-certificate is done with the following command:

yum update -y ca-certificates

If you still have problems with systems being unable to reach our servers after updating ca-certificates, reach out to our support here.

If you would like to address this issue in an alternative way, you could blacklist the certificate manually. However, you do not need to do this if you update the ca-certificates package.

The following commands blacklist the expired certificate:

cp -i /etc/pki/tls/certs/ca-bundle.crt ~/ca-bundle.crt-backup

trust dump –filter “pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10” | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem

sudo update-ca-trust extract

[Source: https://blog.devgenius.io/rhel-centos-7-fix-for-lets-encrypt-change-8af2de587fe4]