We know that frequently updating Linux kernels is critical to the safety of cloud environments – kernels are, after all, a cybersecurity blind spot. But updating kernels is time-consuming and often requires a server restart which can disrupt services.
That’s why live kernel patching is so critical and why Amazon offers the ability to automatically patch Amazon Linux 2 instances. In this article, we explain how live patching of Linux kernels work in AWS, why live patching from Amazon might not be the best solution – and what you can do to apply live patching if your Linux servers are not hosted with AWS.
Contents:
What Is Amazon Kernel Live Patching And Why Do You Need It
Patch management is a major issue for companies large and small. Security vulnerabilities are discovered, exploited, and patched every day of the week. In fact, the Linux kernel is patched countless times in any given year – and failure to apply patches means that rogue actors can exploit a vulnerable kernel.
Indeed, at the enterprise level, kernel patching has evolved into a compliance issue. Large organizations must patch frequently because, if they don’t, a breach can lead to data loss that affects large numbers of people, with significant consequences.
But frequent patching can imply constant service disruptions as Linux servers are taken offline for a restart. These ongoing patching and restart efforts also consume an outsize amount of time. It leads to a difficult standoff. Choosing between expensive, disruptive patching – or living with an unacceptable degree of risk.
Live patching is the answer – and that’s why Amazon has rolled out live patching technology for Linux 2 instances running on AWS. Live patching is supported where you use Amazon Linux 2 and where your kernel version is 4.14.165-131.185, or later.
Note that Amazon Linux 2 Kernel Live Patching only works under x86_64 environments, ARM64 is not supported. ARM-based Linux servers also need live patching, of course, and there is an alternative solution for organizations that run ARM workloads – via KernelCare.
Which Patches Can Amazon Kernel Live Patching Apply
AWS provides the ability to apply two types of kernel patches without restarting the Linux server. First, you can apply the typical bug fixes that Amazon issues, including kernel patches that intend to improve the stability of Amazon Linux 2 servers.
More importantly, thanks to live patching, your organization can ensure that critical security updates are rapidly applied without the need to take servers offline. We’re talking about CVE’s of course, Linux common vulnerabilities and exposures.
Where Amazon Linux Security Advisory rates a CVE as important or critical this update will be included in the live patching regime. There are also cases where Amazon may make available a live patch even when a CVE is yet to be assigned.
Note that there is a time limit to the availability of live patches: you can only live patch a kernel up to three months after it has been released.
How to Activate Amazon Kernel Live Patching
You have a choice when you initiate and use live patching on Amazon Linux 2 instances. First, you can initiate the use of live patches on each individual instance by using the command line. Your other option is to automate live patching via the AWS Systems Manager where you can apply live patching to a group of instances.
If you decide to enable live patching on a particular Linux instance, then Amazon provides a complete guide to enabling live patching via the command line. You’ll need a few things to get started: you must install the yum plugin and also ensure that you have binutils installed.
Note that you also need to first check your kernel version – if it’s more than three months old your first step is to install the latest kernel version.
Once you have the components for live patching in place you can proceed to start the kpatch service. Via the command line, you can then view all the available patches, apply a selected patch live without restarting the instance, and also view a list of all the patches already applied.
Live Patching With the AWS Systems Manager (SSM)
Command line actions can help you ensure that a single Linux instance is kept up to date and patched, but at the enterprise scale, live patching can involve hundreds or thousands of Amazon Linux 2 servers.
Organizations that operate fleets of Amazon EC2 systems cannot afford to manually initiate patching on each and every server by using the command line. Instead, AWS Systems Manager (SSM) helps you to automate the process of managing patches, including live patching.
You apply live patches in the Amazon Systems Manager console by using the Run Command and by choosing a custom Systems Manager document called AWS-ConfigureKernelLivePatching, you can edit this document to meet your requirements.
By using Systems Manager you can live patch fleets of both EC2 instances and servers located on your premises – including virtual machines using the built-in features of Systems Manager – the Run Command and Documents described above, and you can also make use of the Maintenance Window.
Downsides of Amazon Kernel Live Patching
Amazon has made a good effort with live patching, but there are some points to be aware of around using Amazon’s live patching tool to patch Amazon Linux 2 instances – starting with the fact that ARM64 servers are not supported.
However, there is an important issue around the kernel version. Even if you apply all patches via live patching your server’s kernel will not update to the latest version number until you reboot your server. In other words, your server can be right up to date with the latest updates but still display an old kernel version.
Any compliance tools you may be using will reflect this old version – which can be problematic. Furthermore, Amazon’s live kernel patching will interfere with standard kernel tracking functions which can limit the functionality of some of the debugging tools you use including kprobes and SystemTap.
There is also of course the limit of a three-month window – your instance will only receive live patches for three months, if your kernel has not been updated in three months live patches will simply cease to be available.
Alternatives to Amazon Kernel Live Patching
Enterprises that make standard use of Amazon Linux 2 instances on x86_64 (i.e. Intel or AMD silicon) may well find that Amazon’s built-in live patching tool works adequately, permitted the instances involved fit a specific pattern, and that there are no use case requirements that conflict with the restrictions posed by Amazon’s live patching regime.
However, there are alternatives where companies use ARM-based servers or where the live patching offered by Amazon does not meet requirements. One of these, of course, is KernelCare from CloudLinux. CloudLinux is an AWS Partner Network (APN) Advanced Technology Partner.
Support for AWS Graviton2 ARM64 processors is just one of the many advantages of using KernelCare for live patching of Linux instances in AWS. View our complete comparison of live patching tools for more details.
Or, download KernelCare directly from the AWS Marketplace and try it out.
Check out other overviews of live patching services:
Overview of Enterprise Live Patching services: Spotlight on Ksplice
Overview of Enterprise Live Patching services: Spotlight on Canonical Livepatch
Overview of Enterprise Live Patching services: Spotlight on kpatch
