Updating Linux Kernel Without Reboots [Live Patching Tools Overview]

Updating Linux kernels is a routine – as dull as taxes and only slightly less inconvenient than death. New security vulnerabilities in the Linux kernel seem to appear with tedious regularity and even get fancy names. In most but not all cases, the patches needed to fix them follow swiftly after. There is work involved in patching the kernel the latest Linux kernel security updates, and danger if you delay–leave it too long and bad actors might take advantage of the period of vulnerability.

In our previous blog post about, we discussed How to update Linux kernel with 3 different ways, two of which (using command line/yum and kexec) require a server reboot.

It is time to review another way of Linux kernel security updates – rebootless live kernel patching. Read further to learn more about each live patching tool and alternatives.

Applying Linux kernel updates without rebooting

There are times when security patching is super-critical, but so are the processes that stop when you reboot. If you’re running an ‘always-on’ or ‘high-availability’ system, you’ll already be familiar with this dilemma.

Rebootless Linux kernel updates are not a replacement for full kernel upgrades, as it only applies patches for security vulnerabilities or critical bug fixes. But, in many cases, this is all you need, and it is possible to keep a server safe and running for years between reboots using these methods.

A number of leading Linux vendors offer rebootless kernel updates. The one you choose depends on the distribution you run and on your budget. In the remainder of this article we’ll talk about the following products:

• Ksplice by Oracle (for Oracle Linux updates, Ksplice Uptrack for enterprise)
• Kpatch by Red Hat (for RHEL kernel updates and CentOS updates)
• Livepatch by Canonical (for Ubuntu kernel updates)
• Kgraft by SUSE (for SUSE updates only)
• CloudLinux KernelCare (for all major Linux distributions)

Get a FREE 7-Day Supported Trial of KernelCare 

Oracle Ksplice

Ksplice was the first commercially-available implementation of rebootless kernel updating. Ksplice Inc. was eventually acquired by Oracle so that now it is only available (unsurprisingly) on Oracle Linux and RedHat Enterprise Linux distributions, and the deployment needs a license from Oracle.

• No reboot required.
• Automatic updates.
• Available for free on desktop Linux installations, with official support available for Fedora and Ubuntu Linux distributions.

• Works only for Oracle Linux, Red Hat Enterprise Linux, CentOS and Ubuntu.
• Requires a support license, pricing starts from $1,399 per system per year.

To deploy it, run:

sudo wget -N https://ksplice.oracle.com/uptrack/install-uptrack-oc
sudo sh install-uptrack-oc -autoinstall

 

Note, there is no reboot command, and you only need to run the install script once in the lifetime of the server. After that, the Uptrack service will automatically detect new kernel updates and deploy them for you. There’s no scheduling, no downtime, and nothing more to do.

Canonical Livepatch Service 

This is Canonical’s technology for live-patching kernels. You can even create your own patches, although it can be difficult, time-consuming work. Some vendors will create an Ubuntu upgrade kernel for you, for a fee. The service is available for Ubuntu 16.04 and later.

• No reboot required.
• Automatic kernel updatesFree for personal use (up to 3 machines or up to 50 machines for recognised Ubuntu Community Member)

• Non-trivial custom kernel patches.
• Limited distributions supported.
• Limit to the number of updatable hosts.
• Additional hosts for a fee – pricing for Ubuntu Advantage (at least Essential) support subscription starts from $225-$1,500/machine/year on physical servers, and $75-$500/machine/year – on VMs.

It’s deployed like this:

sudo snap install canonical-livepatch
sudo canonical-livepatch enable [TOKEN]

The Canonical Livepatch service is free for up to 3 machines for personal use or up to 50 machines for Ubuntu Community Members. You can sign up for a token here.

Red Hat Kpatch 

This is Red Hat’s own kernel patching tool. It was announced in 2014 and has been ported to work on others in the same family (Fedora, CentOS) as well as for some Debian-based systems (Ubuntu, Gentoo).

• No reboot required.

• Not automated.
• Limited distributions.
• Available on a Premium support subscription for $1299 per year.

Here’s an example of deploying it on RHEL 7:

sudo yum install kpatch
sudo yum install kpatch-patch-X.X.X.el7.x86_64.rpm

Unlike Ubuntu’s Livepatch service or Oracle’s Ksplice, it’s not automatic, and you must manually check for and install each kernel patch as it becomes available.

SUSE Kgraft 

Developed and announced at almost the same time as Red Hat’s solution, Kgraft is SUSE’s live patching offering (known as SUSE Linux Enterprise Live Patching). It’s only for SUSE’s own Linux Enterprise Server 12, and comes preinstalled, so there’s really nothing to do (except pay for it). It works on a different principle to most other approaches but has a feature-set comparable with Kpatch.

• No installation needed.
• No reboot required.

• Single platform support.
• Commercial (but there is a generous 60-day free trial).

CloudLinux KernelCare 

Also launched in 2014, KernelCare’s Linux kernel live patching service stands out among the kernel patching solutions in its OS coverage, which includes CentOS, RHEL, Oracle Linux, Debian, Ubuntu and others. And like Oracle’s solution, KernelCare supports the older 2.6.32 kernels from RHEL 6.

• Easy install.
• No reboot required.
• Wide OS coverage (including one of the most popular Linux flavors, Ubuntu).
• Supports custom and fixed-date patching.
• Good support and industry know-how from CloudLinux.

• Commercial (but there is a free, 7-day trial).
• There is also a free KernelCare license for non-profit organizations.

Here’s how to install KernelCare:

wget -qq -O -- https://kernelcare.com/installer | bash
sudo /usr/bin/kcarectl --register <your key>

For <your key> get your trial key here.

 

KernelCare is an ‘install and forget’ solution. Once installed, KernelCare automatically downloads and applies new kernel security patches, without rebooting the server.

 

But in contrast to its closest competitors, KernelCare can handle some of the more complex patches for vulnerabilities such as Meltdown (CVE-2017-5754), Spectre (CVE-2017-5753 & CVE-2017-5715), and more recently, the Linux kernel buffer overflow flaw known, romantically, as Mutagen Astronomy (CVE-2018-14634). KernelCare supports custom patch configurations, fixed-date patches, delayed patches, and rebootless rollbacks, i.e. patch removals.

Like the other vendors considered here, KernelCare also springs from a good blood line–its creator is CloudLinux, the leading web hosting Linux-based OS vendor.

Conclusion

If your server is non-critical and can endure a period offline, updating the kernel is relatively painless using the standard tools on the command line.

If you’re running an always-on system, (i.e. you can’t or won’t reboot), take a look at live kernel patching solutions. Of these, there are three kinds:

1. Administered–you have to do it yourself. E.g. Kpatch, Kgraft.
2. Fully automatic–it does it for you. E.g. Livepatch, Ksplice.
3. Fully automatic, advanced multi-platform–it does it for you, handling advanced threats on all platforms. E.g. KernelCare from CloudLinux.

If you want to learn more about live patching technology and how it enables your infrastructure security – read our most popular blog posts:

• Reboot Server Now or Later? (Neither, thanks)
• Custom Kernel Patching with Rebootless Updates

Have you ever had a chance to use Linux kernel live patching tools? Which one did you find the most useful for your business? Share your thoughts in comments.