The Ultimate Guide to Linux Kernel Live Patching Software

After the Equifax data breach, which highlighted the consequences of unpatched software, administrators have the delicate task of ensuring that the latest patches are applied especially across public-facing servers while reducing downtime from potential crashes and reboots. Plenty of open-source patching tools are available, but many of them force a reboot of the server. Instead of using resource-intensive open-source patching methods, a better choice is rebootless live patching. Live patching keeps the system updated and protected from the latest vulnerabilities while reducing downtime from reboots of critical production servers.

In this post, we’ll cover everything there is to know about Linux kernel live patching and the ways it can save IT staff time. Whether you’re new to live patching or an expert, we’ll make sure you walk away with a complete understanding of the way live patching can help your organization and keep it compliant with the latest regulatory standards.

The History of Live Patching

The first instance of live patching was seen in 2008 when Jeff Arnold at MIT created an automatic system to apply kernel security patches without rebooting. He named the system Ksplice that was later bought out by Oracle. Arnold created the solution after finding it difficult to update Linux servers at MIT without affecting students. While waiting to update a patch, one of Arnold’s Linux servers was compromised due to an unpatched kernel with known vulnerabilities.

Oracle eventually closed the source code in 2011, so administrators were cut off from live patching conveniences. Because of Oracle’s actions, Linux developers were forced to find alternatives, many of whom developed their own live patching software. Many commercial applications are available, and this solution has now become a critical component of Linux administration and has freed up time for busy IT staff.

What’s in a Kernel Patch and How is It Made?

Updates to any software including Linux and other operating systems are just changes to files from a base version. Every update is given a version number. Large changes are given a full incremented integer such as 1.x to 2.x. Small patched changes might only change the decimal value such as 1.1 to 1.2. Reviewing these version values will help administrators understand if patches are large or small, giving them a general idea on the amount of time it might take to patch a system.

In a Linux distribution patch, an update is a text file containing the delta changes between the currently installed version and the file changes of the new version. Linux uses the diff program to identify these changes. After the text file is created, you use the patch program to apply changes and update operating system files.

As an example of a simple patch procedure, suppose that you have a program with a file named sample.c that must be patched to the latest samplev20.c version. The following diff command will create a patch file with the latest version 2.0 of the program:

diff -u sample.c samplev20.c > sampleupdate.patch

With the patch file created, you can now patch the program. The following patch command will apply the update:

patch < sampleupdate.patch

With larger Linux patches, changes are made based on a code branch in a central repository. Many of the third-party patching tools mentioned in the “Finding the Best Live Patching Software” section eliminate the need for manual patch commands such as the above example, but the basic steps are the same.How to Apply Security Patches to Linux Kernel

Linux security patches are just a way of life for administrators. They are slightly less frequent than other patches, but are probably the most critical of all updates. Administrators must patch the operating system as soon as possible as vulnerabilities are made public to hackers who will immediately develop exploits against unpatched systems. At the same time, it’s a delicate procedure that should be done flawlessly or mistakes result in downtime.

There are 3 ways to patch your Linux system, one of which can be done without rebooting the system. We covered the full details of all three ways already, but here is a quick rundown of the process.

Apply Security Patches to Linux Kernel Using the Command Line

Using the command line is the simplest way to do it. The following Debian command syntax updates the kernel:

sudo apt-get upgrade kernel
sudo reboot

The following command updates Ubuntu:

sudo apt-get upgrade linux-image-generic
sudo reboot

And, the following commands update CentOS or Red Hat Enterprise Linux (RHEL):

sudo yum update kernel
sudo reboot

The pros of this method are that it’s a simple two-line command that requires no third-party installation. The cons, as you can see in the command, is that you need a reboot. With this method, your critical Linux server will be down for a time, meaning it must be scheduled and users notified of the downtime.

Skip the boot loader using kexec

Linux has the ability to skip the boot loader and hardware initialization phase, which significantly reduces the time it takes to reboot. You first need to install kexec, and then run this tool on your server. The following command installs kexec in Ubuntu and Debian:

sudo apt-get install kexec-tools

The following command can be used on CentOS/RHEL:

sudo yum install kexec-tools

With kexec installed, you can now run the tool to install patches using:

sudo yum update kernel
or
sudo rpm -qa kernel

The kexec tool displays the latest version as output. The following commands will install patches:

sudo kexec -l /boot/vmlinuz-3.10.0-862.3.2.el7.x86_64 
-initrd=/boot/initramfs-3.10.0-862.3.2.el7.x86_64.img 
-reuse-cmdline
sudo sync; sudo umount -a; sudo kexec -e

After you install patches, you still need to reboot. The pros of this option are a faster reboot time, but kexec requires much more typing. As most administrators know, a slip up in one command-line option can mean hours of downtime. The con of this option is the increase in the possibility of human errors.

Apply Security Patches to Linux Kernel Using third-party Rebootless Tools

Rebootless Linux patching tools eliminate all of the cons from the previous two options. The only issue with many of the open-source options on the market is that patching is limited to a specific distribution. Jump to this section to see a list of the best live patching software with pros and cons of each.

What is Live Patching and What is Patching Software?

Instead of manually creating a patch file, patching software does all the overhead for you. These programs reduce the time it takes to patch software and eliminates all the typing that could lead to human errors. Live patching takes it all a step further and eliminates the need to reboot, which is a common challenge for Linux administrators.

It’s always a struggle for administrators to determine the right time for patching. It’s critical for security updates to be applied immediately, but updates to production can’t be casually and carelessly installed. Traditionally, updates require a reboot of the system. This means downtime for customers and employees that need to use applications running on the server. With software as a service (SaaS), cloud applications serve users across the globe, so there are no non-working office hours for many administrators. This issue is true for many public-facing web applications.

Live patching lets administrators patch critical Linux servers without a reboot, meaning no more downtime. To see an example of live patching, read our live patching tutorial for Oracle Enterprise Linux 7.

Finding the Best Live Patching Software

You could reinvent the wheel and develop your own live patching software, but the best way to quickly jump to a rebootless process is to use third-party tools. Below is a list of live patching software with the pros and cons of each one.

The following tools are available for rebootless patching:

• Ksplice by Oracle (for Oracle Linux updates, Ksplice Uptrack for enterprise)
• Kpatch by Red Hat (for RHEL kernel updates and CentOS updates)
• Livepatch by Canonical (for Ubuntu kernel updates)
• Kgraft by SUSE (for SUSE updates only)
• KernelCare Enterprise (for all major Linux distributions)

Notice that the last item on the above list is available for all Linux distributions, so it’s the best option for rebootless Linux updates in an environment that has several distributions running in production.

Get a FREE Trial of KernelCare Enterprise 

Ksplice

As mentioned previously, Ksplice is the original rebootless patching application. Now, it’s limited to only Oracle and Red Hat distributions and requires a license to use it. The only pro to this software is that it automatically updates Linux. The following commands will deploy it on your Linux system:

sudo wget -N https://ksplice.oracle.com/uptrack/install-uptrack-oc
sudo sh install-uptrack-oc -autoinstall

Kpatch

Released in 2014, Kpatch is specific to Red Hat distributions only. It’s been ported to work on CentOS and Fedora and some Debian systems such as Ubuntu. The disadvantage of this application is that patches are not automatic. This means that the patching software still requires administrators to trigger the update, which makes it less valuable to enterprise-level systems with dozens of servers to manage.

The following command installs Kpatch:

sudo yum install kpatch
sudo yum install kpatch-patch-X.X.X.el7.x86_64.rpm

Livepatch

Livepatch will automatically update Linux kernels and won’t require a reboot. It’s also customizable allowing administrators to create their own patch instructions. The downside to this live patching software is that it will only work with non-complex patches and supports a limited number of hosts, making this tool unattractive to enterprise administrators as well.

The following command installs Livepatch:

sudo snap install canonical-livepatch
sudo canonical-livepatch enable [TOKEN]

The [TOKEN] argument requires a value supplied during registration. You can obtain a registration token here.

Kgraft

SUSE’s Linux Enterprise Server 12 includes the Kgraft live patching component built-in. No installation is required, so administrators can start using the software without making changes to production servers. It only supports SUSE Linux, however, and it’s commercial, meaning license fees apply after the 60-day trial.

KernelCare

CloudLinux KernelCare has all the advantages administrators want in a live patching application with support of a wide range of Linux distributions. KernelCare supports CentOS, Red Hat, Oracle, Debian, Ubuntu, and several others. Its live patching is automatic and can handle simple and complex updates to the operating system. Administrators can customize their own patch updates to target specific critical vulnerabilities immediately while delaying less severe issues that do not impact the organization.

Some advantages of using KernelCare over other solutions are:

• Rebootless patching for the latest most complex vulnerabilities in the wild.
• Supports enterprise-wide servers with several hosts.
• Rollback options without rebooting.
• Customize patching using dates and versioning.
• Delay specific patches while immediately patching critical vulnerabilities.
• Remove specific patches.

To install KernelCare, use the following commands:

wget -qq -O -- https://kernelcare.com/installer | bash
sudo /usr/bin/kcarectl --register <your key>

The key can be obtained by authenticating into the CloudLinux dashboard and registering the software after the 30-day trial.

Conclusion

Busy Linux administrators responsible for dozens of enterprise servers could potentially spend hours ensuring patches are installed. It’s a full-time job but it doesn’t need to be. Live patching takes care of much of the labor involved with Linux updates, saving administrators time and essentially saving the organization money.

You don’t have to jump right into a solution, but CloudLinux KernelCare has most of the advantages over other solutions. To learn more about live patching, contact us for more information.