kpatch: Overview of Enterprise Live Patching Services

There are currently a whole host of live patching tools on the market. Such options vary significantly in cost, with some significantly more affordable or expensive than others. Moreover, there are live patching tools that are more suitable for only one or two distributions, and then there are distribution-agnostic tools. So much variety can make it challenging to select the perfect option for your business.

One such tool is kpatch, a technology that applies essential security patches to kernels without requiring a system reboot. Although sufficient for basic security and bug fixes, kpatch lacks the persistent patching offered by KernelCare. This article will dissect the pros and cons of kpatch, giving you the information you need to make an informed choice best-fit for your enterprise.

Content:

1. What is kpatch?
2. History of kpatch
3. kpatch Supported Kernel Versions
4. How Does kpatch Work?
   System Components of kpatch
   How to Install kpatch
   How to Uninstall kpatch
   How to Obtain a Patch
   How to Create a Patch
5. Limitations of kpatch
6. Conclusion

What is kpatch?

kpatch is a Linux kernel patching feature created by Red Hat. Once installed, kpatch executes live patching of a running kernel. With kpatch, a Linux kernel patch is applied without necessitating a system reboot or restart. As a result, kpatch empowers system admins to apply crucial security patches to the kernel as soon as they’re needed, without having to worry about users logging-off, scheduling reboot windows, or completing long-running tasks. kpatch effectively amplifies system uptime and availability without forfeiting security or reliability.

History of kpatch

This patching mechanism was developed in-house by Red Hat’s kernel team, released in February 2014, and later submitted for inclusion on the main kernel in May 2014. However, as kpatch was quite similar to Suse’s kGraft, the two teams opted for a unified approach. Merging ideas and components from each tool, basic foundations for live kernel patching were introduced to the Linux kernel mainline in April 2015’s kernel version 4.0.

Such simple infrastructure functionality initially added in Linux kernel 4.0 was limited to patch module loading, unloading, and registration and strictly supported the x86 architecture. And so, there was no mechanism for guaranteeing function-level consistency during the application of hot patches at this juncture, raising concerns as a potential attack vector. Although there has been ongoing work since April 2015 to introduce kpatch to the Linux kernel mainline common live patching core, this remains a work in progress.

kpatch Supported Kernel Versions

Kpatch’s support only extends to Red Hat distributions, particularly RHEL 7.7, RHEL 8.1, and onwards. Users and enterprises with standard subscriptions have restricted access to the most recent minor patch versions only. For example, patches would be available for version 7.7 until version 7.8 launches. Those that require access to patches for older distribution versions must obtain a costly paid subscription.

How Does kpatch Work?

Kpatch operates on top of the ftrace mechanism. Part of the kernel and originally designed to function as an instrumental tool to help perform tracing of function calls via the kernel, the ftrace mechanism is repurposed by kpatch. This enables the interception of old function calls for those featured in kpatch-loaded patch modules.

Using the livepatch kernel subsystem, kpatch diverts old functions to new functions. And when applied to a system, kpatch causes the following to occur.:

1. After being copied to the /var/lib/kpatch/ directory, the kernel patch module is registered for re-application to the kernel on the next booth with systemd.
2. Once loaded into the running kernel, the kpatch module registers patched functions to the ftrace mechanism.
3. After accessing the patched function, the kpatch is redirected by the ftrace mechanism, bypassing the original functions and redirecting the kernel to its patched version.

System Components of kpatch

Kpatch consists of many system components.

The first component is a kernel patch module (.ko). This functions as the delivery mechanism for live patch updates. Designed especially for the kernel that requires patching, the patch module contains code for the sought-after kernel repairs and modifications. “The patch modules register with the livepatch kernel subsystem and provide information about original functions to be replaced, with corresponding pointers to the replacement functions. Kernel patch modules are delivered as RPMs.” And the naming convention for this is kpatch_<kernel version>_<kpatch version>_<kpatch release>.

The second component is the kpatch utility, a command line utility for managing patch models. The kpatch utility is a basic command line that functions quite similarly to standard linux tools for listing running modules: lsmod/insmod/rmmod. This tool inserts new modules to the kernel, removes modules from memory, and applies them to patch modules exclusive to kpatch.

The third component is the kpatch service. This is a system service loaded at boot, which loads the kernel patch modules to memory at boot time.

The final components are contained within kpatch-build, a set of tools used to compare one kernel source code tree with another that has experienced some changes, producing an altered kernel patch module. Although simple in theory, this requires in-depth knowledge of the kernel’s code and inner workings for safe and effective use.

How to Install kpatch

To install kpatch, begin by checking your kernel version. Then search for the existence of a version of kpatch that corresponds to your version of kernel with:

# yum search $(uname -r)

You’ll want to look for a package name that begins with “kpatch-”.

You’ll then install the live patching package with:

# yum install “kpatch-patch = $(uname -r)

This will install and apply only the most recent cumulative live patches that correspond to your particular kernel version.  If the live patching package is a version 1-1 or higher, it will contain a patch module that automatically patches the kernel during installation. This kernel patch module is installed into the /var/lib/kpatch/ directory to be loaded during subsequent reboots by the systemd system and service manager.

How to Uninstall kpatch

To uninstall kpatch, begin by selecting the live patching package with:

# yum list installed | grep kpatch-patch

Then, remove the live patching package with:

# yum remove kpatch-patch-version

The kernel patch module will be eliminated from the disk when a live patching package is removed. However, the kernel will remain patched until the next reboot.

Then, reboot your system. You can verify that you have successfully removed the live patching package with:

# yum list installed | grep kpatch-patch

If this command does not display an output, removal of the package was successful.

How to Obtain a Patch

Once a system has been configured for kpatch, patched modules will be included in yum update/yum upgrade’s normal system update mechanism. kpatch capability is then applied as a kernel module (.ko file) delivered in the form of an RPM package. This is the general method for update installation on Red Hat systems, making kpatch incredibly practical and user-friendly. However, although kpatch live patches are available to all customers through the usual channels, only those who subscribe to an extended support offering will retain access to patches for previous releases as new minor versions are launched.

How to Create a Patch

Kpatch allows users to create custom patches using kpatch-build tools. Comparing a kernel’s clean source-code tree with one that has specific code alterations needed for a particular repair, kpatch-build allows users to produce a module which features the differences between each of those source-code trees. This newly created module can then be loaded by kpatch.

Limitations of kpatch

kpatch is a distribution-specific tool, exclusively compatible with Red Hat’s CentOS/RHEL. As a result, businesses with distributions from several vendors must manage this technology separately. And since kpatch is a kernel-specific tool, it cannot be used for other components like shared libraries. 

Although a useful tool for simple patches, kpatch offers standard Red Hat users minimal support. Standard subscription users are provided limited access solely to the latest minor version release until the subsequent minor version is rolled out.

Per Red Hat’s own admission, kpatch is not able to resolve every crucial CVE and cannot make significant changes to the kernel. The kpatch feature merely applies basic security and bug fix updates when it is impossible to perform system reboots straight away. And so, kpatch undoubtedly fails to offer the support an all-purpose kernel upgrade mechanism would.

Furthermore, enterprises using kpatch must refrain from using debugging tools such as SystemTop or kprobe tools while a patch is loading, as well as after it has loaded. Doing so can result in unpredictable behavior after kpatch patches are loaded, rendering the debugging tools useless. The patch will be prevented from going into effect until such probes have been properly and entirely removed.

KernelCare has no such limitations. Offering support for every type of Linux available, including Red Hat and Oracle, KernelCare gives enterprises best-in-class live patching tools without requiring an expensive yet limiting subscription.

Conclusion

kpatch provides enterprises with a live patching mechanism dependent on kernel included functionality and sufficient for the application of quick, basic fixes without necessitating a reboot. And although kpatch’s reliance on the standard update mechanism for patch delivery makes it a user-friendly tool, it is also heavily limited by its restriction to Red Hat’s own distributions.

Alternatively, KernelCare offers much broader distribution support and can be used to patch system libraries, as well as the kernel. Unlike kpatch, KernelCare provides persistent patching. KernelCare successfully eliminates the need for additional patching software without requiring any system reboots for update applications, protecting your system from hackers.

To learn more about KernelCare and its features, and to review a side by side comparison of KernelCare and kpatch — as well as other live patching software — visit our website and get started using a more agnostic and affordable approach to your Linux kernel updates today!

Check out other overviews of live patching services:

Overview of Enterprise Live Patching services: Spotlight on Ksplice

Overview of Enterprise Live Patching services: Spotlight on Canonical Livepatch

Overview of Enterprise Live Patching services: Spotlight on Amazon Kernel Live Patching