Struggling with maintenance windows, and worried about imperfect patching mechanisms and the impact on your Red Hat security ops? Live Linux kernel patching, also known as hot patching, is the answer to both challenges – but not every live kernel patching tool is created the same.
Most enterprise Linux vendors have rolled out live patching tools – including Red Hat’s tool for Red Hat Enterprise Linux (RHEL), called Kpatch. Adopting live patching is now cybersecurity best practice and using the best tool for your needs is critical.
Kpatch applies changes to the kernel while it is running and thereby mitigates the challenges around maintenance windows. It’s been supporting Red Hat-based server workloads since the earliest version of Kpatch in 2014, but it has its pros and cons. Let’s take a look.
Red Hat’s decision to introduce Kpatch into the Linux kernel mainline came a little late, given that it was developed after KSplice – the MIT project – and after KernelCare, developed by the team at CloudLinux. Like its peers, Kpatch performs live patching by hot swapping a replacement function containing a patched version of the kernel code.
In other words, Kpatch allows you to apply security patches to your Red Hat-based workloads, without the need to immediately restart the server after patching. The tool was developed in-house by Red Hat, and at the outset it was similar to kGraft, a tool developed by the team over at SUSE Linux. Given how closely kGraft is related to Kpatch the two organizations decided to work together and unify their efforts.
Here’s a key point to understand about Kpatch dynamic kernel patching. When performing live patching or as it used to be called, dynamic kernel patching, there are two routes: temporary patching, and persistent patching. Kpatch relies on temporary live patching which is exactly what it says, a temporary way to install critical patches on a running workload without a reboot.
However, temporary patches are not fully integrated, and relying on ongoing temporary patching will eventually degrade performance – until a reboot of the running kernel is performed. Yes, it helps to mitigate the challenges around patching, but relying on temporary patches to kernel modules is not a perfect cure.
A better way to go about patching is persistent patching, where every kernel live patch contains a cumulative fix in one binary. Patch modules are not applied on top of each another, there’s no performance degradation and no need for system reboots. And, when it comes to patching, it can be argued that eliminating the reboots required to update the kernel version is really the primary goal.
Kpatch is an option for anyone that runs Red Hat Enterprise Linux (RHEL). It’s built and maintained by Red Hat developers, after all. However, anyone running RHEL 6, or certain versions of RHEL 7 won’t be covered for hot patches by Kpatch. Only RHEL 8, 7.7, and 7.6 is supported. Some non-RHEL Linux distributions are also supported, including specific versions of Ubuntu, Debian, and Gentoo.
If you’re running a version of RHEL, Ubuntu or Debian that’s not supported by Kpatch, or indeed another Linux distribution such as CentOS or Amazon Linux, you’ll need to look towards KernelCare for live patching support.
Incidentally, if you opted for KernelCare Enterprise, you’ll be looking at budget-friendly pricing that comes down to less than $50/server per year. With persistent patching and support for a broad range of Linux distributions, KernelCare offers a comparatively rich feature set too.
The cost to implement Kpatch is significantly higher unless you’re already signed up for a RHEL support plan. That’s because Kpatch is only available to Red Hat customers with a Premium Support plan, and that’s $1,299 per year, per machine.
|Red Hat Kpatch||KernelCare Enterprise Live Patching|
|Supported distributions||RHEL 8, 7.7, 7.6, Ubuntu, Debian, Gentoo||Amazon Linux 1 & 2, Debian 7, 8, 9 & 10, CentOS 6, 6 Plus, 7, 7 Plus, Ubuntu 14.04 LTS, 16.04 LTS, 18.04 LTS & 20.04 LTS, CloudLinux OS 6, 6 Hybrid, 7 & 8, RHEL 6, 7 & 8, OpenVZ & Virtuozzo, Proxmox VE 2.x, 3.x & 4.x, Xen4CentOS 6 & 7, Oracle UEK 3, 4 & 5, Oracle Enterprise Linux 6, 7 & 8, Yocto, Ubuntu Core|
|Supported kernels older than 3.10||No||Yes|
|Userspace patching||No||Yes (OpenSSL & Glibc)|
|SAP applications patching||Yes||No|
|24/7 support||No, 24×7 for severity 1 and 2 cases, otherwise standard business hours (for premium subscribers)||Yes, online and telephone, 24/7/365 with different priorities for different subscriptions|
|Patchset distribution||No distribution channel, patches are separate||Single patchset for all patches|
|Release timing||None provided||Before or shortly after base distribution|
|Roll-back functionality||Yes||Yes, rebootless|
|Works behind a firewall||No||Yes|
|Available for new clients||Only for RHEL||Yes, more than 40 distributions supported|
|Type of patching||Temporary||Persistent|
|Costs||High cost, Kpatch is only available if you opt for the RedHat premium subscription at $1299 per year||From $3.95 per month per server (under $50 per year, per system). Different add-ons can be included in the subscription. Bulk pricing is available.|
Switching from the Kpatch live kernel patching mechanism to KernelCare Enterprise is straightforward. All it takes is a simple quick start script to install KernelCare, and your Linux systems will enjoy continuous, permanent patching across the board.
Installing KernelCare Enterprise does not disrupt your existing workloads and you maintain the original functions of Kpatch. Yet KernelCare does much more by completely minimizing disruption thanks to permanent patching which means you never need to reboot.
If you’re a Red Hat premium support customer and exclusively use RHEL on your servers, and if temporary kernel patching does not lead to excessive resource drain or downtime, you may want to consider Kpatch, as it’s already included in your agreement with Red Hat.
Organizations that run a mix of Linux distributions or who do not need all the extra bells and whistles included in a Red Hat support contract should seriously consider KernelCare as one of the alternative tools to Kpatch, given the lower cost of KernelCare Enterprise and its broader feature set. Finally, if like many organizations, you simply cannot afford the reboots implied by Kpatch’s temporary live patching method then KernelCare is your best option.
Tell us your challenges and our experts will help you find the best approach to address them with the TuxCare product line.