We need your knowledge! Complete TuxCare’s annual survey on Linux and open source to be entered to win one of many prizes (up to $500)! Get started here.
Struggling with maintenance windows, and worried about imperfect patching mechanisms and the impact on your Red Hat security ops? Live Linux kernel patching, also known as hot patching, is the answer to both challenges – but not every live kernel patching tool is created the same.
Most enterprise Linux vendors have rolled out live patching tools – including Red Hat’s tool for Red Hat Enterprise Linux (RHEL), called kpatch. Adopting live patching is now cybersecurity best practice and using the best tool for your needs is critical.
kpatch applies changes to the kernel while it is running and thereby mitigates the challenges around maintenance windows. It’s been supporting Red Hat-based server workloads since the earliest version of kpatch in 2014, but it has its pros and cons. Let’s take a look.
Content Table
Red Hat Kpatch | KernelCare Enterprise with LibCare add-on | |
---|---|---|
Supported distributions | Red Hat Enterprise Linux | Red Hat Enterprise Linux 6, 7, 8 and 9, as well as Ubuntu, Oracle, AlmaLinux and many others |
Architectures | x86-64 | x86-64, arm64 |
Coverage | Linux kernel | Linux kernel & critical userspace (glibc & openssl) |
Vulnerabilities patched | Subset of High & Critical | All |
Kernel patching lifetime | 6 months | Practically unlimited |
Custom patches | Yes | Yes |
QEMU patching | No | Yes |
24/7 support | No, 24×7 for severity 1 and 2 cases, otherwise standard business hours (for premium subscribers) | Yes, online, 24/7/365 with different priorities for different subscriptions |
Patchset distribution | Single patchset for all patches | Single patchset for all patches |
API available | No | Yes |
Roll-back functionality | Yes, with a reboot | Yes, rebootless |
Available for new clients | Only for RHEL | Yes, more than 60 distro versions supported |
Type of patching | Persistent | Persistent |
Add-ons | – | Custom patches, QEMU, Database patching |
Costs | Bundled with RedHat subscription at $879 per year, per machine. | $49.50 per year, per system. Different add-ons can be included in the subscription. Bulk pricing is available. |
Red Hat’s decision to introduce kpatch into the Linux kernel mainline came a little late, given that it was developed after KSplice – the MIT project – and after KernelCare, developed by the team at CloudLinux. Like its peers, kpatch performs live patching by hot swapping a replacement function containing a patched version of the kernel code.
In other words, kpatch allows you to apply security patches to your Red Hat-based workloads, without the need to immediately restart the server after patching. The tool was developed in-house by Red Hat, and at the outset it was similar to kGraft, a tool developed by the team over at SUSE Linux. Given how closely kGraft is related to kpatch the two organizations decided to work together and unify their efforts.
Red Hat’s kernel ceases to receive live kernel patches after a six-month period from its release. To receive ongoing kpatch updates, customers are required to upgrade the kernel and perform at least two reboots per year. Consequently, customers must align their maintenance windows with the vendor’s release schedule.
In comparison, KernelCare Enterprise offers live patches with a practically unlimited timeframe. This allows customers to enjoy continuous protection for their existing kernels without being bound by the vendor’s release schedule when planning their maintenance windows.
kpatch is an option for anyone who runs Red Hat Enterprise Linux (RHEL). It’s built and maintained by Red Hat developers, after all. However, anyone running RHEL 6, or certain versions of RHEL 7 won’t be covered for hot patches by kpatch. Only RHEL 8, 7.7, and 7.6 is supported.
Some non-RHEL Linux distributions, including specific versions of Ubuntu, Debian, and Gentoo, are technically also supported, but customers who wish to use kpatch with these distributions must manually create the necessary live patches using the tool themselves. It’s important to note that creating live patches most often isn’t an easy process and requires specific expertise. It can have some major pitfalls if you’re not careful.
If you’re running a version of RHEL, Ubuntu or Debian that’s not supported by kpatch, or indeed another Linux distribution such as CentOS or Amazon Linux, you’ll need to look towards KernelCare Enterprise for live patching support.
Another critical aspect to consider is how kpatch handles vulnerability coverage. It addresses only high and critical CVEs (Common Vulnerabilities and Exposures). However, Red Hat can adjust the original severity scores of these CVEs assigned by the National Vulnerability Database based on its proprietary criteria. This means kpatch may not address all vulnerabilities considered high and critical by external rating systems. Moreover, even among the vulnerabilities that Red Hat continues to classify as high and critical, their documentation specifies that not all are guaranteed to be patched using their live patching service. At the same time, some less critical vulnerabilities adjusted by Red Hat might still pose significant risks in environments with specific configurations and security needs.
Should a system administrator choose to, for whatever reason, KernelCare Enterprise allows any patch to be rolled back – a process that also doesn’t involve reboots. This can be particularly useful in situations where the patch has a considerable negative impact on a system’s performance (e.g., Spectre/Meltdown fixes) and there are other mitigations available. kpatch, on the other hand, does not support rebootless rollback functionality. This may cause costly service disruptions compromising the main benefit of live patching.
If a patch doesn’t work as expected, it’s nice to know you can easily revert to an older kernel if you need to. With KernelCare Enterprise, you can always roll back all applied changes by running a special command that does not require your system to be rebooted – removing the disruption out of a potential rollback, which kpatch cannot provide.
Incidentally, if you opted for KernelCare Enterprise, you’ll be looking at budget-friendly pricing that comes down to less than $50/server per year. With support for a broad range of Linux distributions, KernelCare offers a comparatively rich feature set too.
The cost to implement kpatch is significantly higher unless you’re already signed up for a RHEL support plan. That’s because kpatch is only available to Red Hat customers with a Premium Support plan, and that’s $879 per year, per machine.
If you’re a Red Hat Premium Support customer and exclusively use RHEL on your servers, you may want to consider kpatch, as it’s already included in your agreement with Red Hat.
Organizations that run a mix of Linux distributions or who do not need all the extra bells and whistles included in a Red Hat support contract should seriously consider KernelCare Enterprise as one of the alternative tools to kpatch, given the lower cost of KernelCare Enterprise and its broader feature set. Finally, if like many organizations, you simply cannot afford the reboots implied by kpatch’s limited kernel patching lifetime and rollbacks requiring unplanned reboots of your systems, then KernelCare is your best option.
Switching from the kpatch live kernel patching mechanism to KernelCare Enterprise is straightforward. All it takes is a simple quick start script to install KernelCare, and your Linux systems will enjoy ccontinuous live patching across the board.
Installing KernelCare Enterprise does not disrupt your existing workloads and you maintain the original functions of kpatch. Yet KernelCare does much more by completely minimizing disruption thanks to permanent patching which means you never need to reboot. If you’re still unsure, why not give it a try? KernelCare is available as a 30-day trial, with full functionality and no commitment to buy.
Tell us your challenges and our experts will help you find the best approach to address them with the TuxCare product line.