What is Linux Kernel Live Patching?

November 23, 2022 - TuxCare PR Team

Breakthroughs don’t often happen in cybersecurity, but when one does, it can be a real magic bullet.

Linux kernel live patching, which is the ability to apply a Linux kernel security patch to a live, running Linux kernel without rebooting, is one of those incredible cybersecurity breakthroughs that truly changed the game in the fight against threat actors.

But what exactly is Linux kernel live patching, how does it benefit IT teams and the broader organization, and what is the future of live patching? Continue reading our comprehensive breakdown on live patching to find out.

Linux kernel patching is critical – and tough

A patch is a piece of code that fixes a known cybersecurity vulnerability. Whenever a vendor, such as Red Hat or Canonical, is informed of a cybersecurity vulnerability, the vendor develops a patch with modified code that “fixes” that vulnerability so that it cannot be exploited.

Once the code in the patch is integrated into the kernel, the operating system (OS) is “patched”. A patched OS is protected against any attempts to exploit the vulnerability.

Nonetheless, despite patching being a viable solution, known exploits are the entry door for many attacks because organizations do not always patch consistently. In fact, a 2021 Check Point research report suggested that 75% of attacks relied on vulnerabilities published in 2017 and earlier.

Had these vulnerabilities been patched, it would have closed the door to the attacks. So why do some organizations not always patch consistently?

The problem is that patching isn’t always straightforward. Usually, when you patch a service – whether an application, a driver, or the OS itself – you need to restart the service to apply the patch.

That, in turn, implies downtime until the service comes back online. And downtime is expensive on many levels.

A challenge needing a solution

Multiply the patching puzzle across thousands of machines and take into account the emergence of thousands of vulnerabilities every year and you’re quickly stuck with a tough problem, as stakeholders just won’t tolerate disruption and revenue loss every time a patch needs to be applied.

That, in turn, means patches are applied less frequently than the ideal. Organizations schedule downtime for patching at a pace of, say, once a month – if they’re doing a good job. That means there are significant window periods between when a vulnerability is identified in public, and when a patch is finally applied.

However, tech teams are commonly under-resourced, so the situation can quickly get worse, resulting in patches getting delayed for months and months. It’s no wonder then that cybercriminals have such success breaking into networks due to unpatched vulnerabilities: a 2019 study by the Ponemon Institute shows that 62 percent of respondents attributed a data breach to their organization’s failure to apply an available patch.

The history of Linux kernel live patching

Reboots and the associated Linux server downtime lead to ineffective patching. Take away the maintenance window requirement (and the resource drain caused by rebooting) and you solve a big chunk of the patching problem.

It was a researcher whose unpatched Linux server got hacked – Jeff Arnold, at MIT – who started research into live patching in 2006. What motivated him? Well, Arnold’s Linux server was hacked because he delayed a key security update, as he didn’t want to inconvenience other people. It was obvious to him that enterprise Linux users needed an alternative patching method.

Research at MIT was the first stepping stone into live kernel patching. Jeff and some of his colleagues founded a company called KSplice, Inc., which was acquired by Oracle in 2011. Other organizations followed when Red Hat and SUSE also led efforts to develop live patching for Linux servers with their respective solutions Kpatch and Kgraft, which were later combined into a single approach.

Moving to simple, affordable live patching everywhere

Linux kernel live patching, also known as dynamic kernel patching, is typically tied to a single enterprise Linux distribution and is often only available with expensive support contracts for Linux servers.

At TuxCare, we’ve independently developed the patented KernelCare live patching solution for almost a decade, supported by the team at CloudLinux. We made sure that live kernel patches can be applied across all the major enterprise Linux distributions, leading to a universal and affordable live patching service that’s not restricted to a single distribution such as Red Hat or SUSE Linux.

What’s more, KernelCare introduced persistent live patching. The original method for live patching, called temporary live patching, led to reduced performance over time because patches are applied using a technique that accumulates patches on top of one another.

In contrast, persistent live patching contains a cumulative fix in one binary, so there’s no performance drag. Thanks to several patented and patent-pending technological innovations, KernelCare delivers rebootless live patching with no performance hit.

And, as a next major step in live patching, we’ve now expanded the ability to live patch beyond the Linux kernel to also include live patching of shared libraries, database servers, and virtualization environments.

How Linux kernel live patching works in practice

Live patching is a simple premise, but there is a bit of complicated, technical gymnastics going on in the background.

Here’s a simplified explanation. The live patching service creates a patch from the corrected code so that the corrected code can be loaded into the running kernel memory. Code execution is then redirected away from the vulnerable version of the code to the corrected version of the code while the kernel is running.

It means that the kernel does not have to be restarted, which also means that the whole system avoids a reboot. Nonetheless, even though the system was never rebooted, the corrected version of the code is now used and the system is safe from the vulnerability.

The patch can be as complex or as simple as necessary. From a single one-line fix to multiple functions, the process is always the same. It’s a bit like magic, and live kernel patching has major practical implications for day-to-day, on-the-ground patching efforts.

Without live patching, system administrators must plan and schedule maintenance windows to take enterprise Linux servers offline. For large technology estates, it is a complicated process that is bound to upset someone, somewhere along the way, because nobody likes downtime. Yes, load balancing and redundancy can mitigate downtime, but it still leaves performance degradation as a side effect.

However, when you bring live patching software into the picture, you simplify security updates by running just a few command line steps in order to configure live patching. Servers and applications, such as databases, no longer need to go offline for patching. It means that system administrators do not need to worry about maintenance schedules and stakeholders never hear the words “downtime” or “disruption”.

As soon as a patch is released in response to a vulnerability, the patch is applied by the live patching technology. Thanks to the ability to apply a kernel live patch, there’s no waiting period at all and no interruption in the running service.

Live patching is critical for compliance

Because live patching is so responsive (remember, no maintenance schedules and no downtime) it means that patches are applied consistently and that there are no window periods where attackers can exploit a published vulnerability.

The consistent, gapless nature of live patching has an important implication in today’s highly regulated world, and live patching is a game-changer for organizations that have compliance obligations.

In fact, deploying security patches through a live patching mechanism is the fastest way currently available anywhere to secure a system against new threats.

For example, take the PCI DSS (Payment Card Industry Data Security Standard), which applies to organizations that process payments.

PCI DSS demands in Section 6.2 of its rules that critical security patches are applied within a month of release. Patch later than that and you fall foul of PCI DSS requirements – which can have significant consequences, including heavy fines, or worse, if a breach occurs. Standards such as ISO 27001 and the NIST Cyber Security Framework have similar requirements for patching.

Organizations that use kernel live patches reduce the risk that they break compliance rules to the minimum because patching no longer relies on manual intervention or unreliable and lengthy maintenance windows. Whatever compliance timeframe you’re required to meet is automatically achieved.

It makes a big difference for IT teams too

At the C-level, compliance really matters, but live patching has big, positive practical implications for tech teams as well. Because kernel updates happen automatically in the background, system administrators can spend their time doing higher-level tasks that contribute to overall security posture.

It implies happier stakeholders too, because maintenance windows have a way of upsetting everyone in the organization. Downtime means disrupted revenue streams, unhappy customers, and frustrated employees. These problems are all history thanks to live patching because live patching reduces the need for maintenance windows.

And there is, of course, the headline benefit of greatly improved security. Use live patching to minimize the patching window to just a couple of days (or even hours) and you minimize the opportunity for attackers to exploit a vulnerability and cause damage along the way.

Are there any negatives to Linux kernel live patching?

Live patching isn’t free, so there’s a cost factor to keep in mind when signing up for a live patching service. For Kpatch and other vendor-sponsored live patching services, the costs can be significant, as live patching is included in support plans. Consider the price tag of $1,299 per machine, per year for Red Hat Premium Support, for example.

That said, TuxCare’s KernelCare live security patching is affordable, at just $59.50 per year, per machine. It’s a small price to pay when you consider the time saved trying to organize maintenance windows and patch vulnerabilities manually. That’s not to mention the potential cost of a successful cyber-attack that occurred because you failed to patch.

While performance used to be a concern with temporary live patching, the emergence of KernelCare’s patented persistent live patching has removed the performance concern, as systems are simply patched while they’re running – and then continue running just as before, with no performance hit.

Conclusion

Live security patching is a major win for cybersecurity. It makes light work out of applying kernel patches to Linux enterprise servers, a task that’s typically a tough, time-consuming practice. What’s more, TuxCare’s live patching solution has now expanded to include a wide range of live patching services – covering libraries with LibCare and databases with DBCare.

TuxCare even covers you for virtualization, as we now offer QEMU live patching via our QEMUCare service. Every step of the way, we bring you closer to lighter workloads and full compliance.

If you run enterprise Linux workloads like Red Hat or Oracle, it is imperative that you consider live patching to see how it can benefit your cybersecurity posture. With easy integrations across tools including Nessus, Qualys, Rapid7, Puppet, Ansible, Chef, Datadog, and Crowdstrike, there’s simply no excuse. Read more about our range of live patching solutions here.

Summary