ClickCease Developer Tutorial: Live patching Debian 10 Linux kernel-Kpatch

Join Our Popular Newsletter

Join 4,500+ Linux & Open Source Professionals!

2x a month. No spam.

Developer Tutorial: Live patching Debian 10 Linux kernel with Kpatch

by

January 27, 2023 - TuxCare PR Team

Live patching is a method of updating a Linux kernel without restarting the kernel – and therefore without the need to reboot the machine. Live patching is often used to patch severe Linux kernel vulnerabilities without delay, because live patching doesn’t cause disruption and doesn’t require a maintenance window.

There are a few live patching tools out there and users of the Debian operating system sometimes turn to kpatch to implement live patching – or to tweak the kernel.

This tutorial is a practical demonstration of kpatch. We will demonstrate how to use kpatch to change the behavior of a running Debian 10 kernel without stopping it, changing the contents of /proc/uptime (and the uptime command) so that the system’s reported uptime is 10 years greater.

kpatch was created by Red Hat and works on Red Hat Enterprise Linux (RHEL) and its derivatives. Red Hat offers this commercial live patching service for RHEL customers. At TuxCare, we offer KernelCare Enterprise – which is able to live patch RHEL as well as every other popular enterprise Linux distribution.

I have chosen kpatch for this tutorial because its source code is freely available and regularly updated.

Prerequisites

Here are the system prerequisites for following this tutorial:

  • A test (non-production) system running Debian Bullseye (11.6 was used for this demo) on an x86_64/amd64 architecture
  • 20 Gb of free disk space (the Linux kernel source code takes up around 909 mb on disk, growing to 17 gb when compiled)
  • Your kernel has not been customized; you are using the standard kernel supplied by Debian
  • Your kernel has live patching built in. Use this command and expect to see two values set to y for CONFIG_HAVE_LIVEPATCH and CONFIG_LIVEPATCH:
grep LIVEPATCH /boot/config-$(uname -r)
  • Version of gcc installed matches that used to build the original kernel (The kpatch-build command will fail if the versions don’t match. This can be overridden with the option –skip-gcc-check, although use of it is discouraged.)
    • To see the version of gcc installed: gcc –version
    • To see the version of gcc used to compile the current kernel: cat /proc/version
  1. Install Dependency Packages

  1. Install and configure sudo

      As root:

  apt-get install sudo 

  (sudo should already be installed, but this makes sure of it)

  adduser <user> sudo
  where <user> is the username for a normal user (all subsequent commands should be done as this user)

     2.Install packages

  3.sudo apt-get -y update

  4.sudo apt-get -y upgrade

  5.sudo apt-get -y install build-essential devscripts ccache gawk libelf-dev libssl-dev    linux-source linux-image-$(uname -r)-dbg 

  1. Install kpatch

As is common with Debian’s packages, there is a delay between the current versions and the versions offered as updates through regular apt. Let’s take the most current kpatch code and build it from source.

git clone https://github.com/dynup/kpatch.git
cd kpatch && make && sudo make install
  1. Get a Copy of the Linux Kernel Source Code

  1. (Optional) Make and move into a working directory
mkdir kernel && cd $_
  1. Extract the Linux kernel source code
tar xaf /usr/src/linux-source-5.10.tar.xz

Note: 5.10 is the Linux kernel version for Debian 11.6 at the time of writing. You should check and substitute the most recent version present in /usr/src

  1. Create The Linux Kernel Configuration File

  1. The Linux kernel is compiled using settings in a configuration file supplied with your distribution. Take a copy and change some settings so kpatch-build can compile a Linux kernel with the same settings as your running kernel.
  2. cd linux-source-5.10

cp /boot/config-$(uname -r) .config

  1. Check that the required kernel settings are enabled for using kpatch. All should return ‘y’.
  2. scripts/config -s DYNAMIC_FTRACE_WITH_REGS
  3. scripts/config -s FUNCTION_TRACER
  4. scripts/config -s HAVE_DYNAMIC_FTRACE_WITH_REGS
  5. scripts/config -s HAVE_FENTRY
  6. scripts/config -s HAVE_LIVEPATCH
  7. scripts/config -s KALLSYMS_ALL
  8. scripts/config -s KALLSYMS_ALL
  9. scripts/config -s LIVEPATCH
  10. scripts/config -s MODULES
  11. scripts/config -s MODULE_SIG
  12. scripts/config -s SYSFS
scripts/config -s SYSTEM_TRUSTED_KEYRING
  1. Change the value of one kernel configuration item
scripts/config --set-str SYSTEM_TRUSTED_KEYS ""
  1. Leave the source directory
cd .
  1. Create a Patch

A patch source file is the output from the diff command run on the original and the changed source code files.

The patching example shown in the ‘Quick start’ section of the kpatch github page changes the output of /proc/meminfo. Many other kpatch articles reproduce this example, so I wanted something different and a little more interesting, yet still safe.

This example changes the output of the uptime command to give the illusion that your server’s uptime has increased by a decade.

  1. Still in your working directory, copy a file
cp linux-source-4.19/fs/proc/uptime.c .
  1. Edit it. On line 33, change:
(unsigned long) uptime.tv_sec,

to

(unsigned long) uptime.tv_sec + 315576000,

Save the file
  1. Create the patch file
diff -u linux-source-5.10/fs/proc/uptime.c ./uptime.c > uptime.patch
  1. Create the patch module. (The first time you do this, it will take some time, as the kernel source code must be compiled. Subsequent builds are significantly faster, in the order of minutes. This is one of those operations where throwing more (virtual) CPUs at the problem will help speed the process along. It is a CPU-bound task).
kpatch-build -s linux-source-5.10 -v /usr/lib/debug/lib/modules/$(uname -r)/vmlinux uptime.patch
  1. When done, you will have a Linux Loadable Kernel Module file (.ko) for the patch
ls -l *.ko
  1. Test the Patch

  1. Before loading the patch module, check the current uptime
cat /proc/uptime && uptime -p
  1. Load the patch module
sudo kpatch load livepatch-uptime.ko
  1. Check the uptime again
cat /proc/uptime && uptime -p

You should see your uptime is ten years better (the internal value hasn’t been changed, only what is printed)

  1. Unload the patch module
sudo kpatch unload livepatch-uptime.ko
  1. Check that the uptime has returned to its former value
cat /proc/uptime && uptime -p

Conclusion

Live patching the Linux kernel with kpatch isn’t hard. The difficulty lies in writing a patch that won’t crash the system and that will work with other patches that come later. Most patches are created from more than a simple diff and need thorough testing across multiple kernel versions running on various distributions.

Patch writers need to be both accomplished C programmers and experienced Linux kernel developers. Compiling the kernel and testing patches for every kernel version requires a large investment in hardware and automation tools. 

The rarity of skills and the cost of infrastructure means that vendors must charge for live patching services, which are often provided through expensive maintenance programs.

These charges add up to large sums, but that’s not the case with TuxCare’s live patching service – which starts from just $49.50/server/year for Linux. Click here to read more about TuxCare’s range of live patching services for the Linux kernel, databases, shared libraries, IoT devices, and virtualization systems.

And don’t forget to read our explicit guide on How to Apply Linux Kernel Security Patches: 3 Different Ways. The article explains how to update Linux kernels without rebooting, covering three different methods for some of the most popular Linux kernels.

Summary
Developer Tutorial: Live patching Debian 10 Linux kernel-Kpatch
Article Name
Developer Tutorial: Live patching Debian 10 Linux kernel-Kpatch
Description
This tutorial is a practical demonstration of kpatch. We will demonstrate how to use it to change the behavior of a running Debian 10 kernel
Author
Publisher Name
TuxCare
Publisher Logo

Looking to automate vulnerability patching without kernel reboots, system downtime, or scheduled maintenance windows?

Become a TuxCare Guest Writer

Mail

Help Us Understand
the Linux Landscape!

Complete our survey on the state of Open Source and you could win one of several prizes, with the top prize valued at $500!

Your expertise is needed to shape the future of Enterprise Linux!