Developer Tutorial: Live Patching Oracle Enterprise Linux 7 with Kpatch

Live patching allows you to update Linux kernel without rebooting your system. This approach is definitely a perfect solution for huge enterprises with interruption-sensitive services when pauses can cause toxic effects to the whole business. But kernel update is a must in today’s world full of vulnerabilities, security issues and gaps. Only a forehanded kernel update can guarantee trouble-free smooth operation of your business.

You might have already read our live kernel patching tutorials: Live patching Debian 10 Linux kernel with Kpatch and Live patching Ubuntu 20.04 LTS Focal Fossa Linux kernel with Kpatch. And if not, consider them as useful materials to share with your colleagues and friends.

In this tutorial, I’ll show you how to patch Oracle Enterprise Linux 7 – UEK4 Linux kernel using Kpatch without rebooting it.

Kpatch is a Red Hat project and works on RHEL and its derivatives. Red Hat offers a commercial live patch service for RHEL customers, as do the following companies, who each focus on different distributions:

• Canonical Livepatch Service for Ubuntu;
• KernelCare for most major Linux distributions;
• Ksplice for Oracle Linux;
• SUSE Live Patching (previously known as Kgraft) for SUSE Enterprise Linux.

We have chosen Kpatch for this tutorial because its source code is available for free and is regularly updated.

Prerequisites

Here are the system prerequisites for following this tutorial.

• A test (non-production) system running Oracle7 UEK4 on an x86_64/amd64 architecture.
• 20 Gb of free disk space. (The Linux kernel source code takes up around 909 Mb on disk, growing to 17 Gb when compiled.)
• Your kernel has not been customized; you are using the standard one supplied by Debian
• Your kernel has live patching built in. Use this command and expect to see two values set to y for CONFIG_HAVE_LIVEPATCH and CONFIG_LIVEPATCH:

grep LIVEPATCH /boot/config-$(uname -r)

• Version of gcc installed matches that used to build the original kernel. (The kpatch-build command will fail if the versions don’t match. This can be overridden with the option --skip-gcc-check, although use of it is discouraged.)
  • To see the version of gcc installed: gcc --version (if gcc not installed run ”sudo apt install gcc”)
  • To see the version of gcc used to compile the current kernel: cat /proc/version

1. Install dependency packages

1. Install and configure sudo.

   As root:

   apt-get install sudo
   adduser <user> sudo

   where <user> is the username for a normal user. (All subsequent commands should be done as this user.)

2. Install packages.

   sudo yum install elfutils elfutils-devel
   sudo yum install pesign yum-utils zlib-devel binutils-devel newt-devel python-devel 
   perl-ExtUtils-Embed audit-libs numactl-devel pciutils-devel bison patchutils bc rpm-build glib2-devel

3. Download and install libdtrace:

   wget https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/getPackage/libdtrace-ctf-0.8.0-1.el7.x86_64.rpm
   wget https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/getPackage/libdtrace-ctf-devel-0.8.0-1.el7.x86_64.rpm
   rpm -ivh libdtrace-ctf-0.8.0-1.el7.x86_64.rpm libdtrace-ctf-devel-0.8.0-1.el7.x86_64.rpm

4. Download and install kernel sources and kernel-devel packages:

   wget https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/getPackageSource/kernel-uek-4.1.12-124.40.6.3.el7uek.src.rpm
   wget https://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/getPackage/kernel-uek-devel-4.1.12-124.40.6.3.el7uek.x86_64.rpm
   rpm -ivh kernel-uek-4.1.12-124.40.6.3.el7uek.x86_64.rpm --nodeps
   rpm -ivh --nodeps kernel-uek-devel-4.1.12-124.40.6.3.el7uek.x86_64.rpm

5. Download and install debuginfo packages:

wget https://oss.oracle.com/ol7/debuginfo/kernel-uek-debuginfo-4.1.12-124.40.6.3.el7uek.x86_64.rpm
wget https://oss.oracle.com/ol7/debuginfo/kernel-uek-debuginfo-common-4.1.12-124.40.6.3.el7uek.x86_64.rpm
rpm -ivh kernel-uek-debuginfo-4.1.12-124.40.6.3.el7uek.x86_64.rpm kernel-uek-debuginfo-common-4.1.12-124.40.6.3.el7uek.x86_64.rpm

2. Install Kpatch

git clone https://github.com/dynup/kpatch.git
cd kpatch && make && sudo make install
cd kmod && make
mkdir /lib/modules/`uname -r`/extra/kpatch
cp core/kpatch.ko /lib/modules/`uname -r`/extra/kpatch
cp core/Module.symvers /lib/modules/`uname -r`/extra/kpatch

3. Get a copy of the Linux kernel source code

Unpack kernel sources:

cd /root/rpmbuild/SOURCES
tar xfj linux-4.1.12.tar.bz2

4. Create the Linux kernel configuration file

The Linux kernel is compiled using settings in a configuration file supplied with your distribution. Take a copy and change some settings so kpatch-build can compile a Linux kernel with the same settings as your running kernel.

cd linux-source-5.4.0/
cp /boot/config-$(uname -r) .config

Check that the required kernel settings are enabled for using Kpatch. All should return ‘y’.

scripts/config -s DYNAMIC_FTRACE_WITH_REGS
scripts/config -s FUNCTION_TRACER
scripts/config -s HAVE_DYNAMIC_FTRACE_WITH_REGS
scripts/config -s HAVE_FENTRY
scripts/config -s HAVE_LIVEPATCH
scripts/config -s KALLSYMS_ALL
scripts/config -s KALLSYMS
scripts/config -s LIVEPATCH
scripts/config -s MODULES
scripts/config -s MODULE_SIG
scripts/config -s SYSFS
scripts/config -s SYSTEM_TRUSTED_KEYRING

Change the value of one kernel configuration item.

scripts/config --set-str SYSTEM_TRUSTED_KEYS ""

Leave the source directory.

cd ..

5. Create a patch

A patch source file is the output from the diff command run on the original and the changed source code files.

The patching example shown in the ‘Quick start’ section of the kpatch github page changes the output of /proc/meminfo. Many other Kpatch articles reproduce this example, so I wanted something different and a little more interesting, yet still safe.

This example changes the output of the uptime command to give them illusion that your server’s uptime has increased by a decade.

1. Still in your working directory, copy a file.

cd /root/rpmbuild/SOURCES
cp linux-4.1.12/fs/proc/uptime.c .

2. Edit it. At line 26, change:

(unsigned long) uptime.tv_sec,

to

(unsigned long) uptime.tv_sec + 315576000,

Save the file.

3. Create the patch file.

diff -u linux-source-5.4.0/fs/proc/uptime.c ./uptime.c > uptime.patch

4. Create the patch module. (The first time you do this, it will take some hours, as the kernel source code must be compiled. Subsequent builds are significantly faster, in the order of minutes.)

kpatch-build -t vmlinux -v /usr/lib/debug/boot/vmlinux-5.4.0-37-generic uptime.patch

5. When done, you will have a Linux Loadable Kernel Module file kpatch-uptime.ko for the patch.

6. Test the patch

1. Before loading the patch module, check the current uptime.

   cat /proc/uptime && uptime -p

2. Load the patch module.

   # kpatch load kpatch-uptime.ko
   loading patch module: kpatch-uptime.ko

3. Check the uptime again.

   cat /proc/uptime && uptime -p

4. Unload the patch module.

   # kpatch unload kpatch-uptime.ko
   disabling patch module: kpatch_uptime
   unloading patch module: kpatch_uptime

5. Check the uptime has returned to its former value.

   # cat /proc/uptime && uptime -p
   7215.13 27930.41
   up 2 hours

Conclusion

We have worked through the Oracle Enterprise Linux 7 – UEK4 Linux kernel live patching process with Kpatch. You can see that the process itself is quite simple but it is not easy. And a patch development presents the main difficulty. The patch should not break the system down, it should be compatible with the previous and next packages. Add here different kernel distributions and versions, testing and support, and you can end up with a notable price for a live patching tool.

Get a FREE 7-Day Supported Trial of KernelCare 

You can explore three different ways of applying Linux kernel security patches in our explicit guide. The article explains how to update Linux kernels without rebooting, covering three different methods for some of the most popular Linux kernels.