An update on “Retbleed” work (Updated Dec 21, 2022)

Joao Correia

December 22, 2022 - Technical Evangelist

As part of developing and testing new patches, the KernelCare team has reevaluated the impact of the Retbleed patches. We have serious concerns that the patches may introduce instability and performance degradation when applied. As such, we will not be offering any patches for this specific, and exceptional, situation. Our assessment is that the risks outweigh the benefits of live patching this highly unusual case, so we are recommending if you must remediate it you should apply the latest regular disk updates then reboot your instances to obtain a fresh kernel.
The same applies to Extended Lifecycle Support patches.

The Vulnerability

Retbleed is a hardware-level vulnerability conceptually similar to Spectre V2. It is a speculative execution attack targeting predictive branch functionality found in modern CPUs. A local unprivileged user can exploit this vulnerability to access otherwise out-of-reach memory locations.

Cloud virtual machine environments, or multi-tenant hypervisors, are especially vulnerable, as one compromised virtual machine could access memory in use by other virtual machines in the same virtualization host.

There are three different CVEs relevant to this issue (CVE-2022-23816, CVE-2022-29901, CVE-2022-23825). Some advisories may also refer to CVE-2022-29900, which is a duplicate of CVE-2022-23816.

Mitigations

Existing Indirect Branch Restricted Speculation (IBRS) on Intel CPUs mitigates the issue for Enterprise Linux 7 based distributions.

If you can afford the reboot, then, for Enterprise Linux 8 based distributions, enabling spectre_v2 mitigations with:

spectre_v2=ibrs

as a kernel parameter on boot will also mitigate the problem on Intel CPUs.

Performance impact

Mitigations for previous speculative execution-based vulnerabilities like Spectre and Meltdown introduced considerable performance impacts, ranging from 5% to 30% depending on the specific workload of the system and the brand of the CPU.

Retbleed, unfortunately, continues this trend and the mitigations for it also have a severe performance impact. On average, the performance cost varies from 14% to 39%. In some scenarios, the performance regression was as much as 70%, impacting not just raw CPU performance, but I/O-bound operations as well (storage and networking performance both suffer).

TuxCare’s work on patches

Retbleed patches are being worked on by both the KernelCare Enterprise live patching and Extended Lifecycle Support teams, as it impacts not just recent Linux distributions, but older ones as well.

The main challenges of this patch come from the scope – it impacts a vast majority of the kernel code – and the complexity of the patched code. We already have working live patching fixes and are working on the extensive tests that something of this scope requires to ensure patches that not only fix the underlying issue, but that minimize the performance impact at the same time while avoiding the addition of unexpected behaviors to running systems.

To see the up-to-date status of the patches for our products, check out the following links:

Conclusion

There are mitigations in place that will prevent the problem for most kernels, so if you have the ability to perform a reboot, it is a possibility. Please note that, to exploit the problem, an attacker would need to have local system access, so not all systems are equally at risk. If you have systems that you cannot afford to reboot, KernelCare live patches are in progress and will be available soon.

Summary