Security Debt, or When Bugs Go Bad

Joao Correia

March 11, 2024 - Technical Evangelist

Occasionally, the IT world is gripped by a resurgence of concern – sometimes a fleeting trend, other times a significant issue. Lately, the term “Security Debt” has been making the rounds across IT news platforms. This article aims to unpack the essence of Security Debt and its implications.

From Technical to Security Debt

The concept of “Technical Debt” is familiar to many; it refers to the challenges that arise as projects expand beyond manageable proportions. This “size” could be measured in lines of code, modules, functions, or other metrics, reflecting a team’s capacity to simultaneously add new features and simultaneously maintain existing code.

For example, it is orders-of-magnitude more difficult to add new functionality to a mature application than to that same application when development is only starting. There are less moving parts, and less interactions between different parts of the code, which makes adding new features less likely to break the existing ones. As the application evolves, every change can bring about an unexpected interaction somewhere else in the code. That just means “you added a bug” in devspeak. Faced with limited time and resources, developers must juggle new development with bug fixes, slowing overall progress. This maintenance backlog constitutes “technical debt.”

Some – if not all – of those bugs, when left in the code for long enough, can turn into security problems. For example, an overflowing, or underflowing, variable somewhere may simply mean a report gets a bad value, or, if someone finds a way to trigger it, can lead to memory corruption on unrelated parts of the application. This is the “Security Debt” – that is, the security bugs that are in the code, sometimes for months or even years, until someone gets around to fixing them. Left unattended, these vulnerabilities may be exploited before they are rectified.

So Why Is It Making a Comeback?

The primary culprit is complexity. Code bases are only ever getting larger. You always see new features being added to software, never the other way around. That means more code gets added, not removed. Our human brains struggle to fully comprehend the intricacies of our digital creations.

Consider the Linux Kernel code base which grew from 2.4 million lines of code in 2001 to over 20 million in 2015. No one, or no team for that matter, is able to adequately grasp all of its aspects. So whenever someone adds a new line of code to it, it can work just fine or it can trigger a meltdown somewhere else. Tossing a coin would give you equal chances.

This isn’t exclusive to the Kernel, or even to projects of this magnitude, or even to open source projects at all. This complexity increase happens in closed source proprietary software just the same, we simply don’t have the code size metrics to show for it. If you look at binary sizes for a comparison, Windows 95 shipped on 13 floppy disks 1.68 MB (not a typo, those disks used a different track layout to fit more data, and it dual-purposed as a copy protection mechanism). Windows 11 ships as an approximately 5GB iso.

So yes, the code base increased.

It’s also not because developers suddenly got worse. The typical developer may be able to accurately understand a few dozen lines of code from memory. A good one about double or triple that (I am being intentionally vague, as this is difficult to gauge). For all, looking at code they didn’t see in a few days or weeks is like reading a foreign language for the first time, and the context switch takes time to do.

Understanding entire projects is no longer an expected prospect. The fact that we code bugs is a stark reminder of this fact.

Is That All?

The issue is compounded by “hidden complexity,” introduced with each added library or third-party component. A recent report highlighted that the average Java project had approximately 150 dependencies, a significant increase from the previous year. This complexity adds to the already challenging task of managing first-party code.

Addressing Security Debt

All this code inherently includes bugs, many of which are security problems directly. Handling them correctly and in a timely manner is difficult. How difficult depends on several factors:

How quickly they are detected, which in turn depends on the testing that happens in the development and build pipeline: If your security checks happen at the end of a long merge window with multiple devs committing code over an extended period of time, when the alerts are triggered, the effort of the original developer in addressing the problem is much higher, as he or she will no longer be able to recall the reason something was implemented the way that it was.
How comprehensive your testing is: Are you only testing for specific problems? Maybe static analysis is only looking for particular patterns, like wrongly typed variables, or buffer overflows, or NULL usage. Something else may slip unnoticed until it’s found in the wild.
The overall number of identified problems: If you have a large backlog of security problems to address, do you have the resources – time, know-how, manpower – to fix them all? Are you prioritizing? What if something happens elsewhere that makes a low priority issue suddenly become more important? Is your system flexible enough to react to that?

There are many aspects to Security Debt, and this article only looks at some core concerns. Its goal was to sound an alarm about the problem of technical debt turning into security debt when complexity grows past a developer or team’s ability to manage it properly.

There are some tactics to employ when addressing the issue: better security testing, earlier security testing, security testing before committing code (as opposed to only checking after everyone commits), peer review, outsourcing third-party library checks to a third party, or even hiring more developers as code bases grow. Alone, none of these suggestions will solve the problem entirely, but put together they might just hold the complexity at bay for a while longer and mitigate its impact.

Note: If you’d like to know more about the problem, the podcast “Enterprise Linux Security” has an episode, 84, on the subject. You can find it on all the podcast platforms or on YouTube with a live recording session.

Summary