Fixing the security implications of open source technical debt

Major progress is usually made step by step – building capabilities, layer by layer. That’s the case for free and open-source (FOSS) software too, with today’s incredibly capable and very complex solutions emerging from countless building blocks, some of which are decades old.

That’s why much of the modern software we rely on commonly contains extremely old code that works reasonably well, if not perfectly. This code isn’t updated simply because it takes too much time to do so, and in the case of FOSS, this time is often contributed voluntarily.

It leads to what’s called technical debt – technical work that should have been done, but never was. This old, indebted code has drawbacks, one of which is security vulnerabilities that have proven time and time again to be extremely dangerous.

In this article, we discuss the issues of technical debt around open-source code including what exactly technical debt is, how it accumulates and why it is so difficult to address. We also cover why technical debt can harbor security dangers, why nobody is immune – and what can be done about it.

CONTENT
1- What is open-source technical debt?
– Defining technical debt
– Causes of technical debt
– Symptoms of technical debt
– What are the consequences?
2- No one is immune to the security impact of technical debt
3- Ways to address the issue
– Building awareness
– Fixing and preventing debt
– Vulnerability management
4- Current efforts
– OpenSSF
– SLSA
5- Change is needed – but deploy protective measures in the meantime

What is open-source technical debt?

We’ve hinted at the source of technical debt: old building blocks, consisting out of old code. But let’s take a closer look at exactly how technical debt is defined.

The term technical debt was first used by a well-known programmer called Ward Cunningham, one of the authors of the Agile Manifesto. According to Cunningham, technical debt is a lot like financial debt – you can borrow against the future, but at some point, you must pay back the debt.

With finances, incurring a little bit of debt to launch your business can help you build something faster. That’s just the same with technical debt – ignoring important development tasks can help you get to market quicker.

But, just like financial debt, the technical debt must be repaid at some point in time – or you’ll face the consequences. With financial debt, it’s interest, a claim on your assets – or bankruptcy. With technical debt, the consequences include poorly performing code, glitches – and security vulnerabilities.

Defining technical debt

 Let’s take a closer look at a formal definition of technical debt. According to Wikipedia, technical debt “is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer.”

That fits in with the financial analogy – technical teams decide to postpone hard work for a later, but eventually, the needed work mounts up, leading to a debt – that causes problems, including security risks.

Causes of technical debt

For commercial organizations, the development process involves a monetary outlay: developers must be paid in some shape or form whether it’s by the hour or via a salary. There’s typically a finite pot of money available for development, alongside pressure to complete projects. That leads to tough choices and often teams might decide to take shortcuts, incurring technical debt.

Looking at open-source software the reasons behind the accumulation of technical debt is a little different. FOSS owes its existence to a volunteer effort that’s usually unpaid. In other words, it’s up to volunteers to progress open-source projects, and it’s easy to see why contributors may end up working on projects in a way that moves projects forward quickly – but that also leads to an accumulation of technical debt.

A Linux Foundation study looking at the way in which FOSS is maintained found that contributors said that fixing security issues in old code is a “sole-withering chore”. There’s little motivation for open source developers to try and fix known problems, never mind to hunt through code to try and find possible weaknesses.

The result is that the effort to fix technical debt is simply never made and that projects continue to take shortcuts that grow the debt pile.

Symptoms of technical debt

So, when technical debt accumulates, what exactly are the symptoms? How does technical debt impact your application workloads? Whether commercial or open-source, here are some of the signs:

• Maintaining the code base gets harder and harder, as your developers are required to compensate for decisions made in the past by working with a complex, inflexible code base that contains legacy code.
• Finding developers becomes hard as you struggle to convince developers to join the team and improve the code simply because the code in the project is so old that today’s developers either can’t deal with it – or won’t deal with it.
• Increasing problems with compatibility and reliability because your software solutions contain components that are so far out of sync with modern software that making old and new work together is a challenge.
•  Security issues showing up more and more, which – while addressable – becomes costly to fix, and leaves your systems unnecessarily open to exploits.

You may spot these signs in both your custom, purpose-written software that you deploy for your unique applications – but also in the components this software relies on, and on supporting software including the operating systems you use.

What are the consequences?

Technical debt has a range of long-term consequences – while you will have saved development “money” at the initial stages, you will almost certainly repay it, and do so with interest.

For example, because your codebase is harder to maintain and because you find it more difficult to hire developers, your development projects will simply cost you more in the long run, and will take longer to complete. That counts for developments teams behind FOSS too. Technical debt will also reduce flexibility as you have fewer components and compatible systems to choose from.

From a security perspective, technical debt is particularly disconcerting. Coding errors made years and years ago can accumulate in the code base, leaving an unknown number of security holes in your solution – all representing opportunities for attackers.

No one is immune to the security impact of technical debt

Our last point referred to the security impacts of technical debt. How serious is this threat really? After all, technical debt is extremely commonplace. You should assume any software you rely on contains old, exploitable code. If this code is everywhere, surely the impacts wouldn’t be that serious – because the open-source community would have acted by now?

The reality is that the risks are extremely serious. Old code in the OpenSSL libraries, for example, was behind one of the biggest IT security threats of the last decade – the Heartbleed Bug. As it turns out, a mistake made decades ago was never fixed. Few people were looking at OpenSSL in any detail, yet countless websites relied on it – including major sites such as Yahoo.

The problem is that while the friendly guys are not doing anything about technical debt, the wrong guys are looking at old code, hunting for opportunities. The more of this technical debt is carried forward, the more places there are to hunt for vulnerable code.

Yes, technical debt creates security issues for FOSS – but it does so for commercial software as well. Technical debt can hide just about anywhere. Take, for example, the discovery in 2019 that many real-time operating systems that drive always-on devices were reliant on decades-old networking protocols with security flaws.

That said, given the precarious state of motivation behind open-source development, an argument could be constructed that enterprises that deploy open-source code should be particularly cautious about the technical debt that open-source code contains.

Notably, even where companies do not knowingly deploy open-source software (by relying on a Linux distribution, for example) there is a high probability that open-source code is contained even within commercial software solutions.

Ways to address the issue

Technical debt is a problem that’s been around since the computing revolution started. After all – programming resources have always been restricted either by budget or by motivation. Addressing it won’t be easy, but here we outline some steps that could help – and steps that could tide your organization over while the challenges are fixed.

Building awareness

Understanding the scope of the problem is the first step. As we explained, really old code can hide vulnerabilities that may well be found by the wrong people. In other words, understand that old code isn’t innocent code – though it may work, it could also harbor dangers. These old vulnerabilities can come to the foreground at an unexpected moment.

Also, consider the unique and relatively precarious position of FOSS. With no clear motivation to fix security issues, you may well find that there’s a unique aspect to the security risks posed by FOSS. And, even if you’re not explicitly relying on open-source code, beware that your commercial solutions may nonetheless contain open-source code. In other words, try to map out your debt to understand where you’re vulnerable.

Fixing and preventing debt

When it comes to dealing with technical debt in commercial solutions – well, if the code was written by your team, it’s your prerogative to manage the development process better or to fund your team in a way that ensures that technical debt doesn’t emerge.

A proactive development approach can reduce existing technical debt – and reduce future technical debt. It’s something worth considering given the obvious benefits to security. You’re somewhat stuck if you’re using an off-the-shelf solution that contains a lot of technical debt, however – and vulnerability management, discussed below, might be your better option.

Open-source technical debt is another matter. Given that, more often than not, no one has the ultimate responsibility to fix open-source technical debt, a more concerted effort is needed. In fact, a redraw of the map might be required.

Instead of fixing old code, developers could consider completely rewriting code portions or indeed entire components of open-source projects – particularly for those most prone to vulnerabilities, or where the project or component structure is so convoluted that it’s costly to maintain.

Finding a way to incentivize FOSS developers is key as the work might not get done otherwise. One way to do it would be to deliver a bounty-hunting program, not dissimilar to bug bounty programs – that’s indeed what the Linux Foundation suggests in its FOSS contributor survey.

For both commercial software and FOSS, better development practices can also help prevent the accumulation of technical debt. For example, improved pointer management and correct allocation of resources including improved clean-up after use. In the case of FOSS, a higher barrier for code quality could also help – making it harder for poor quality code to get accepted into open-source projects.

Vulnerability management

Efforts made to mitigate technical debt as it stands will never be able to remove all the security risks associated with technical debt – improving the current situation will take time. In the meantime, addressing these security risks will require a security-first approach.

Automation helps: running vulnerability scanners, for example, that scan your solutions for known problems can help you plug the security holes left by technical debt. Likewise, relying on automated patching solutions ensures that security vulnerabilities caused by technical debt are remediated pretty much as soon as the vulnerability becomes public.

Current efforts

There are a few initiatives that aim to tackle this issue specifically, and open source software security in general. They are industry-driven projects that introduce guidelines to development. More than simple best practices, they generally tend to create auditable environments, where a project can be checked for compliance with the guidelines set forth in each case.

OpenSSF

Under Linux Foundation guidance, OpenSSF – Open Source Security Foundation – is a cross-industry collaboration effort, bringing together the Core Infrastructure Initiative and the Open Source Security Coalition, as well as new working groups on vulnerability disclosures, security tooling, metrics and community efforts.

The introduction of SPDX, Software Package Data Exchange, and the SBOM, a Software Bill Of Materials, can be traced back to the OpenSSF work. This SBOM includes every dependency that a given project has, including specific version numbers and information about those dependencies’ SBOMs. This makes everything traceable and facilitates the discovery of dependencies that may not be obvious (for example libcurl’s inclusion into software that does not announce it anywhere).

SLSA

Somewhat complimentary to SBOM, Google’s SLSA (pronounced Salsa), “is a set of security guidelines being established by industry consensus” [quoted from the project’s description]. The end goal is the creation of auditable metadata that can be incorporated into the packages or build platforms and give a “SLSA certification”.

To think of it abstractly, it is a way to generate a “certificate” that a given project/application/package has not been tampered with at any stage of the development process, including through any linked/embedded library that is used by it. This provides a “Supply Chain Integrity” check mechanism.

The SLSA effort is part of OpenSSF’s Digital Identity Attestation Working Group, and you can follow the discussion there.

Change is needed – but deploy protective measures in the meantime

There’s no easy way to fix the issues behind technical debt. Many of our suggestions will make life significantly harder for developers, going as far as to raise the barrier to entry: both into the developer world, and for the progression and release of new, inventive projects.

But, with a daily stream of CVE announcements, it’s obvious something needs to happen. A set of standards could be a way forward – where open-source projects could, for example, adhere to a standard the way a source license is attached to a project. And this is an effort that is best undertaken when the project is in its early stages, so developers looking to start their great new idea should consider adhering to guidelines like the ones established under OpenSSF or Google’s SLSA.

While these initiatives have a high level view of the projects and the security aspects, they fall short of recognizing individual developers’ efforts. So far, it is impossible to distinguish the efforts of a developer who takes the, arguably, harder task of going back through old code refactoring code smells and fixing bugs, from another who adds some new feature to a project. They are obviously both important, but the proportion of these two categories of developers leans heavily towards the second one – while the first one is the one keeping the code safe from vulnerabilities.

On the other hand, adding administrative work or setting the admission bar so high that newcomers will be afraid to contribute is also detrimental to the success of open source projects, so there is a need to carefully balance the matter.

Either way, awareness is a start – and until the risk of open-source technical debt is better addressed, vulnerability management will remain a critical component of keeping your organization safe.