Owner-initiated Cybersecurity Supply Chain Attacks Posted on September 23, 2022September 23, 2022 by Linux Live Patching Expert Team Supply chain attacks come in all forms and shapes. One example is taking over legitimate accounts to deploy malicious code into widely used libraries. Another is deploying changes during compilation through malicious tooling. We could carry on – think about mislabeled content intent on tricking unwary users, or additional files added that subvert application behavior. But, another particularly malicious supply chain attack perpetrated by the actual code owner or the original developer behind the code is emerging – and it’s proving to be incredibly disruptive. A new, dangerous supply chain threat This attack vector emerged early in 2022 when developers realized that a couple of very widely used libraries (“faker.js” and “colors.js”) had been tampered with to include code that was unrelated to the original intent of these libraries. Worse, the “additional” code was breaking the applications relying on it by creating an infinite loop in the code. Because the code for these libraries is open and hosted in public repositories it was possible to analyze the commits and identify who had added that malicious code. The developer community was disconcerted to find that the code was added by the original developer. Apparently unhappy with not being able to monetize the time he expended in developing the code, and how it was being used by so many developers, the developer decided to sabotage his own libraries. Sabotage that impacts millions It might sound like it’s nothing – it’s the developer’s code, after all. But to put it all into context, the affected libraries had an aggregate of 25 million downloads… not across their useful lifetime, but every single week. The politics of open source and how open source is monetized – including the steps a developer can take to address it – is really an entire article in and of itself, and opinions can vary a lot. But, for obvious reasons, the backlash around a developer tampering with code used by millions was immense. In fact, GitHub, where the code is hosted, suspended the developer’s account (though GitHub restored it a few days after). The developer claimed to have hundreds of projects, not only in those two libraries but of course, the two libraries in question have an immense following. Malicious action with real impact on developers The libraries ended up being blocked from distribution, the code was reverted, and new libraries were created to replace the functionality. What this meant for developers using those libraries was the extra work of checking code, updating it, and changing dependencies. One individual’s actions led to extra testing and all the other tasks that are necessary to ensure proper software execution. Countless work hours were spent on responding to a malicious actor, hours that could have been more productively used elsewhere. It was deliberate action by a single developer that affected thousands of applications. But there are other situations where it’s not deliberate, but it nonetheless comes from the source, the developer. Or, at least, the developer’s account. A compromised git server Just a couple of months after that, in March, the git server holding the code for the most popular web server programming language, PHP, was compromised. Yes, you read the right, the PHP Team’s official PHP git repository was tampered with. Attackers managed to change the source code for PHP 8.1 and introduce malicious commits into the system under the guise of “typo fixes”, which hid malicious code that, if successfully integrated into the PHP codebase, would disseminate to millions of PHP deployments and create a backdoor in millions of internet-facing websites and services written in PHP. How did it happen? Git is a great tool, but it has a flaw where it is possible to sign code commits as someone else. Another person reviewing the code edits may then be fooled into thinking that the edits were made by, say, the main project developer. The result is that any review could rely heavily on trust – the trust that the right person edited the code and made acceptable changes. Reviews can therefore be less stringent, allowing malicious code to slip under the radar undetected. And it’s not just PHP… The malicious code added to PHP 8.1 was detected and the commits were rejected, so the code never made it into the PHP codebase millions use every day. Yet the problem it created led the team to move from its own git repository to GitHub. In May, Python’s ctx library and PHP’s phpass library were both compromised in an operation that the attacker claimed was simply “for research purposes”, a claim that was ultimately impossible to prove or disprove. In a convoluted series of steps, the attacker took over the repositories for these libraries by impersonating the original developers – taking over email addresses that no longer existed, then requesting password recovery for those accounts, and creating fake domains to validate ownership of expired ones. This tricked the repository hosts into giving repository administration to the attacker, who was posing as the original developer who had lost access to his own code. The attacker proceeded to include credential-stealing code in those libraries, including AWS credentials. The attacker’s claims that all the gathered credentials were deleted were also impossible to verify. Threat actors are investing heavily in supply chain attacks These examples shine a light on the need for a quick response to threats in the code supply chain. The threat to code supply chains is not new in and of itself. However, the attacks we’re seeing lately are different because the time invested goes much beyond the usual attack turnaround time of days or weeks. The latest examples of compromises involve a huge amount of time to execute the compromise. In many ways it is a much more serious example of a supply chain attack, striking at the heart of code that is – blindly – used by millions of organizations around the world. In another point that makes the matter more serious: it’s worth considering the large amount of work required to change library versions to respond to a compromise. New libraries take a considerable amount of time to execute, test and deploy each time such an event takes place. This is where a service like Extended Lifecycle Support for PHP and Extended Lifecycle for Python can help. TuxCare quickly provides security updates for language level issues and for related modules, which help close holes created by some of the attacks we described faster and with less work than the alternatives.