Large Scale Security Incidents in Open Source
Version: 0.9, April 10, 2024
License: Creative Commons Lizenz CC BY-SA 4.0
Definition
An Large Scale Security Incident is a security incident affecting many projects and possible production sites caused by a bug in an Open Source project. They have in common to have a huge blast radius endangering a lot of sites or nearly the entire internet. Examples are Heartbleed, Log4Shell and the new XZ backdoor.
Heartbleed
The Heartbleed bug affected the OpenSSL library leaking secret keys to an attacker. The blast radius was more or less the entire internet.
Log4Shell
Log4Shell is a zero-day vulnerability in Log4j, a popular Java logging framework, involving arbitrary code execution. The blast radius was nearly every Java server application.
XZ backdoor
The XZ backdoor only targeted Debian- and RPM-based systems on x86-64 architecture. The backdoor was discovered before being deployed on a big number of production systems.
Common factors
All incidents have in common, that the actual number of developers is very small, from one to less of a handfull people without professional positions.
According to the Wikipedia article on the OpenSSL project history there have been … seventeen developers with commit access … There are only two full-time employees (fellows) and the remainder are volunteers.
The entire Apache logging team only had 16 unpaid developers according to infoworld. The Log4j team is now supported by the German Sovereign Tech Fund funding two full time positions.
The XZ package has been maintained by a single developer who fell victim to a well orchestrated social media attack and passed the project to a offender in a burnout situation as described by The New Stack
There are probably a lot or more projects which are in danger to get infested by similar bugs.
Typical damage
The typical damage of a large scale security incident is hard to estimate. A guess based on the time to analyze and mitigate the loss goes into several Millions at the first day. This sums up to hundreds of millions over time.
Solution proposal
This is a three stage approach for the immediate response, the clean up phase and the long term solution.
Funding for immediate response
Create a fund for immediate response covering the expenses to support a team in an incident situation. As it does not make sense to add new developers in an emergency situation this covers only the core team. Together with experienced communications experts whose one an only task is to keep the backs of the dev team clear this response needs a budget of 20.000€-100.000€ depending on the period needed to do hot fixes. This amount of money should be spent without conditions and immediately.
The decision could be done by any institution with proven record of supporting projects as the OpenSSF or the Sovereign Tech Fund
Forensic analysis an clean up
After the hot fixes have been released there should be a forensic analysis and financial support for the team. The range of money can be estimated to 100.000€-1.000.000€ and should be spent in a structured way by a committee which has to be defined. Communication is also a very important part to get rid of the deployments still using the vulnerable versions.
Long term measures
This is the most expensive part as it might need fundamental structural changes.
The OpenSSL library suffers from the focus of being a consultant business supporting every hardware (private communication to a OpenSSL consultant) and not primarily the internet and should probably deprecated and replaced.
Log4j suffers from the deployment of JNDI which has originally been created to load Java packages to deploy software on Blue Ray players.
The XZ packages is dynamically linked into other software like systemd even if the functionality is not needed.
In general all software systems suffer from a bloated and unnecessary mess of unused software. Niklaus Wirth has mourned the bloat of software for decades and nothing has happened.
At the moment it is hard to ignore that we as a community vendors are not doing our job to protect the customers as we do work like engineers but when we take software from the commons, we are like raccoons digging through a dumpster to find something useful.
Nevertheless, the only highly promising approach can be to clean up all important software packages.
- remove unnecessary dependencies
- value the removal of code more than adding new features
- identify by SBOMs the most central and widest spread libraries
- look for developers and maintainers in burnout situations
- offer the creation of full time positions to the core developers (don’t add arbitrary new people)
- this effort definitely needs the support of a big number of commercial vendors using the software.
Foto by Christina Welsh under CC BY-SA 2.0 DEED