Revisiting security debt: Are we ready to have a discussion yet?
Back in March 2012 James Vaughn and I wrote a little-read paper titled Software Security Austerity – Software security debt in modern software development while working together at Recx. I then went on to present on the topic at 44CON in 2012 as part of NCC Group.
The purpose of the paper and presentation was fourfold:
- Highlight that such a thing (debt) exists in software and systems to a wider audience beyond those familiar with the technical debt concepts and applying it specifically to security.
- Discuss where it comes from, why it is a reality, and that free software and systems are not without cost.
- Provide pragmatic strategies and tactics to consider when managing the problem based on experience.
- Encourage wider discussion and debate.
While the last of these did not happen outside of some small pockets, we partially achieved our first three goals.
During a recent discussion with peers at the UK’s National Cyber Security Centre (NCSC), it was pointed out that the paper and its concepts were a little ahead of their time.
Also, it is important to call out that while software is the focus of the original paper, the concepts apply equally to systems.
So what has changed? Spoiler: A lot
So what has changed in the last six years? Looking at the seismic shifts in software and system development along with their operation, there is a long list.
- From a technical perspective (i.e. system and software development and operations) this includes:
- Prevalent use of cloud and virtualization.
- Increasingly huge diversity in open source.
- Serverless architectures.
- Continuous integration and delivery pipelines (i.e. heavy automation).
- Policy as code.
- Frameworks (and lots of them).
- High frequency release cycles (i.e. the rate at which both systems and code are produced and deployed).
- General complexity of the software supply chain exploding between inter-dependencies and package managers etc.
- Reliance on third parties (SaaS and components) to provide security functionality.
From the perspective of vulnerability discovery we have moved on from a world of humans plus some static or dynamic analysis and fuzzing to one that includes:
- Scaled threat modelling.
- Secure development training.
- Language improvements.
- Compiler improvements.
- Crowdsourced vulnerability discovery by humans (i.e. bug bounties).
- Taint analysis improvements.
- Tooling improvements (e.g. in the domains of fuzzing and vulnerability trapping such as AddressSanitizer).
- Algorithmic improvements to all of the above.
- Systems and software telemetry improvements to allow the detection of attacks and vulnerability exploitation attempts.
- Automation around dependency tracking and known vulnerabilities.
- Plus various others.
From the perspective of vulnerability exploitation, a smaller change is that it has become more common to chain together many vulnerabilities. Previously, attackers used one or maybe two vulnerabilities to achieve an objective. However, because of improved mitigations such as sandboxing, containerization and microservices, it is not uncommon to see chains of six, seven, eight, or even more vulnerabilities with various severities to achieve the ultimate outcome.
From the perspective of security incentives the world has changed, but arguably not radically enough quite yet. As of 2018 there is now:
- Increasing personal liability for directors and officers in relation to cyber security.
- Sector level regulators waking up (e.g. FCA/PRA, DNB, MAS and HKMA in financial services).
- Sector level regulation such as the Second Payment Services Directive (PSD2).
- Regional legislation such as General Data Protection Regulation (GDPR).
- Legislation around disclosure.
- Risk transference (i.e. insurance that comes with its own stipulations).
- Large organizations demanding more from their supply chains.
- Continued debate and contemplation around software liability (i.e. the vendors being liable, which works in a paid world, but what about open source?).
One important point here is that debt increases along with regulation, as you potentially now have to reach a new level of security which was not there previously.
Another area to consider is the immense security debt involved in “the new normal” development processes, via third party code with a known vulnerability. Some of the large breaches that occurred due to known vulnerabilities in the last couple of years are key examples of this.
So what hasn’t changed?
A lot has changed between 2012 and 2018, but what has stayed the same?
- Start-ups, as well as new development within established organisations, often still incur massive amounts of technical (and security) debt. We have yet to solve this problem. Although, with the correct investment in continuous integration and continuous delivery (CI/CD) pipelines this can be often be minimised.
- Large and established organisations continue to carry unfathomable amounts of security debt, but rarely know the scale of it or have robust organisational strategies and team tactics in place to deal with it.
- Relatively high costs to discover the level of security debt remain.
- Hidden and unpayable debt in embedded systems continues.
- Humans with experience and skill, which are a limited resource, are still needed to prioritise and address identified debt.
- Not enough appreciation exists for the value of refactoring and modernisation to address security debt in a systematic fashion.
- Lack of understanding of security and technical debt as a concept at senior levels persists.
So what are the silver bullets?
During an internal peer review of this post I was challenged with the idea that, and I paraphrase, “talk is cheap and our customers often want to buy off-the-shelf solutions to make the pain go away.”
However, there really isn’t one single solution to the problem. Instead, it is often a mix of tools, culture, education and capability. While this may sound like a boring consultant answer, it is sadly the reality of today.
While the market has positive innovations such as Irius Risk and BDD-Security from Continuum Security, ThreadFix from Denim Group, Engineering Analytics from Semmle and CodeSonar from GrammaTech, they are only work aids. Without the appropriate culture, education and capability, the legacy technical debt problems will not be magically fixed by the sole adoption of these technological solutions.
So are we ready to have a discussion yet?
Based on conversations with clients of all sizes, it increasingly feels that organizations are warming up to the concept of security debt and the need to address it. This is in part because of prominent events such as WannaCry and NotPetya highlighting the exposure of debt in legacy systems, but also due to a general maturity in understanding of what cyber security resilience is and the desire to live in a less fragile ecosystem, coupled with the previously outlined incentives.
We need to research and explore various supporting approaches, tooling and solutions at a business, process, technology and people level to get the required leap forwards.
If you want to discuss any of these concepts or just have a general conversation on the topic, feel free to email me on firstname.lastname@example.org.
Published date:  16 March 2018
Written by:  Ollie Whitehouse