Six AppSec Learnings from SolarWinds

The SolarWinds incident is a rapidly evolving situation as more and more organizations realize they’ve been breached. We don’t know exactly what the original source of the attack was, and rather than describe the attack as many others have already done, we’ll use this post to think out loud on initial lessons learned.

Whatever we assume is secure will become insecure (Part 1)

As an industry, we’ve assumed that a binary signed using a valid certificate is safe. For many years it was but clearly we can no longer make that assumption. Attackers tend to go after the weakest link, which creates a natural evolution of which types of attacks are most common. 

In the pre-cloud days the weak point was often the network. However, AWS, GCP and Azure have done a good job protecting the network so the weak point has become the app itself. As organizations shift their AppSec efforts left, and embrace devsecops, AppSec is improving, and the applications we build are more secure. This shifts the weakest link again. 

Development infrastructure is the new weakest link

Although we can’t say where or how the breach occurred (but we do have a guess — see below), it is most likely that the integrity of SolarWinds’ development pipeline was compromised such that attackers could tamper with SolarWinds’ code undetected.

Security conscious organizations deploy an entire alphabet’s worth of AppSec tools to protect the contents and output of their development pipelines. Whether its SAST, SCA, IAST, DAST, WAF, RASP or container security organizations invest heavily in protecting their applications in production and finding vulnerabilities in development. Yet, investments in securing the development infrastructure itself, (code repositories, CI/CD systems, Infrastructure-as-Code, etc.) are few and far between. As an example, build servers like Jenkins have become a major target for crypto miners and malware and code repo access controls are notoriously lax so as not to impact developer agility or velocity.

Whatever we assume is secure will become insecure (Part 2)

Another example of the presumption of security creating weak links is the explosion of hard-coded secrets in source code, which may have played a major role in the SolarWinds breach. Changes in the SDLC, such as the adoptions of cloud-computing, microservices architectures, and APIs, rapidly increased the need to store secrets in code. Developers believed this was secure because only insiders had access to source code. However, at the same time, enterprises embraced open-sourcing some of their own code, which increased the complexity of managing repositories. These two trends combined to create a dangerous security situation in which many organizations have accidentally exposed legitimate credentials in publicly accessible repositories. For example, Uber exposed 57M records via a breach that started when the attackers compromised Uber’s GitHub instance and found exposed AWS admin credentials. It may simply be a coincidence, but SolarWinds had a similar issue last year.

Everything-as-Code makes repos high value targets

Each new phase of the cloud-computing era has led to more consolidation of the SDLC into code. First computing became code, then infrastructure followed, and now orchestration and GitHub Actions mean code is dominating the entire SDLC. 

Source code was always valuable as both the core of any software’s intellectual property and the blueprint of how applications work (and therefore how to exploit them). However, code repositories now also contain key CI/CD settings and Infrastructure as-Code (Terraform, Cloudformation, Kubernetes, etc.). Combined with hard-coded secrets, it’s clear that attackers no longer need to target production applications directly. By breaching the code repository attackers can own the whole pipeline.  

Integrity is the new black

Tampering with code will likely become AppSec’s buzzword de jour (DevIntegOps 😂😂😂) as a result of the Solar Winds breach. Inserting back doors into the supply chain has precedent, but best practices on identifying and preventing them haven’t really coallessed yet. For example, 5 years later, it’s still not clear if Juniper ever knows how their backdoor breach occurred.

To date most efforts at integrity have been focused on a single domain and trusting the authenticity of signed certificates is a perfect example of why we need cross domain integrity checks. When we think about integrity in the dev cycle we think about multiple developers signing pull/merge requests. We think about SAST and SCA scanning for vulnerabilities in that code. We run IAST and DAST in test environments. And the emerging IaC security scanning tools focus on what settings should be.

The problem is that each of these tools assumes that code always flows in one direction from dev to prod w/o skipping a step. This was true in the waterfall development paradigm, where the environments were segregated and access was limited to a mere few. In truth, there are multiple places in the SDLC that code can be tampered with and as we evolve our learnings from SolarWinds we must begin to tie what happens in one phase of the SDLC to what happens in the next to ensure the integrity of the entire funnel.

Ultimately, we must ensure that the output of the pipeline is as expected. For example, how do we eliminate drift between the desired IaC settings and actual production configurations? What good are our efforts to find and fix vulnerabilities in production if source code files on production systems can be tampered with? In a world where every developer can influence the production environment do we really secure the process enough? 

Who’s the target in the age of high entropy supply chain attacks?

As attacks grow in sophistication, likely aided/funded/directed by nation-states, attackers are recognizing that patience is a major asset. Indirect supply chain attacks are very difficult to detect in part because the attackers have traded off their control and time for increases in success-rates. Stuxnet is an example of a high entropy attack that had a defined target, but authors of Stuxnet gave up control to increase the likelihood of success at the expense of time. Was SolarWinds a target or a vehicle? Was FireEye a target or collateral damage? For SolarWinds and FireEye, does it really matter? As supply chain attacks evolve and embrace entropy, there may be no target at all. Instead attackers simply wait to see which targets their exploits land on.

6 steps you can take to reduce your organization’s risk

1. Harden your infrastructure’s access controls. 
   1. First, inventory your infrastructure assets.
   2. Ensure all of your pipeline services are not publicly accessible. 
   3. Audit all systems to remove default credentials. 
   4. Then require MFA for all users. 
   5. Lastly, enforce least privilege policies across the entire process. 
2. Search public repositories for your organization’s proprietary code. 
3. Search your repositories for hard-coded secrets (especially any code in the public domain).
4. Adopt integrity checks to confirm that the output of each phase of the SDLC is as expected in the next.
5. Adopt commit signing practices to ensure that code can always be traced back to when it was merged and who wrote it. 
6. Ensure that source code files cannot be tampered with in production. 

How Cycode can help

Crises are a time for the community to come together. As such, Cycode is offering a free code repository security assessment to any organization that is looking to harden their development infrastructure. Alternatively, if there is some other way we can help your organization, please get in touch with us.