Introduction
On April 15th, Codecov disclosed a major breach when an attacker compromised its infrastructure allowing them to export sensitive information like tokens or access keys stored in continuous integration (CI) pipeline environments of multiple organizations around the world. It’s now clear that developers and development infrastructure are under attack.
We’ve ignored security, in favor of developer agility and release velocity, for so long that development infrastructure has become most organizations weakest links, which attackers are now rapidly exploiting.
This post will cover the Codecov breach – What is Codecov, how the breach occurred (as much as we know as of today), remediation steps as Codecov suggested and strategic steps to reduce your build vulnerability to supply chain attacks.
The Codecov Breach
To begin with, it is important to emphasize that at the time of writing this blog, specific details of this breach are not yet fully known, therefore I’ll outline what is known based on Codecov findings.
Details
Codecov is a code analysis tool with which users can group, merge, archive, and compare coverage reports. Code coverage describes which lines of code were executed by the test suite and which ones were not.
The security breach took place on January 31th, 2021, and was discovered on April 1st by one of its customers comparing the hash of the formal bash-uploader script and one downloaded.
As published in Codecov’s security update:
“On Thursday, April 1, 2021, we learned that someone had gained unauthorized access to our Bash Uploader script and modified it without our permission. The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.”
From what was revealed so far, the attacker gained access to Codecov’s Google cloud storage by leveraging a misconfiguration in Codecov’s docker image creation process, where the attacker was able to extract Codecov’s Google Cloud (GCP) Storage key. The way the exact GCP key was obtained is not yet known, but we can guess several possibilities including a developer simply and mistakenly leaving the GCP exposed in plain text in the dockerfile.
Upon gaining the access, the attacker altered the bash-uploader script file,which helps uploading the coverage report which Codecov generates.
This attack effectively created a targeted backdoor that automatically extracted some of the most sensitive information in code, such as secrets, username-password or tokens, and published it at a remote url.
From here on, Codecov customers could be affected by the attack through downloading and explicitly using the bash-uploader OR by integrating the bash-uploader into their CI tools, i.e. GitHub action, CircleCI orb or Bitrise step.
According to Codecov, the altered version of the Bash Uploader script could potentially affect:
- Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed.
- Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
- The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.
The attack was detected by one of Codecov customers comparing the hash of the formal bash-uploader script and one downloaded.
Impact
While impact and attack spread are still ongoing, we know that Codecov customers who used Codecov in their CI pipeline between January 31st,2021 and April 1st, 2021 are potentially at risk. Codecov claims 29,000 enterprise customers on their website. Atlassian, Kubernetes, Python, Node.js, Ansible, etc.. Where does this attack end? If Python, BitBucket or Kubernetes are compromised how many other orgs got hacked?
By some estimates approximately 15,000 files using the bash-uploader script in hundreds of different open source projects.
Actions to take for potentially affected users
If one suspects to be affected by this breach in any way, Codecov official instruction is to “immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.” This means revoking manually existing secrets, token and any additional sensitive data, and generating new ones to replace them with. However, this is no easy task. For any Codecov customer scrambling to find and replace hard coded secrets in their development infrastructure, Cycode will make its software available for free(*) to help identify secrets that may have been exposed. In addition, it is suggested to run an authenticity check on bash-uploader script before actually using it.
* Common sense limits apply. Cycode’s intention is to help non-competitive Codecov customers recover from the breach. Cycode reserves the right to determine what is free and for how long on a case-by-case basis.
Hardening Your Build Environment
The Codecov incident is yet another instance of a supply chain attack, where hackers gain access to the company’s product and use it to attack its customers. The fallout from this breach will most likely come to light in the following months, as the attacker uses the stolen credentials to attack other companies and steal their data.
To reduce your build vulnerability, we recommend considering the following mitigation steps:
- Scan for secrets that can be obtained, if compromised. If found, revoke and remove them . It is recommended to do this earlier in the development lifecycle prior to the build or deployment to production stages. This applies on:
- Your docker images.
- Your source code repositories.
- Adopt integrity validation across the entire pipeline – Correlate the output of the different stages with the actual values used. For example, confirm that the bash script you are distributing/hosting in your cloud is equal to the bash script you have in code.
- Enforce secure development best practices
- Ensure code is signed in development and validated in build and production environments to prevent tampering.
- Don’t let developers pull and use open-source code straight from the internet. Before any software is updated, the changes have to go through a code checking review and signing process by another party, something that can guard against both unintentional oversights and insider threat.
How Cycode Can Help
Crises are a time for the community to come together. As such, Cycode is offering a free code repository security assessment to any organization that is looking to harden their development infrastructure. Alternatively, if there is some other way we can help your organization, please get in touch with us.
