Detecting and Mitigating Source Code Compromise in the Wake of the CircleCI Incident

On January 4, 2023, CircleCI reported a data breach that affected some of their customers (for more information on that please see our previous blog post). In an effort to ensure transparency and provide their clients with a clear understanding of the event, CircleCI released an updated detailed report on January 13, 2023. In this report, they explain the cause of the breach, the measures taken to remediate the situation and secure their systems, and the recommended actions for those who may have been impacted. 

CircleCI has confirmed a data breach occurred on December 16, 2022, where an employee’s laptop was compromised by malware, allowing the attackers to steal the employee’s two-factor authentication-backed credentials and breach the company’s systems. The malware went undetected by the company’s antivirus software

The attacker used the compromised credentials to impersonate the employee and gain access to a subset of the company’s production systems. The attacker then exfiltrated data from a subset of CircleCI’s databases and stores, including customer environment variables, tokens, and keys. The attack was carried out on December 22, 2022. The data exfiltrated was encrypted at rest, but the attacker was able to extract the encryption keys from a running process, allowing them to access the encrypted data.

According to the incident report, the first indication of a compromise was an alert on unusual activity related to GitHub OAuth tokens, brought to CircleCI’s attention by one of their customers. This prompted an in-depth investigation by CircleCI’s security team in collaboration with GitHub. 

This incident highlights the importance of enhancing security and auditing capabilities around our source code management systems. In this blog post, we will examine the current capabilities and options available to improve security and oversight of these systems.

Initial access: How to detect stolen or misused credentials in Github

As recommended by CircleCI, one way to check for compromise is to review the GitHub audit log for unexpected commands. The example CircleCI provides looks for the repo.download_zip event, which is sent when a repository zip is downloaded. In addition to this event, we recommend looking at the following ones as well:

• The git category for clone activity – specifically the clone event 
• The integrations installation category for any unusual installed applications
• The add_member event across orgs, teams, and repos for suspicious access grant
• The protected_branch category to detect any suspicious activity related to security configuration around your repositories

It is important to look at the location of the events, the frequency and anomalies between past historical events, and activity within the breach window.

In Cycode, we continuously monitor these events and alert on any suspicious activity, such as an anomalous location or frequent activity:

Mitigation recommendation – Reducing the blast radius of compromised credentials

This incident emphasizes the need for good hygiene around your source code management platform, and in a wider sense, a strong and secure CI/CD posture. This would reduce the impact of such a breach and decrease the blast radius.

The main steps to take to achieve a secure CI/CD posture are:

• Harden authentication – Add an additional layer of security to GitHub accounts such as two-factor authentication or SAML single sign-on. In addition, it is also recommended to leverage IP range restrictions and IP allowlisting to access your GitHub environment.
• Audit for excess privileges – Stopping insiders is very difficult so it’s important to establish least privilege policies to minimize their radius and ease of lateral movement. However, a least privilege policy is only as good as its last audit for excess privileges, so now is the time to ask yourself when your last excess privilege audit was? For most organizations, the honest answer is that privileges were set when the account was provisioned and never reviewed again. Now is the time to get caught up.
• Eliminate hardcoded secrets – Just as removing excess privileges is a key part of reducing insider risk, so is eliminating hardcoded secrets. Hardcoded secrets roll out the welcome mat to attackers to access other valuable resources. Not only should hardcoded secrets be removed from repositories, build logs, container images, K8s, your members’ personal repos and more, but once removed, authentication-related secrets should be revoked and recycled.
• Continuously monitor for anomalous user behavior – As we mentioned earlier, auditing and monitoring the activity within the tools is key to identifying compromise, and there are several aspects to which events you should audit. Keep monitoring for any unusual activity on your Github account and take appropriate action in case of any suspicious activity.

An update on CircleCI context secrets

Finally, in CircleCI’s report, they explain that they have added the updated_at field to the context API so that customers can verify the successful rotation of these variables. (More on the various types of secrets in CircleCI in our previous blog post.) 

Cycode’s clients can query this data within the knowledge graph to make sure they have rotated all the relevant secrets within CircleCI:

If you need help securing your software supply chain tools and infrastructure, or are concerned about the CircleCI breach and possible exposure resulting from it, we are here to help. For more details, schedule a demo today.