Cycode Discovers a Supply Chain Vulnerability in Bazel

Executive Summary

The Cycode Research Team discovered a software supply chain vulnerability in one of Google’s open source flagship products, Bazel.
We found that a GitHub Actions workflow could have been injected by a malicious code due to a command injection vulnerability in one of Bazel’s dependent Actions. This vulnerability directly impacts the software supply chain, potentially allowing malicious actors to insert harmful code into the Bazel codebase, create a backdoor, and affect the production environment of anyone using Bazel. This vulnerability could have affected millions of projects and users who use Bazel, including Kubernetes, Angular, Uber, LinkedIn, Databricks, DropBox, Nvidia, Google, and many more.

We reported the vulnerability to Google via its Vulnerability Reward Program, where they acknowledged our discovery and proceeded to address and fix the vulnerable components. Bazel subsequently updated the workflow base permissions following the Principle of Least Privilege and modified the dependent Action, effectively resolving the command injection vulnerability.

Bazel users don’t need to take any action since the vulnerability fix has already been applied.

Introduction

GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline as part of the software development lifecycle (SDLC).
This is achieved through the creation of workflows, which are customizable, automated processes defined by the creation of YAML files within the .github/workflows directory. These workflow files contain instructions organized into jobs and steps, allowing users to specify and customize various tasks. These instructions can execute code or trigger other components, providing users with a versatile and adaptable structure to tailor workflows based on their specific needs.

GitHub Custom Actions

GitHub Actions provides a versatile approach to streamlining your workflow by combining individual tasks called “Actions”. Users can create their own custom actions or use actions created by other users. Custom actions can be compared to functions being called within code, where we use our own functions and import third-party ones.

The useskeyword allows us to import different actions, along with the with keyword, which is used to specify inputs.

- uses: cycodelabs/example-action
  with:
  token: ${{ github.token }}

There are three types of custom actions: Node, Docker, or Composite action. Let’s delve into each type to understand how each one interacts with the build pipeline.

Docker Actions

• Action runs inside a Docker container.
• Can be configured using a Dockerfile or with an Image.

name: example
description: Example of a Docker Action
...
runs:
  using: docker
  image: Dockerfile
  env:
    INPUT_NAME:  ${{ inputs.name }}
    INPUT_VERSION:  ${{ inputs.version }}

JavaScript Actions

• Similar to NodeJS programs that execute code and call different functions.
• Utilizes the Github Actions Toolkit to interact with the workflow.

name: example
description: Example of a JavaScript Action
...
runs:
  using: "node16"
  main: "dist/index.js"

Composite Actions

• Combines multiple workflow steps within one action.
• Each step can invoke shell commands or call additional actions.

name: example
description: Example of a Composite Action
...
runs:
  using: "composite"
  steps:
    - run: printenv
      shell: bash

Vulnerabilities Out Of Sight

Custom actions add a significant burden on the organization’s software supply chain. A few lines of code in the top-level workflow can translate into thousands or even millions of lines of code, many of which we may not even be aware of. Also, they contain many components that undergo frequent changes and updates, potentially addressing new vulnerabilities on a daily basis.

When creating new workflows, we heavily rely on various third-party dependencies, including custom actions that, in turn, use other actions. These actions execute programs written in different languages, such as JavaScript or Python, and leverage libraries from various package managers like NPM or PyPI, forming an extensive chain of dependencies.

Securing this complex web of dependencies is a complicated task, as vulnerabilities could emerge at any point, posing a threat to the entire build pipeline. So far, we’ve identified dozens of vulnerabilities in open source projects, mainly focusing on vulnerabilities within the workflows themselves. However, this current research has focused on uncovering vulnerabilities in indirect dependencies, such as custom actions. These are often more challenging to identify since they may be located in different repositories, in other ecosystems, and managed by other maintainers.

Examining workflows in public repositories reveals the extent of the attack surface. Out of 3.4 million workflows, nearly all of them (~98.75%) incorporate one or more custom action. This blog focuses on abusing vulnerable composite actions.

The Story of Bazel

Bazel is an open source software tool used for the automation of building and testing software similar to Make, Maven, and Gradle. Bazel is widely used, has more than 21K stars on GitHub, and is used by some of the most significant projects and companies.

As part of its open source management, Bazel is utilizing GitHub Actions to test and build new code, label issues, and run scheduled tasks.
We managed to identify a command injection vulnerability in the cherry-picker workflow. Let’s break it down part by part and explore how the vulnerability formed.

Part 1: Triggers & Permissions

name: cherry-picker

on:
  …
  issues:
    types: [closed, milestoned]

env:
  GH_TOKEN: ${{ secrets.BAZEL_IO_TOKEN }}

…

The workflow runs every time an issue is being closed/milestoned. By investigating one of the workflow’s runs, it looks like the workflow is granted with full Read/Write permissions.

This is caused by the following workflow permissions setting that can be configured either in the repository or organization settings panel in GitHub.

Combining the permissive permissions at the repository level without modifying the default workflow permissions results in complete access granted to the GITHUB_TOKEN.

Part 2: The Innocent Workflow

cherry-picker-on-milestoned:
  if: github.event.action == 'milestoned'
  runs-on: ubuntu-latest
  steps:
      ...
    - if: startsWith(github.event.issue.body, '### Commit IDs')
      name: Run cherrypicker on demand
      uses: bazelbuild/continuous-integration/actions/cherry_picker@7ac...
      with:
        triggered-on: ondemand
        milestone-title: ${{ github.event.milestone.title }}
        milestoned-issue-number: ${{ github.event.issue.number }}
        issue-title: ${{ github.event.issue.title }}
        issue-body: ${{ github.event.issue.body }}
        is-prod: True

The job cherry-picker-on-milestoned will be executed if a maintainer milestones our issue. All we need to do is write a valuable issue that will be part of any of the milestones defined by the Bazel project.

Let’s assume we tracked the Bazel repository for a while and managed to write a valuable issue that will be milestoned.
Once that happens, the mentioned job checks the issue body (which is controlled by the user), and if it starts with ### Commit IDs, it will call the composite action located at the Bazel continues integration repository.

Part 3: The Injectable Composite Action

Until this point, the workflow appears to be just another one that could have been run with fewer permissions, but without a visible attack vector. That’s the moment the composite action comes into the game 🏀.

name: "Cherry-picker when comment is created or issue/pr is closed"
...

runs:
  using: "composite"
  steps:
    ...
    - name: Pass Inputs to Shell
      run: |
        ...
        echo "INPUT_ISSUE_BODY<<EOF" >> $GITHUB_ENV
        echo "${{ inputs.issue-body }}" >> $GITHUB_ENV
        echo "EOF" >> $GITHUB_ENV

        echo "INPUT_ISSUE_TITLE=${{ inputs.issue-title }}" >> $GITHUB_ENV

The Pass Inputs to Shell step passes the issue-body and issue-title inputs directly to an inline bash script. As a reminder, those inputs are received from the original workflow with the details of the issue, which are controlled by the user and can be manipulated.

By leveraging a shell feature called command substitution, we can include a shell command using the $( ) characters. Anything within these brackets will be treated as a system command and will be executed.
In our scenario, one of the inputs we control is the issue body. Let’s say the issue body looks like this:

This is a new issue $(ls)

The workflow (that runs on a Linux system) will interpret ls as a command and will execute it. You can learn more about this attack vector in our previous research.

Part 4: Attack in Action

We have created a full replica of the Bazel repository to reproduce this vulnerability and show its huge impact.
Let’s start by creating a new issue containing our malicious payload. In this case, we inject a base64 encoded curl command that will fetch a script from a remote server and execute it.
We will hide our malicious payload within system logs that typically don’t undergo close inspection.

Once our valuable issue is milestoned, the cherry-picker workflow will start, and our malicious code will be executed. In this case, our script will dump the secrets from the runner’s memory and will send it to the attacker’s server.

Both the BAZEL_IO_TOKEN and the GITHUB_TOKEN will be sent to the attacker’s server.

How Did We Find It? Raven!

In our ongoing pursuit of identifying new vulnerabilities, we used RAVEN to scan public repositories with high star ratings. Our mission was to uncover critical CI/CD vulnerabilities that could potentially impact millions of end users.

Following this research, we created the following query to help find a similar class of vulnerabilities, that can’t be found using the standard tooling. This query directed our research toward Google Bazel and Apache Camel, both of which were identified, reported, and subsequently addressed to fix the vulnerabilities.

Disclosure Timeline

• November 1, 2023 – We sent a detailed report explaining the vulnerability to Google through the Google Bug Bounty Program.
• November 7, 2023 – Google opened a new issue addressing the high permissions of the cherry-pick workflow.
• November 8, 2023 – Google pushed a new commit addressing the high permissions of the cherry-pick workflow.
• December 5, 2023 – A pull request containing a fix for the vulnerable Composite Action has been merged.
• December 12, 2023 – Google granted us a generous $13,337 Bounty, acknowledging the critical importance of this vulnerability.

How Cycode Can Help

Cycode’s platform secures your software supply chain by providing complete visibility into enterprise DevOps tools and infrastructure.
With a simple GitHub integration, Cycode’s platform detects command injection vulnerabilities and additional configuration issues in your GitHub Actions workflows and help you remediate them. Such violations look like the following:

Learn More About Cycode’s Complete ASPM Platform

Cycode’s complete ASPM platform secures CI/CD pipelines by providing unparalleled visibility, prioritization, and remediation advice for your software supply chain. Our Risk Intelligence Graph (RIG) provides visibility and intelligent context for all your alerts. We help security and development work better together by allowing teams to prioritize and remediate the most critical vulnerabilities so developers can focus on what they do best: create innovative features that help differentiate your business.

Want to learn more about Cycode’s complete ASPM platform? Book a demo now to find out how we can help you achieve faster time to value, reduce critical vulnerabilities, and remediate faster.