Enhancing CI/CD Pipeline Security with OIDC Tokens for Cloud Authentication

As the demand for faster and more efficient application deployment grows, the use of pipelines in the CI/CD process has become increasingly common. Pipelines enable the automation of testing, building, and deployment of applications, resulting in quicker time-to-market and higher efficiency.
Authentication to cloud systems within the CI refers to the process of verifying the identity of users who access cloud-based resources during the CI process. In simple terms, it is the way to ensure that only authorized users can access the cloud resources, applications, and data. To achieve authentication, the most popular method is by saving secrets in the secret manager of the CI system.
A secret manager is a tool that securely stores and manages secrets, such as API keys, access keys, passwords, and other sensitive data. By using a secret manager, developers can avoid hardcoding sensitive information directly into their code, which is prone to leakage or unauthorized access. Instead, they can fetch the required secrets from the secret manager during runtime, ensuring only authorized users can access the necessary resources.
With growing applications, services, and tools requiring authentication credentials and access keys, the number of secrets stored in CI systems can quickly grow to an unmanageable level.

The article will present that the popular method to access cloud resources, which is also suggested in all best practices, can lead to the exposure of sensitive information and compromise the production environment, as seen in the latest CodeCov and CircleCI breaches. The article will explain the problem and give practical tools for applying better solutions to mitigate it.

Real World example

Consider a common scenario where a GitHub or GitLab pipeline is utilized to upload a configuration file to a storage bucket on GCP. The common way to implement this scenario is the following:

• Create a service account in GCP with Storage Object Creator permissions to a bucket.
• Generate a new long-lived secret key that will be used for authentication and authorization with that service account.
• Usually, the secret key will be stored inside the CI system secret manager.
• Configure the pipeline to authenticate using that secret key.
• The pipeline will authenticate with GCP and will upload the configuration file.

The Problem: Long-lived Credentials

The following illustrates creating a pipeline in GitHub and GitLab using long-lived credentials. In both examples, the pipeline will authenticate with GCP using the service account secret key and then will upload a configuration file to the bucket. The following snippet shows how the GitHub Actions pipeline will implement such a scenario:

jobs:
  upload-cloud-storage:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - id: 'auth'
      uses: 'google-github-actions/auth@v1'
      with:
        credentials_json: '${{ secrets.GCP_CREDENTIALS }}'

    - id: 'upload-conf'
      uses: 'google-github-actions/upload-cloud-storage@v1'
      with:
        path: 'conf.json'
        destination: cycode_demo_bucket/conf.json

GCP service account key is a JSON format containing data such as: the service account email, the environment, and a private key.

{
  "type": "service_account",
  "project_id": "example",
  "private_key_id": "00000",
  "private_key": "-----BEGIN PRIVATE KEY-----\n00000\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "example",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/..."
}

When using GitLab,  a suggested best practice is to store secrets in a HashiCorp vault. In environments without a vault, the JSON credential file must be saved as a masked variable. Attempting to mask the JSON variable will result in an error warning due to unmatched regular expressions. As a result, GCP long-lived credentials cannot be masked, and credentials could be visible in the job logs, potentially compromising the application’s security.

While there are some workarounds to this issue, such as base64 encoding the JSON key (which can be masked), it is not the optimal solution.
The following pipeline will execute the same procedure through GitLab CI. The functionality remains unchanged, but it will use gcloud client CLI instead:

gcp-auth-without-oidc:
    stage: build
    image: google/cloud-sdk:slim
    script: 
    - echo ${GCP_CREDENTIALS} > token.json
    - gcloud auth login --cred-file=token.json
    - gcloud storage cp conf.json gs://cycode_demo_bucket/conf.json

The Risks

Unauthorized parties can act on behalf of the authenticated account if the service account secret key is compromised. The recent examples are CodeCov and CircleCI breaches.
In the CodeCov case, malicious code was injected into a bash uploader script that runs inside customers’ pipelines, enabling the attackers to steal all the pipeline secrets.
In the CircleCI case, stored secrets in the CircleCI system were compromised due to the CircleCI breach, risking production environments and additional sensitive assets of any CircleCI user.
What is the solution? OIDC! Open ID Connect tokens provide the capability to authenticate without relying on long-lived access keys.

Open ID Connect Authentication

An Open ID Connect token is a digitally-signed JWT (JSON Web Token) that serves as proof of identity. An Open ID Connect token used in a pipeline holds relevant information about the pipeline itself, such as the user who triggered it, the namespace, the workflow path, and more.
Once verifying the signature, a short-lived credential is returned by the cloud provider, and access is granted based on configured service account permissions.
In our scenario, the short-lived credentials will be used to upload the configuration file to the bucket instead of relying on long-lived access keys.
The diagram below demonstrates the process of retrieving short-lived credentials from the cloud provider.

OIDC Token Claims

OIDC token contains claims that are signed by the selected identity provider. Some claims are standardized, like issuer (iss), audience (aud), and subject (sub). In addition, the OIDC token contains some custom claims that the identity provider defines. Each identity platform serves the OIDC discovery document under the issuer path combined with .well-known/openid-configuration. GitHub default issuer is: https://token.actions.githubusercontent.com, and the default issuer for GitLab is https://gitlab.com.

Setting Up OIDC Environment

Firstly, we will need to configure a new workload identity pool. A Workload Identity Pool allows for managing external identities. For each identity pool, we can add several providers.
Each provider can be customized to a specific issuer, audiences, attribute mapping, and attribute conditions. Let’s break it down by configuring OIDC-based token authentication in GCP, for GitHub and GitLab.
In the GCP panel, we will search Workload Identity Federation and choose the CREATE POOL option.

Configuring Providers

Next, we can proceed to the ADD PROVIDER section and select the option to add this provider to the identity pool we have just created.
We will have to configure the provider name and the issuer as we described.

Workload identity provider

Finally, the identity pool should include two providers, one for GitHub Actions and one for GitLab CI.

We will also use the service account we created with Storage Object Creator permissions to a storage bucket.
The following diagram describes the environment we set up.

Using OIDC in GitLab CI & GitHub Actions

We can now add a new GitHub Action that authenticates to GCP using Open ID Connect. The id-token: write allows the JWT to be requested from GitHub’s OIDC identity provider. We no longer need to use the service account service key. Instead, we simply need to specify the identity pool path and the service account email.

GitHub Actions Workflow

permissions:
 contents: 'read'
 id-token: 'write'


jobs:
 upload-cloud-storage-oidc:
   runs-on: ubuntu-latest
   steps:
   - name: Checkout
     uses: actions/checkout@v3


   - id: 'auth'
     uses: 'google-github-actions/auth@v1'
     with:
       workload_identity_provider: 'projects/123456789/locations/global/workloadIdentityPools/demo-pool/providers/github-provider'
       service_account: '[email protected]'
       access_token_lifetime: '300s'


   - id: 'upload-file'
     uses: 'google-github-actions/upload-cloud-storage@v1'
     with:
       path: 'conf.json'
       destination: cycode_demo_bucket/conf.json

GitLab CI Workflow

With GitLab, an OIDC token can be generated using the id_tokens keyword, which will then be saved within the OIDC_TOKEN variable. Finally, the pipeline will authenticate using the gcloud CLI client.

gcp-auth:
   stage: build
   image: google/cloud-sdk:slim
   id_tokens:
     OIDC_TOKEN:
       aud: https://cycode.com


   variables:
       IDENTITY_PROVIDER: projects/123456789/locations/global/workloadIdentityPools/demo-pool/providers/gitlab-provider
       SERVICE_ACCOUNT: '[email protected]'


   script:
   - echo ${OIDC_TOKEN} > .ci_job_jwt_file
   - cat .ci_job_jwt_file
   - gcloud iam workload-identity-pools create-cred-config
       ${IDENTITY_PROVIDER}
       --service-account=${SERVICE_ACCOUNT}
       --output-file=.gcp_temp_cred.json
       --credential-source-file=.ci_job_jwt_file
   - gcloud auth login --cred-file=`pwd`/.gcp_temp_cred.json
   - gcloud storage cp conf.json gs://demo_bucket/conf.json

Attribute Conditions

In GCP, attribute conditions are expressions in CEL that allows verifying the source environment that the OIDC token was created on. If an attribute condition evaluates to True for an expression, it is accepted. Otherwise, it is rejected. All valid authentication credentials will be accepted if the attribute condition is left empty.

We can use the assertion keyword to select claims from the OIDC token. The assertion method differs between cloud providers.

Each claim (standard or custom) can be used to restrict access. As GitHub says: “you must add conditions that filter incoming requests, so that untrusted repositories or workflows can’t request access tokens for your cloud resources”. Google mentions that: “The maximum length of the condition expression is 4096 characters. If unspecified, all valid authentication credentials are accepted.”

Avoiding misconfigurations

As described earlier, the OIDC tokens include standard and custom claims. Any claim can be used to restrict access, but we will have to be very careful about the claims we trust and the conditions we apply.

Strict conditions:

assertion.repository_id == “100”
assertion.sub == 'repo:octo-org/octo-repo:ref:refs/heads/demo-branch'

Easily bypassed conditions:

assertion.namespace_path.contains(“octo-org”)
assertion.repository.startsWith(“octo-repo”)

Attackers can leverage misconfigured attribute conditions by creating a new repository/namespace path to match the condition.
Let’s say a DevOps team member uses the pipeline we described earlier to authenticate to GCP using the OIDC token. The team member also defined the following attribute condition to protect the organization against attacks.

assertion.namespace_path.contains(“some-org”)

By creating a new pipeline that uses OIDC authentication. Attackers can create a new GitLab account containing the company name: some-org-temp-123. The pipeline will request a new OIDC token, and the namespace_path claim will contain the correct namespace path. The configured attribute condition will evaluate as True, and the request will be accepted.

Summary

Recent attacks on CodeCov and CircleCI have raised the need to minimize the number of secrets stored inside the pipeline. The amount of tools requiring token authentication is enormous – source control, build tools, artifact registries, cloud providers, monitoring tools, and more. They all use unique tokens that need to be saved inside the CI system or a secret manager. OIDC token authentication is one step towards removing long-lived secrets and using a better authentication method that allows easy management, enhanced security, and reduced pipeline exposure risk.

Originally published: April 21, 2023