While many developers understand the risks associated with hardcoding credentials into code, when it comes to containers, understanding that risk is the exception, not the rule. As a result, it’s not uncommon for developers to hardcode secrets into container images.
The good news is that it’s a largely preventable risk, one that Cycode can mitigate through its Docker Image Secret Scanner, which just became Generally Available as part of the complete secret detection solution.
In this blog, we will provide a deep dive into how our Docker Secret Scanning works, but before we jump in, let’s do a quick review of why we need container secret scanning in the first place.
What Are Containers?
Containers are a lightweight, portable and cost-effective alternative to virtual machines (VMs). While VMs split physical resources by using hypervisors to emulate discrete, fully fledged operating systems, containers share the kernel of the host operating system with other containers. This provides a much more streamlined environment for each workload, providing the resources needed (i.e., dependencies and code) to build the application. Containers have been around since 1979, but it wasn’t until Docker launched in 2013 that containers became viable for modern application development.
What is a Docker Container Image – How is it Built?
Docker is an open source framework for building containers that run on the Docker platform. A Docker image is a read-only template that contains instructions for creating a container that can run on the Docker platform. Docker images consist of multiple layers — filesystem content represented as multiple independent layers stacked on top of each other.
The build process of a Docker Image is defined by the Dockerfile type file. Each line in that file represents a build step that forms a layer in the final image. To avoid rebuilding the same steps in subsequent builds, Docker also caches known layers and uses them when possible. A more thorough explanation can be found here. In our Secret Scanner we treat each layer as a ZIP file, and the completed image as a ZIP file of ZIP files.
Once the image is built, it can be deployed for private use or uploaded into a public registry – such as Docker Hub, Amazon Elastic Container Registry, Google Container Registry, Azure Container Registry, JFrog, from which users can deploy containers, test or share images.
What Are Secrets And How Do They Get Into Container Images?
Secrets, like passwords, access tokens and API keys, are discrete building blocks of a container. They are the glue that binds pieces of functionality together. However, there are secure — and insecure — ways of working with secrets. Working securely with secrets is an ongoing effort by the entire development community, as such, it’s not uncommon for developers to hardcode unencrypted secrets in images, which is a highly insecure practice.
Secrets in Image Layers
There are two common ways in which secrets become embedded in container image layers. The first way is through hard-coded secrets in the code, which can end up showing in one of the image layers during the build process. Another way is by unintentionally copying sensitive files, such as credentials, during the image build process. As an example, if in the Dockerfile we use the command COPY . /app, it will copy all the contents of the current working directory on the host machine to the /app directory in the Docker image. If there are files in the current working directory like the .git folder, which most likely contains credentials to push code to a private source control management system, the entire folder, including any secrets contained within it, will be copied to the image. Both of these ways will lead to the secrets being present in the layers of the images.
Secrets Present in Manifest File
The second way secrets end up in Docker images is through misuse of certain commands in the Dockerfile, such as ARG and ENV and could lead to the exposure of secrets in the manifest file
ARG commands are only available when building the image – they allow developers to ‘inject’ variable names in the build process. A common misuse is to set secret variables at the build process as they are exposed in the manifest file. For example, let’s build the next Dockerfile:
FROM alpine:latest ARG secret RUN echo "value in build time: $secret" CMD echo "value in run time: $secret"
This is how we build it:
$ docker build --build-arg secret=password . --tag build-time .. => [1/2] FROM docker.io/library/alpine:latest => [2/2] RUN echo "value in build time: password"
The manifest file generated by docker:
Notice the value of our secret `password` is exposed.
ENV parameters are available to the application containers during the build and when the container is running. When done correctly, the ENV command is the correct way to incorporate secrets because it enables the secret to be accessed at Runtime. However, developers often use it in a way that hard-codes the secret during the build phase, right when the ENV command is issued. For example in the following Dockerfile:
FROM alpine:latest ENV secret=password CMD echo "value in run time: $secret"
The ENV command exposes the secret value password in the manifest file.
Once an image with secrets is loaded into a public registry, it’s in the public domain. Anyone can download the image, including attackers, and look for credentials, access tokens, and other sensitive information. This could lead to a potential breach similar to the CodeCov incident, where they used secrets found in the container image to update the bash installer with malicious script.
Container Secrets Create Yet Another Scary Software Supply Chain Attack Vector
In August 2022. researchers from cloud security firm Wiz discovered a software supply chain vulnerability in IBM Cloud Databases for PostgreSQL they named Hell’s Keychain.
As stated in their blog, the vulnerability consists of a chain of three exposed secrets coupled with overly permissive network access to internal build servers. They describe it as a “first-of-its-kind supply-chain attack vector impacting a cloud provider’s infrastructure.”
While the blog details how they were able to use the vulnerability to compromise IBM Cloud, Wiz carried out similar attacks across several providers as well. In other words, this was not a one off vulnerability in one cloud provider’s infrastructure, but an industry-wide concern.
AppSec teams can’t fix these issues in a vacuum. We need to make it easy for developers to implement best security practices across the SDLC. Image scanning is a fundamental best practice. Our goal is to create a frictionless experience for developers so that the images remain secure.
How Cycode Scans and Protects From Exposing Secrets in Containers
Cycode’s container scanning functionality scans each layer and manifest (for ARG and ENV variables) of an image to find secrets buried deep within the intermediate layers of an image where few (except for attackers) would bother to look. All current users need to do is enable the feature and configure it based on their needs.
In fact, Cycode is the only SDLC security vendor that offers the ability to automatically and seamlessly scan for secrets in containers as a native part of our platform (In other words, no more manual scans using CLI tools).
We aim to prevent hard-coded secrets from being reintroduced and help developers to adopt best-practices of secret usages when building Docker images. We encourage our users to enable scanning upon every push to a registry. If secrets were found, they could be prioritized and remediated through the violation tracking in Cycode’s platform dashboard.
Ultimately, the risk of secrets stems from three types of exposure: compromised insiders, malicious insiders and code leakage. A complete secrets detection solution must include comprehensive secrets scanning and address the ways secrets are exposed.
Best Practices for Secure Secrets Management – Don’t Place in Build, Execute in Runtime!
As we’ve detailed in an earlier blog, while secrets in source code result from developer mistakes and missing best practices, secrets in the build systems are essential for creating meaningful workflows that communicate and authenticate with various services, such as cloud providers, artifact registries, package managers, ticket handling systems, messaging apps, etc. Usually, when you store these secrets, they are well-encrypted and revealed inside the specific builds authorized to access them.
While these secrets are securely stored in the build environment, they can be misused, leading to their exposure. When injecting these secrets to container images during the build phase, they are unencrypted, available to everyone, and shouldn’t be there in the first place.
Instead of populating the secret in the build state, developers should supply the secret in the run state, that way the image itself doesn’t contain the secret.
For example assume we are using the next Dockerfile:
FROM alpine:latest ARG secret RUN echo "value in build time: $secret" CMD echo "value in run time: $secret"
That’s how we build it:
$ docker build --build-arg secret=password . --tag build-time .. => [1/2] FROM docker.io/library/alpine:latest => [2/2] RUN echo "value in build time: password"
And That’s how we run it:
$ docker run build-time value in run time:
Notice the value of our secret is only present in the build stage, this means it is exposed at the manifest of the docker image as shown earlier.
Instead, we would want to write it as follow:
FROM alpine:latest ENV secret= RUN echo "value in build time: $secret" CMD echo "value in run time: $secret"
Build the following way:
$ docker build . --tag run-time .. => [1/2] FROM docker.io/library/alpine:latest => [2/2] RUN echo "value in build time: "
And run the following way:
$ docker run -e secret=password run-time value in run time: password
Notice this time it is present only in runtime, Thus not exposed in the manifest file.
It’s a simple change, in theory, but may affect some developer behavior. We understand that, and our goal is to help facilitate that change using automated, developer-friendly tools that deepen ties between AppSec and DevOps teams.
Learn More
Want to learn more about how Cycode provides visibility, security, and integrity across all phases of the SDLC? Schedule a demo today!
Originally published: February 2, 2023
