February 28, 2022 | 21 MIN

Jenkins Security Best Practices

Head of Security Research

Jenkins is one of the most well-known tools for creating automation pipelines and integrating them with the rest of your CI/CD tools. It has an active community that has contributed thousands of plugins to extend Jenkins’ core functionality, which is the main reason why Jenkins is the industry standard for creating build, test, and deployment processes. Part of Cycode’s role as a leader of supply-chain security is securing build systems like Jenkins. This article shares our real-life experience of applying Jenkins security best practices to avoid the mentioned risks.

Jenkins can connect and automate many users’ SDLC processes across their cloud and on-premise infrastructure by utilizing many of its plugins. However, Jenkins could also become your organization’s weakness without proper controls. Besides the obvious supply-chain threats like the SolarWinds hack, a compromised Jenkins system can threaten the entire production environment and be an entry point for inserting malicious software and exfiltrating sensitive data.

Note – As you will see below, we suggest installing several popular plugins that add to Jenkins’s security controls. Even though the recommended plugins are popular with thousands of downloads, it is vital to maintain and update them as they may contain vulnerabilities requiring patching.

Update Vulnerable Core and Outdated Plugins as a Jenkins Security Best Practice

When searching for known vulnerabilities, we can split Jenkins into two components: Jenkins core automation platform and Jenkins plugins which come on top. According to cvedetails.com, since Jenkins launched in 2011, almost a thousand vulnerabilities have been reported, and the majority are from the past three years:

Keeping a vulnerable version is extremely risky because malicious actors could exploit your server using publicly available exploits. While updating a core version is a manual process, updating plugins is straightforward and done through a few clicks on Jenkins UI. Most of the published vulnerabilities are plugin-related; thus, the latter will close most security issues.

Operating System Hardening

Even the most hardened Jenkins application won’t help if installed on a non-secure server or in a non-secure manner. Most Jenkins servers will be probably Linux-based, so we would suggest applying general Linux recommendations to deny malicious access and modification to the server, such as:

Verify that only needed ports are opened (such as HTTP). If you need additional protocols, such as SSH, define proper firewall rules to block unnecessary access.
Run Jenkins as a non-administrator user.
Harden permissions for JENKINS_HOME directory only for the proper Jenkins user.
Verify that the Jenkins user cannot elevate his permissions through the sudo command. You can do this by modifying the /etc/sudoers file.
Configure Jenkins with HTTPS protocol and trusted certificate like explained here.
Due to its sensitivity, consider blocking your Jenkins server from public access by either installing it in the on-premise network or through a tight security group policy.

The rule is simple – secure the server as much as you secure your Jenkins application. Now let’s dive into Jenkins configuration a bit.

Authentication

Jenkins supplies several built-in authentication methods (these are also called “Security Realms” in Jenkins) – “Delegate to servlet container” and “Jenkins’ user database”. The Jenkins security best practice is not to use the built-in methods and instead use a centralized 3rd party vendor to authenticate against, such as GitLab, Github, LDAP, SAML, Google. By utilizing these methods, certain policies could be applied to the passwords, like password complexity, which helps deny malicious access to the server.

If you did choose “Jenkins’ user database” as a temporary solution, we suggest disabling the “Allow users to sign up” option and controlling the registered users manually.

Authorization

Similar to the previous section, Jenkins supplies the following built-in authorization methods – “Anyone can do anything”, “Legacy mode” or “Logged-in users can do anything”. We suggest not to use these built-in methods but to use plugins for more advanced authorization methods.

The most recognized plugins for this purpose are Matrix Authorization Strategy and Role-based Authorization Strategy, which allow great flexibility for implementing PoLP (Principle of least privilege) by defining the privileges of anonymous users, authenticated users, or specific ones. In addition, you can also define privileges per project or assign created roles for each user. This tutorial explains the process of defining roles through these plugins.

Using GitHub-based and GitLab-based authorization is also possible (using previously mentioned plugins). Still, we discourage that due to the lack of granularity of permission definition and possible misconfiguration like presented here.

Security (Mis)configurations

Jenkins architecture comprises a controller server that holds build configurations, encrypted secrets, and agents that run the builds. To enable Jenkins security best practices, we should protect the controller node by denying or limiting any code execution.

No Executors on the Controller Node

Pipelines run on the controller node have direct access to the Jenkins filesystem, the configured workspace, encrypted secrets, and additional assets. Running builds on the controller node is extremely risky and should be disabled. You can do this by going “Manage Jenkins” → ”Manage Nodes and Clouds” → ”Built-In Node” → ”Configure” → setting “Number of executors” to 0.

Hardening Inbound Connections

The default agent-to-controller communication is done by agents initiating TCP connections to the controller through a dedicated port. This method is called an inbound agent.

If you are not using this method, then the inbound port on the controller node should be completely disabled. If your environment requires inbound connections, it is crucial to ensure the connections are encrypted. Jenkins offers a setting called “Inbound TCP Agent Protocol/4 (TLS encryption)” that should be turned on to run inbound connections through the secured TLS protocol.

These configurations could be found at “Manage Jenkins” → “Configure Global Security” → ״Agents״.

Disabling SSHD

If the SSH Server plugin is installed, verify that SSH is turned off by configuring “Manage Jenkins” → “Configure Global Security” → “SSH Server” → “Disable”

Limiting the Agent Permissions

By default, build pipelines will be running with the internal SYSTEM user permissions. This gives builds the ability to run code on any node, create and delete jobs, start and cancel other builds, and more. Running builds with such permissions can cause serious security issues if, for example, Jenkins pulls malicious build pipelines from the SCM platform, which the Jenkins administrator doesn’t monitor. Here is an example for a default privileged build log showing the permissions at the second line:

By using Authorize Project plugin, you can configure which user, hence what permissions, will run the build. The rule of thumb would be setting the least privilege for every project. Here is an example for the build log after modifying its permissions at “Manage Jenkins” → “Configure Global Security” → “Access Control for Builds” to have the same as the user who triggered the build.

Running Containerized Agents

There are many methods and guides for running the agents – physical machine, virtual machine, container, Kubernetes cluster, and more. From a security perspective, we want to minimize the impact of a compromised build over other builds or the entire system. Thus, we prefer that each build environment be created as a new container from scratch.

As explained here, this could be done by creating a container image with all needed dependencies and deploying the job to a remote docker service. Doing that will ensure that each build will run on a separate clean container.

The main drawback of the previous solution is that you need to create manually the agents who would run the containers. The next talk walks you step-by-step on how you can use the Kubernetes plugin to provision the agents dynamically, thus creating a complete ephemeral environment that depends solely on the SCM system. Example for such Jenkinsfile configuration:

pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        metadata:
          labels:
            some-label: some-label-value
        spec:
          containers:
          - name: maven
            image: maven:alpine
            command:
            - cat
            tty: true
        '''
    }
  }
  stages {
    stage('Run maven') {
      steps {
        container('maven') {
          sh 'mvn -version'
        }
      }
    }
  }
}

Manual Plugin Installation

Jenkins security best practices include installing plugins manually through the UI by uploading a *.hpi file. You can access this menu in the UI by going “Manage Jenkins” → “Manage Plugins” → “Advanced” → “Deploy Plugin”:

We heavily discourage installing plugins outside the Jenkins plugin manager. Non-verified plugins could contain malicious behavior, and installing them could lead to security breaches and system compromise.

We suggest verifying the package’s source or downloading it from the official plugin repository if required.

Safe HTML Rendering

The default behavior in Jenkins is text rendering – this means that all the descriptions are treated as text, and all HTML tags are escaped for XSS (Cross-site scripting) protection. You can also configure HTML rendering with the help of OWASP Markup Formatter, which implements an HTML subset without risky tags such as <script>. After installing this plugin, you should configure “Manage Jenkins” → “Configure Global Security” → “Markup Formatter” → “Safe HTML”.

Secure Credentials

The Jenkins core application doesn’t give reasonable solutions for limiting credentials exposure for specific users and builds, but several popular plugins are doing that exceptionally.

Each added credential into Jenkins is defined either as “Global” – makes it available for Jenkins, nodes, items, all child’s items, basically everything, or as “System” – makes it available for Jenkins and nodes only. In addition, you can create “Domains” for credentials which helps to organize and access them.

Creating credentials in “Domains” doesn’t give any security benefits and leaves them with the same amount of exposure – means any build can access these credentials. For example, developers who modify and commit to a Jenkinsfile in an SCM platform could exfiltrate sensitive secrets without accessing Jenkins. This article demonstrates some of the methods attackers could use to exploit this concept.

To counter it, we can use the popular Folders plugin to define credentials under specific folders, which would be accessed only by the pipelines under that folder. To use this feature, we will create a new folder (through “New Item” → “Folder”) or use an existing one, go to “Credentials” → “Folders”, and create our credentials there.

To complete it, we’ll also want to limit users from accessing these pipelines. For that, we can use the aforementioned Role-based Authorization plugin by creating authorization roles for that folder using regular expressions.

Audit Logs

Plugins like Audit Trail allow writing or sending logs to a remote server (Syslog server or Elastic Search). Enabling this feature allows investigating security incidents or creating anomaly rules to detect malicious activity and prevent breaches. After you install the plugin, you should go to “Manage Jenkins” → “Configure System” → “Audit Trail” and configure your desired logging method.