The Ultimate Guide to Kubernetes Secrets: Types, Creation, and Management

More and more companies are now running their applications on Kubernetes (aka K8s) than ever before. Due to the rapid adoption of Kubernetes, developers and security teams need to focus more on the security of Kubernetes since K8s provides various mechanisms to store sensitive data.

Now consider all the credentials your applications require — database passwords, API tokens, SSH keys, and certificates. In Kubernetes, these are all secrets that require proper management. If not handled correctly, they are sitting ducks for threat actors attempting to get into your systems.

Containerized environments bring their own set of security challenges. Secrets are often found directly within container images, infrastructure code, or even application code. Access credentials are shared across multiple services among teams. Developers sometimes ship code with default configurations that leave secrets out in the open. Moreover, as the cluster grows it becomes increasingly difficult to track who has access to which secrets.

In this blog post, we will discuss everything you need to know about Kubernetes secret management. We will start with discussing the basic concepts and dive into more complex concepts like creating custom secret policies.

What are Secrets in Kubernetes (K8s)?

Kubernetes stores and manages many forms of sensitive data, from simple API keys to complex certificates. That’s where secrets come in. Secrets are special objects in Kubernetes that are intended to store and manage sensitive information that your applications need to run.

Kubernetes stores secrets in etcd – the database of the cluster, as base64 encoded strings by default. Though this encoding delivers a way to deal with binary data and special characters, it is not encryption. It’s really about moving your sensitive data into a format that Kubernetes can easily handle.

Now, if you have used Kubernetes before, you might ask, why not use ConfigMaps? After all, they’re for storing configuration data, too. The primary distinction comes from security. ConfigMaps are used to store the environment variables as plain text (in most cases), and because these ConfigMaps are often version-controlled, they become a security risk (due to secrets getting pushed to GitHub/GitLab). While Secrets are designed for storing sensitive data, they are base64-encoded by default when stored in etcd. However, secrets can be enhanced with additional security features. They work with encryption providers, have memory-only storage options, and give you fine-grained access control.

Keep in mind, though, the default setup isn’t foolproof. Unless you add more protection, anyone who has access to etcd can read your secrets. That’s why many teams use additional tools like encryption providers or external secret stores.

Also Read: Encryption vs Encoding: The Real Deal

Types of Kubernetes Secrets

There are various Kubernetes secret types, depending on what you want to store and how you will be using your Kubernetes deployment.

Let’s start by discussing the most basic Kubernetes secret types – Opaque secrets. These are your default secrets, which can store any kind of data. Most teams use these to store database passwords, API keys, or other custom data their apps require.

Docker registry secrets allow your nodes to pull images from private registries. You save your Docker Hub or AWS ECR credentials as secrets rather than placing them in plain sight. Kubernetes automatically uses these credentials when pulling private images.

For basic user authentication, auth secrets are used. They store usernames and passwords your apps may need for HTTP basic authentication. Think of internal tools that need simple password protection.

SSH authentication secrets store SSH credentials. You can use these when you need your pods to clone private git repositories or securely connect to other servers.

Service account tokens are a special kind of secret that uses BoundServiceAccountToken by default since Kubernetes 1.24, with legacy automatic token creation being disabled.  Each pod receives one by default, and they’re used to communicate with the Kubernetes API. Bootstrap tokens are similar but temporary; they enable new nodes to securely join your cluster.

How to Create Secrets in Kubernetes

Kubernetes provides the means to manage secrets, but it must be balanced. Before we start to dive into creating secrets, we need to understand how RBAC (Role-Based Access Control) fits into the process.

Role-Based Access Control setup is often your first line of defense. You want to ensure that only the correct person or process has access to read or update secrets. One common mistake that teams make is granting broad access to secrets across namespaces. Instead, you scope the access to the secret to a particular namespace and limit who is allowed to create or alter them.

Let’s start with setting up RBAC rules. This role will let us manage secrets in a specific namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-manager
  namespace: cycode-app
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "create", "update"]
  resourceNames: ["app-credentials", "api-keys"]

Here, we’re creating a Role named secret-manager in the cycode-app namespace. The rules allow three basic operations on specific secrets named ‘app-credentials’ and ‘api-keys’: getting, creating, and updating them. The rules allow three basic operations on secrets: getting, creating, and updating them. Notice we didn’t include delete – this is a common practice to prevent accidental secret deletion.

Now for creating secrets, kubectl gives us a straightforward way (using CLI):

kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=super-secret-pwd

# creating secret from files
kubectl create secret generic tls-cert \
  --from-file=tls.crt \
  --from-file=tls.key

The first command creates a secret named db-creds with two key-value pairs. The second creates a secret from TLS certificate files – useful for HTTPS setup.

Note: For running the above command, you will need to install kubectl (Kubernetes command-line tool).

If you prefer YAML (many teams do for version control), here’s the format:

apiVersion: v1
kind: Secret
metadata:
  name: api-keys
type: Opaque
stringData:
  api_key: api-key
  api_secret: api-secret

We’re using stringData instead of data here – this lets us write the values as plain text. Kubernetes handles the base64 encoding for us. However, be cautious with this approach – never store actual secret values in YAML files that could be committed to version control. Instead, use this format as a template and inject the real secrets through secure methods like CI/CD pipelines or external secret management tools.

For existing secrets or env files:

# read from the env file
kubectl create secret generic prod-creds --from-env-file=.env.prod

# from existing secret in another namespace
kubectl get secret myapp-secret --namespace=dev -o yaml | \
  sed 's/namespace: dev/namespace: prod/' | \
  kubectl apply -f -

The first command reads a .env file (.env.prod file is the file name and can be changed as per the needs) and creates secrets from it. The second shows how to copy secrets between namespaces (handy for promoting configurations across environments).

Note: Remember to never commit secrets to version control. Always use a secure way to pass secrets to your cluster.

How to Scan for Secrets in Kubernetes

Finding exposed or mismanaged secrets in your Kubernetes clusters is crucial for security. Let’s look at different tools and approaches to scan and discover secrets across your clusters.

Kubernetes audit logs are your starting point. They track who accessed which secrets and when. This is how it can be done:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata  # Logs all metadata about secret operations
  resources:
  - group: ""
    resources: ["secrets"]

Note: The example shared is only logs metadata about secret operations. Additional configurations are needed to capture full details securely.

SAST (static analysis) of the manifests in the K8s is another important protection layer. This can be achieved by using Cycode IaC or an open-source tools like kube-score to scan the K8s YAML files. 

# using kube-score for manifest analysis
kube-score score manifest.yaml --check-container-security

# using kubesec for security scanning
kubesec scan deployment.yaml

These tools (kube-score and kubesec) can detect:

  • Hardcoded credentials in ConfigMaps
  • Base64-encoded secrets in plain YAML
  • Environment variables containing sensitive data
  • Volume mounts exposing secret data

Runtime scanning is even more critical. Using admission webhooks, you can catch secrets during deployment:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: cycode-secret-scan
webhooks:
- name: scan.secrets.security.io
  clientConfig:
    service:
      name: secret-scan-service
      namespace: security
      path: "/validate-secrets"
    caBundle: <base64-encoded-CA-cert>
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["secrets"]
    operations: ["CREATE", "UPDATE"]
  admissionReviewVersions: ["v1"]
  sideEffects: None

 

This example configures a webhook named cycode-secret-scan that points to the secret-scan-service in the security namespace. It validates secrets during creation and updates by calling the /validate-secrets endpoint.

While open-source tools like kube-score and kubesec are a great start for manifest and runtime scanning, specialized platforms take security a step further. Cycode integrates secret scanning throughout the software development lifecycle—from pre-commit hooks and pull requests in your version control system to runtime monitoring in production clusters. This proactive approach helps catch secrets before they ever enter your Kubernetes environment while also detecting exposed or mismanaged secrets in existing deployments.

How to Manage Your Kubernetes Secrets

Secrets management in Kubernetes is much more than just how to store and retrieve sensitive data. In a growing cluster, you need solid strategies to manage K8s secrets between teams, environments, and applications. The following is the essential information about Kubernetes secret management that you need to build a secure and scalable system.

Secret Rotation Strategy

Companies usually take a simple approach to secrets – create them once and forget about them. This results in stale credentials that could be compromised without the knowledge. Intelligent secret rotation strategies are changing this. To implement this effectively in Kubernetes, you can use native tools like kubectl rollout restart to refresh pods after secret updates, or integrate with external secret management platforms like HashiCorp Vault for dynamic secret generation and rotation. The key is to make rotation automated and predictable so you can keep the secrets secure in your apps without having to handhold them all the time.

External Secret Stores

Native K8s secrets are great for simple use cases, but as your needs continue to mature, an external secrets store like HashiCorp Vault becomes extremely useful. These tools do not only encrypt secrets; they become your security command center. They provide deep insights into secret utilization, automate complex rotation patterns, and help meet stringent compliance requirements. Also, they serve as a ‘single source of truth’ to ease the management of secrets and sensitive information throughout your entire infrastructure.

Role-Based Access Control

RBAC is the foundation of secure secret management. Without proper RBAC configurations, sensitive data will be exposed to unauthorized users or services. The key is to implement granular access control based on the principle of least privilege. This dictates that all applications and users have as few secrets as possible. Regular RBAC audits ensure those permissions are up-to-date and correct.

Environment-Specific Management

Development environments might prioritize ease of use, while production environments demand stricter security controls. This balance requires careful planning. Many teams use different secret stores for different environments, implement varying rotation schedules, and maintain separate access policies. This separation helps prevent development credentials from accidentally making their way into production systems.

Monitoring and Auditing

Monitoring the logs regularly can help you catch unusual access patterns or potential security breaches in their early stages. This involves logging who is accessing K8s secrets when they are being accessed, and how they are being used. Additionally, audit logs must record all operations related to secrets for potential security reviews and compliance. Implementing proper monitoring also helps teams to know how secrets are being used throughout their applications.

Creating Your Own Secret Policy in Kubernetes

Secret policies in Kubernetes help your team maintain consistent security standards. Kubernetes comes with basic secret management capabilities, but custom policies allow you to enforce specific rules that align with your security requirements and compliance needs.

Before you start writing a policy, remember policies exist at different levels. Some policies govern the creation and storage of secrets, while others determine who can access them and how they are used. You need a policy framework that covers all of these points but also can be flexible enough to allow your teams to conduct their business.

At the heart of your secret policies are Custom Resource Definitions (CRDs). It augments Kubernetes native functionality to allow you to define and enforce custom rules. CRDs can be seen as a kind of template that defines what a valid secret for your organization is. They can define requirements such as required encryption, rotation schedules, and naming conventions.

Here’s a basic CRD that enforces secret rotation and encryption requirements:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: secretpolicies.security.k8s
spec:
  group: security.k8s
  names:
    kind: SecretPolicy
    plural: secretpolicies
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                rotationDays:
                  type: integer
                  minimum: 1
                requireEncryption:
                  type: boolean

Open Policy Agent (OPA) Gatekeeper takes these CRDs further by actively enforcing your policies. It acts as a security guard, checking every secret against your rules before allowing it into the cluster. This prevents non-compliant secrets from being created or modified, ensuring consistent security standards.

Consider this Gatekeeper constraint template that validates secrets:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: secretvalidation
spec:
  crd:
    spec:
      names:
        kind: SecretValidation
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package secrets
        
        violation[{"msg": msg}] {
          input.review.kind.kind == "Secret"
          not input.review.object.metadata.annotations["encryption"]
          msg := "Secrets must specify encryption type"
        }

Remember that policies shouldn’t just block operations – they should guide users toward secure practices. 

Best Practices for Kubernetes Secrets Management

When working with secrets in Kubernetes, following security best practices is crucial. Let’s explore the key practices that can help secure your cluster’s sensitive data.

Encryption at Rest

Kubernetes uses etcd as a default data store and stores the secrets as base64-encoded strings, which is unsuitable for sensitive data. Use encryption providers to enable encryption at rest. This adds another layer of security between your secrets and those who might attack them. Rotate encryption keys regularly and follow secure key management best practices.

Implementing Least Privilege Access

RBAC access control should adhere to the principle of least privilege. Secrets should provide applications and users with the minimal access required. Regularly review and audit access permissions. Prune unused permissions and old access rules. Also, use access tokens that are timed for a few days of needed access.

Regular Secret Rotation

Use automated secret rotation so that the impact of potential exposures is reduced. Establish explicit rotation schedules according to secret sensitivity and compliance thresholds. Implement a scheduled task to rotate (either manually or automatically) at some specified interval, like monthly, quarterly, etc.

External Secret Management

For external secret management solutions, consider using solutions like HashiCorp Vault or AWS KMS. These tools offer advanced features such as centralized audit logs, fine-grained access control, and automated secret rotation. They are also better at encryption and can integrate with existing security infrastructure.

Monitoring and Alerting

Set up comprehensive monitoring of secret access and changes. Watch for unusual patterns like sudden spikes in access attempts or off-hours activities. Create alerts for suspicious behaviors and unauthorized access attempts. Regular monitoring helps detect potential security incidents early.

CI/CD Pipeline Security

Secure your secrets throughout the deployment pipeline. Never store raw secrets in git repositories. Use tools like Cycode that enable secure secret management in your CI/CD workflows. Implement checks to prevent secret leakage during builds and deployments.

Regular Audits and Reviews

Regularly audit your secret management processes. Find unused secrets that need to be cleaned up. Confirm that access controls are functioning as intended. Practice your rotation and recovery procedures. Document and take action on any findings quickly.

Secret Recovery Procedures

Maintain clear procedures for secret recovery in case of emergencies. If a secret has been exposed, document what the process is for rotating it. Make backup processes for key secrets. Regularly test these processes to make sure they would function when required.

Conclusion

Securing Kubernetes secrets isn’t just about creating and storing them – it’s about building a complete security strategy. From proper storage and access controls to regular monitoring and rotation, each aspect plays a vital role in protecting your cluster’s sensitive data.

Secrets management, secrets scanning and secrets detection tools like Cycode bring automation and security to your Kubernetes environment. They provide continuous secret scanning across your software development lifecycle, help prioritize risks, and integrate directly into developer workflows. This proactive approach catches potential security risks before they become real problems.

As the adoption of Kubernetes grows, the importance of proper secrets management can’t be overstated. Start with the basics – encryption, RBAC, and monitoring. Then, gradually build up to more advanced practices like automated rotation and external secret stores.

For more information about securing your Kubernetes clusters and preventing secrets exposure, visit cycode.com.

About Cycode

Cycode is the leading Application Security Posture Management (ASPM) providing Peace of Mind to its customers. Its Complete ASPM platform delivers safe code, faster. That means stopping application risk before it starts, reducing developer productivity tax and lowering the total cost of ownership.

The platform can replace existing application security testing tools or integrate with them while providing cyber resiliency through unmatched visibility, risk driven prioritization and just in-time remediation of code vulnerabilities as scale. Cycode’s Risk Intelligence Graph (RIG), the ‘brain’ behind the platform, provides traceability across the entire SDLC through natural language.