Helm has become the package manager of choice for deploying and managing applications among the Kubernetes orchestration tools. Yet, this simplified standard also comes with an essential responsibility, i.e., the secure handling of sensitive data. Secrets management in Helm is not merely a capability but a necessity to secure production-grade Kubernetes deployments.
In practice, Helm charts frequently require the use of sensitive data such as API keys, database passwords, OAuth tokens, and TLS certificates. If not handled correctly, these secrets will act as the weak link in your Kubernetes security chain. Helm does an excellent job of making application deployments repeatable and configurable, but it doesn’t focus on security by default when storing sensitive data. This is where the need for proper secrets management strategies comes into play.
In this guide, we’ll cover the different strategies for managing secrets in Helm charts, from the basic provision of Kubernetes secrets to more advanced solutions such as external secrets operators. Whether you are familiar with Helm or just want to improve the security of your existing charts, this guide will help you with actionable strategies and best practices for securely managing your secrets.
Why You Should Deal with Secrets in Helm
The struggle with proper secrets management in Helm is more than a matter of good security hygiene. A key overall aspect that you should consider when developing your application is that it will directly affect your app’s security posture, operational efficiency, and application compliance requirements. Let’s explore why this is so important.
Handling sensitive data in Kubernetes is a challenge, especially with Helm. While both traditional and containerized applications deal with multiple environments, Kubernetes introduces unique challenges for secret management. The distributed nature of Kubernetes, with workloads spread across clusters, namespaces, and environments, creates additional complexity in how secrets need to be stored, transmitted, and accessed.
Let’s take a common example: you’re deploying a microservices application that requires database credentials, external service API keys, and SSL certificates. With improper secrets management:
- Your values.yaml file can contain clear text credentials, which are added to the version control.
- Development teams might exchange secrets over insecure channels such as email or chat.
- Multiple environments (dev, staging, production) might share the same secrets.
Common Security Challenges in Helm Charts
Secret management in Helm charts comes with a few big challenges that organizations often face. Such issues arise from both technical shortcomings and general operating practices that place convenience over security.
The most common problem is using hardcoded secrets in values files. During development, organizations often rely on hardcoded credentials for the sake of rapid iteration, and such practices often make their way into production environments. This poses a major security risk because values files are typically stored in version control, and thus, sensitive data may fall into the hands of anyone with access to a repository.
However, even when teams understand the risks associated with hardcoded secrets, they still often fall into the Base64 encoding trap. There is often a misconception that Base64 encoding is a way of securing the information, but it is just a form of encoding that can be reverted easily. This provides a false sense of security, as teams think their secrets are protected when they’re not.
Now, it gets more complex if the applications deploy in multiple environments. Most organizations have multiple environments (development, staging, production) that will have different security needs. Writing secrets across these environments creates a number of challenges.
# Example of problematic multi-environment secret handling
secrets:
{{- if eq .Values.environment "production" }}
apiKey: "production-key-here"
{{- else }}
apiKey: "dev/staging-key-here"
{{- end }}
Such an approach exposes sensitive data in logs by default, creates maintenance overhead, and increases the risk of inadvertently using development credentials in production.
Access control adds another dimension of complication. Helm requires a careful approach to deal with Kubernetes RBAC (Role-Based Access Control). RBAC should enforce the principle of least privilege, but teams instead create broad and permissive roles for the sake of getting things working quickly (which can compromise security). For instance:
# Overly permissive RBAC configuration rules: - apiGroups: [""] resources: ["secrets"] verbs: ["*"]
This configuration allows secrets to be accessed freely, going against the least privilege principle.
Also Read: Kubernetes Security Best Practices: 8 Tips to Secure K8s
How to Manage Secrets in Helm
Sensitive data must be handled properly in Helm deployments, with a balance between security and operational efficiency. There are many solutions, but each of these solutions has different tradeoffs in terms of overall complexity, security, and ease of use.
Native Kubernetes Secrets
The simplest one uses built-in secrets management in Kubernetes. This approach retains secrets in the form of base64-encoded (not encrypted) strings in the Kubernetes cluster itself. Although easy to implement, this scheme offers limited security capabilities. The passwords are available to anyone who has access to the cluster, and they are only Base64 encoded.
# Create a secret using kubectl
kubectl create secret generic db-creds \
--from-literal=username=insecure-admin \
--from-literal=password=insecure123
# Reference in Helm template
apiVersion: v1
kind: Deployment
metadata:
name: {{ .Release.Name }}
spec:
template:
spec:
containers:
- name: app
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: password
Helm Plugin Secrets
The helm-secrets plugin extends Helm’s built-in functionality by supporting encrypted storage of secrets. It uses Mozilla SOPS (Secrets OPerationS) to enable file-level encryption of sensitive data. This solution strikes a balance between developer-driven workflows and security needs by allowing you to version control encrypted secrets alongside your Helm charts.
Together with cloud provider KMS (Key Management Services), helm-secrets uses a strong mechanism for teams wishing to remain secure while ensuring they can get all the benefits of Helm as a package manager.
# secrets.yaml (encrypted) secrets: dbPassword: ENC[AES256_GCM,data:abcd...]= apiKey: ENC[AES256_GCM,data:wxyz...]= helm secrets install myapp ./mychart -f secrets.yaml
Note: SOPs are not plug-and-play solution but require proper key management setup first.
External Secrets Operator
The External Secrets Operator integrates external vaults and secret management systems for enterprise-grade secrets management. This breaks the application’s concern of secret storage from the secret consumption while providing teams the ability to grow on an existing security infrastructure and maintain Kubernetes-native workflows.
The operator pulls secrets from external providers into Kubernetes secrets, providing automatic rotation and centralized audit logging. This is especially useful for organizations that are required to adhere to complex security compliance policies or share consistent secret management across multiple clusters and environments.
Helm Secrets Management Best Practices
Let us discuss some of the best practices that should be followed for effective secret management.
Template Organization and Structure
Secure Helm deployments start at the template level. Your overall structure should contain folders for secrets-related configuration that are separate from the overall chart, thereby maintaining isolation of sensitive and non-sensitive content. The organization you create should follow a hierarchy that makes sense around what you consider the security boundaries of your application.
In addition to just the basic directory structure, the template inheritance hierarchy should be designed to not leak secrets between parent and child charts, especially in cases of complex dependencies. Wherever you automatically upload template files, introduce appropriate validation hooks so that when a misconfigured template file makes its way to production, it can be caught before it is ever deployed.
Secret Lifecycle Automation
From the creation of secrets to their rotation and deletion, automate the entire lifecycle of your Helm deployments. Define automated secret rotation processes and the ability to reinject secrets; this can be done without restarting the various deployments. This requires careful orchestration of rolling updates and application readiness probes.
Use automated validation systems to conduct checks, such as ensuring secrets are present and have a valid format before proceeding with any deployments. Be sure to build your automation with appropriate error handling if secret retrieval fails or values are malformed.
Patterns of Isolation in the Environment
If you must use the various environments, make sure that you design your Helm charts to maintain significant separation between them. Rather than just sticking with simple namespace separation, build validation checks to prevent production secrets from being inadvertently used in lower environments.
Your chart should include hooks that depend on the environment and respond based on whether a secret is stored in the secret store backend (wherever that might be specific to the deployment context).
Access Control Implementation
Create granular RBAC policies based on the least privilege principle. Your charts must be creating the right ServiceAccounts, Roles, and RoleBindings that restrict secret access only to the pods that really need it.
Draft network policies that prevent pod-to-pod and pod-to-service communication of secrets. If you are on Kubernetes < 1.25, enforce Pod Security Policies (PSP) to deny privilege escalation and unauthorized access to secrets via volume mount. For newer versions, configure Pod Security Admission (PSA) with the strictest policies that align with your security needs.
Architecture for Audit and Monitoring
Implement complete auditing in your secret management system. Log all the attempts made to access the secrets (even the failed attempts). Develop systems to monitor and detect anomalies in secret access or modifications.
Your charts must include configurations for audit log aggregation and define clear alert thresholds with reactions to suspicious activity. You can automate responses for possible security events like temporary access revocation or deployment rollbacks.
Security Controls for Distribution
Use secure channels for distributing charts that ensure sensitive configurations are secure and protected. All secrets required to be shipped with charts should be strongly encrypted. Develop automatic validation systems to validate the integrity of charts and their secrets during their distribution.
Also, design your distribution pipeline to support different ways of injecting secrets (deployment environment and security requirements). Ensure you can quickly revoke secret access for everyone across your deployment targets (system and application).
Conclusion
Effective secrets management in Helm represents a critical aspect of secure Kubernetes deployments. Moving beyond basic storage solutions to implement comprehensive secret handling requires careful consideration of security, scalability, and operational efficiency. The tools and practices we’ve discussed, from native Kubernetes secrets to external operators, provide a spectrum of options for organizations to match their security requirements with their operational capabilities.
Organizations must understand that proper secrets management extends beyond tool selection; it requires a holistic approach encompassing secret scanning, template organization, access controls, automation, and monitoring.
For organizations looking to enhance their Kubernetes security posture, Cycode offers advanced security solutions that complement Helm’s native capabilities. Visit Cycode to learn more.
About Cycode
Cycode is the leading Application Security Posture Management (ASPM) providing Peace of Mind to its customers. Its Complete ASPM platform delivers safe code, faster. That means stopping application risk before it starts, reducing developer productivity tax and lowering the total cost of ownership.
The platform can replace existing application security testing tools or integrate with them while providing cyber resiliency through unmatched visibility, risk driven prioritization and just in-time remediation of code vulnerabilities as scale. Cycode’s Risk Intelligence Graph (RIG), the ‘brain’ behind the platform, provides traceability across the entire SDLC through natural language.
