gotchabashkubernetesCritical
Secrets management: avoid storing plaintext secrets in Kubernetes Secrets
Viewed 0 times
secretsbase64encryption at restsealed secretsexternal secrets operatorvaultetcd encryptiongitopskubesealaws secrets manager
Problem
Kubernetes Secrets are only base64-encoded, not encrypted. Anyone with read access to etcd or the right RBAC permissions can decode all secrets in the cluster.
Solution
Layer defense-in-depth for secrets:
At minimum: restrict RBAC access to secrets, enable audit logging, and never print secrets in logs.
- Enable etcd encryption at rest (EncryptionConfiguration in kube-apiserver)
- Use Sealed Secrets (Bitnami) to store encrypted secrets in Git safely
- Use external secret stores (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) with the External Secrets Operator
# Seal a secret for GitOps (Sealed Secrets)
kubectl create secret generic db-creds \
--from-literal=password=supersecret \
--dry-run=client -o yaml | \
kubeseal --format yaml > sealed-db-creds.yaml
# Commit sealed-db-creds.yaml to Git safelyAt minimum: restrict RBAC access to secrets, enable audit logging, and never print secrets in logs.
Why
Base64 is an encoding, not encryption — it is trivially reversible. Without etcd encryption at rest, secrets are stored in plaintext in the etcd database. Additionally, anyone with
kubectl get secret permission can read them.Gotchas
- RBAC: separate roles for pods that mount secrets (no
geton secrets) vs operators who manage secrets - kubectl get secret -o yaml | base64 --decode reveals the value — audit who has this access
- Sealed Secrets encryption is tied to the cluster's controller keypair — rotating keys requires re-sealing all secrets
- External Secrets Operator syncs external secrets into Kubernetes Secrets on a schedule — still ends up as a k8s Secret in etcd
- Never commit raw Kubernetes Secret YAML to Git — use Sealed Secrets, SOPS, or a secret management tool
Context
Managing sensitive credentials and configuration in Kubernetes clusters
Revisions (0)
No revisions yet.