Backend says 200, Caddy says 503: the health check trap

The browser shows 503 Service Unavailable, yet a direct curl to the backend returns 200 OK. The first instinct is to restart the application. Most of the time, though, the problem isn't the application at all; it's how the proxy decides whether the backend is "healthy".

Symptoms

• Going straight to the backend, bypassing the proxy, everything works.
• Every request that goes through the proxy comes back as 503.
• The Caddy logs contain lines along the lines of no upstreams available.

When an active health check is configured, Caddy's reverse_proxy module polls the backend at regular intervals. If the response doesn't meet the expected criteria, the backend is marked "unhealthy" and no traffic is sent to it. Once there are no healthy backends left, the client gets a 503. Even if the application is up and running.

Why does it happen?

In this case, the health check was configured to expect a 4xx response from the backend. Most likely the endpoint being checked once required authentication and returned 401, and the rule was written around that. Later the endpoint was fixed and started returning 200, so the check began to fail. A perfectly healthy backend became "sick" in the eyes of the proxy.

Other causes that lead to the same result:

• The path given as health_uri no longer exists (404) or redirects (301/302).
• The health check endpoint requires authentication.
• health_timeout is shorter than the endpoint's actual response time.
• The health check goes out with a different port or Host header, and the backend rejects it.

Diagnosis

First, see what the proxy sees: call the same path with the same headers the health check uses.

# Directly to the backend, using the health check's path
curl -i http://127.0.0.1:8080/health

# Status code and timing on a single line
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' http://127.0.0.1:8080/health

Then compare it with what the Caddyfile expects:

example.com {
    reverse_proxy 127.0.0.1:8080 {
        health_uri      /health
        health_interval 10s
        health_timeout  3s
        health_status   4xx   # ← if the backend now returns 200, every check fails
    }
}

If the admin API is enabled, you can also query the upstream status:

curl -s localhost:2019/reverse_proxy/upstreams

Fix

Align the expectation with the real response, and design the check deliberately:

reverse_proxy 127.0.0.1:8080 {
    health_uri      /health
    health_interval 10s
    health_timeout  3s
    health_status   2xx
}

A good health check endpoint should:

• Require no authentication, be lightweight and not redirect.
• Show that the application can actually do its job (e.g. a database connection), without running a heavy query on every check.
• Have its expected status code documented. If the endpoint changes, the proxy rule gets updated in the same change.

Lesson

A 503 doesn't mean "the backend is down"; it means "the proxy has no healthy backend to send to". First, take a look: check what the proxy is checking, from the proxy's point of view.