Reverse Proxies, Logs, and Debugging Production Failures

The short answer

Production debugging is the work of locating the last boundary that behaved as expected and the first boundary that did not.

A reverse proxy is one such boundary. It accepts a client-facing request on behalf of an upstream service and may route, secure, limit, cache, or balance that request. If it returns 502 Bad Gateway, the browser reached the proxy, but the proxy did not obtain a response it could use from an upstream.

The status narrows the search; it rarely names the root cause. Logs record events, metrics show numeric behavior over time, traces connect operations along a request path, and request identifiers help correlate evidence. Debugging becomes reliable when each hypothesis is tested at the boundary it describes.

The complete request path revisited

The ten lessons now fit into one qualified path:

Do not diagnose this as one server. Each arrow can cross a process, network, trust, deployment, or ownership boundary. Some requests skip several stages; a service worker or CDN may answer early, and a static page may never reach application code.

What a reverse proxy is

RFC 9110 calls a gateway—also known as a reverse proxy—an intermediary that appears as an origin server on the client-facing connection, then translates or forwards requests to one or more inbound servers.The proxy becomes an HTTP participant rather than a transparent wire.¹

From the browser's perspective, the proxy can be the server that completes TLS and produces the HTTP response. Behind it, the proxy opens or reuses another connection to an upstream application.

Nginx is one software product that can perform this role. Its proxy_pass directive names the protocol, address, optional port, and URI mapping for the proxied server.Nginx can run on a physical host, VM, container, or managed environment; it is not itself an EC2 instance or data center.²

Reverse proxy versus application server

The reverse proxy handles client-facing transport and HTTP infrastructure concerns. The application server executes product behavior.

A proxy may:

• terminate TLS;
• select an upstream from the host or path;
• distribute requests among instances;
• enforce size, rate, or timeout policy;
• add or remove request fields;
• compress or cache responses;
• serve static content;
• generate an error without application code running.

The application commonly performs route handling, validation, identity and permission checks, business logic, and data access.

These responsibilities can coexist in one process or managed product. A framework application can serve static files, while a proxy can apply application-adjacent policy. The names describe roles, not an absolute software boundary.

Downstream client and upstream service

Upstream and downstream are relative to the component being discussed. For a reverse proxy:

• the browser or preceding gateway is downstream on the client-facing side;
• the application service the proxy calls is upstream.

From a database client's perspective, the database may be downstream in a data-flow diagram, even though proxy documentation uses “upstream” for the application server. Always name the reference point.

Why reverse proxies are used

One stable public endpoint can hide changing application instances. The proxy can centralize TLS certificates, route /api and /assets differently, protect private application ports, attach forwarding context, and balance requests.

Load balancing is the act of selecting among eligible destinations. Reverse proxying is the broader role of receiving and forwarding on behalf of upstream services. Products often overlap: Nginx documents HTTP load balancing as part of its reverse-proxy implementation.The terms are related, but not universally identical across products and network layers.⁴

Not every application has a separately managed reverse proxy. A platform ingress, cloud load balancer, CDN, gateway, or application server may fill some or all of the role.

What 502 Bad Gateway means

RFC 9110 defines 502 Bad Gateway as a gateway or proxy receiving an invalid response from an inbound server it accessed while attempting to fulfill the request.In common architecture language, that inbound server is the proxy's upstream.¹

Depending on proxy behavior and configuration, investigations may include:

• no process listening at the configured upstream host and port;
• a refused or blocked connection;
• the application listening only on another interface;
• HTTP sent to an HTTPS port or the reverse;
• upstream TLS identity or trust failure;
• the upstream closing early or producing unusable protocol data;
• a timeout classified by that implementation as a bad gateway;
• all selected upstream instances being unusable.

The 502 proves that an HTTP-speaking gateway or proxy produced a response. It does not prove that the application process handled the request, and it does not mean every backend is down. Identify the responder and inspect its exact error evidence.

Other useful failure categories

• DNS resolution failure: the client did not obtain usable destination information.
• Connection refused: a destination actively rejected the transport connection, commonly because nothing accepts it or policy rejects it.
• TLS certificate error: connection setup reached TLS identity validation, but the requested identity was not accepted.
• 404 Not Found: the responding HTTP component did not find or disclose a current representation. A proxy or application may generate it.
• 401 Unauthorized: under HTTP semantics, valid authentication credentials are absent for the target resource.
• 403 Forbidden: the server understood the request and refuses to fulfill it.
• 500 Internal Server Error: a server encountered an unexpected condition preventing fulfillment.
• 502 Bad Gateway: a gateway or proxy received an invalid upstream response.
• 503 Service Unavailable: the responding server is temporarily unable to handle the request, commonly due to overload or maintenance.
• 504 Gateway Timeout: a gateway or proxy did not receive a timely response from a needed upstream.

RFC 9110 defines these HTTP categories.The status plus the identity of the responder locates a boundary; neither automatically reveals the underlying defect.¹

Access logs

An access log records requests handled by an HTTP server or proxy. Useful fields can include time, method, path, status, response size, duration, upstream address, upstream status, and request identifier.

Nginx's HTTP log module writes request logs in a configurable format and records the request in the location where processing ends, which can differ after an internal redirect.A configured field must be present before you can rely on it during an incident.³

An access log can prove that the request reached this component and show the response it emitted. It does not necessarily explain why application code made a decision or why an upstream failed.

Application logs

Application logs record discrete events from business and technical execution: route selection, validation rejection, dependency timeout, transaction conflict, or classified exception.

Useful logs include time, environment, service and version, severity, event name, safe context, and a correlation identifier. They should not contain plaintext passwords, session identifiers, access tokens, private keys, database connection strings, or complete payment data.

OWASP recommends consistent event fields, interaction identifiers, sanitization, access control, retention policy, and explicit exclusion of sensitive values.Logs are security-sensitive data and need protection against reading, tampering, loss, and unbounded retention.⁷

An exception stack is evidence, not always root cause. It may show where a timeout surfaced rather than why the dependency stopped responding.

Metrics

Metrics are numeric measurements aggregated or sampled over time: request rate, error rate, latency distributions, queue depth, memory use, active connections, or database-pool saturation.

They answer “when did behavior change?” and “how broad is the impact?” A sudden rise in proxy 502 responses alongside zero healthy upstreams suggests a different boundary from one user's isolated 401.

Aggregation loses individual-request detail. A healthy average can hide a slow tail or one failing route. Histograms, labels, and alert thresholds need careful design to remain useful and affordable.

OpenTelemetry treats metrics, logs, and traces as distinct telemetry signals that describe a system from different angles.No single signal replaces the others.⁵

Traces

A trace represents the path of one request or operation through instrumented components. Each span records one segment, such as proxy ingress, application handler, database query, or external API call.

Traces help answer where time was spent and which downstream call failed. Missing spans are ambiguous: the request may not have reached the component, instrumentation may be absent, context may have been lost, or sampling may have excluded it.

Tracing must also avoid sensitive attributes and uncontrolled cardinality. Instrumentation and sampling policy determine what evidence exists.

Request and correlation identifiers

A request identifier is a value used to connect records for the same interaction. A proxy can accept or generate one, include it in an upstream request, and return it in a response. The application can attach it to logs.

Distributed tracing uses propagated context such as trace and span identifiers so operations across process and network boundaries can belong to one trace. OpenTelemetry describes context propagation as the mechanism that carries this correlation between services.A local request ID and a trace ID can coexist; their trust and generation rules should be explicit.⁶

Do not trust arbitrary client-provided identifiers as authoritative without validation. Do not place secrets or user data inside identifiers merely because they appear in headers and logs.

A layered debugging method

1. Reproduce reliably. State what action fails and whether it fails every time.
2. Record the observation. Capture exact URL, method, time with zone, environment, request context, response status, and safe request ID.
3. Confirm DNS. Verify which destination information the failing client receives.
4. Confirm reachability. Determine whether a connection reaches the expected address and port where the environment permits that test.
5. Confirm TLS. Inspect service identity, trust, validity, and protocol errors.
6. Inspect HTTP evidence. Record status, fields, content, timing, and any responder clues.
7. Identify the responder. Decide whether the response came from a CDN, load balancer, reverse proxy, application, or another layer.
8. Inspect infrastructure records. Use proxy or platform access and error logs at the same time and request ID.
9. Confirm process and listener. Verify that the intended version runs and listens on the configured interface and port.
10. Inspect application evidence. Follow correlated logs or traces through routing and dependencies.
11. Check required dependencies. Test database, cache, queue, and external-service boundaries with safe, scoped diagnostics.
12. Change one hypothesis at a time. Preserve before-and-after evidence.
13. Verify from the public boundary. A working internal endpoint alone does not prove the user's path is repaired.

This sequence is not a rigid command list. Environments expose different diagnostics and permissions. Skip a test only when stronger existing evidence already proves that boundary.

Debugging tools at a high level

• The browser Network panel records requests, responses, timing, initiators, redirects, and browser policy failures. Chrome documents these inspection workflows in its DevTools Network guide.Browser console errors alone do not show every network boundary.⁸
• curl can issue an HTTP request outside application UI code and show response fields or verbose connection details. The official manual documents --head, --verbose, output, timeout, and protocol options.Use a safe method and approved target; verbose output can expose credentials.⁹
• DNS lookup tools show resolver answers.
• Process and socket tools show whether the intended program is running and listening.
• Platform dashboards show deployment state, health checks, restarts, and routing targets.
• Proxy access and error logs identify the response and upstream attempt.
• Application logs and traces follow execution inside the service.
• Database connectivity and plan tools test the data boundary with least privilege.

An illustrative safe request against the reserved documentation domain is:

curl --verbose https://example.com/

Verbose output can include headers and connection details. Do not paste real authorization fields, cookies, or secrets into shared incident reports.

Capstone failure scenario

Common debugging mistakes

• changing DNS, proxy configuration, application code, and firewall rules together;
• calling an HTTP status the root cause rather than a symptom at a boundary;
• comparing logs from the wrong environment, instance, or time zone;
• looking only at frontend console output;
• assuming every server error came from application code;
• treating a running process as proof of readiness;
• logging credentials or copying them into tickets;
• redeploying repeatedly before preserving evidence;
• testing only a private endpoint and declaring the public path fixed;
• stopping at the first plausible explanation instead of verifying it.

One controlled change may still affect several systems. The discipline is to state the hypothesis and define which evidence would confirm or reject it before acting.

Common misconceptions

Knowledge check

Learning-path synthesis

You can now trace a user action through the full system without collapsing it into “the frontend calls the server”:

• the frontend gathers input and decides whether a request is needed;
• the URL identifies a scheme, host, path, and query;
• DNS supplies destination information;
• TCP or QUIC provides transport while TLS protects HTTPS and HTTP defines the exchange;
• an API contract gives paths, fields, and representations application meaning;
• backend routing, validation, identity, permission, and business rules execute;
• caches, databases, and other services participate only when the architecture requires them;
• session or token evidence connects authenticated state to a request;
• the deployed runtime, configuration, process, and health boundaries make the application reachable;
• proxies and observability signals expose where production traffic succeeds or fails;
• the response returns, and the browser parses, loads, executes, lays out, and paints.

The durable model is a chain of responsibilities with optional branches. Debugging means proving the chain one boundary at a time.

When that chain reaches its capacity limit, How Scalable Systems Actually Work begins with requirements and measurements before adding more infrastructure.

What to learn after this path