Nginx 502 Bad Gateway: find and fix the upstream failure
A 502 usually means nginx is reachable but the application behind it is not returning a valid response. Diagnose the upstream before restarting anything.
Direct answer
To fix an nginx 502 Bad Gateway, read the nginx error log first and identify the upstream address in the failing request. Then verify that the upstream process or container is running, listening on the configured port or socket, and able to answer directly. Check application logs and memory pressure before changing state. mttrly packages the first-pass evidence collection as its read-only server-errors recipe; any restart, container restart, or nginx reload is a separate action that waits for explicit human approval.
What you see
A browser only shows “502 Bad Gateway,” but the nginx error log normally narrows the failure to one of a few concrete patterns:
nginx error log: “connect() failed (111: Connection refused) while connecting to upstream”
nginx error log: “upstream timed out (110: Connection timed out) while reading response header from upstream”
nginx error log: “no live upstreams while connecting to upstream”
The application service is failed, inactive, crash-looping, or listening on a different port
A Docker container is Exited or unhealthy while nginx itself remains active
502 errors started immediately after a deploy, configuration change, or resource spike
How to diagnose an nginx 502 manually
Work from the proxy toward the application. The first seven steps are read-only; reload or restart only after the evidence identifies the failed layer.
Confirm the response really comes from nginx
Capture the response headers and timestamp so you can match the request with the correct log entry.
curl -i https://example.com/health
Read the nginx error log around the failure
The error line usually names the upstream address and distinguishes connection refused, timeout, and malformed response failures.
tail -n 150 /var/log/nginx/error.log journalctl -u nginx -n 150 --no-pager
Find the configured upstream
Check whether nginx proxies to a TCP port, Unix socket, FastCGI service, or named upstream group.
nginx -T 2>/dev/null | grep -nE 'upstream|proxy_pass|fastcgi_pass'
Test the upstream directly
Bypass nginx. A refused connection means nothing is listening; a timeout means the process or its dependency may be stuck.
ss -ltnp | grep ':3000' curl -i --max-time 5 http://127.0.0.1:3000/health
Check the application process or container
Confirm whether the upstream is active, failed, exited, or unhealthy before deciding whether a restart is appropriate.
systemctl status myapp --no-pager docker ps -a
Read application logs
Look for startup errors, dependency failures, database timeouts, missing environment variables, and out-of-memory signals.
journalctl -u myapp -n 150 --no-pager docker logs myapp --tail 150
Check resource pressure and nginx syntax
Memory exhaustion can kill an upstream. A syntax test protects you from applying a broken nginx configuration.
free -h dmesg -T | grep -Ei "oom|killed process" | tail -20 nginx -t
Apply the smallest evidence-backed fix
Fix the application or configuration first. Only then restart the failed upstream or reload nginx, and verify both the upstream and public URL.
systemctl restart myapp systemctl reload nginx curl -i https://example.com/health
Do not reload nginx when nginx -t fails, and do not treat every 502 as a reason to restart nginx. nginx is often healthy and correctly reporting that its upstream is unavailable. A blind restart can hide evidence without fixing the application, database, network, or configuration problem.
How mttrly handles it
mttrly has a diagnostic recipe named server-errors for 500/502/503/504 incidents. The recipe is evidence collection, not an automatic fix: it reads nginx and application logs, checks memory, checks container state, and asks the AI layer to summarize the most likely cause.
You describe the 502; mttrly selects server-errors
From Telegram or an MCP-enabled IDE, a 502/5xx request can invoke the server-errors diagnostic recipe through mttrly_run_diagnostic.
log_tail reads nginx logs (read-only)
The recipe requests the latest 150 nginx log lines to find the upstream address and the proxy error pattern.
log_tail reads application logs (read-only)
It then requests the latest 150 application log lines, where startup failures and dependency errors usually appear.
memory_check and container_status add server context (read-only)
Memory pressure and Docker state help distinguish an application crash, an OOM event, and a healthy process with another dependency failure.
Follow-up checks stay scoped
When needed, the assistant can request separate read-only playbooks such as service_status, check_port, nginx_test, or docker-logs before proposing a remediation.
Example workflow
A realistic MCP session using the actual server-errors sequence (values illustrative):
You -> Claude Code: "Production returns 502 from nginx. Find the failed layer before changing anything." Claude -> mttrly: calls mttrly_run_diagnostic for the 502 symptom mttrly selects server-errors and returns: - nginx log: connect() failed (111) while connecting to upstream 127.0.0.1:3000 - app log: startup failed — DATABASE_URL is missing - memory_check: no memory pressure - container_status: api container Exited (1) Claude -> You: "nginx is reachable. The api container exited because its required DATABASE_URL is missing. Restarting now would produce the same failure. Restore the configuration first; then I can request restart_container." You: restore the missing setting in your deployment configuration Claude -> mttrly: requests restart_container for api mttrly: creates a pending action and waits You: review and approve the action mttrly: inspects and restarts the container, reports its new state, and records the result
Remediation under runtime policy
restart_service — restarts an allowlisted systemd upstream and checks its resulting status. It is approve-required and does not run until a human confirms the pending action.
restart_container — inspects and restarts a named Docker container, then lists container state. It is also approve-required; mttrly does not silently restart crash-looping workloads.
nginx_reload — runs nginx -t before reloading nginx and checks nginx status afterward. The playbook is approve-required, and a failed config test prevents a successful reload.
mttrly does not rewrite application code, restore missing secrets, or edit nginx configuration as part of the server-errors recipe. It gathers evidence and keeps state changes behind approval.
Diagnostics are read-only. Risky fixes follow runtime policy.
- +mttrly investigates on its own: diagnostic recipes and read-only playbooks inspect the server without changing anything.
- +Approval-required fixes create a pending action for confirmation from Telegram, the dashboard, or the IDE. Runtime policy — not the AI — decides the action class.
- +A separately authorized Investigation session can let mttrly_execute_command skip per-action approval only within its server, time, and action limits. That bounded exception is visible and auditable.
- +mttrly does not give an AI client an unrestricted production shell. Diagnostics, configured preauthorization, approval gates, and bounded sessions remain explicit.
These incident guides use the same scoped MCP action layer and can route approval decisions through the Telegram mobile workflow.
Detection tools tell you something is wrong. mttrly is the agent on your server that diagnoses the incident and prepares the fix — through scoped MCP tools, diagnostic recipes, and remediation playbooks. Approval-required actions normally wait for a human; any narrowly configured preauthorization remains bounded and audited. It complements monitoring like Grafana, Datadog, or UptimeRobot; it does not replace it.
When mttrly is not enough
The on-server evidence can identify many 502 causes, but some failures live outside the agent’s control boundary:
- -A managed load balancer, CDN, Kubernetes ingress, or external reverse proxy may be the component generating the 502.
- -Application bugs, broken database migrations, and missing secrets require a code or deployment configuration fix before a restart can help.
- -Network policy or provider firewall failures between nginx and a remote upstream must be fixed in that external control plane.
- -If the server or mttrly agent is offline, mttrly cannot collect live logs or run playbooks; use the provider console or a controlled SSH recovery path.
Frequently asked questions
What does nginx 502 Bad Gateway mean?
It means nginx received the request but could not obtain a valid response from the configured upstream. Common causes are a stopped application, wrong port or socket, timeout, container exit, resource exhaustion, or an application that closes the connection before returning headers.
Does the mttrly server-errors recipe fix a 502 automatically?
No. server-errors is a diagnostic recipe: it gathers nginx logs, application logs, memory state, and container state for AI analysis. Any restart, container restart, or nginx reload is a separate approve-required playbook that waits for explicit human confirmation.
Should I restart nginx first?
Usually not. nginx is often healthy and correctly reporting a failed upstream. Read the nginx error line, test the upstream directly, and inspect the application first. If an nginx reload is actually needed, mttrly uses the approval-gated nginx_reload playbook, which tests the configuration before reload.
Can mttrly inspect a Docker upstream behind nginx?
Yes. container_status reports exited or unhealthy containers, and docker-logs reads a selected container’s recent logs. These checks are read-only. Restarting the container uses restart_container and requires approval.
How is this different from the website-down playbook?
The website-down guide starts broad with reachability, ports, nginx configuration, SSL, and general health. This guide starts from a confirmed nginx 502 and follows the proxy path to the failed upstream, using the server-errors recipe as the product-specific first pass.
Related
Diagnose the next 502 before restarting production
Connect your Linux server, collect read-only evidence from chat or your AI IDE, and keep approval-required remediation behind a visible human gate.