Production broke after deploy: diagnose before you restart or roll back
Correlate the release with recent application logs, health, and memory, then choose a bounded recovery action with explicit human approval.
Direct answer
When production breaks after a deployment, first confirm the timing and collect read-only evidence: recent application logs, external health, service state, and memory pressure. Do not restart or roll back just because the incident followed a deploy. mttrly packages the server-side core as the post-deploy-issue diagnostic recipe: it reads up to 200 recent application log lines, runs the required server healthcheck, performs a dedicated memory check when available, and asks AI to summarize the likely failure. Any restart or rollback remains a separate, approval-gated action.
What you see
A deployment regression may be obvious, or it may look like a generic infrastructure incident. Typical post-deploy signals include:
The deployment reports success, but the public health endpoint returns an error
The application process starts and then enters a crash loop
5xx responses or application exceptions spike immediately after the release
Memory climbs sharply after the new process starts, followed by OOM kills or latency
The site is reachable, but a newly deployed route or background worker fails
Recent logs mention a missing environment variable, incompatible schema, failed import, or binding error
How to diagnose a post-deploy incident manually
Preserve the evidence and move from the outside in. These checks are read-only; the final recovery choice is a separate state-changing step.
Establish the release window and current revision
Record what changed and when. Correlation with a release is useful evidence, but not proof that the release caused the incident.
git -C /var/www/your-app log -1 --format="%h %cI %s"
Check the application health endpoint
Test the same endpoint your load balancer or deploy pipeline uses, including its status code and response body.
curl -i --max-time 10 https://your-app.example.com/health
Read only the recent application logs
Start at the deployment timestamp. The first error after process startup is usually more useful than thousands of repeated failures.
journalctl -u your-app --since "15 minutes ago" -n 200 --no-pager
Check service and container state
Confirm whether the process is running, restarting, exited, or waiting on another dependency.
systemctl status your-app --no-pager docker ps -a --no-trunc
Check memory pressure and recent OOM signals
A release can introduce a leak or raise baseline memory enough to trigger the kernel OOM killer.
free -h ps aux --sort=-%mem | head journalctl -k --since "15 minutes ago" | grep -i oom
Compare configuration and migration assumptions
Look for new environment variables, service dependencies, or migrations that make the new code incompatible with the current production state.
git -C /var/www/your-app diff --name-only HEAD~1..HEAD
Choose the narrowest recovery action and verify
Restart only if the evidence shows a wedged process. Roll back only if the release is the likely regression and its prior state is compatible. Re-run health and log checks afterward.
A deploy timestamp is not a root cause. Blindly restarting can erase useful evidence, and blindly rolling back can make a database or configuration mismatch worse. Capture logs, health, revision, and migration context first.
How mttrly handles it
mttrly ships a diagnostic recipe named post-deploy-issue. It runs a small, code-backed sequence — recent app logs, required server health, and memory — then produces an AI-assisted summary without changing the server.
You describe what broke after the deploy
From Telegram or via mttrly_run_diagnostic in an MCP-enabled AI IDE, select the affected server and ask for post-deploy diagnosis.
Step 1 — log_tail for the app (read-only)
The recipe requests up to 200 recent lines for the app service. This step is optional when the service or its logs are unavailable.
Step 2 — healthcheck (read-only, required)
mttrly checks server uptime, disk, and memory. The recipe cannot complete successfully without this core server-health step; check the public application endpoint separately when needed.
Step 3 — memory_check (read-only)
Memory usage provides evidence for leaks, pressure, and OOM-related failures after the release. This step is optional when unavailable.
AI summarizes the likely regression
The analysis correlates logs, health, and memory. It can narrow the recovery path, but it does not claim to prove an application-code root cause from infrastructure signals alone.
You approve any recovery action separately
A restart_service request or deployment rollback creates its own pending approval. Diagnostics never silently turn into a production change.
Example workflow
A realistic incident flow from an MCP-enabled AI IDE or Telegram (values illustrative):
You -> Claude Code:
"Prod started returning 500s right after the deploy. Diagnose it before changing anything."
Claude -> mttrly:
calls mttrly_run_diagnostic { recipe: "post-deploy-issue" }
mttrly returns:
- log_tail: first exception appeared two minutes after the release
- healthcheck: host uptime, disk, and baseline memory are normal
- memory_check: memory is stable at 48%; no pressure signal
Claude -> You:
"The host is healthy and memory is normal. The failure begins in the new
application logs immediately after the release, so a deployment regression is
more likely than a resource incident. Review the captured rollback target?"
You: request rollback for prod-web-01
mttrly: shows the saved revision and creates a deploy_rollback approval
You: review and approve from Telegram
mttrly: runs the configured rollback path, then you run post-deploy-verification
to confirm health and inspect recent logsRemediation under runtime policy
post-deploy-issue recipe — read-only diagnosis using app log_tail, required healthcheck, optional memory_check, and an AI-assisted summary.
restart_service playbook — appropriate only when the evidence points to a wedged service rather than bad code. It is approve-required and does not run until a human confirms it.
Deployment rollback — natural-language rollback intent goes through the configured deploy pipeline and captured snapshot. It creates a deploy_rollback approval before any restore starts.
post-deploy-verification recipe — read-only server-health and recent-log checks after a restart, rollback, or corrected release. Confirm the public application endpoint separately when required.
Diagnostics are read-only. Risky fixes follow runtime policy.
- +mttrly investigates on its own: diagnostic recipes and read-only playbooks inspect the server without changing anything.
- +Approval-required fixes create a pending action for confirmation from Telegram, the dashboard, or the IDE. Runtime policy — not the AI — decides the action class.
- +A separately authorized Investigation session can let mttrly_execute_command skip per-action approval only within its server, time, and action limits. That bounded exception is visible and auditable.
- +mttrly does not give an AI client an unrestricted production shell. Diagnostics, configured preauthorization, approval gates, and bounded sessions remain explicit.
These incident guides use the same scoped MCP action layer and can route approval decisions through the Telegram mobile workflow.
Detection tools tell you something is wrong. mttrly is the agent on your server that diagnoses the incident and prepares the fix — through scoped MCP tools, diagnostic recipes, and remediation playbooks. Approval-required actions normally wait for a human; any narrowly configured preauthorization remains bounded and audited. It complements monitoring like Grafana, Datadog, or UptimeRobot; it does not replace it.
When the recipe cannot resolve the incident
The recipe observes server and application signals; it does not understand every application or reverse every release side effect. Boundaries include:
- -An application logic bug still needs a code fix. mttrly can surface the failing log and correlate timing, but it is not a substitute for debugging the codebase.
- -Database migrations, corrupted data, and incompatible schemas need an application-specific recovery plan. A code rollback may not restore data compatibility.
- -External dependencies such as payment APIs, managed databases, DNS, or cloud platforms may fail while the server itself remains healthy.
- -If the app service is not configured or its logs are not readable, the optional log step may be unavailable and the diagnosis will have less evidence.
- -If the agent is offline, mttrly cannot collect live server evidence until connectivity is restored through your provider console or another recovery path.
Frequently asked questions
What does the post-deploy-issue recipe check?
It requests up to 200 recent lines from the app service, runs the required healthcheck, checks memory when available, and uses AI analysis to summarize the signals. These steps are read-only.
Will mttrly automatically restart or roll back after the diagnosis?
No. The diagnostic recipe does not change production. A restart_service playbook or a standalone rollback request creates a separate pending action and waits for explicit human approval.
Can mttrly prove that the new code caused the failure?
Not from server signals alone. mttrly can correlate the deployment time with logs, health, and memory and identify a likely regression. Application-level root cause analysis may still require the codebase, traces, database context, or an external service investigation.
How do I verify the server after a restart or rollback?
Run the post-deploy-verification recipe. It performs a read-only server healthcheck and reads recent application logs. Then check the public application endpoint or another user-visible success signal before closing the incident.
Related
Diagnose the next bad deploy before changing production
Connect a server and keep recent logs, health, memory, approval-gated recovery, and post-deploy verification in one incident path.