Monitoring alerts that matter (and how to stop ignoring them)
Design alerting so every alert is worth reading, with severity, routing and runbooks.
An alert everyone ignores is worse than no alert: it hides the real one.
Alert on symptoms
Your customers experience "the site is slow" and "the site is down" — not "CPU is 82%". Alert on:
Availability — the service does not respond.
Latency — responses far slower than normal.
Error rate — 5xx responses climbing.
Capacity — disk/memory/bandwidth past a safe threshold.
Keep resource alerts as early warnings, not pages, and reserve waking someone for the things users feel.
Severity levels
| Level | Meaning | Route |
|---|---|---|
| Critical | Service down, data at risk | Page immediately (push/phone/chat) |
| Warning | Trending toward a problem | Email/chat, next working period |
| Info | Notable but normal | Dashboard only |
Reduce noise
Thresholds with a duration — alert only if the condition holds for N minutes, killing single-sample flaps.
Group related alerts — one "web tier degraded" beats twenty individual host alerts.
Suppress during planned maintenance — or you will get paged for your own deploy.
Tune after every false alarm. An alert that fired wrongly is a bug in your alerting; fix it.
Every alert needs a runbook
An alert should say what is wrong and what to do, not just "something is red". Link each alert to a short procedure: "Check X, then Y; if Z, escalate." Your future self at 3 a.m. will not remember the architecture — write it down. Our troubleshooting guides exist for exactly this reason.
Review periodically
Once a month, look at which alerts fired and whether each was actionable. Delete the ones that never are. Alerting is a garden, not a one-off installation.
Related reading