Signal · rule · alert · response
FIG. 20 — SIGNAL, RULE, ALERT, RESPONSE AN ALERT IS A DECISION, NOT A NUMBER RUNNING SYSTEM · LIVE SIGNALS ALERT RULE ENGINE NOTIFICATION · ON-CALL PRODUCTION · checkout-api Running, and reporting Three instances serving real users, emitting the same few numbers every few seconds. emitting… LIVE SIGNALS · WHAT IS MEASURED AVAILABILITY Requests succeeding 99.98% Steady ERROR RATE 5xx responses 7.4% Above threshold LATENCY p95 Time to respond 612 ms Rising with errors SATURATION CPU headroom used 61% Healthy RULE ENGINE · EVALUATES CONTINUOUSLY ALERT RULE A condition, not a number error_rate > 5% for 5m The signal reports. The rule decides whether that reading deserves a person. THRESHOLD + WINDOW Two brief spikes discarded · third held 5m ALERT STATE OK FIRING RESOLVED An alert is a small state machine: it opens when the condition holds and closes itself when the signal recovers. HUMAN RESPONSE Mitigate, then understand Roll back the deployment that introduced the errors. Restore service first — the review of why it happened comes afterwards. NOTIFICATION · DELIVERED Paged the on-call engineer Firing is a state inside the system. The notification is how it reaches a human. CRITICAL → phone WARN → chat ACTIONABLE ALERT checkout-api · errors 7.4% WHAT Checkout requests failing for 5 minutes IMPACT ≈ 1 in 13 purchases affected NEXT Compare with the 14:02 deployment Everything needed to start, in one message. RECOVERY · LOOP CLOSED RESOLVED error_rate 0.4% Errors fell back under the threshold and stayed there, so the rule evaluates false again and the alert closes itself — no one has to remember to tidy up. SEVERITY · ROUTING Who gets woken CRITICAL · pages on-call WARNING · team channel INFO · dashboard only Impact decides the channel. ANTI-PATTERN Alert fatigue Twenty pages a day and the one that mattered is missed. Alert on symptoms, require a window, delete rules nobody has ever acted on. THE LOOP Signal → rule Rule → alert state Alert → notification Human → recovery Monitoring watches known signals; alerting turns one sustained, meaningful condition into an action for a person — and closes itself when the signal recovers.