The Alert Fatigue Death Spiral
High alert volume drives analysts to ignore alerts, which lets real attacks go unnoticed, which leads to post-incident reviews asking why nobody noticed. The fix isn't more analysts. It's fewer, better alerts.
The Four Alert Tiers
I categorise every SIEM rule into one of four tiers:
- High-fidelity automated response — so reliable, automated actions are safe
- High-fidelity analyst review — fires rarely, almost always real
- Medium-fidelity investigate — fires regularly, some FPs expected
- Informational/hunting — not an alert, stored for hunting and correlation
Most shops have everything in tier 3 and wonder why their analysts burn out.
Measuring What You Have
index=notable
| stats count as alerts, dc(event_id) as unique_events by search_name
| eval tp_estimate = round(unique_events / alerts * 100, 1)
| sort -alerts
Tuning Process
For each rule with FP rate > 70%:
- Add suppression with expiry date and documented justification
- Raise thresholds or add contextual filters
- Consider retiring if the technique is no longer relevant
Suppression vs. Exclusion
- Suppression — temporary, with expiry date. Good for maintenance windows.
- Exclusion — permanent, requires documented justification and review date.
Defensive Takeaways
- If an analyst closes 90% of a rule's alerts without investigation, that rule needs tuning
- Track TP rate per rule, not just alert volume
- Deduplicate before alerting — 1000 failed logins is one alert, not 1000
- Quarterly rule review sessions beat ad-hoc tuning for maintaining coverage
Keep going
Get the next writeup in your inbox
New posts delivered when I publish. No spam.