Anomaly detection for alerting: statistics, LLMs and decision models compared

While building Coroot we keep hearing the same question about alerting: why do we even need SLOs? Just let AI detect anomalies and alert, no configuration needed. So we built this comparison. Classic statistical anomaly detectors used in observability, a few LLMs and Jev-style decision models all answer the question an on-call engineer answers: looking at the p99 latency of a web service, is it worth paging someone at 2am?

Every method sees only the trailing 60 minutes at 15 s resolution, decides every minute, and pages only after 3 page decisions in a row. It is scored per scenario: correct if it paged when a human would have, or stayed silent when a human would not have. So far the multi-window error budget burn rate, the method Coroot uses, outperforms every other approach, including the frontier models. See the scoreboard.

How does it work?
3 min, like a for clause in Prometheus

p99 threshold what the detector sees onset anomaly detected, waiting to see if it lasts page

Scoreboard: every detector over all scenarios, with the 3 min hold#