NOC teams are discovering that ML-driven anomaly detection isn't a futuristic abstraction — it's a deployable reality. We break down the patterns that separate the implementations that actually reduced downtime from those that added complexity.
AIOps has an unusually wide gap between what is promised and what is deployed. The promise is a system that predicts failures before they happen and resolves them without human involvement. What is actually running in production at most organisations is considerably less glamorous and considerably more useful: statistical models that reduce alert noise so that on-call engineers can find the signal.
That distinction is worth holding onto, because it determines whether a project succeeds. The teams we have seen get real value started by treating machine learning as a filtering problem, not a prediction problem.
Alert fatigue is the actual problem
Ask an on-call engineer what makes incidents worse and the answer is almost never 'we did not detect it'. It is that the detection arrived buried in two hundred other alerts, most of which were symptoms of the same root cause and none of which said which one mattered.
Static thresholds cause this. A CPU alert set at 80% fires every afternoon during a batch job and gets muted. A disk alert set for a database is wrong for a cache node. Over time the alerting configuration accumulates exceptions until nobody trusts any of it, and the meaningful alert arrives with the same visual weight as the noise.
An alert that fires when nothing is wrong is not a false positive. It is a broken alert, and it costs you the next real one.
Three patterns that work
Baselining instead of thresholds
Learning what normal looks like per service, per hour of day, per day of week, and alerting on deviation from that baseline is unglamorous and effective. It handles the batch job automatically. It handles the difference between a cache node and a database automatically. Most of the reported reduction in downtime attributed to AIOps comes from this one change, because engineers start responding to alerts again.
Correlation and grouping
When a dependency fails, everything downstream of it alerts. Clustering related alerts into a single incident, ordered by the dependency graph, turns two hundred notifications into one notification with a probable cause at the top. This is the single largest improvement to time-to-diagnosis we have measured in practice.
Change correlation
The overwhelming majority of incidents are caused by a change — a deployment, a config edit, a feature flag, a certificate expiry that was technically scheduled months ago. Automatically surfacing 'what changed in this service in the last hour' alongside the alert answers the first question every responder asks, before they have to ask it.
What to measure
If you deploy this, measure the things that reflect whether engineers' lives improved, not model accuracy in isolation.
- Alerts per on-call shift, and what fraction were actioned rather than dismissed.
- Time from alert to correct diagnosis — not time to resolution, which is dominated by the fix itself.
- Incidents detected by monitoring versus reported by customers. This ratio is the honest measure of whether your observability works.
- How often the automated top-ranked probable cause was in fact the cause.
Start small, deliberately
Pick one noisy service. Baseline it, correlate its alerts, wire in change data, and run the new system alongside the old one for a month without acting on it. You will learn quickly whether the output is trustworthy, and you will have evidence rather than a vendor claim when you decide whether to expand it.