Gray failures are partial, slow-spreading outages that standard health checks miss. Databricks’ RADAR pipeline turns user-facing error signals into fast, metric-agnostic detection, alerting, and root-cause steps to find those failures in minutes.
RADAR is a four-stage pipeline — reliability metrics, anomaly detection, alerting, and root-cause analysis — designed to flag unusual error spikes by user slice and other dimensions.
A practical, high-signal input is spikes in user-facing errors that would normally look like user fault (e.g., INVALID_ARGUMENT) but occur at scale for many users at once.
This post is about catching them early with anomaly detection — how we do it at Databricks with a system called RADAR, and how you can build the same thing for whatever metric matters most to your business. Catch gray failures with anomaly detection | Databricks Blog Skip to main content Summary Gray failures slip past green dashboards, quietly costing you customers and revenue before anyone notices. You can build the same system on Databricks for any metric — billing, conversion, or model performance — using native components and an AI-agent scaffold.
From the outside the house looks fine, but inside the damage is spreading — and the longer you wait, the bigger the blast radius. Get the RADAR scaffold on GitHub Because the best outcome isn't a faster response to angry customers — it's that your customers never have to discover your incidents for you. Some of the most damaging outages are the ones your monitoring never flags: a slice of your customers quietly fails while every health check still reads normal.
RADAR is a four-stage, metric-agnostic pattern — reliability metrics, anomaly detection, alerting, and root-cause analysis — that Databricks runs on itself to catch these failures in minutes, at over 90% precision and 95% faster discovery. When everything is green but nothing is fine Picture a normal Wednesday. By every signal your team watches, the system looks perfect.
Conventional screening asks: does this unit violate a limit we set? Supervised machine learning asks: does this unit resemble a defect we’ve seen before? Both questions require us to already know what we’re hunting.
Anomaly detection has become one of the most critical technologies of the modern connected world, quietly guarding everything from hospital patients to water treatment plants. Every heartbeat monitored by a wearable device, every vibration sensor on an industrial assembly line, and every traffic flow measurement in a s
A demo that nails every question in the interview room, then embarrasses itself in production a month later. Here’s why that happens — and… Continue reading on Medium »
How to verify your app aligns with your intent without ever reading a line of generated code. The post Coding Agents Keep Shipping Silent Failures — Here Is How to Catch Them appeared first on Towards Data Science .
I spent the first six months of a project convinced we had a model quality problem. Our anomaly detection system for manufacturing telemetry was missing obvious defects; things a human operator would catch in seconds. We tried bigger models, better embeddings, more training data. Nothing moved the needle.
Business intelligence dashboards can fail silently, showing blank, stale, or wrong data even when every infrastructure monitor reports healthy. Learn how an AWS team built an automated, AI-powered content validation solution on Amazon Bedrock that scans hundreds of dashboards and alerts owners, cutting mean time to det
Loading more related stories...
Open the app view to save this story, compare related coverage, and continue from the same source.
