- hardest three, quiet machine; encouraging, not proof
- 18 / 18
- faults never missed
- 11 / 15
- AI models
- 4
- findings, read by hand
- 53
Why we did this
When a data or ML pipeline fails, the alert says where it hurts: a forecast is late, a dashboard is stale, a job keeps failing. It rarely says why. An engineer then spends the first hour checking pipelines, databases, deploys and vendors, one at a time. Lumis is built to do that first hour. It forms possible causes, checks each one against the system’s own evidence, and says which cause the evidence supports, showing every step. When the evidence can’t decide, it says so.
In Did it look, or did it guess? we tested our open proof of concept. This piece is about what came next: Lumis Core, the engine we’re building for real estates. We ran it on the same testbed, again and again, on several models, and read every investigation by hand. The question was simple: is it right, and is it right every time?
The testbed: GridCast
GridCast is an electricity-demand forecasting system we built to test Lumis honestly. It’s synthetic but operationally real: ten services on Kubernetes; data pipelines, a feature service, model training and serving; external data vendors; and the observability a real team would have: metrics, logs, traces, pipeline runs and the database. We can break it in fifteen ways, the kinds of failure real data teams hit.
| What breaks | Why it’s tricky | |
|---|---|---|
| A | A feature-service release makes each build run ~600× more database work | The release looks harmless; only the database load gives it away |
| B | The weather vendor keeps answering, with stale data under fresh timestamps | Nothing errors; the data just stops being new |
| C | A database password is rotated, and one service keeps the old one | The database refuses it, but the database isn’t the fault |
| D | A memory limit is cut below what the forecast service needs | The service crash-loops; an automated bot made the change |
| E | A much slower model is promoted to production | Nothing breaks; inference just gets ~100× slower |
| F | Two releases land within a minute, and only one caused it | Timing alone can’t pick the guilty one |
| G | The grid data vendor ships an incompatible API version | Every demand batch is rejected |
| H | The grid data vendor silently switches units (MW to kW) | Values are accepted, then fail a check downstream |
| I | The main weather vendor goes down | A fallback vendor exists and is healthy |
| J | A service is left scaled to zero after maintenance | A known pattern: an engineer’s check catches it in seconds |
| K | A timeout is lowered below a vendor that was already slow | Two things are true at once; the change, not the vendor, is the trigger |
| L | Grid data stops arriving, right next to an unrelated release | Nothing errors anywhere; the obvious suspect is the decoy |
| M | A bot cuts a service’s CPU, slowing its builds | Looks like a database problem; only shows when the system is busy |
| N | A release changes a unit the model was trained on | Every check passes; only the forecasts drift |
| O | One small zone’s data quietly stops | The overall data still looks fresh |
How each test runs:
- We break one thing. Lumis isn’t told what.
- We wait for the system’s own alert, as an on-call engineer would.
- Lumis investigates with the evidence the team would have.
- We compare its answer with what we actually broke, and judge the mechanism by hand.
The answer key is written before the run and kept where the investigation can’t read it. We check that separation on every run.
How Lumis reaches an answer
The model proposes; the evidence decides.
- The model writes testable claims. It proposes possible causes, each with what should be true if it’s right and what would rule it out.
- Lumis gathers the evidence itself, through queries the team has approved.
- Lumis judges each claim against what it observed. The model never marks its own work.
- One answer: when the evidence supports one cause, Lumis concludes, and every step can be traced back to an observation.
- When it can’t decide, Lumis says so, and says what would decide it.
Over these three days we added a third kind of answer between those two: a leading cause. When the evidence supports more than one cause, or rules out all but one, Lumis names the most likely, says plainly how it chose, and says what would confirm it. It’s labelled so; it is not presented as proven.
Here is one investigation of K, the timeout fault, on 14 October, in its own words.
Fault K · 14 Oct · DeepSeek V4 Pro, run 2
- Proposed
Ingestion’s per-request vendor HTTP timeout was reduced from 15s to 2s by GitOps commit 636032a (08:10:19Z). grid-telemetry’s demand reads take longer than 2 seconds, so every demand batch now aborts with an httpx ReadTimeout and fails, while weather vendor calls (which return within 2s) keep succeeding.
- True if right
demand batch errors (10 min) > 0 · weather batch errors (10 min) = 0 · zones reporting demand (20 min) = 4 · demand freshness > 300 s. Rules it out: demand batch errors (10 min) = 0.
- Observed
Demand freshness 354 s (normal 46). Demand batch p95 17.77 s (normal about 0.13 s). Demand errors 5.28 (normal 0). Weather errors 0. 4 zones reporting.
- Lumis
supported: its predictions held (demand-freshness, ingestion-demand-errors, ingestion-weather-errors, demand-zones-reporting). Nine of the team’s checks were ruled out on the same board, each with “a falsifier held”.
- Open question
I could not directly measure grid-telemetry /v1/load latency, so whether the endpoint itself degraded at ~08:10 (independent of the timeout cut) is unconfirmed — but the timeout reduction is the only change in the window…
- Would decide it
Direct per-request latency for grid-telemetry’s /v1/load endpoint…
The results
Each sitting is one run of the experiment on a frozen version of the code.
| When | What ran | Right | Wrong | No conclusion |
|---|---|---|---|---|
| 12 Oct | All faults, twice, one model (our first measured run) | 23 of 26 | 1 | 2 |
| 13 Oct | The hardest three (C, K, L), twice, four models | 19 of 24 | 0 | 5 |
| 13 Oct | M, twice, one model | 0 of 2 | 2 | 0 |
| 13 Oct | K and L, twice, three models, after fixes, on a loaded machine | 7 of 12 | 0 | 5 |
| 14 Oct, morning | C, K and L, twice, three models, current engine, quiet machine | 18 of 18 | 0 | 0 |
| 14 Oct, afternoon | C and K, twice, three models, with CPUs deliberately kept busy | 11 of 12 | 1 | 0 |
| 14 Oct, evening | K and L, twice, three models, with one CPU kept busy | 10 of 12 | 2 | 0 |
- Eleven faults have never been missed: A, B, D–J, N and O.
- K: 5 of 6 on the 13th, then 6 of 6 on a quiet machine, then 5 of 6 and 4 of 6 on busy ones. K is still the hardest case.
- L: 3 or 4 of 6, then 6 of 6 on a quiet machine and again 6 of 6 on a busy one. One model had never solved L before (it gave no conclusion each time); it was right in all four runs.
- The wrong answers are kept in the table. Each one taught us something.
How the models behaved
We gave the same broken system to several models, and to the same model twice. Lumis was identical in every run. Here’s what we saw:
- Same incident, same model, different answer. Two runs can diverge in their first move: which observation they look at first, which cause they write down first, how they phrase a test. Most of the inconsistency we saw starts there, not in a wrong fact.
- The model writes its own tests, and sometimes gets a detail wrong. One wrong detail inside a right idea is the most common way a right answer was lost.
- Speed varied about tenfold, and accuracy didn’t follow it. The quickest model was also right in every valid run on the current engine; the slowest was the most methodical. Load roughly doubled the slowest model’s time.
- Each model has a temperament. Some would rather say “I can’t tell” than guess; some reach for a two-part explanation; some keep an answer even when asked about something it doesn’t explain.
- The machine matters as much as the model. On a busy machine every service slows down. That noise looks like evidence, and it pulled models toward wrong second causes.
What made results more consistent wasn’t a better model. It was making the evidence tell the causes apart: every number shown with its normal value; tests written relative to that normal; a second look at what the conclusion leaves unexplained; and a clear way to change your mind.
| Sitting | Same component first | Both runs right |
|---|---|---|
| 12 Oct | 10 of 13 | 10 of 13 |
| 13 Oct, four models | 10 of 12 | 9 of 12 |
| 13 Oct, loaded machine | 5 of 6 | 2 of 6 |
| 14 Oct, quiet | 9 of 9 | 9 of 9 |
| 14 Oct, busy (C, K) | 5 of 6 | 5 of 6 |
| 14 Oct, one CPU busy | 4 of 6 | 4 of 6 |
Model by model
How each behaved, not a ranking: the samples are small, and the engine changed between sittings.
- DeepSeek V4 Pro: the most-run and the most methodical, with many queries, and the slowest per investigation. Its two wrong answers on K came under load: it wrote a detail into its own hypothesis that ruled out the right cause.
- Claude Sonnet 5.5: the quickest to a conclusion, often in one to two minutes. It revises its own hypotheses mid-investigation, withdrawing a version and proposing a better one, and readily uses checks against normal.
- GPT-6.1 Sol: cautious. On the earlier engine it gave no conclusion on L every time; on the current engine it solved L by ruling rivals out by measurement. Once, under load, it withdrew its own supported cause and settled on the slow vendor instead.
- Gemini 3.8 Flash: run only on the earlier engine: right on C and K in every run, no conclusion on L both times, and the most model requests per investigation (about 40).
What we learned
Many “model failures” were ours
Reading every investigation by hand, not just the totals, turned up 53 findings. A surprising number were Lumis’s bugs, not the model’s: an “outage” that was a request limit shared by every investigation; a time shown as exact when we only knew a change happened at or before it; our own failed login read as evidence; and a test harness that trusted a port. None of these shows up in a total. Each shows up in the first investigation you read.
A number needs its normal
On M, the true cause predicted “builds over five seconds”. They took 2.35 seconds: twenty-four times their normal, but under a number the model made up. The right cause was ruled out by a guess. Every observation now carries its normal value, and a model can say “three times above normal” instead of guessing a threshold. In the latest runs, models did exactly that.
Fault M · 13 Oct · DeepSeek V4 Pro, run 1
- Proposed
Feature-service CPU throttling (hard 50m CPU limit) is the dominant cause of the pipeline slowness…
- Its own test
True if feature build p95 > 5 s; ruled out if ≤ 4 s.
- Observed
Feature build p95: 2.35 s, many times its normal (about 24×).
- Lumis
contradicted: a falsifier held (feature-build-p95)
Let it change its mind, on the record
A model would revise its own idea and then have two versions of itself on the board, tying with each other. Now it can withdraw the earlier version and say why. The withdrawn idea stays visible, and it no longer competes.
The strict rules earned their place
We took the rules apart one at a time and replayed every recorded investigation under each version. Loosening the strict ones made things worse: more ties, or wrong answers. The gains were in what Lumis does when it would otherwise give up: the model’s own account usually named the right cause, and Lumis refused to say so. The labelled leading cause came from that.
“What changed just before?”
Our first two wrong answers on K were the same mistake, two days apart: the true cause, a config commit that cut a timeout two minutes before the alert, was proposed and then ruled out by a wrong detail in the model’s own test, and the answer named the vendor, which was slow but only a background condition. Lumis now asks that question before it concludes. When a conclusion still leaves such a change unexplained, it concludes and labels it: concluded, with an open question; check this before acting. The answer stays useful, and the doubt is visible. That label is built and has been replayed on recorded runs; it has not yet been tested live.
What this doesn’t show
- One estate, built by us. GridCast is realistic, but it isn’t a customer’s system, and we built the faults. Public benchmarks (OpenRCA, ORCA-bench) come next.
- Small samples. Most rows are two runs per model per fault, and each fault was broken once per sitting. “18 of 18” is encouraging, not proof.
- The machine matters. Our best run was on a quiet machine. We now run busy on purpose, as a setting, to measure that.
- Some changes were tested on the runs that motivated them, then confirmed with fresh runs. Replays can’t show how a model would have acted differently.
- We judge mechanisms ourselves. The component is scored automatically; whether the explanation is right is judged by hand, by us.
- Two things are still open: M has alerted only once in four tries (it needs a busy system), and K on a busy machine split again, 4 of 6.
What’s next
- The labelled “open question”, tested live on K.
- Checks against normal, tested live on M.
- Public benchmarks: OpenRCA and ORCA-bench.
- The console, where every investigation above can be replayed step by step (in progress).
We’ll keep publishing results, including when they’re not good. What Lumis Data & ML v0.0.1 is, part by part, is in the companion piece.