A system that can break in more than one way.
To make the test fair, we needed exactly that. So we rebuilt the testbed from Anthropic’s Site Reliability Agent cookbook into something harder: a running app (FastAPI, Postgres, Prometheus, Grafana) that stages four independent faults, each hidden behind a deliberately vague deploy. The cookbook is a great demo, but as a benchmark it has two properties that make NavFlow look good for the wrong reasons:
- One lever, one incident. Its api-server exposes a single knob (the DB connection pool size), so it can only ever stage the same pool-exhaustion incident. A win on one fault could just be luck.
- The deploy names the bug. Its changelog says
"Reduce DB connection pool size for staging parity". The commit message is the answer. An agent doesn’t have to reason; it reads the diagnosis off the deploy log.
Our rebuild keeps the same clean Prometheus/Grafana shape but fixes both. The api-server now exposes four independent fault levers, injectable at runtime with no redeploy, each producing a distinguishable signature in the metrics and logs:
| incident | what the agent sees |
|---|---|
| DB pool exhaustion | pool-exhausted logs, db_connections pinned at 1, 500s on /api/users |
| Latency regression | p99 blows past the request timeout, pool itself healthy |
| Error spike (bad flag) | app-exception logs (KeyError: user_tier), DB healthy |
| Dependency outage | dependency_up = 0, upstream-unreachable on /api/orders |
The whole platform (app, fault injection, both agent variants, the grader) is in the cookbook repo under cookbooks/01_sre_incident_response, with one command to stand it up and one per run.
Commit messages that never name the knob.
Each fault is preceded by a deploy in the changelog, the correlation signal a real SRE leans on. But unlike the cookbook, the commit messages never name the knob. They read like real commits, so the agent has to connect the deploy to the symptoms instead of reading the answer:
| fault | deploy commit (what the agent sees) | what it hid |
|---|---|---|
| db_pool exhaustion | “Align resource limits with staging environment” | db_pool_size 20 → 1 |
| latency regression | “Add synchronous audit-log write to user lookup” | inject_latency_ms 0 → 800 |
| error spike | “Ship new pricing-tier feature flag” | error_rate 0 → 0.30 |
| dependency outage | “Roll out v2 checkout integration” | payments-api forced down |
This matters because it tests the thing that’s easy to fake: whether a correlated read actually helps the agent reason, or just lowers a call count. The agent gets the same generic prompt every time (“something is wrong, investigate”) with no hint of which fault is live.
Identical agents. Only the tools change.
Same runtime, same model, same task prompt, same grading. The only difference is the tools they’re handed. Baseline gets the provider-style read path: each signal is its own tool, and the agent fans out across them every investigation. NavFlow is the same agent with the fan-out tools replaced by one query. NavFlow ingests the same five signals into its own store and serves them back as one time-ordered, correlated timeline in a single read.
Eight runs, every number. Nothing averaged away.
One run per incident per path (eight runs total) on claude-sonnet-5, with the platform reset and NavFlow’s objects torn down between every run, so no run inherits another’s state. Cost is computed from token usage × published pricing.
| incident | baseline (fan-out) | NavFlow (one read) | root cause | ||||
|---|---|---|---|---|---|---|---|
| reads | turns | cost | reads | turns | cost | ||
| db_pool_exhaustion | 6 | 3 | $0.028 | 1 | 2 | $0.018 | both |
| latency_regression | 9 | 4 | $0.037 | 1 | 2 | $0.024 | both |
| error_spike | 4 | 2 | $0.019 | 1 | 2 | $0.018 | both |
| dependency_outage | 5 | 3 | $0.026 | 1 | 2 | $0.024 | both |
| total | 24 | 12 | $0.11 | 4 | 8 | $0.08 | 8/8 |
Across all four faults: 6× fewer reads, ~1.3× cheaper, and the same root cause found in all eight runs.
The number that holds up isn’t any single row, it’s the shape. NavFlow is dead flat at 1 read / 2 turns on every incident, no matter which fault is live. The baseline swings from 4 to 9 reads depending on how hard the fault is to disambiguate. The correlated query removes the search variance, not just the call count.
A vague commit, connected end to end.
Here is the NavFlow agent on the latency regression, the one whose deploy said "Add synchronous audit-log write to user lookup", never inject_latency_ms = 800. From one query, it reasoned the whole chain:
It connected a vague commit to the p99 metric and the timeout config, and explicitly ruled out the pool-exhaustion red herring. This is the kind of correlation the baseline reassembles by hand across a dozen separate reads.
What we claim. And what we don’t.
Reads are the real result.
Cache-independent and dead stable: 24 → 4 across the set, with NavFlow fixed at one read every time. The baseline swings between 4 and 9 depending on how hard the fault is to disambiguate. Lead with this.
Turns move less than you'd think.
A modern runtime fires the baseline's read tools in parallel, so turns go 12 → 8 (1.5×), not the dramatic collapse a sequential loop would show. NavFlow sits at two turns. We don't oversell it.
Cost is a supporting figure.
~1.3× cheaper, and lower on every single incident, but a few cents either way at this scale, and computed from token usage × published pricing, not a billing statement. Never the headline.
Tokens usually drop, but not always.
~1.4× fewer input tokens overall, but on the cheapest fault (error_spike) the baseline's four small reads beat NavFlow's one fat correlated read. Context size isn't a reliable axis, so we treat it as directional.
Accuracy is the floor.
If the cheaper path got the wrong answer, none of the above would matter. It didn't: 8/8, both paths, including the three faults the original cookbook can't even stage.
You don’t rebuild the agent. You give it a saner read path.
Across four independent faults, hidden behind commit messages that never name the bug, the same SRE agent reached the same root cause whether it fanned out across a handful of tools or made one NavFlow query. The read path went from 24 calls to 4 over the set, and NavFlow held a flat 1 read / 2 turns regardless of how messy the incident was. The investigation logic never changed; what changed was only where the data comes from.
Run it yourself
Everything on this page (the platform, the fault injection, both agent variants, and the grader) is open source in cookbooks/01_sre_incident_response. Stand it up, stage a fault, and compare the two read paths on your own API key.
Get a NavFlow project at cloud.navflow.ai, or self-host: open source, MIT.