Faultline NOC
faultline-noc
A router that turns one request into an ordered plan over three specialist agents, and the harness that scores each plan before anything runs.
Built
A typed RoutePlan contract: ordered steps, the context each step receives, which step may write to the network and whether it needs confirmation. Asking the user to clarify is an outcome, not a fourth agent. A keyword baseline and two Claude routers answer through a forced submit_ tool call, from a prompt checked on a separate 8-item dev set (8/8 on the first draft, unchanged) and frozen by SHA-256. Underneath sits a seeded 5G SA simulator where every telemetry record carries an evidence_.
Evaluated by
4 deterministic detectors (misroute, missing handoff, unsafe write, missed clarification) and set-level metrics. The harness is itself tested: 4 mutant routers each carry one planted defect and must trip their own detector. A malformed model answer is scored as wrong and never repaired. CI replays every recorded model call with no API key.
- Define
- Build
- Trace
- Observe
Numbers, limits (6), sources
A router that turns one request into an ordered plan over three specialist agents (knowledge, testing, incident response), and the harness that scores that plan before anything runs: which agent, in what order, what context is handed on, and which step may write to the network. It runs on a seeded, simulated 5G SA core, next to an evaluation harness for root cause analysis (RCA) agents.
All numbers
- 45/52 exact routes for both Claude Haiku 4.5 and Sonnet 5, against 40/52 for the keyword baseline. The intervals overlap ([0.75, 0.93] against [0.64, 0.86]), so this is a comparison, not a quality claim.
- 1 of 6 injection items followed: Haiku granted the restart a pasted ticket asked for;
unsafe_caught it.write - 4 malformed Sonnet answers (3 sent
stepsas a string, 1 an empty object), each scored as wrong. - 8 of 8 RCA mutants trip their own detector across 8000 seeded, scripted runs (rule baseline, oracle, 8 single-defect mutants), and nothing outside their declared side effects. The baseline also scores 800/800, so these runs test the harness, not the agent.
Limits (6)
- Plans are scored, not executed: the knowledge and testing specialists exist only as contract labels; the incident side is the separately evaluated RCA harness.
- 52 items written by me, one recorded answer per model per item.
- The telemetry is simulated, not emulated: no protocol stack runs.
- Write targets are not checked, only which step writes.
- Handoff refs exist before step one runs, so wiring one step's output into the next is not tested.
- RCA confidence is not calibrated.
Define, Build, Trace, Observe
- Define Covered Typed
RoutePlan, 52 authored items, 4 detectors - Build Covered Keyword baseline and two Claude routers
- Trace Covered Every plan, recorded response and detector trip committed per item
- Observe Covered Accuracy with intervals, unsafe writes, calibration and cost; replayed in CI
Sources
- 45/52
faultline-noc README.md:142-144 - 1 of 6
faultline-noc docs/adr/ 003-llm-router.md:35, :40 (injection resistance 0.833 over 6 injection items) - 4
faultline-noc docs/adr/ 003-llm-router.md:41 - 8 of 8
faultline-noc README.md:89, :104, :110
Stack: Python 3.11+, pydantic, mypy strict, pytest, Anthropic tool use, recorded cassettes, Vite + TypeScript project page