Skip to content
Research project · The Debug Agent · Autonomous RCA

An agent that finds
the root cause

When production breaks, an LLM agent investigates — grounded in a FalkorDB world-model, disciplined by hard evidence, and honest about what it does not know. It only opens a pull request when the evidence supports it. Research-grade RCA at a €50/month homelab budget.

Graph-grounded Correlation-first MAST-aligned reasoning €50 / month

How it works

An alert fires; the agent recalls similar incidents, ranks suspects from the graph, runs a bounded tool-using investigation, and decides by confidence — never auto-remediating. Five research ideas make that possible, each written up as a paper below.

The investigation pipeline

  1. 1

    Detect

    An Alertmanager webhook is de-duplicated, then deterministic passes gather traces, logs and code context before any model call.

  2. 2

    Recall & rank

    Personalized PageRank + BARO change-point scoring rank candidate causes; hybrid memory recalls related past incidents — all without an LLM.

  3. 3

    Investigate

    A bounded ReAct loop queries real tools. A verification-gate FSM — not the model — decides when evidence is sufficient to stop.

  4. 4

    Decide

    A confidence ladder, soft-capped by graph grounding, chooses the action: notify, open an issue, attach patches, or open a PR for a human.

Five ideas behind it

01

The StackGraph

A persistent neuro-symbolic world-model in FalkorDB that fuses code, infra, runtime telemetry, change history and past RCA outcomes into one property graph the agent reasons over.

FalkorDBProperty graphNeuro-symbolic
02

Correlation-first blame propagation

Personalized PageRank over an error-weighted dependency graph, blended with parameter-free BOCPD change-point scoring (BARO) — deterministic graph priors that seed the LLM's hypothesis space. No causal discovery.

PageRankBOCPD / BAROChange-point
03

Don't grade your own homework

An externalized verification-gate FSM governs loop termination; hypothesis state is derived only from hard tool evidence (never LLM-asserted); a grounding-based soft-cap keeps thin conclusions below the auto-PR tier.

MASTVerification gateHypothesis tree
04

Eval-first agentic RCA

Every change passes a deterministic AC@1 / MTTR replay gate with distractor injection, behind a shadow → active rollout — so an LLM-in-the-loop system ships safely against a labelled ground-truth set.

AC@1 / MTTRRegression gateShadow→active
05

Bi-temporal incident memory

A hand-rolled Graphiti-on-FalkorDB temporal knowledge graph with no-LLM-at-retrieval hybrid recall (semantic + BM25 + graph-BFS, fused by RRF) and Reflexion-style lessons — the agent learns from every incident.

Temporal KGRRF recallReflexion

Each idea is written up as a paper below.

Anatomy of an investigation

What happens when Alertmanager fires: a deterministic pre-LLM stage seeds suspects from the FalkorDB world-model, a bounded ReAct loop reasons against real tools through an MCP fleet — Loki, Tempo, Prometheus, Grafana, Kubernetes and more — with the LLM reached over a Cloudflare AI Gateway, and a verification gate decides when the evidence is enough to act.

BlueRobin Debug Agent — alert-triggered root-cause-analysis flow When Alertmanager fires a webhook, the Debug Agent (a .NET service on the Hetzner cloud worker) de-duplicates it, runs a deterministic no-LLM stage (Personalized PageRank plus BARO change-point scoring) seeded by a FalkorDB StackGraph query and Qdrant hybrid recall, then enters a bounded ReAct loop: an LLM turn (Anthropic Claude, egressing through the Cloudflare AI Gateway to frontier providers) alternates with an externalized Verification-Gate FSM that governs termination. The loop calls tools through an MCP server fleet that fans out to the LGTM observability and platform backends — Loki logs, Tempo traces, Prometheus metrics, Grafana, Kubernetes and Flux state, NATS events, GitHub commits and code search. Tool evidence builds a hypothesis tree under a grounding-based confidence soft-cap, and a confidence-tiered action ladder resolves to a notification (ntfy), a GitHub issue, attached patches, or an auto-PR a human merges. Debug Agent · Hetzner cloud worker · FEAT-008 ReAct loop · AgenticChatLoop Observability · LGTM Alertmanager alert webhook fires Debug Agent · receiver de-dup · FastEndpoints Pre-LLM seeds — no model PageRank + BARO change-point FalkorDB StackGraph · query Qdrant hybrid recall · RRF LLM turn — reason + act Anthropic · Claude Verification-Gate FSM deterministic stop · MAST External LLM APIs frontier providers Cloudflare AI Gateway LLM egress proxy MCP server fleet tool transport Loki logs Tempo traces Prometheus metrics Grafana dashboards K8s · Flux cluster state NATS events GitHub commits · code Code repo search Evidence → HypothesisTree grounding-based soft-cap Action ladder · by confidence notify · issue · patch · PR ntfy · operator notify GitHub auto-PR / issue human merges cloud data external / I-O dashed = zone boundary · ⇄ tool calls via the MCP fleet

See it in action

One agent, three incidents — the same StackGraph traversal and verification gate each time. Not an LLM guessing: evidence-grounded reasoning over a live world-model, honest enough to only notify when confidence is below the auto-PR bar.

The Detective

An alert fires. The agent reads signals from three observability sources, traverses the FalkorDB StackGraph to find what changed, and proposes a fix — all before an on-call engineer opens their laptop.

3 Signals
⚠ Alertmanager — P1 CRITICAL 📋 Loki — error spike 23% ⏱ Tempo — P99 latency 4.2s
StackGraph traversal — FalkorDB world-model
StackGraph traversal 7 nodes · 9 edges · live query
SIGNALS RCA CHAIN TRIGGERED ×3 IMPLICATED CHANGED_BY CAUSED CONFIDENCE EVIDENCE Alertmanager P1 CRITICAL 📋 Loki error rate 23% Tempo P99 latency 4.2s StackGraph WORLD-MODEL · hub api-gateway SERVICE · implicated timeout=10s COMMIT a8f3d91 pool exhausted ROOT CAUSE confidence 0.87
Signal / alert Service / change StackGraph hub → typed relationship
Debug agent — bounded ReAct loop + verification gate
debug-agent · generated StackGraph traversal query
MATCH (svc:Service {name:"api-gateway"})-[:DEPENDS_ON]->(dep:Service)
MATCH (dep)-[:LAST_CHANGE]->(c:Commit)
WHERE c.deployed_at >= $alert_time - duration("PT2H")
  AND dep.error_rate > 0.05
RETURN dep.name, c.sha[0..7], c.message, dep.error_rate
ORDER BY dep.error_rate DESC
⊥ StackGraph  FalkorDB traversal + Tempo trace correlation
Alert
api-gateway: 502 error rate 23% · P99 latency 4.2s · fired 09:14 UTC
Agent verdict: Root cause: postgres-primary connection pool exhausted (94% utilization). Cause: commit a8f3d91 (Victor Robin, 2h before alert) increased query_timeout from 5s to 10s, doubling average connection hold time under load. Draft PR #127 opened to revert. Confidence: 0.87.
Evidence: Tempo trace waterfall Evidence: CNPG pool metrics

Research Papers

Papers

Each paper follows a scientific structure — abstract, related work, method, evaluation, discussion — and is published here and on ArXiv.

Observability Expert 18 min

Correlation-First Blame Propagation: Personalized PageRank and Parameter-Free Change-Point Detection as Pre-LLM Graph Priors for Root-Cause Analysis

A deterministic, training-free pre-LLM scoring layer that seeds an RCA agent's hypothesis space: error-weighted personalized PageRank blame propagation, parameter-free BOCPD change-point onset detection, a BARO-style robust scorer, and a five-term re-normalizing additive blend with provable fail-open collapse — and a principled, CI-enforced refusal of causal discovery.

AI/ML Expert 20 min

Do Not Grade Your Own Homework: Externalized Verification Gates, Evidence-Derived Hypothesis Trees, and Grounding-Based Confidence Capping for LLM Root-Cause Agents

A systems paper on the reasoning-safety core of the BlueRobin Debug Agent: a deterministic verification-gate FSM outside the model, hypothesis state derived only from real-tool evidence, and a grounding-based soft-cap that bounds LLM self-confidence below the action tiers — all driven by the MAST failure taxonomy.

CI/CD Expert 18 min

Eval-First Agentic RCA: A Deterministic Accuracy/MTTR Regression Gate and Shadow-to-Active Rollout for Shipping LLM-in-the-Loop Reliability Tooling

How the BlueRobin Debug Agent makes every RCA claim measurable and safe to ship: a deterministic offline replay harness over a labelled incident set emitting an AC@1 + MTTR scorecard, a per-commit regression gate that blocks merges, distractor-injected honest-accuracy arms, and a shadow-to-active live-activation rollout — at a 50 EUR/month homelab budget.

AI/ML Expert 21 min

Bi-Temporal Incident Memory on a Property Graph: Hand-Rolled Graphiti-on-FalkorDB with No-LLM-at-Retrieval Hybrid Recall and Reflexion Lessons

How the BlueRobin Debug Agent gives an LLM RCA agent durable, time-aware memory: a bi-temporal MemoryEpisode/MemoryEntity subgraph on the existing FalkorDB, a three-leg hybrid recall (semantic + BM25 + graph-BFS) fused by Reciprocal Rank Fusion with no LLM at retrieval, and Reflexion/Voyager-style post-incident lessons — all inside a ~50 EUR/month homelab budget.

Security Intermediate 15 min

Managing Secrets with Infisical and Kubernetes

A step-by-step guide to setting up Infisical for secret management, integrating with External Secrets Operator in Kubernetes, and using secrets in .NET applications.

Built with

FalkorDB Qdrant Ollama Claude Cloudflare AI Gateway .NET 10 Tempo Loki Prometheus