How We Built AI Root Cause Analysis for Kubernetes

Root cause analysis isn't magic. It's correlation at scale.

When an incident fires, you have hundreds of signals: metrics spiking, logs flooding, traces timing out. The challenge isn't collecting data — it's connecting it.

The Signal Graph

At the core of Tagent's RCA engine is a directed graph of your services and their dependencies. Every service is a node. Every call between services is an edge. Every metric, log pattern, and trace anomaly is a weight on those edges.

payment-service → postgres-db (weight: 0.94 anomaly score)
checkout-service → payment-service (weight: 0.87)
orders-service → payment-service (weight: 0.82)

When something breaks, we don't look at individual alerts. We look at the graph.

Why LLMs Alone Aren't Enough

Large language models are good at reasoning. They're not good at real-time signal correlation across thousands of metrics.

Our approach: use deterministic graph algorithms for correlation, use LLMs for explanation.

# Deterministic layer
root_node = find_highest_anomaly_score(graph)
blast_radius = bfs_from_node(root_node, threshold=0.7)

# LLM layer  
explanation = llm.explain(root_node, context, history)

The Baseline Problem

You can't detect anomalies without knowing what normal looks like. Tagent builds per-service baselines over a 72-hour learning window, accounting for time-of-day patterns and weekly cycles.

What's Next

We're working on cross-cluster correlation and historical pattern matching — so Tagent can say "this looks like the incident from November 3rd" and apply the known fix automatically.