Case study 07
An on-call engineer that never acts alone
An AI SRE agent inside the cluster: it investigates every alert with read-only tools, reports cause and evidence in chat, and changes nothing without a human approval checked in code.
- AI agents
- Observability
- Kubernetes
- Security
The problem
Small teams cannot staff round-the-clock on-call, and most alerts need the same first hour of work: which service, since when, what changed, what the logs say. The risk with an AI doing that work is not that it is slow; it is that it acts with too much power on too little evidence.
The approach
The agent runs in the cluster with its own identity. It reads metrics, logs, traces, Kubernetes objects, GitOps state and the code, but not secrets. It answers in a fixed shape: verdict, observed facts with the query that produced each one, hypotheses and how each was tested, blast radius, and one next action. A measurement that is the same in the incident window and a healthy window is a precondition, not a cause.
Approval by blast radius
Reading needs no approval. A runbook action, such as restarting a deployment or resuming a reconciliation, runs under a separate operator identity after one approval in chat; the approver is checked in code against the chat identity, not in the prompt, and no answer within thirty minutes counts as a rejection. Anything else is a pull request that goes through CI and review. Secret values are redacted from every message and refused in commits, and every run has a spend limit.
What changed
Alerts arrive already investigated. People decide; the agent does the legwork and leaves an audit trail of every tool call and every decision.
Described as a pattern: names, numbers of systems and identifiers from the environments where this was built are left out.
Start a conversation
Bring your infrastructure
into focus.
Tell me what you are building, operating, or trying to improve.