soft guideINFRASTRUCTURE & ADVISORY

Case study 07

An on-call engineer that never acts alone

An AI SRE agent inside the cluster: it investigates every alert with read-only tools, reports cause and evidence in chat, and changes nothing without a human approval checked in code.

Role
Designed the agent's safety model and permissions, built its tools, skills and approval flow, and its evaluation suite.
Available as
Autonomous package · also sold for an existing cluster
Alert, sweep or chatInvestigateread-only, own identityReport in chatcause · evidence · proposed fixRunbook actionone approval, auditedPull requestCI, review, GitOpsNO APPROVAL → NOTHING CHANGES · THE AGENT NEVER APPROVES ITSELF

The problem

Small teams cannot staff round-the-clock on-call, and most alerts need the same first hour of work: which service, since when, what changed, what the logs say. The risk with an AI doing that work is not that it is slow; it is that it acts with too much power on too little evidence.

The approach

The agent runs in the cluster with its own identity. It reads metrics, logs, traces, Kubernetes objects, GitOps state and the code, but not secrets. It answers in a fixed shape: verdict, observed facts with the query that produced each one, hypotheses and how each was tested, blast radius, and one next action. A measurement that is the same in the incident window and a healthy window is a precondition, not a cause.

Approval by blast radius

Reading needs no approval. A runbook action, such as restarting a deployment or resuming a reconciliation, runs under a separate operator identity after one approval in chat; the approver is checked in code against the chat identity, not in the prompt, and no answer within thirty minutes counts as a rejection. Anything else is a pull request that goes through CI and review. Secret values are redacted from every message and refused in commits, and every run has a spend limit.

What changed

Alerts arrive already investigated. People decide; the agent does the legwork and leaves an audit trail of every tool call and every decision.

Described as a pattern: names, numbers of systems and identifiers from the environments where this was built are left out.

Start a conversation

Bring your infrastructure
into focus.

Tell me what you are building, operating, or trying to improve.

Let’s talk