An on-call engineer that acts within your rules

What the agent is for

Most alerts need the same first hour of work: which service, since when, what changed, what the logs say. An agent can do that hour. The risk is not that it is slow; it is that it acts with too much power on too little evidence.

Built on proven pieces

• Behaviour: oncall-kit. Its setup mines the team's incident history into triage playbooks, ends every phase at a human gate, and requires a blind replay eval and a shadow period before anyone relies on it.
• Conversation: Claude Tag in the team's Slack on-call and incident channels.
• Execution: self-hosted runners inside the cell. Hosted sandboxes cannot reach private endpoints, so sessions run next to the data. An orchestrator holds the environment key and starts one Kubernetes Job per session with a single-use work order; the key never reaches a pod that runs a session.

# spawn-runner hook: one Job per order, named from the order ID so redelivery creates nothing new
name="runner-$(printf '%s' "$CLAUDE_RUNNER_ORDER_ID" | tr 'A-Z_' 'a-z-' | cut -c1-50)"
kubectl -n agents get job "$name" >/dev/null 2>&1 && exit 0
# ... create the Job, store the work order in a Secret owned by it, exit without waiting

Acting, not only advising

oncall-kit keeps its agent read-only. That is the right start, but an agent that can only describe the restart still wakes someone up to type it. So the agent can act, through one door: an action broker in the cell. The agent's own identity stays read-only; to change something it asks the broker, with a reason.

act restart deployment/checkout -n shop \
  --reason "OOMKilled 6 times since 02:10 after the 1.42 rollout; memory flat on 1.41"
# pending a1b2c3: waiting for approval in #platform-approvals

The broker knows five typed actions: restart a workload, scale it up to a ceiling, delete one pod, reconcile or resume a Flux object. There is no action that runs a command or applies a manifest. A policy in Git says which namespaces are open, who may approve, and which actions may run without asking, up to a number per hour. Everything starts behind approval: the broker posts the request to Slack with Approve and Reject buttons and runs it only when someone on the approver list presses one. When a team has approved the same restart for the tenth time, they move that action to run on its own with a pull request, not a prompt.

Lasting fixes, such as a memory limit, a configuration value or a version, stay pull requests. The DevSecOps agent writes only pull requests; branch protection keeps merging with people.

Guarantees that do not depend on the model

• The agent's own RBAC is read-only, with no Secrets, no exec, no write verbs; managed settings in the image deny direct write commands.
• The broker can run its five actions and nothing else. Its role is bound only in the namespaces the policy opens, and platform namespaces (Flux, Kyverno, the gateway, the agents themselves) are refused in its code as well.
• Approvers are checked by the broker against the chat identity; a request nobody answers expires as rejected; enabled: false stops every action.
• Network: the model API, GitHub, the in-cluster observability endpoints and the broker, nothing else.
• Every request, decision and result is logged with the agent's reason and the approver.

Text the agent reads from logs, alerts and tickets is data, never instructions; that rule is in its prompt, and the credentials make sure it does not matter if the prompt is ignored.

What it changes

Alerts arrive already investigated. People decide; the agent does the legwork and leaves an audit trail of every tool call and every decision.