soft guideINFRASTRUCTURE & ADVISORYWork / Case studies
Infrastructure that
holds under pressure.
How the platform behind Soft Guide was built: the problems, the decisions, and what changed. Patterns and results, without client or employer details.
A new region is a new cell, not a bigger cluster
A platform that grows by adding identical, isolated cells: one VPC and one cluster each, built from one module and handed to GitOps through a single ConfigMap.
- AWS
- Terraform
- Terramate
- Kubernetes
- GitOps
- Flux
One way out to the internet
Every outbound connection from a cluster leaves through a single gateway that allowlists hostnames, behind three independent fences: routing, CNI policy and an L7 proxy.
- AWS
- Kubernetes
- Cilium
- Envoy
- Security
Network access a service declares, not one it inherits
Default-deny networking where each workload declares its dependencies by label, each destination admits the same label, and a separate, narrowly privileged pipeline applies the policies.
- Kubernetes
- Cilium
- GitOps
- Flux
- Security
Guardrails an account inherits on day one
Organization-wide SCP and RCP guardrails consolidated from per-account copies into one baseline per OU, with offline tests and a dry run against real CloudTrail history.
- AWS
- Security
- Terraform
Admission policies that are proven to block
Kubernetes admission policy written in CEL, tested with good and bad fixtures on every pull request, reported in development and enforced in production.
- Kubernetes
- Kyverno
- Security
- GitOps
Nobody applies from a laptop
Infrastructure delivery through Atlantis: a plan for every changed stack on the pull request, checked against security policy, applied only after approval, nightly drift detection, and a supply chain signed end to end.
- CI/CD
- Atlantis
- Terraform
- Terramate
- AWS
- Security
An on-call engineer that never acts alone
An AI SRE agent inside the cluster: it investigates every alert with read-only tools, reports cause and evidence in chat, and changes nothing without a human approval checked in code.
- AI agents
- Observability
- Kubernetes
- Security
North-south, east-west, egress: three directions, three tools
Notes from choosing a Kubernetes networking stack: the strongest component for each traffic direction instead of one product that does all three adequately.
- Kubernetes
- Cilium
- Envoy
- AWS
No case study for this topic yet.
Case study 01 / 08
A new region is a new cell, not a bigger cluster
A platform that grows by adding identical, isolated cells: one VPC and one cluster each, built from one module and handed to GitOps through a single ConfigMap.
- AWS
- Terraform
- Terramate
- Kubernetes
- GitOps
- Flux
The problem
One large cluster per environment concentrates the blast radius. A bad admission policy, a CNI upgrade or a noisy tenant takes everything down at once, and every upgrade becomes a high-stakes event that nobody wants to schedule.
The approach
The unit of deployment became a cell: one account and region, one VPC with private, egress and public tiers, one EKS cluster, its own keys and identities. A new region or a customer that needs isolation is a new cell. Terramate globals cascade from the organization through the account to the cell, so a new cell is one small stack file; the backend, providers and module call are generated and committed so reviewers read real Terraform.
The bridge
Terraform knows the AWS facts that manifests need: role ARNs, queue names, key ARNs, the VPC. It writes them once into a cell-metadata ConfigMap, and every Flux Kustomization substitutes ${variables} from it. Manifests carry no account, region or ARN, so one GitOps tree serves every cell. Terraform installs only what must exist before Flux: Cilium, the egress gateway and Flux itself.
What changed
Blast radius became one cell. Upgrades roll cell by cell, development first. Creating a cell went from a project to a pull request, and the same reviewed code builds every one of them.
Case study 02 / 08
One way out to the internet
Every outbound connection from a cluster leaves through a single gateway that allowlists hostnames, behind three independent fences: routing, CNI policy and an L7 proxy.
- AWS
- Kubernetes
- Cilium
- Envoy
- Security
The problem
In most clusters any pod can open a connection to anywhere. Exfiltration, a cryptominer or a compromised dependency calling home needs nothing more. Network policies alone are one mistake away from open: a broad rule, a missing selector, an agent left in audit mode.
The approach
Node and pod subnets have no default route. Only a dedicated egress tier routes to NAT, and only the egress gateway runs there. The gateway is an Envoy CONNECT proxy that forwards allowlisted host:port pairs and answers everything else with a 403 and a log line. Around it, a Cilium toFQDNs policy repeats the allowlist at the CNI layer, and a security group lets only that tier send HTTPS out of the VPC. AWS APIs go through VPC endpoints and never touch the gateway.
The details that matter
Wildcards only for a vendor's own zone, never a public suffix or a multi-tenant storage domain: those are exfiltration channels, not allowlist entries. Nodes pull container images through the same gateway, so even image pulls follow the policy. The allowlist doubles as documentation of everything the platform talks to.
What changed
Leaving the network became a decision someone reviews, not a default. A new external dependency is a one-line change in a pull request, and an unexpected destination shows up as a 403 and a dropped flow instead of a successful connection.
Case study 03 / 08
Network access a service declares, not one it inherits
Default-deny networking where each workload declares its dependencies by label, each destination admits the same label, and a separate, narrowly privileged pipeline applies the policies.
- Kubernetes
- Cilium
- GitOps
- Flux
- Security
The problem
Hand-written network policies drift. Callers and the allow-lists of what they call are edited in different places, a copied rule grants more than intended, and a "temporary" broad rule becomes permanent because nobody can prove what removing it would break.
The approach
Every tenant namespace is deny-by-default. A shared dependency is a capability: a reusable component the namespace lists and a label the workload carries. The destination admits callers by that same label, so the two sides cannot drift apart. A destination only one workload uses, such as its own bucket, is a private numbered policy, so a label can never grant access to someone else's data.
Guardrails on the guardrails
Policies live in their own path, applied by a service account that can manage Cilium policies and nothing else. CI rejects world, cluster-wide or 0.0.0.0/0 allows, broad FQDN patterns, CIDR rules without ports, and the cross-namespace selector that silently means "this namespace only". Dropped flows are exported and alerted on, which turns a missing rule into a ticket instead of a mystery timeout.
What changed
Who may talk to whom became readable in one directory and reviewable in one pull request. Broad rules were removed only after real traffic showed the replacement covered every flow: parity first, removal second, evidence always.
Case study 04 / 08
Guardrails an account inherits on day one
Organization-wide SCP and RCP guardrails consolidated from per-account copies into one baseline per OU, with offline tests and a dry run against real CloudTrail history.
- AWS
- Security
- Terraform
The problem
Guardrails had grown account by account: the same rule copied with a different list of ARNs, some accounts at the five-policy quota, new accounts with no guardrails at all. Nobody could say which controls an account actually had without reading every policy on its path.
The approach
One rule lives in one place, at the highest level where it is true. Root protects the audit trail, the root user and the tags that grant privilege. The workloads OU requires permission boundaries, locks regions, enforces IMDSv2 and encryption, and removes interactive host access. Production adds write-path guards: only CI publishes images, only autoscalers start instances, only code changes cluster access and network edges. A resource control policy keeps S3, Secrets Manager and KMS closed to principals outside the organization and to plain HTTP.
Changing guardrails safely
SCPs have no audit mode, so a broken one is found by an outage. Every change passes offline tests for quota, Deny-only content and break-glass exemptions, then a replay of ninety days of CloudTrail against the candidate policy lists every principal it would have denied. New policies attach alongside the old ones for 48 hours before anything is removed, OU by OU, production last.
What changed
Dead and duplicated policies went first, cutting the custom set by a third with no change in effective access. The target layout takes it from nineteen policies to six and from thirteen account-level attachments to one deliberate exception, rolled out OU by OU. The dry run has already paid for itself: it caught a change that would have stopped node autoscaling in production before it shipped.
Case study 05 / 08
Admission policies that are proven to block
Kubernetes admission policy written in CEL, tested with good and bad fixtures on every pull request, reported in development and enforced in production.
- Kubernetes
- Kyverno
- Security
- GitOps
The problem
A policy that looks right in review can block nothing: a verb that does not exist, a check on a field an attacker controls, a match that never fires. Those failures are silent by nature, and they surface during an incident, not before it.
The approach
Pod Security Admission covers privileged pods and host access natively; Kyverno CEL policies add what it does not: images only from the platform registry and signed by the release workflow, pinned versions, requests and memory limits, probes, read-only root filesystems, no default service accounts, no public load balancers without an approval reference and source ranges, cluster-admin bound only by GitOps, and a namespace syncing only its own secrets.
Tested both ways
Every policy ships with fixtures that must be admitted and fixtures that must be rejected by that policy by name. The same policies run in Audit in development, so teams see violations before production rejects them. Exceptions exist, but each names its owner, its reason and its expiry, and exempts a workload, never a namespace.
What changed
A policy that blocks nothing can no longer pass review, because its test fails. Admission policy became something teams can predict, rather than a wall they discover at deploy time.
Case study 06 / 08
Nobody applies from a laptop
Infrastructure delivery through Atlantis: a plan for every changed stack on the pull request, checked against security policy, applied only after approval, nightly drift detection, and a supply chain signed end to end.
- CI/CD
- Atlantis
- Terraform
- Terramate
- AWS
- Security
The problem
Applies from laptops mean unreviewed changes, credentials on laptops and no record of what was applied when. Console fixes become the new normal until the next apply silently reverts them.
The approach
Atlantis plans every changed stack and posts the plan to the pull request, in the order the stacks depend on each other. Each plan is checked with conftest before it may be applied: protected resources cannot be destroyed, and open ingress, wildcard IAM, unencrypted storage and public buckets fail the check. Apply needs an approving review, a mergeable and up-to-date branch, and applies exactly the plan that was reviewed. The policies are baked into the Atlantis image, so a pull request cannot weaken the policy that checks it.
Identity
No keys anywhere. Atlantis runs with its own workload identity, which may only assume a plan role and an apply role in each account; CI publishes images through OIDC roles trusted only from release tags. A nightly plan of every stack opens one issue per drifted stack.
Supply chain
Third-party actions are pinned to commit SHAs; tool versions come from one file for laptops and CI alike. Images are built, scanned, given an SBOM and signed with keyless cosign; the cluster admits only images signed by the organization's release workflow. Build steps run with an allowlisted environment, so a dependency's install script never sees cloud credentials or the job token.
What changed
The plan a reviewer approves is the plan that runs, and security rules are enforced on the plan itself rather than remembered by reviewers. Drift is found the next morning instead of during the next incident.
Case study 07 / 08
An on-call engineer that never acts alone
An AI SRE agent inside the cluster: it investigates every alert with read-only tools, reports cause and evidence in chat, and changes nothing without a human approval checked in code.
- AI agents
- Observability
- Kubernetes
- Security
The problem
Small teams cannot staff round-the-clock on-call, and most alerts need the same first hour of work: which service, since when, what changed, what the logs say. The risk with an AI doing that work is not that it is slow; it is that it acts with too much power on too little evidence.
The approach
The agent runs in the cluster with its own identity. It reads metrics, logs, traces, Kubernetes objects, GitOps state and the code, but not secrets. It answers in a fixed shape: verdict, observed facts with the query that produced each one, hypotheses and how each was tested, blast radius, and one next action. A measurement that is the same in the incident window and a healthy window is a precondition, not a cause.
Approval by blast radius
Reading needs no approval. A runbook action, such as restarting a deployment or resuming a reconciliation, runs under a separate operator identity after one approval in chat; the approver is checked in code against the chat identity, not in the prompt, and no answer within thirty minutes counts as a rejection. Anything else is a pull request that goes through CI and review. Secret values are redacted from every message and refused in commits, and every run has a spend limit.
What changed
Alerts arrive already investigated. People decide; the agent does the legwork and leaves an audit trail of every tool call and every decision.
Case study 08 / 08
North-south, east-west, egress: three directions, three tools
Notes from choosing a Kubernetes networking stack: the strongest component for each traffic direction instead of one product that does all three adequately.
- Kubernetes
- Cilium
- Envoy
- AWS
North-south
Gateway API as the interface, the AWS Load Balancer Controller as the implementation. The data plane stays in the AWS load balancer, so no extra proxy runs in the cluster just to receive traffic. The trade-off: target registration is an external control-plane operation and reacts more slowly than an in-cluster proxy, so test it against your availability targets.
East-west
Cilium handles L3 and L4 in the kernel with eBPF, and policy is by workload identity, not IP. Where L7 is genuinely needed (gRPC routing, retries, circuit breaking, L7 policy), traffic is redirected to a node-local Envoy, not a sidecar in every pod. The Gateway API mesh profile describes those routes in a standard API; it standardises routing, not mutual TLS, which is a separate decision.
Egress
The requirement is stronger than a stable source IP: a central point that reads the request and asks business logic before letting it out. A dedicated Envoy gateway does that, with external authorization served over a Unix socket inside the same pod, so the decision costs no network hop. Cilium's egress gateway is a different tool: L3 and L4 source-address control, useful underneath, not a replacement.
Mental models
eBPF is not L7, and the kernel cannot parse HTTP. Transparency is ergonomics; latency is hops. Gateway API is a specification, not a product. Priority classes order scheduling; they do not make one pod wait for another.
Start a conversation
Bring your infrastructure
into focus.
Tell me what you are building, operating, or trying to improve.