Note 03
One way out to the internet, three fences around it
How the Secure package makes a cluster unable to reach the internet except through one allowlisting gateway, and why one control is never enough.
- AWS
- Kubernetes
- Cilium
- Envoy
- Security
The problem
Most clusters let any pod open a connection to anywhere. A compromised dependency calling home, a cryptominer, data leaving through an innocent-looking HTTPS request: none of them need more than that. Network policy alone does not fix it, because it is one mistake away from open: a broad rule added in a hurry, a selector that matches nothing, an agent left in audit mode.
Three independent fences
- Routing. Node and pod subnets have no default route. Only a dedicated egress tier routes to NAT, and only the egress gateway runs there. A pod that ignores every policy still has nowhere to send internet traffic.
- CNI policy. The gateway pods may reach only the allowlisted hostnames, enforced by Cilium's DNS proxy with
toFQDNs. Application pods may reach the gateway only if they carry the capability label. - L7 proxy. The gateway is an Envoy CONNECT proxy. It forwards only allowlisted
host:portpairs and answers everything else with a 403 and a JSON log line.
A security group lets only the egress tier send HTTPS out of the VPC: a fourth, coarse fence. AWS APIs never touch the gateway; they are reached through VPC endpoints.
virtual_hosts:
- name: allowed
domains: ["github.com:443", "*.githubusercontent.com:443", "api.stripe.com:443"]
routes:
- match: { connect_matcher: {} }
route: { cluster: dynamic_forward_proxy, upgrade_configs: [{ upgrade_type: CONNECT, connect_config: {} }] }
- name: denied
domains: ["*"]
routes:
- match: { connect_matcher: {} }
direct_response: { status: 403 }The details that matter
- Wildcards only for a vendor's own zone.
*.github.iowould let anything hosted on GitHub Pages through;*.s3.amazonaws.comis an exfiltration channel, not an allowlist entry. - Nodes pull container images through the same gateway, by a fixed ClusterIP, because they do not use cluster DNS. Even image pulls follow the policy.
- The gateway is installed by Terraform, before Flux, because Flux reaches GitHub through it.
- Workloads that must call hosts nobody can list (webhooks to customer URLs) get an egress pool with its own fixed IP and a named exception, instead of a wildcard on the gateway.
What changes
Leaving the network becomes a decision someone reviews. A new dependency is a one-line allowlist change in a pull request; an unexpected destination shows up as a 403 and a dropped flow instead of a successful connection. And the allowlist is, for free, the documentation of everything the platform talks to.
Start a conversation
Bring your infrastructure
into focus.
Tell me what you are building, operating, or trying to improve.