Cilium from the first node, and why not the VPC CNI

The default, and why it is not enough

An EKS cluster comes with the Amazon VPC CNI for pod networking and kube-proxy for Services. They work, and for many clusters they are the right choice. The platform replaces both with Cilium, for reasons that are about security and operations rather than taste.

Why Cilium

• Policy by identity, not by IP. Cilium enforces network policy on workload identity derived from labels, in the kernel with eBPF. Pod IPs change constantly; identities do not. The platform's default-deny model with declared capabilities depends on this.
• FQDN policy. toFQDNs lets a policy say "this workload may reach api.example.com on 443" and have Cilium's DNS proxy enforce it. The VPC CNI's network policy support has no equivalent; the egress gateway's second fence is built on it.
• Seeing what was dropped. Hubble records every flow with its verdict. "Why can't A reach B" becomes a log query instead of a guess, and dropped flows become an alert.
• kube-proxy replacement. Service load balancing in eBPF with Maglev hashing instead of iptables chains that grow with every Service, and node processes can reach ClusterIPs, which the platform uses so nodes pull images through the egress gateway.
• L7 only where needed. When a route needs retries or gRPC-aware balancing, Cilium sends that traffic through a node-local Envoy; everything else stays in the kernel. No sidecar in every pod.

Why from the first node

The common path is to create the cluster with the VPC CNI, install Cilium later, and remove the old CNI with a script. That means two applies, a period where nodes run two datapaths, and a cleanup step that has to be right on every cluster.

The platform instead creates EKS with no self-managed add-ons, installs Cilium with Terraform right after the control plane, and only then creates node groups. Nodes carry a node.cilium.io/agent-not-ready taint until Cilium is running on them, and CoreDNS tolerates it. One apply builds a working cluster.

resource "helm_release" "cilium" {
  chart = "cilium"
  wait  = false # nodes join after this release
  values = [yamlencode({
    ipam                 = { mode = "eni" }
    eni                  = { enabled = true, awsEnablePrefixDelegation = true }
    routingMode          = "native"
    kubeProxyReplacement = true
  })]
}
module "node_group_system" { depends_on = [helm_release.cilium] }

Why ENI mode

In ENI mode Cilium gives each pod an address from its node's subnet, as the VPC CNI does. That keeps pods routable in the VPC without overlays, and it is what makes subnet tiers mean something for pods: a pod on a private-tier node is in the private tier and has no route to the internet; a pod on an egress-pool node leaves through that pool's NAT and fixed IP.

What it costs

• You own the CNI. Cilium upgrades are platform changes, planned and rolled cell by cell, not an AWS-managed add-on version bump.
• Support boundary. AWS supports the VPC CNI; for Cilium you rely on the project, a commercial distribution, or your own team.
• IP planning still matters. ENI mode uses VPC addresses; prefix delegation makes it efficient, but subnets must be sized for pods, not only nodes.

For a platform whose security model is built on identity-aware policy, FQDN egress and flow visibility, those costs are worth paying. For a single small cluster with no network policy, the default is fine.