Skip to content

ADR-0013: Enable Cilium Prefix Delegation on AWS EKS

Date: 2026-07-05 Status: accepted Deciders: Tom Morelly (ClearRoute platform team)

Context

Cilium (ADR-0003) runs in AWS ENI IPAM mode: every pod gets a real VPC IP from an ENI attached to its node, so a node's pod count is bounded by its instance type's ENI/IP limit — as low as 12 on m3.medium, 35 on t3.large. Because ~6–9 platform DaemonSets (Cilium, cilium-envoy, EBS/EFS CSI, node-exporter, Alloy, …) run on every node, small nodes fill up before application pods can schedule. When the Alloy DaemonSet landed (ADR-0012), its pods sat Pending on full m3.medium nodes — and Karpenter (ADR-0002) does not rescue a pinned DaemonSet pod, so they never recovered.

Two structural limits caused this:

  1. Per-node IP density is tied to ENI limits. AWS supports prefix delegation — assigning each ENI a /28 prefix (16 IPs) instead of individual secondary IPs — which raises the ceiling to ~110 pods/node. But it requires Nitro instances and contiguous /28 blocks in the pod subnet, and kubelet's max-pods must be raised to advertise the extra capacity.
  2. The pod subnets were too small. Pods drew IPs from the two /24 private subnets (~512 IPs cluster-wide), which are too fragmented for AWS to carve contiguous /28 prefixes — so prefix delegation silently fell back to individual IPs ("Subnet might be out of prefixes").

Decision

Enable prefix delegation properly via four coordinated changes:

  1. Dedicated pod subnets (infra/pod_subnets.tf): add secondary VPC CIDR 100.64.0.0/16 (RFC 6598) with one /18 per AZ, tagged cilium.io/pod-subnet=<env>. They reuse the module's per-AZ private route tables (existing NAT gateways) for egress; intra-VPC routing to 10.0.0.0/16 (RDS, services) is covered by the local route AWS auto-adds for the secondary CIDR. This gives the contiguous space /28 prefix delegation needs.
  2. Point Cilium at those subnets (applications/argocd/dev/cilium.yaml): eni.subnetTagsFilter: ["cilium.io/pod-subnet=<cluster>"], with eni.awsEnablePrefixDelegation: true. The node's primary ENI (eth0) is in the untagged /24, so this operator-level filter also keeps pods off eth0 — pod ENIs are created only in 100.64. Pod ENIs inherit the node security group by default, so SG-based rules (RDS ingress from the node SG, infra/rds.tf) keep working.
  3. Nitro-only NodePools: karpenter.k8s.aws/instance-hypervisor In ["nitro"] on spot, on-demand, and orbit — prefix delegation requires Nitro, and this excludes the pathological non-Nitro m3.medium.
  4. Raise the pod ceiling to 110 on the default EC2NodeClass and the baseline EKS MNG (infra/eks.tf, via an AL2023 nodeadm NodeConfig merge — no custom ami_id, which an EKS-optimized node group rejects). Because eth0's /24 is excluded, per-node pod capacity is (maxENIs − 1) × IPv4perENI × 16; 2-ENI instances like m7a.medium top out at ~64 IPs, so the spot/on-demand pools also require instance-size NotIn [nano,micro,small,medium] (≥ large). The baseline MNG's t3 (3 ENIs, ~192 IPs) is already safe.

This narrows the broad instance allowlist from ADR-0002.

Alternatives Considered

Restrict by instance generation (instance-generation Gt 4) instead of hypervisor

  • Cons: generation doesn't map to Nitro — t3/t3a are gen-3 but Nitro, so Gt 4 would wrongly exclude the t3a instances the orbit pool relies on.
  • Why not: instance-hypervisor In ["nitro"] states the actual hardware requirement precisely.

Re-carve the primary 10.0.0.0/16 into larger pod subnets

  • Cons: existing subnets can't be resized; recreating them recreates the cluster's networking (nodes, NAT, RDS subnet group) — highly disruptive on a live cluster.
  • Why not: a secondary CIDR is additive and non-disruptive.

Move nodes into the secondary CIDR too (not just pods)

  • Cons: node primary IPs in CGNAT space; churns NLB target registration, RDS SG source, and node placement for no extra benefit.
  • Why not: only pods need the IP headroom; keeping nodes put minimizes blast radius.

Switch Cilium to overlay/tunnel mode (pod IPs off the VPC)

  • Cons: gives up native VPC routing and per-pod VPC reachability; a disruptive CNI-level change.
  • Why not: dedicated pod subnets solve density while keeping the native routing we rely on.

Consequences

Positive

  • ~110 pods/node: the fixed DaemonSet overhead drops from ~60% of a small node to under 10%, and DaemonSets (Alloy, node-exporter) schedule everywhere.
  • Cluster-wide pod-IP ceiling rises from ~512 to tens of thousands (roomy 100.64 subnets).
  • Additive and non-disruptive: no change to existing subnets, nodes, NAT, or RDS; SG-based access is preserved because pod ENIs inherit the node security group.

Negative

  • Pod IPs are now in CGNAT space (100.64.x); anything that allow-lists pods by CIDR (rather than by SG or Cilium identity) must include 100.64.0.0/16. Today's RDS access is SG-based, so this is a no-op, but future CIDR-based rules must account for it.
  • The instance set is narrower (Nitro, ≥ large on spot/on-demand), marginally reducing spot breadth.
  • One more VPC construct (secondary CIDR + per-AZ pod subnets) to keep aligned.

Risks & operational notes

  • Ordering: pod subnets (terraform apply) must exist before the Cilium subnetTagsFilter syncs — otherwise Cilium has no subnet to allocate from and pod IP allocation fails cluster-wide.
  • Keep the three levers together: Nitro + ≥ large + maxPods: 110. Relaxing Nitro or the size floor without lowering maxPods lets kubelet advertise pods a node can't get IPs for.
  • prod pins its Karpenter/management sync to a commit, so this does not auto-apply there; prod adopts by bumping its pin after its own terraform apply creates prod pod subnets. Prod adopted this by adding eni.subnetTagsFilter to applications/argocd/prod/cilium.yaml and bumping the applications/argocd/prod/karpenter.yaml pin to pick up the Nitro/≥ large/maxPods: 110 NodePool changes; the pod subnets and baseline-MNG maxPods are env-agnostic Terraform applied to both accounts by Atlantis.
  • Verification (dev): a recycled node shows a /28 prefix in 100.64 on its pod ENI, eth0 with 0 pod IPs, maxPods: 110, and no "out of prefixes" warnings in the cilium-operator log.