Compute

Karpenter

We use Karpenter for Cluster Autoscaling.

Karpenter dynamically adds and removes nodes in the EKS cluster based on:

• Pending pod requirements
• Node utilization
• Available AWS instance types
• Current AWS pricing

Node Types

Generally, Constellation has three types of nodes:

1. EKS Managed Node Group (workload-tier: baseline)

These are EKS-managed NodeGroups created by Terraform. EKS MNG Nodes are tainted, preventing non-critical workloads from being scheduled on them.

2. On-Demand Instances (workload-tier: on-demand)

Regular EC2 instances billed per-second/hour with no commitment. They provide guaranteed capacity (within AWS service limits) and are not interrupted.

3. Spot Instances (workload-tier: spot)

Spot instances use spare AWS capacity at a large discount (typically 60–90% cheaper), but AWS may interrupt and reclaim the instance with 2 minutes notice.

Tip

Using Spot instances significantly reduces compute costs for Constellation clusters.

Pod Density & Nitro Instances

Constellation runs Cilium in AWS ENI IPAM mode with prefix delegation: pods draw real VPC IPs, and each ENI is assigned /28 prefixes (16 IPs) from dedicated 100.64.0.0/16 pod subnets (infra/pod_subnets.tf), selected by Cilium via eni.subnetTagsFilter. This gives ~110 pods/node, so kubelet maxPods is set to 110 on both the Karpenter nodeclass and the baseline MNG.

Because ~6–9 platform DaemonSets (Cilium, cilium-envoy, EBS/EFS CSI, node-exporter, Alloy, …) run on every node, a very small node (e.g. the 12-slot m3.medium) can fill up before application pods — or even other DaemonSets — can schedule. Karpenter does not rescue a Pending DaemonSet pod: the DaemonSet controller pins each pod to one existing node, so Karpenter never provisions a new node on its behalf.

All NodePools require Nitro hardware — karpenter.k8s.aws/instance-hypervisor: ["nitro"] (prefix delegation needs Nitro). The spot/on-demand pools also require ≥ large instances (instance-size NotIn [nano,micro,small,medium]): pods use secondary ENIs only (the primary eth0 is in the untagged /24), so capacity is (maxENIs − 1) × IPv4perENI × 16, and 2-ENI instances like m7a.medium top out at ~64 IPs — too few to back maxPods: 110.

Raising maxPods depends on the pod subnets

maxPods: 110 is only safe because pods come from the 100.64.0.0/16 pod subnets where /28 prefix delegation works. Do not enable it on a cluster whose pod subnets aren't applied yet (e.g. prod until its terraform apply runs), and keep the ≥ large NodePool guardrail — without it, small instances overcommit IPs and pods hang in ContainerCreating. Verify prefix delegation is live (ENI Ipv4Prefixes populated) before relying on it. See ADR-0013.

Constellation Compute Strategy

Our workload placement strategy is as follows:

• Cluster Critical Workloads (ArgoCD, CoreDNS, Cilium, Karpenter, Traefik) -> workload-tier: baseline
• Business Critical Workloads (Passport, etc.) -> workload-tier: on-demand
• All Other Applications -> workload-tier: spot

This strategy enables us to reduce compute costs while maintaining stability and predictability for critical services.

Workload Pinning

Using spec.template.spec.nodeSelector

Add a spec.template.spec.nodeSelector to your deployments to specify the workload-tier:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
spec:
  replicas: 1
  template:
    spec:
      nodeSelector:
        workload-tier: spot # or on-demand, baseline

Using kustomize

Alternatively, apply it via a kustomize patch:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

patches:
  - target:
      kind: Deployment # applies to all Deployments
    patch: |-
      - op: add
        path: /spec/template/spec/nodeSelector
        value:
          workload-tier: on-demand

PodDisruptionBudgets (PDBs)

PodDisruptionBudgets ensure Kubernetes does not evict too many pods at once during events such as:

• Node scale-down
• Karpenter consolidation
• Spot interruptions
• Planned maintenance

Example: For a Deployment with app=my-service and 2+ replicas:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-service-pdb
spec:
  selector:
    matchLabels:
      app: my-service
  minAvailable: 1

• This ensures at least 1 pod is always running.
• It allows voluntary disruptions (Karpenter draining, node upgrades, etc.) as long as that condition stays true.