Security

TLS

Use the following Ingress Annotations to 1. expose your services port, 2. get a route53 record, 3. acquire a TLS cert:

Info

cert-manager only watches networking.k8s.io/v1 Ingress resources so do not use Trafiks IngressRule CR.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: host
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    cert-manager.io/acme-challenge-type: dns01
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    external-dns.alpha.kubernetes.io/hostname: host.dev.clearroute.io
spec:
  ingressClassName: traefik
  rules:
    - host: host.sandbox.clearroute.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: host
                port:
                  number: 80
  tls:
    - hosts: [host.sandbox.clearroute.io]
      secretName: host-cert

IAM

IRSA

If your Kubernetes Application Service Account needs to assume an IAM role you can leverage IRSA.

Here are to example to create an IAM role & policy and map it to the specified namespace & service_account_name. Once created you can annotate your ServiceAccount with the role_arn (see example below)

1. Create IAM role & policy and map it to a namespace + ServiceAccount:

# infra/cluster/<cluster_name>.tfvars
irsa = {
  cluster_autoscaler = {
    namespace            = "kube-system"
    service_account_name = "cluster-autoscaler"
    iam_policy = {
      # # https://docs.aws.amazon.com/eks/latest/best-practices/cas.html
      "ClusterAutoscalerAll" = {
        actions = [
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeAutoScalingInstances",
          "autoscaling:DescribeLaunchConfigurations",
          "autoscaling:DescribeTags",
          "autoscaling:SetDesiredCapacity",
          "autoscaling:TerminateInstanceInAutoScalingGroup",
          "ec2:DescribeLaunchTemplateVersions",
          "ec2:DescribeInstanceTypes",
          "eks:DescribeNodegroup"
        ]
        resources = ["*"]
      }
    }
  }
  external_secrets_operator = {
    namespace            = "external-secrets"
    service_account_name = "external-secrets"
    iam_policy = {
      "ExternalSecretsOperatorAll" = {
        actions = [
          "secretsmanager:ListSecrets",
          "secretsmanager:BatchGetSecretValue",
          "secretsmanager:GetResourcePolicy",
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret",
          "secretsmanager:ListSecretVersionIds"
        ]
        resources = ["*"]
      }
    }
  }
}

2. Annotate your applications ServiceAccount

That steps depends on your application annotation

# applications/{external,internal}/<app-name>/overlays/{sandbox,dev,prod}/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

[...]

patches:
  - target:
      kind: ServiceAccount
      name: serviceaccount
    patch: |
      - op: replace
        path: /metadata/annotations/eks.amazonaws.com~1role-arn # kustomize uses ~1 to escape / in json pointer paths
        value: <role_arn> # e.g arn:aws:iam::799468650620:role/clearcomply-backend-dev

IRSA - Cross Account

Similarly, if your applications ServiceAccount needs to assume a role in ClearRoutes main account (e.g for Route53 operations), you must use the irsa_clerroute_account variable:

Example:

irsa_clearroute_account = {
  external_dns = {
    namespace            = "external-dns"
    service_account_name = "external-dns"
    iam_policy = {
      # https://kubernetes-sigs.github.io/external-dns/latest/docs/tutorials/aws/#iam-policy
      "ChangeResourceRecordSets" = {
        actions   = ["route53:ChangeResourceRecordSets"]
        resources = ["arn:aws:route53:::hostedzone/*"]
      }
      "Route53Records" = {
        actions = [
          "route53:ListHostedZones",
          "route53:ListResourceRecordSets",
          "route53:ListTagsForResources"
        ]
        resources = ["*"]
      }
    }
  }
  cert_manager = {
    namespace            = "cert-manager"
    service_account_name = "cert-manager"
    iam_policy = {
      "ChangeResourceRecordSets" = {
        actions = [
          "route53:ChangeResourceRecordSets",
          "route53:ListResourceRecordSets"
        ]
        resources = ["arn:aws:route53:::hostedzone/*"]
        conditions = {
          "ForAllValues:StringEquals" = {
            variable = "route53:ChangeResourceRecordSetsRecordTypes"
            values   = ["TXT"]
          }
        }
      }
      "ChangeResourceRecordSets" = {
        actions = [
          "route53:ListHostedZones",
          "route53:ListHostedZonesByName",
          "route53:GetChange",
          "route53:GetHostedZone",
        ]
        resources = ["*"]
      }
    }
  }
}

Secrets Management

Synchronizing Secrets from AWS SSM

Here is an example for how to create (and synchronize) a Kubernetes Secret from an AWS Secrets Manager Secret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: <name>
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secretsmanager
  target:
    name: <secret-name> # the kubernetes secret name
  data:
    - secretKey: username # the kubernetes secret key
      remoteRef:
        key: constellation/external-secrets/test # change to your path
        property: username # the AWS secret key
    - secretKey: password
      remoteRef:
        key: constellation/external-secrets/test
        property: password 

if you instead just want to map every key-value pair from your AWS secret you can use dataFrom (this method is preferred):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: <name>
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secretsmanager
  target:
    name: <secret-name> # the kubernetes secret name
  dataFrom:
   - extract:
      key: constellation/external-secrets/test

You can then mount and consume the secret in your Pods using env or envFrom.

ImagePullSecrets

If you need to pull from a private registry, you need to create an ImagePullSecret, here is an example:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: clearroute-image-pull-secret
  annotations:
    argocd.argoproj.io/hook: PreSync
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secretsmanager
  target:
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "ghcr.io": {
                "username": "{{ .username }}",
                "password": "{{ .password }}",
                "auth": "{{ printf "%s:%s" .username .password | b64enc }}"
              }
            }
          }  
  dataFrom:
    - extract:
        key: constellation/sandbox/clearroute-pullsecret # contains username & password