Security

SSO using Passport

You can protect your app with SSO via EntraID by attaching the traefik-forward-auth-passport@kubernetescrd middleware to your Ingress.

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    # TLS via cert-manager
    cert-manager.io/cluster-issuer: letsencrypt
    cert-manager.io/acme-challenge-type: dns01

    # Traefik Ingress Annotations
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.middlewares: traefik-forward-auth-passport@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
    - host: "" # Provided via kustomize overlays patches
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                # Should match service.yaml name
                name: example-app
                port:
                  # Should match service.yaml port
                  number: 80
  tls:
    - hosts: [] # Provided via kustomize overlays patches
      secretName: example-app-cert

Don't forget to also have the required Passport Network Policies in place.

TLS

Use the following Ingress annotations to: 1. Expose your service's port. 2. Create a Route53 record. 3. Acquire a TLS certificate.

Info

cert-manager only watches networking.k8s.io/v1 Ingress resources, so do not use Traefik's IngressRule CR.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    # Use cert-manager to attach TLS certificates for the specified hosts
    cert-manager.io/cluster-issuer: letsencrypt
    cert-manager.io/acme-challenge-type: dns01
spec:
  ingressClassName: traefik
  rules:
    - host: host.sandbox.clearroute.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: host
                port:
                  number: 80
  tls:
    - hosts: [host.sandbox.clearroute.io] # TLS Cert CN
      secretName: host-cert

IAM

IRSA

If your Kubernetes Application Service Account needs to assume an IAM role, you can leverage IRSA.

The following example shows how to create an IAM role & policy and map it to the specified namespace & service_account_name. Once created, you can annotate your ServiceAccount with the role_arn (see example below).

1. Create IAM Role & Policy and Map it to a namespace + ServiceAccount

# infra/cluster/<cluster_name>.tfvars
irsa = {
  cluster_autoscaler = {
    namespace            = "kube-system"
    service_account_name = "cluster-autoscaler"
    iam_policy = {
      # # https://docs.aws.amazon.com/eks/latest/best-practices/cas.html
      "ClusterAutoscalerAll" = {
        actions = [
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeAutoScalingInstances",
          "autoscaling:DescribeLaunchConfigurations",
          "autoscaling:DescribeTags",
          "autoscaling:SetDesiredCapacity",
          "autoscaling:TerminateInstanceInAutoScalingGroup",
          "ec2:DescribeLaunchTemplateVersions",
          "ec2:DescribeInstanceTypes",
          "eks:DescribeNodegroup"
        ]
        resources = ["*"]
      }
    }
  }
  external_secrets_operator = {
    namespace            = "external-secrets"
    service_account_name = "external-secrets"
    iam_policy = {
      "ExternalSecretsOperatorAll" = {
        actions = [
          "secretsmanager:ListSecrets",
          "secretsmanager:BatchGetSecretValue",
          "secretsmanager:GetResourcePolicy",
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret",
          "secretsmanager:ListSecretVersionIds"
        ]
        resources = ["*"]
      }
    }
  }
}

2. Annotate your Application's ServiceAccount

This step depends on how your application is managed.

# applications/{external,internal}/<app-name>/overlays/{sandbox,dev,prod}/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

[...]

patches:
  - target:
      kind: ServiceAccount
      name: serviceaccount
    patch: |
      - op: replace
        path: /metadata/annotations/eks.amazonaws.com~1role-arn # Kustomize uses ~1 to escape / in JSON pointer paths
        value: <role_arn> # e.g., arn:aws:iam::799468650620:role/comply-backend-dev

IRSA - Cross Account

Similarly, if your application's ServiceAccount needs to assume a role in ClearRoute's main account (e.g., for Route53 operations), you must use the irsa_clearroute_account variable:

Example:

irsa_clearroute_account = {
  external_dns = {
    namespace            = "external-dns"
    service_account_name = "external-dns"
    iam_policy = {
      # https://kubernetes-sigs.github.io/external-dns/latest/docs/tutorials/aws/#iam-policy
      "ChangeResourceRecordSets" = {
        actions   = ["route53:ChangeResourceRecordSets"]
        resources = ["arn:aws:route53:::hostedzone/*"]
      }
      "Route53Records" = {
        actions = [
          "route53:ListHostedZones",
          "route53:ListResourceRecordSets",
          "route53:ListTagsForResources"
        ]
        resources = ["*"]
      }
    }
  }
  cert_manager = {
    namespace            = "cert-manager"
    service_account_name = "cert-manager"
    iam_policy = {
      "ChangeResourceRecordSets" = {
        actions = [
          "route53:ChangeResourceRecordSets",
          "route53:ListResourceRecordSets"
        ]
        resources = ["arn:aws:route53:::hostedzone/*"]
        conditions = {
          "ForAllValues:StringEquals" = {
            variable = "route53:ChangeResourceRecordSetsRecordTypes"
            values   = ["TXT"]
          }
        }
      }
      "ChangeResourceRecordSets" = {
        actions = [
          "route53:ListHostedZones",
          "route53:ListHostedZonesByName",
          "route53:GetChange",
          "route53:GetHostedZone",
        ]
        resources = ["*"]
      }
    }
  }
}

Secrets Management

Synchronizing Secrets from AWS SSM

Here is an example of how to create (and synchronize) a Kubernetes Secret from an AWS Secrets Manager Secret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: <name>
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secretsmanager
  target:
    name: <secret-name> # The Kubernetes secret name
  data:
    - secretKey: username # The Kubernetes secret key
      remoteRef:
        key: constellation/external-secrets/test # Change to your path
        property: username # The AWS secret key
    - secretKey: password
      remoteRef:
        key: constellation/external-secrets/test
        property: password

If you prefer to map every key-value pair from your AWS secret, you can use dataFrom (this method is preferred):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: <name>
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secretsmanager
  target:
    name: <secret-name> # The Kubernetes secret name
  dataFrom:
   - extract:
      key: constellation/external-secrets/test

You can then mount and consume the secret in your Pods using env or envFrom.

ImagePullSecrets

If you need to pull from a private registry, you must create an ImagePullSecret. Here is an example:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: clearroute-image-pull-secret
  annotations:
    argocd.argoproj.io/hook: PreSync
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secretsmanager
  target:
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "ghcr.io": {
                "username": "{{ .username }}",
                "password": "{{ .password }}",
                "auth": "{{ printf "%s:%s" .username .password | b64enc }}"
              }
            }
          }
  dataFrom:
    - extract:
        key: constellation/sandbox/clearroute-pullsecret # contains username & password