Networking

Network Policies with Cilium

We use Cilium to enforce NetworkPolicy on Constellation Workloads.

Warning

Every workload deployed in a app-* namespace (e.g specified under applications/{external,internal}) is blocked per default (except for CoreDNS & KubeAPI requests). This is done by labeling the namespace with deny-default-all: "true" in the App ApplicationSet (Namespace labels are inherited to all workloads within the namespace).

This means that your workload will not be able to connect to anything unless you explicitly allow it

Available NetworkPolicies

Cilium matches NetworkPolicies to Identities (= Kubernetes Labels), all you have to do is to label your app appropriately and you app should be able to connect to external or internal workloads.

The following network policies are available:

• deny-default-all -> Blocks all traffic except CoreDNS & KubeAPI. Enabled by default, enforced on namespace level
• allow-egress-rds -> allow your app to connect to RDS
• allow-ingress-traefik -> allow traefik to route incoming requests to your app
• allow-ingress-prometheus -> allow prometheus to scrape your app
• allow-egress-world -> allow outgoing connections to any endpoints outside of the cluster

Usage

Simply add the desired label for the correct network policy to your apps kustomization.yaml:

# applications/{external,internal}/<app>/overlays/<env>/kustomization.yaml
labels:
  - pairs:
      app: clearcomply
      # network policies
      allow-ingress-traefik: "true"
      allow-egress-rds: "true"
    includeSelectors: true

Allow Pod-to-Pod Communication (e.g Frontend to Backend)

In order to allow your frontend communicate with your backend, you will need to allow your frontend egressing to your backend and your backend ingressing from your frontend pod. You will need a distinct label sets to uniquely identify your backend and frontend pods.

Tip

Even if your frontend-app uses the service FQDN of your backend (e.g backend-service.app-namespace.svc.cluster.local) you will need to explicitly allow your frontend Pod to communicate with your backend Pod

Example Network Policy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-ingress-backend-from-frontend
  namespace: app-clearcomply
spec:
  endpointSelector:
    matchLabels:
      k8s:component: clearcomply-backend
      k8s:io.kubernetes.pod.namespace: app-clearcomply
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:component: clearcomply-frontend
            k8s:io.kubernetes.pod.namespace: app-clearcomply
      toPorts:
        - ports:
            - port: "8080" # Pod/Container Port
              protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-egress-frontend-to-backend
  namespace: app-clearcomply
spec:
  endpointSelector:
    matchLabels:
      k8s:component: clearcomply-frontend
      k8s:io.kubernetes.pod.namespace: app-clearcomply
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:component: clearcomply-backend
            k8s:io.kubernetes.pod.namespace: app-clearcomply
      toPorts:
        - ports:
            - port: "8080" # Pod/Container Port
              protocol: TCP

Allow External Endpoints

Here is an example to allow outgoing external endpoints using toFQDNS keyword:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-egress-endpoints
spec:
  endpointSelector:
    matchLabels:
      k8s:component: passport-backend
      k8s:io.kubernetes.pod.namespace: app-passport
  egress:
    - toFQDNs:
        - matchName: slack.com
          rules:
            dns:
              - matchName: slack.com
        - matchName: api.bamboohr.com
          rules:
            dns:
              - matchName: api.bamboohr.com
        - matchName: clearroute.jamfcloud.com
          rules:
            dns:
              - matchName: clearroute.jamfcloud.com
        - matchName: vogsy.io
          rules:
            dns:
              - matchName: vogsy.io

Enable Passport

We purposely choose a mutual trust workflow to allow apps using passport, you will have to allow passport communication on your app side (egress), as well as on the passport side (ingress).

Here is an example for ClearComply:

Application Side

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-egress-passport
spec:
  endpointSelector:
    matchLabels:
      k8s:component: clearcomply-backend
      k8s:io.kubernetes.pod.namespace: app-clearcomply
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:component: passport-backend
            k8s:io.kubernetes.pod.namespace: app-passport
      toPorts:
        - ports:
            - port: "3001"
              protocol: TCP

Passport Side

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-clearcomply
spec:
  endpointSelector:
    matchLabels:
      k8s:component: passport-backend
      k8s:io.kubernetes.pod.namespace: app-passport
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:component: clearcomply-backend
            k8s:io.kubernetes.pod.namespace: app-clearcomply
      toPorts:
        - ports:
            - port: "3001"
              protocol: TCP

DNS with Route53

Simply add the external-dns.alpha.kubernetes.io/hostname annotation to your Ingress for an external resolvable route53 record:

Info

external-dns only watches networking.k8s.io/v1 Ingress resources so do not use Trafiks IngressRule CR.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: engineering
  annotations:
    # creates a route53 record of engineering.sandbox.clearroute.io pointing to Constellations NLB Public IP 
    external-dns.alpha.kubernetes.io/hostname: engineering.sandbox.clearroute.io
spec: ...

Traefik

How to expose my App externally?

Warning

Be extra careful when exposing your App, as the App will be available in internet

1. Expose the port your container/pod is serving its UI:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: <app-name>
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: app
        image: clearroute/app
        ports:
            - containerPort: <Port> # change to your applications port
            ...

2. create a service for your Pod + Port:

apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  type: ClusterIP
  selector:
    app: <app-name>
  ports:
  - port: 80
    protocol: TCP
    targetPort: <container port>

3. add an Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    cert-manager.io/acme-challenge-type: dns01
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect-https@kubernetescrd # http to https redirect
spec:
  ingressClassName: traefik
  rules:
  - host: app.dev.clearroute.io # your hostname
    http:
      paths:
      - backend:
          service:
            name: app # your service name
            port: 
              number: 80 # your service port
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - engineering.dev.clearroute.io #your hostname
    secretName: engineering-cert