Install Cilium / network policy support

Use the cilium package to make Kix responsible for the Cilium CNI and the CiliumNetworkPolicy API.

This guide uses a new Kind cluster. Cilium must be the cluster’s CNI from the start, so create Kind without its default CNI before deploying the example.

Create the Kind cluster

Create this Kind configuration beside the kix-examples checkout:

kind-cilium.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: "10.244.0.0/16"

Create the cluster:

❱ kind create cluster --config kind-cilium.yaml

The control-plane node remains NotReady until Cilium is deployed. That is expected for a cluster with no CNI.

Make the CNI role available

Tell Kix that the Kind environment is not providing a CNI, then enable network policy generation:

how-to/package-stacks/cilium-cluster.nix

      # The Kind cluster for this example is created without kindnet, so the
      # CNI role is available for a managed package.
      cluster.roles.cni = {
        binding = "absent";
        provider = null;
      };

      networkPolicy.enable = true;

View source on GitHub ↗

The role setting must describe the cluster you actually created. Do not mark the role absent on a Kind cluster that is already running kindnet.

Add the Cilium instance

Install Cilium in kube-system:

how-to/package-stacks/cilium-cluster.nix

      # Keeps Kind's kube-proxy; the IPAM pool is the kind flavour's
      # cluster.network.podCidr, the podSubnet in kind-cilium.yaml.
      instances.kube-system.cilium = {
        package = packages.cilium;
      };

View source on GitHub ↗

The instance needs no values. It keeps Kind’s kube-proxy (the chart default) and takes its IPAM pool from cluster.network.podCidr, which the kind flavour sets to Kind’s default podSubnet, 10.244.0.0/16. A Kind configuration with another podSubnet sets cluster.network.podCidr to match. With network policy on, the package sets Cilium’s policyEnforcementMode to always, so a pod without an allow rule loses all traffic except cluster DNS. The package also writes two CiliumClusterwideNetworkPolicies: cilium-cluster-dns-egress keeps cluster DNS open for every pod, and cilium-health-probes lets Cilium’s health probes reach the other nodes.

Check the generated resources

Evaluate the cluster before deploying it:

kix-examples/

❱ kix check how-to-package-cilium
 TOOL       RESULT  DETAILS                       
 eval       pass    36 manifests evaluated        
 scorecard  pass    0 errors, 36 warnings, 4 info

Inspect the main Cilium components and a generated default-deny policy:

kix-examples/ output excerpt

❱ kix build how-to-package-cilium --output json
[
  {
    "apiVersion": "apps/v1",
    "kind": "DaemonSet",
    "metadata": {
      "name": "cilium",
      "namespace": "kube-system"
    },
    "containers": [
      {
        "name": "cilium-agent",
        "image": "quay.io/cilium/cilium:v1.19.6@sha256:0df5b2750b64c49843aba1d649e9eaf61467cb0645ad3171db6f6962c095ac92"
      }
    ]
  },
  {
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "metadata": {
      "name": "cilium-operator",
      "namespace": "kube-system"
    },
    "containers": [
      {
        "name": "cilium-operator",
        "image": "quay.io/cilium/operator-generic:v1.19.6@sha256:0db4ca4e06969d8904ee036617795d0e9c3228cf7b8d902ba74fc2bb98d2d665"
      }
    ]
  },
  {
    "apiVersion": "cilium.io/v2",
    "kind": "CiliumNetworkPolicy",
    "metadata": {
      "name": "kube-system-default-deny",
      "namespace": "kube-system"
    },
    "spec": {
      "egress": [
        {}
      ],
      "endpointSelector": {},
      "ingress": [
        {}
      ]
    }
  }
]

Kix builds the Cilium DaemonSet and operator from the package, then connects generated policies to Cilium’s custom resource definitions in the deployment graph.

Deploy and verify Cilium

Deploy the cluster and wait for the Cilium pods to become ready:

kix-examples/

❱ kix deploy how-to-package-cilium

❱ kubectl rollout status -n kube-system daemonset/cilium

❱ kubectl get nodes

The Kind node should report Ready after the Cilium DaemonSet is running. Check the generated policies with:

❱ kubectl get ciliumnetworkpolicies --all-namespaces

If the node remains NotReady, compare the pod CIDR in the Kind configuration with cluster.network.podCidr, then inspect the Cilium pod logs in kube-system.

When RKE2 installs Cilium

RKE2 installs its own Cilium chart when its server config sets cni: cilium. Tell Kix the same thing with the rke2 flavour, and declare the instance in kube-system:

cluster.flavor.rke2.cni = [ "cilium" ];
instances.kube-system.cilium.package = packages.cilium;

The flavour declares RKE2’s Cilium as the platform import kube-system/platform-cilium, and the instance takes it as a dependency. The instance does not install a second Cilium. It writes its values into the HelmChartConfig kube-system/rke2-cilium, which the import names and RKE2’s helm-controller merges into the bundled chart: policy enforcement, operator replicas, and rollouts. RKE2’s chart uses kubernetes IPAM, so the package sets no pool. A cluster that wants cluster-pool IPAM sets values.ipam.mode = "cluster-pool" and the pool together.

To replace kube-proxy on RKE2, set values.kubeProxyReplacement = true with values.k8sServiceHost and values.k8sServicePort, and configure every RKE2 server with cni: cilium and disable-kube-proxy: true. Kix cannot read the server config, so it cannot check that the two agree.

With network policy on, the rke2 flavour declares platform intents for the pods RKE2’s bundled charts run in kube-system (CoreDNS, metrics-server, the ingress controllers, the snapshot controller, and the chart install Jobs).

Kix then owns kube-system/rke2-cilium, which has two consequences:

• RKE2 also applies any rke2-cilium HelmChartConfig in the server’s manifests directory (/var/lib/rancher/rke2/server/manifests). Remove that file, or RKE2 and Kix write the same object.
• Removing the instance makes the next kix deploy --prune delete the HelmChartConfig, and helm-controller then reinstalls RKE2’s Cilium with its chart defaults.

When the environment installed Cilium

Where something outside Kix installed and configures Cilium, declare that the environment provides it, as a platform import the cilium instance depends on, and leave its configuration alone:

cluster.roles.cni.binding = "shadowable";
instances.cilium-policy = {
  cilium.package = packages.cilium;
  platform-cilium.package = kix.mkImport {
    roles = [ "cni" ];
    out.helmChartConfig = null;
  };
};

out.helmChartConfig = null says the environment merges no HelmChartConfig into its Cilium. Set it to { name; namespace; } when it does, and the instance writes its values there, as on RKE2.

Declare the instance in a namespace of its own, not in kube-system. With network policy on, every namespace Kix manages gets a default-deny policy. Nothing here declares platform intents for the environment’s kube-system pods, so a default deny there would cut off CoreDNS, and with it DNS for every pod.

The instance then emits only the CRD references that generated policies wait on, and the evaluation fails if it is given values.