From d29e93acbb22343419288ffec8d861a217035653 Mon Sep 17 00:00:00 2001 From: Pushkar Joglekar Date: Mon, 8 Nov 2021 17:19:45 -0800 Subject: [PATCH] Tutorial for pod security admission Refer blog post for v1.23 + suggestions from code review --- content/en/docs/tutorials/_index.md | 5 + content/en/docs/tutorials/security/_index.md | 5 + .../tutorials/security/cluster-level-pss.md | 319 ++++++++++++++++++ .../docs/tutorials/security/ns-level-pss.md | 160 +++++++++ 4 files changed, 489 insertions(+) create mode 100644 content/en/docs/tutorials/security/_index.md create mode 100644 content/en/docs/tutorials/security/cluster-level-pss.md create mode 100644 content/en/docs/tutorials/security/ns-level-pss.md diff --git a/content/en/docs/tutorials/_index.md b/content/en/docs/tutorials/_index.md index fdc62e11fb..34a91b6863 100644 --- a/content/en/docs/tutorials/_index.md +++ b/content/en/docs/tutorials/_index.md @@ -57,6 +57,11 @@ Before walking through each tutorial, you may want to bookmark the * [Using Source IP](/docs/tutorials/services/source-ip/) +## Security + +* [Applying Pod Security Standards at Cluster level](/docs/tutorials/security/cluster-level-pss/) +* [Applying Pod Security Standards at Namespace level](/docs/tutorials/security/ns-level-pss/) + ## {{% heading "whatsnext" %}} If you would like to write a tutorial, see diff --git a/content/en/docs/tutorials/security/_index.md b/content/en/docs/tutorials/security/_index.md new file mode 100644 index 0000000000..fbb8140dfa --- /dev/null +++ b/content/en/docs/tutorials/security/_index.md @@ -0,0 +1,5 @@ +--- +title: "Security" +weight: 40 +--- + diff --git a/content/en/docs/tutorials/security/cluster-level-pss.md b/content/en/docs/tutorials/security/cluster-level-pss.md new file mode 100644 index 0000000000..75a77b346a --- /dev/null +++ b/content/en/docs/tutorials/security/cluster-level-pss.md @@ -0,0 +1,319 @@ +--- +title: Applying Pod Security Standards at the cluster level +content_type: tutorial +weight: 10 +--- + +{{% alert title="Note" %}} +This tutorial applies only for new clusters. +{{% /alert %}} + +Pod Security admission (PSA) is enabled by default in v1.23 and later, as it [graduated +to beta](/blog/2021/12/15/pod-security-admission-beta/). Pod Security Admission +is an admission controller that applies Pod Security Standards when pods are +created. This tutorial shows you how to enforce the `baseline` Pod Security +Standard at the cluster level which applies a standard configuration +to all namespaces in a cluster. + +For applying pod security standards one namespace at a time, please [follow this +tutorial](/docs/tutorials/security/ns-level-pss). + +## {{% heading "prerequisites" %}} + +Install the following on your workstation: + +- [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) +- [kubectl](https://kubernetes.io/docs/tasks/tools/) + +## Choose the right Pod Security Standard to apply + +[Pod Security Admission](/docs/concepts/security/pod-security-admission/) +lets you apply built-in [Pod Security Standards](/docs/concepts/security/pod-security-standards/) +with the following modes: `enforce`, `audit`, and `warn`. + +To gather information that helps you to choose the Pod Security Standards +that are most appropriate for your configuration, do the following: + +1. Create a cluster with no Pod Security Standards applied: + + ```shell + kind create cluster --name psa-wo-cluster-pss --image kindest/node:latest + ``` + The output is similar to this: + ``` + Creating cluster "psa-wo-cluster-pss" ... + ✓ Ensuring node image (kindest/node:latest) đŸ–ŧ + ✓ Preparing nodes đŸ“Ļ + ✓ Writing configuration 📜 + ✓ Starting control-plane 🕹ī¸ + ✓ Installing CNI 🔌 + ✓ Installing StorageClass 💾 + Set kubectl context to "kind-psa-wo-cluster-pss" + You can now use your cluster with: + + kubectl cluster-info --context kind-psa-wo-cluster-pss + + Thanks for using kind! 😊 + + ``` + +2. Set the kubectl context to the new cluster: + + ```shell + kubectl cluster-info --context kind-psa-wo-cluster-pss + ``` + The output is similar to this: + + ``` + Kubernetes control plane is running at https://127.0.0.1:61350 + + CoreDNS is running at https://127.0.0.1:61350/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + ``` + +3. Get a list of namespaces in the cluster: + + ```shell + kubectl get ns + ``` + The output is similar to this: + ``` + NAME STATUS AGE + default Active 9m30s + kube-node-lease Active 9m32s + kube-public Active 9m32s + kube-system Active 9m32s + local-path-storage Active 9m26s + ``` + +4. Use `--dry-run=server` to understand what happens when different Pod Security Standards + are applied: + + 1. Privileged + ```shell + kubectl label --dry-run=server --overwrite ns --all \ + pod-security.kubernetes.io/enforce=privileged + ``` + The output is similar to this: + ``` + namespace/default labeled + namespace/kube-node-lease labeled + namespace/kube-public labeled + namespace/kube-system labeled + namespace/local-path-storage labeled + ``` + 2. Baseline + ```shell + kubectl label --dry-run=server --overwrite ns --all \ + pod-security.kubernetes.io/enforce=baseline + ``` + The output is similar to this: + ``` + namespace/default labeled + namespace/kube-node-lease labeled + namespace/kube-public labeled + Warning: existing pods in namespace "kube-system" violate the new PodSecurity enforce level "baseline:latest" + Warning: etcd-psa-wo-cluster-pss-control-plane (and 3 other pods): host namespaces, hostPath volumes + Warning: kindnet-vzj42: non-default capabilities, host namespaces, hostPath volumes + Warning: kube-proxy-m6hwf: host namespaces, hostPath volumes, privileged + namespace/kube-system labeled + namespace/local-path-storage labeled + ``` + + 3. Restricted + ```shell + kubectl label --dry-run=server --overwrite ns --all \ + pod-security.kubernetes.io/enforce=restricted + ``` + The output is similar to this: + ``` + namespace/default labeled + namespace/kube-node-lease labeled + namespace/kube-public labeled + Warning: existing pods in namespace "kube-system" violate the new PodSecurity enforce level "restricted:latest" + Warning: coredns-7bb9c7b568-hsptc (and 1 other pod): unrestricted capabilities, runAsNonRoot != true, seccompProfile + Warning: etcd-psa-wo-cluster-pss-control-plane (and 3 other pods): host namespaces, hostPath volumes, allowPrivilegeEscalation != false, unrestricted capabilities, restricted volume types, runAsNonRoot != true + Warning: kindnet-vzj42: non-default capabilities, host namespaces, hostPath volumes, allowPrivilegeEscalation != false, unrestricted capabilities, restricted volume types, runAsNonRoot != true, seccompProfile + Warning: kube-proxy-m6hwf: host namespaces, hostPath volumes, privileged, allowPrivilegeEscalation != false, unrestricted capabilities, restricted volume types, runAsNonRoot != true, seccompProfile + namespace/kube-system labeled + Warning: existing pods in namespace "local-path-storage" violate the new PodSecurity enforce level "restricted:latest" + Warning: local-path-provisioner-d6d9f7ffc-lw9lh: allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile + namespace/local-path-storage labeled + ``` + +From the previous output, you'll notice that applying the `privileged` Pod Security Standard shows no warnings +for any namespaces. However, `baseline` and `restricted` standards both have +warnings, specifically in the `kube-system` namespace. + +## Set modes, versions and standards + +In this tutorial, you apply the following Pod Security Standards to the `latest` version: + + * `baseline` standard in `enforce` mode. + * `restricted` standard in `warn` and `audit` mode. + +The `baseline` Pod Security Standard provides a convenient +middle ground that allows keeping the exemption list short and prevents known +privilege escalations. + +Additionally, to prevent pods from failing in `kube-system`, you'll exempt the namespace +from having Pod Security Standards applied. + +When you implement Pod Security Admission in your own environment, consider the +following: + +1. Based on the risk posture applied to a cluster, a stricter Pod Security + Standard like `restricted` might be a better choice. +1. Exempting the `kube-system` namespace allows pods to run as + `privileged` in this namespace. We recommend that you apply strict RBAC + policies that limit access to `kube-system`, following the principle of least + privilege. + +1. Create a configuration file that can be consumed by the Pod Security +Admission Controller to implement these Pod Security Standards: + +``` +mkdir -p /tmp/pss +cat < /tmp/pss/cluster-level-pss.yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: AdmissionConfiguration +plugins: +- name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1beta1 + kind: PodSecurityConfiguration + defaults: + enforce: "baseline" + enforce-version: "latest" + audit: "restricted" + audit-version: "latest" + warn: "restricted" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: [kube-system] +EOF +``` + + +1. Configure the API server to consume this file during cluster creation: + + ``` + cat < /tmp/pss/cluster-config.yaml + kind: Cluster + apiVersion: kind.x-k8s.io/v1alpha4 + nodes: + - role: control-plane + kubeadmConfigPatches: + - | + kind: ClusterConfiguration + apiServer: + extraArgs: + admission-control-config-file: /etc/config/cluster-level-pss.yaml + extraVolumes: + - name: accf + hostPath: /etc/config + mountPath: /etc/config + readOnly: false + pathType: "DirectoryOrCreate" + extraMounts: + - hostPath: /tmp/pss + containerPath: /etc/config + # optional: if set, the mount is read-only. + # default false + readOnly: false + # optional: if set, the mount needs SELinux relabeling. + # default false + selinuxRelabel: false + # optional: set propagation mode (None, HostToContainer or Bidirectional) + # see https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation + # default None + propagation: None + EOF + ``` + + {{}} + If you use Docker Desktop with KinD, the `/tmp` + directory is added as a Shared Directory under + **Preferences > Resources > File Sharing** on Mac OS. + {{}} + +2. Create a cluster that uses Pod Security Admission to apply + these Pod Security Standards: + + ```shell + kind create cluster --name psa-with-cluster-pss --image kindest/node:latest --config /tmp/pss/cluster-config.yaml + ``` + The output is similar to this: + ``` + Creating cluster "psa-with-cluster-pss" ... + ✓ Ensuring node image (kindest/node:latest) đŸ–ŧ + ✓ Preparing nodes đŸ“Ļ + ✓ Writing configuration 📜 + ✓ Starting control-plane 🕹ī¸ + ✓ Installing CNI 🔌 + ✓ Installing StorageClass 💾 + Set kubectl context to "kind-psa-with-cluster-pss" + You can now use your cluster with: + + kubectl cluster-info --context kind-psa-with-cluster-pss + + Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂 + +3. Point kubectl to the cluster + ```shell + kubectl cluster-info --context kind-psa-with-cluster-pss + Kubernetes control plane is running at https://127.0.0.1:63855 + CoreDNS is running at https://127.0.0.1:63855/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + ``` + +4. Create a Pod with minimal configuration in the default namespace: + + ``` + cat < /tmp/pss/nginx-pod.yaml + apiVersion: v1 + kind: Pod + metadata: + name: nginx + spec: + containers: + - image: nginx + name: nginx + ports: + - containerPort: 80 + EOF + ``` +5. Create Pod after pod security is enabled at cluster level: + + ```shell + kubectl apply -f /tmp/pss/nginx-pod.yaml + ``` + The output is similar to this: + ``` + Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") + pod/nginx created + ``` +## Clean up + +Run `kind delete cluster -name psa-with-cluster-pss` and +`kind delete cluster -name psa-wo-cluster-pss` to delete the clusters you +created. + +## {{% heading "whatsnext" %}} + +- Run a + [gist](https://gist.github.com/PushkarJ/9f7a0045f4bec31097bdd1e9db0f2f6e) + to perform all the preceding steps at once: + 1. Create a Pod Security Standards based cluster level Configuration + 2. Create a file to let API server consumes this configuration + 3. Create a cluster that creates an API server with this configuration + 4. Set kubectl context to this new cluster + 5. Create a minimal pod yaml file + 6. Apply this file to create a Pod in the new cluster +- [Pod Security Admission](/docs/concepts/security/pod-security-admission/) +- [Pod Security Standards](/docs/concepts/security/pod-security-standards/) +- [Applying Pod Security Standards at the namespace level](/docs/tutorials/security/ns-level-pss/) diff --git a/content/en/docs/tutorials/security/ns-level-pss.md b/content/en/docs/tutorials/security/ns-level-pss.md new file mode 100644 index 0000000000..5f975418c5 --- /dev/null +++ b/content/en/docs/tutorials/security/ns-level-pss.md @@ -0,0 +1,160 @@ +--- +title: Applying Pod Security Standards at Namespace level +content_type: tutorial +weight: 10 +--- + +{{% alert title="Note" %}} +This tutorial applies only for new clusters. +{{% /alert %}} + +Pod Security admission (PSA) is enabled by default in v1.23 and later, as it [graduated +to beta](/blog/2021/12/15/pod-security-admission-beta/). Pod Security Admission +is an admission controller that applies Pod Security Standards when pods are +created. In this tutorial, we will enforce `baseline` Pod Security Standard, +one namespace at a time. + +# Pre-requisites + +Install the following on your workstation: + +- [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) +- [kubectl](https://kubernetes.io/docs/tasks/tools/) + +# Create cluster + +1. Create a `KinD` cluster as follows: + + ```shell + kind create cluster --name psa-ns-level --image kindest/node:latest + ``` + The output is similar to this: + ``` + Creating cluster "psa-ns-level" ... + ✓ Ensuring node image (kindest/node:latest) đŸ–ŧ + ✓ Preparing nodes đŸ“Ļ + ✓ Writing configuration 📜 + ✓ Starting control-plane 🕹ī¸ + ✓ Installing CNI 🔌 + ✓ Installing StorageClass 💾 + Set kubectl context to "kind-psa-ns-level" + You can now use your cluster with: + + kubectl cluster-info --context kind-psa-ns-level + + Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/ + ``` + +2. Set the kubectl context to the new cluster + ```shell + kubectl cluster-info --context kind-psa-ns-level + ``` + The output is similar to this: + ``` + Kubernetes control plane is running at https://127.0.0.1:50996 + CoreDNS is running at https://127.0.0.1:50996/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + ``` + +# Create Namespace + +Create a new namespace `example` for this tutorial: + +```shell +kubectl create ns example +``` +The output is similar to this: +``` +namespace/example created +``` + +# Applying one Pod Security Standard + +Enable Pod Security Standards on this namespace using labels supported by +built-in Pod Security Admission. In this step we will warn on baseline pod +security standard as per the latest version (default value) + +```shell +kubectl label --overwrite ns example \ + pod-security.kubernetes.io/warn=baseline \ + pod-security.kubernetes.io/warn-version=latest +``` + +# Applying multiple Pod Security Standards + +Multiple pod security standards can be enabled on any namespace, using labels. +Following command will `enforce` the `baseline` Pod Security Standard, but +`warn` and `audit` for `restricted` Pod Security Standards as per the latest +version (default value) + +``` +kubectl label --overwrite ns example \ + pod-security.kubernetes.io/enforce=baseline \ + pod-security.kubernetes.io/enforce-version=latest \ + pod-security.kubernetes.io/warn=restricted \ + pod-security.kubernetes.io/warn-version=latest \ + pod-security.kubernetes.io/audit=restricted \ + pod-security.kubernetes.io/audit-version=latest +``` + +# Create Pod + +1. Create a minimal pod in `example` namespace: + + ```shell + cat < /tmp/pss/nginx-pod.yaml + apiVersion: v1 + kind: Pod + metadata: + name: nginx + spec: + containers: + - image: nginx + name: nginx + ports: + - containerPort: 80 + EOF + ``` +2. Apply the pod spec to the cluster in `example` namespace: + ```shell + kubectl apply -n example -f /tmp/pss/nginx-pod.yaml + ``` + The output is similar to this: + ``` + Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") + pod/nginx created + ``` + +3. Apply the pod spec to the cluster in `default` namespace: + ```shell + kubectl apply -n default -f /tmp/pss/nginx-pod.yaml + ``` + Output is similar to this: + ``` + pod/nginx created + ``` + +As you can see the Pod Security Standards were applied only to `example` +namespace. For `default` namespace, pod was created without any warnings. +To apply pod security standards to multiple namespaces at once at cluster +level, please +[follow this tutorial](/docs/tutorials/security/cluster-level-pss). + +# Clean up + +Run `kind delete cluster -name psa-ns-level` to delete the cluster created. + +## {{% heading "whatsnext" %}} + +- Run a +[gist](https://gist.github.com/PushkarJ/c694bac35c2d100f906861667474afb5) +to perform all the preceding steps all at once. + 1. Create KinD cluster + 2. Create new namespace + 3. Apply `baseline` Pod Security Standard in `enforce` mode while applying + `restricted` Pod Security Standard also in `warn` and `audit` mode. + 4. Create a new pod with the following pod security standards applied +- [Pod Security Admission](/docs/concepts/security/pod-security-admission/) +- [Pod Security Standards](/docs/concepts/security/pod-security-standards/) +- [Applying Pod Security Standards at the cluster level](/docs/tutorials/security/cluster-level-pss/) \ No newline at end of file