From 220a7b201b42c919824c679cbf08ef7997ce78cd Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Tue, 10 Nov 2020 00:10:11 +0000 Subject: [PATCH 1/4] ExecProbeTimeout feature gate introduction --- .../feature-gates.md | 2 ++ ...igure-liveness-readiness-startup-probes.md | 23 +++++++++++++++---- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 303c90aae5..b16b4d722b 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -227,6 +227,7 @@ different Kubernetes components. | `EvenPodsSpread` | `false` | Alpha | 1.16 | 1.17 | | `EvenPodsSpread` | `true` | Beta | 1.18 | 1.18 | | `EvenPodsSpread` | `true` | GA | 1.19 | - | +| `ExecProbeTimeout` | `true` | GA | 1.20 | - | | `GCERegionalPersistentDisk` | `true` | Beta | 1.10 | 1.12 | | `GCERegionalPersistentDisk` | `true` | GA | 1.13 | - | | `HugePages` | `false` | Alpha | 1.8 | 1.9 | @@ -450,6 +451,7 @@ Each feature gate is designed for enabling/disabling a specific feature: - `EphemeralContainers`: Enable the ability to add {{< glossary_tooltip text="ephemeral containers" term_id="ephemeral-container" >}} to running pods. - `EvenPodsSpread`: Enable pods to be scheduled evenly across topology domains. See [Pod Topology Spread Constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/). +- `ExecProbeTimeout`: Ensure kubelet respects exec probe timeouts. This feature gate exists in case any of your existing workloads depend on a now-corrected fault where Kubernetes ignored exec probe timeouts. See [readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes). - `ExpandInUsePersistentVolumes`: Enable expanding in-use PVCs. See [Resizing an in-use PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim). - `ExpandPersistentVolumes`: Enable the expanding of persistent volumes. See [Expanding Persistent Volumes Claims](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims). - `ExperimentalCriticalPodAnnotation`: Enable annotating specific pods as *critical* so that their [scheduling is guaranteed](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/). diff --git a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md index 150fe720e1..e55f8cb789 100644 --- a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md +++ b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md @@ -336,6 +336,25 @@ liveness. Minimum value is 1. try `failureThreshold` times before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready. Defaults to 3. Minimum value is 1. +{{< note >}} +Before Kubernetes 1.20, the field `timeoutSeconds` was not respected for exec probes: +probes continued running indefinitely, even past their configured deadline, +until a result was returned. + +This defect was corrected in Kubernetes v1.20. You may have been relying on the previous behavior, +even without realizing it, as the default timeout is 1 second. +As a cluster administrator, you can disable the feature gate `ExecProbeTimeout` (set it to `false`) +on kubelet to restore the behavior from older versions, then remove that override +once all the exec probes in the cluster have a `timeoutSeconds` value set. + +With the fix of the defect, for exec probes, on Kubernetes `1.20+` with the `dockershim` container runtime, +the process inside the container may keep running even after probe returned failure because of the timeout. +{{< /note >}} +{{< caution >}} +Incorrect implementation of readiness probes may result in an ever growing number +of processes in the container, and resource starvation if this is left unchecked. +{{< /caution >}} + ### HTTP probes [HTTP probes](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#httpgetaction-v1-core) @@ -406,7 +425,3 @@ You can also read the API references for: * [Pod](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) * [Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core) * [Probe](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#probe-v1-core) - - - - From d81ee2342ad67342516ef4082f5a4b35a2bf8658 Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Wed, 11 Nov 2020 01:38:09 -0800 Subject: [PATCH 2/4] Update content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md Co-authored-by: Tim Bannister --- .../configure-liveness-readiness-startup-probes.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md index e55f8cb789..e4add99e2a 100644 --- a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md +++ b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md @@ -345,7 +345,10 @@ This defect was corrected in Kubernetes v1.20. You may have been relying on the even without realizing it, as the default timeout is 1 second. As a cluster administrator, you can disable the feature gate `ExecProbeTimeout` (set it to `false`) on kubelet to restore the behavior from older versions, then remove that override -once all the exec probes in the cluster have a `timeoutSeconds` value set. +once all the exec probes in the cluster have a `timeoutSeconds` value set. +If you have pods that are impacted from the default 1 second timeout, +you should update their probe timeout so that you're ready for the +eventual removal of that feature gate. With the fix of the defect, for exec probes, on Kubernetes `1.20+` with the `dockershim` container runtime, the process inside the container may keep running even after probe returned failure because of the timeout. From 1a13c6ba45e1110edd03dcd4fc1ca578d6918f10 Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Wed, 11 Nov 2020 01:38:45 -0800 Subject: [PATCH 3/4] Update content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md Co-authored-by: Tim Bannister --- .../configure-liveness-readiness-startup-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md index e4add99e2a..4138fd7a3a 100644 --- a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md +++ b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md @@ -343,7 +343,7 @@ until a result was returned. This defect was corrected in Kubernetes v1.20. You may have been relying on the previous behavior, even without realizing it, as the default timeout is 1 second. -As a cluster administrator, you can disable the feature gate `ExecProbeTimeout` (set it to `false`) +As a cluster administrator, you can disable the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `ExecProbeTimeout` (set it to `false`) on kubelet to restore the behavior from older versions, then remove that override once all the exec probes in the cluster have a `timeoutSeconds` value set. If you have pods that are impacted from the default 1 second timeout, From 1f306541c7fde10ccf8a725221cf6a3e6d79c057 Mon Sep 17 00:00:00 2001 From: Sergey Kanzhelev Date: Wed, 11 Nov 2020 01:39:07 -0800 Subject: [PATCH 4/4] Update content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md Co-authored-by: Tim Bannister --- .../configure-liveness-readiness-startup-probes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md index 4138fd7a3a..cdbcddb5d0 100644 --- a/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md +++ b/content/en/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes.md @@ -344,7 +344,7 @@ until a result was returned. This defect was corrected in Kubernetes v1.20. You may have been relying on the previous behavior, even without realizing it, as the default timeout is 1 second. As a cluster administrator, you can disable the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `ExecProbeTimeout` (set it to `false`) -on kubelet to restore the behavior from older versions, then remove that override +on each kubelet to restore the behavior from older versions, then remove that override once all the exec probes in the cluster have a `timeoutSeconds` value set. If you have pods that are impacted from the default 1 second timeout, you should update their probe timeout so that you're ready for the