From 290a652991534e83d12b27ab679bc1ecff95ddd2 Mon Sep 17 00:00:00 2001
From: Abdullah Gharaibeh <ahg@google.com>
Date: Wed, 10 Mar 2021 14:31:32 -0500
Subject: [PATCH] Docs for pod deletion cost feature

---
 .../workloads/controllers/replicaset.md       | 30 +++++++++++++++++++
 .../feature-gates.md                          |  3 ++
 .../reference/labels-annotations-taints.md    | 10 +++++++
 3 files changed, 43 insertions(+)

diff --git a/content/en/docs/concepts/workloads/controllers/replicaset.md b/content/en/docs/concepts/workloads/controllers/replicaset.md
index cade83afa5..4c75b20234 100644
--- a/content/en/docs/concepts/workloads/controllers/replicaset.md
+++ b/content/en/docs/concepts/workloads/controllers/replicaset.md
@@ -321,6 +321,36 @@ prioritize scaling down pods based on the following general algorithm:
     
 If all of the above match, then selection is random.
 
+### Pod deletion cost 
+{{< feature-state for_k8s_version="v1.21" state="alpha" >}}
+
+Using the [`controller.kubernetes.io/pod-deletion-cost`](/docs/reference/command-line-tools-reference/labels-annotations-taints/#pod-deletion-cost) 
+annotation, users can set a preference regarding which pods to remove first when downscaling a ReplicaSet.
+
+The annotation should be set on the pod, the range is [-2147483647, 2147483647]. It represents the cost of
+deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion
+cost are preferred to be deleted before pods with higher deletion cost. 
+
+The implicit value for this annotation for pods that don't set it is 0; negative values are permitted.
+Invalid values will be rejected by the API server.
+
+This feature is alpha and disabled by default. You can enable it by setting the
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+`PodDeletionCost` in both kube-apiserver and kube-controller-manager.
+
+{{< note >}}
+- This is honored on a best-effort basis, so it does not offer any guarantees on pod deletion order.
+- Users should avoid updating the annotation frequently, such as updating it based on a metric value,
+  because doing so will generate a significant number of pod updates on the apiserver.
+{{< /note >}}
+
+#### Example Use Case
+The different pods of an application could have different utilization levels. On scale down, the application 
+may prefer to remove the pods with lower utilization. To avoid frequently updating the pods, the application
+should update `controller.kubernetes.io/pod-deletion-cost` once before issuing a scale down (setting the 
+annotation to a value proportional to pod utilization level). This works if the application itself controls
+the down scaling; for example, the driver pod of a Spark deployment.
+
 ### ReplicaSet as a Horizontal Pod Autoscaler Target
 
 A ReplicaSet can also be a target for
diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md
index 2c181ca331..180a9aab7f 100644
--- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md
+++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md
@@ -141,6 +141,7 @@ different Kubernetes components.
 | `NodeDisruptionExclusion` | `true` | Beta | 1.19 | |
 | `NonPreemptingPriority` | `false` | Alpha | 1.15 | 1.18 |
 | `NonPreemptingPriority` | `true` | Beta | 1.19 | |
+| `PodDeletionCost` | `false` | Alpha | 1.21 | |
 | `PodDisruptionBudget` | `false` | Alpha | 1.3 | 1.4 |
 | `PodDisruptionBudget` | `true` | Beta | 1.5 | |
 | `PodAffinityNamespaceSelector` | `false` | Alpha | 1.21 | |
@@ -682,6 +683,8 @@ Each feature gate is designed for enabling/disabling a specific feature:
 - `NonPreemptingPriority`: Enable `preemptionPolicy` field for PriorityClass and Pod.
 - `PVCProtection`: Enable the prevention of a PersistentVolumeClaim (PVC) from
   being deleted when it is still used by any Pod.
+- `PodDeletionCost`: Enable the [Pod Deletion Cost](/docs/content/en/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost)
+   feature which allows users to influence ReplicaSet downscaling order.
 - `PersistentLocalVolumes`: Enable the usage of `local` volume type in Pods.
   Pod affinity has to be specified if requesting a `local` volume.
 - `PodDisruptionBudget`: Enable the [PodDisruptionBudget](/docs/tasks/run-application/configure-pdb/) feature.
diff --git a/content/en/docs/reference/labels-annotations-taints.md b/content/en/docs/reference/labels-annotations-taints.md
index 0399bd561e..a4a9a78d5a 100644
--- a/content/en/docs/reference/labels-annotations-taints.md
+++ b/content/en/docs/reference/labels-annotations-taints.md
@@ -56,6 +56,16 @@ The Kubelet populates this label with the hostname. Note that the hostname can b
 
 This label is also used as part of the topology hierarchy.  See [topology.kubernetes.io/zone](#topologykubernetesiozone) for more information.
 
+
+## controller.kubernetes.io/pod-deletion-cost {#pod-deletion-cost}
+
+Example: `controller.kubernetes.io/pod-deletion-cost=10`
+
+Used on: Pod
+
+This annotation is used to set [Pod Deletion Cost](/docs/content/en/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost)
+which allows users to influence ReplicaSet downscaling order. The annotation parses into an `int32` type.
+
 ## beta.kubernetes.io/instance-type (deprecated)
 
 {{< note >}} Starting in v1.17, this label is deprecated in favor of [node.kubernetes.io/instance-type](#nodekubernetesioinstance-type). {{< /note >}}