[Do Not Merge] Release 1.12 (#10292)
* Update docs for fields allowed at root of CRD schema (#9973) * add plugin docs and examples (#10053) * docs update to promote TaintNodesByCondition to beta (#9626) * HPA Specificity Improvements (#8757) Updated the HPA docs to reference the `autoscaling/v2beta2` API version, and added documentation about the new fields. * adjust docs for pod ready++ (#10049) * Remove --cadvisor-port - has been deprecated since v1.10 (#10023) Change-Id: Id2a685473a243aef492a98ff450759f39e362557 * Add Documentation for Snapshot Feature (#9948) * Add documentation for snapshot feature * Update volume-snapshots.md * Add dry-run to api-concepts (#10033) * kubeadm-init: Update the offline support section (#10062) The update includes the following things (in mind with Kubernetes 1.12): - Remove the 1.8 image versions - Add the 1.10 image versions that were missing until now - Include a comment for the missing arch suffixes in 1.12 Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com> * Say bye to `DynamicProvisioningScheduling` (#10157) The mentioned feature gate is now collapsed into `VolumeScheduling`. xref: kubernetes/kubernetes#67432 * Update ResourceQuota per PriorityClass state for 1.12 (#10229) * TokenRequest and TokenRequestProjection now beta (#10161) xref: kubernetes/kubernetes#67349 * Change feature state for kms provider to beta. (#10230) KMS Provider will be graduating to beta in v1.12, reflecting this change on the website. * coredns default (#10200) * Promote ShareProcessNamespace to beta in docs (#9996) * Add CoreDNS details to DNS Debug docs (#10201) * add coredns details * address nits, add query logging section * Update docs with topology aware dynamic provisioning (#9939) * Document topology aware volume binding feature * update for readability * Update storage-classes.md * comma splice * don't abbreviate * HPA Algorithm Information Improvements (#9780) * Update HPA docs with more algorithm details The HPA docs pointed to an out-of-date document for information on the algorithm details, which users were finding confusing. This sticks a section on the algorithm in the HPA docs instead, documenting both general behavior and corner cases. * Add glossary info, HPA docs on quantities People often ask about the quantity notation when working with the metrics APIs, so this adds a glossary entry on quantities (since they're used elsewhere in the system), and a short explantation in the HPA walkthough. * Information about HPA readiness and stabilization This adds information about the new changes to HPA readiness and stabilization from kubernetes/features#591, and other minor changes that landed in Kubernetes 1.12. * Update horizontal-pod-autoscale.md * Audit 1.12 doc (#9953) * audit 1.12 document * remove legacy audit feature https://github.com/kubernetes/kubernetes/pull/65862 * update feature gate doc * MountPropagation is now GA (#10090) * RuntimeClass documentation (#10102) * RuntimeClass documentation * Update runtime-class.md * Add documentation for Scheduler performance tuning (#10048) * Add documentation for Scheduler performance tuning * Update scheduler-perf-tuning.md * TTL controller for cleaning up finished resources (#10064) * TTL controller for cleaning up finished resources * Address comments * Update ttlafterfinished.md * Bump quota configuration api version (#10217) * Incremental update from master (#10278) * fix invalid href of cloud controller manager (#10240) * fix invalid yaml format (#10238) * update storage-limits doc with Azure disk part (#10224) update storage-limits doc with Azure disk part fix comments * Update kubelet-config-file.md (#10222) Update link to KubeletConfiguration struct. * fix a trivial misspelling (#10244) * Fix cassandra-statefulset.yaml indent level (#10243) * Mention minimum etcd versions (#10208) Source: https://groups.google.com/d/msg/kubernetes-dev/jMPA4JzKiY4/HIx2ugvLBAAJ * fix 404 error (#10250) * Small verb tweak (#10190) Present participle, ftw. * Add AnchorJS logic for header links (#10155) * Add AnchorJS JavaScript * Remove existing inpage_heading logic * Remove underline from anchor tags * Use single icon and add touch visibility * Use paragraph link icon for AnchorJS * Update Sass to use code formatting in docsContent headers * Update header size coverage to H3-H6 * fix broken link in kubefed.md (#10254) * Update the version numbers for the X-Remote-Extra- and Impersonate-Extra- key fixes (#9827) The fix was cherry picked into 1.11.3, 1.10.7, and 1.9.11: https://github.com/kubernetes/kubernetes/pull/67162 https://github.com/kubernetes/kubernetes/pull/67163 https://github.com/kubernetes/kubernetes/pull/67164 * fix typo (#10168) * fix typo * addressing comments. * Update setup-ha-etcd-with-kubeadm.md * fix typos (#10252) * fix description of contribute guide (#10253) * describe truncate feature about advanced audit (#10236) * describe truncate feature about advanced audit * Update audit.md * docs update to promote ScheduleDaemonSetPods to beta (#9923) * Dynamic volume limit updates for 1.12 (#10211) * add a placeholder commit * Update docs for csi volume limits * Update storage-limits.md * Add "MayRunAs" value among other GroupStrategies (#9888) * Add CoreDNS details to the customize DNS doc (#10228) * Add CoreDNS details to the customize DNS doc Rewrite the document to include more details about CoreDNS, since it's now the default from v1.12 * Address comments * Improve doc wording * Fix link * Update dns-custom-nameservers.md * Update dns-custom-nameservers.md * Fix secrets docs in 1.12 branch (#10056) * Fix secrets docs * Update secret.md * Revert CoreDNS Docs (#10319) * Revert "Add CoreDNS details to DNS Debug docs (#10201)" This reverts commit462817a674. * Revert "Add CoreDNS details to the customize DNS doc (#10228)" This reverts commite7319eeb8c. * Revert "coredns default (#10200)" This reverts commit698e93b441. * Add CRI installation instructions page Added cri-installation page with CRI installation instructions Referenced it from kubeadm-init and install-kubeadm pages. * kubeadm: update API types documentation for 1.12 (#10283) v1alpha2 -> v1alpha3 MasterConfiguration -> [new-api-types] * TokenRequest feature documentation (#10295) * AdvancedAuditing is now GA (#10156) xref: kubernetes/kubernetes#65862 `AdvancedAuditing` feature is GA in 1.12. This PR adjusts the related docs. * update runtime-class.md (#10332) * update runtime-class.md * Update runtime-class.md * Document cross-authorizer permissions for creating RBAC roles (#10015) * Document cross-authorizer permissions for creating RBAC roles * Update rbac.md * kubeadm: update authored content for 1.12 (reference docs and cluster creation) (#10348) * kubeadm: update authored content in reference docs for 1.12 * kubeadm: add time frame in create-cluster-kubeadm for 1.12 * add AllowedProcMountTypes and ProcMountType to docs (#9911) Signed-off-by: Jess Frazelle <acidburn@microsoft.com> * kubeadm: add new command line reference (#10306) Add: - placeholder files - include place holder files - include "renew" sub command - add missing tabs for "alpha phase kubelet" * Documenting SCTP support in Kubernetes (#10279) * Documenting SCTP support in Kubernetes Service, Endpoint, NetworkPolicy and Pod * Updates based on comments on the PR * kubectl expose update with SCTP support * Updated according to comments in the PR * Revert "kubectl expose update with SCTP support" This reverts commit 0d5a1e6720a012390cf100c83e16b4a8c0782356. * TLS Bootstrap and Server Cert Rotation feature documentation (#10232) * TokenRequest feature documentation * line wrapping to make review not insane * update content for GA without major refactor * Update kubelet-tls-bootstrapping.md * Add clarifications for volume snapshots (#10296) * Update kubadm ha installation for 1.12 (#10264) * Update kubadm ha installation for 1.12 Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * update stable version Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * Update stacked control plane for v1.12 (#2) * use v1alpha3 Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * more v1alpha3 (#4) * updates Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * Document how to run in-tree cloud providers with kubeadm (#10357) Change-Id: Iab6b996a830503d74a6eb0c507c5f8ca7a39235b * kubeadm reference doc for release 1.12 (#10359) * Revert "Revert "Add CoreDNS details to DNS Debug docs (#10201)"" This reverts commit bb30f4d1fcd6fba2fe6190778ead99f8010033b7. * Revert "Revert "Add CoreDNS details to the customize DNS doc (#10228)"" This reverts commit bc23d45c09d7b83cac130fe22a0bd91e72435862. * Revert "Revert "coredns default (#10200)"" This reverts commit 7f4350d6ab7fc554ee53126d3875e845d2e43d1f. * add missing instruction for ha guide (#10374) Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * kubeadm - Ha upgrade updates (#10340) * Update HA upgrade docs * Adds external etcd HA upgrade guide Signed-off-by: Chuck Ha <ha.chuck@gmail.com> * copyedit * more edits * add runasgroup in psp (#10076) * update KubeletPluginsWatcher feature gate (#10205) * generated 1.12 docs * Building Multi-arch images with Manifests (#10379) In 1.12, a variety of images used in a typical kubernetes installation have started to using manifests to better support environments with arm or ppc64le architectures. For example all images used with kubeadm by default have manifests, another would be all the tests in the conformance test suite. Here we capture the best practices for everyone to start using manifests in their own workflows. Change-Id: I5ba4c5fe55ffc9486a8251760f3352be4f2e1494 * Upgrade docs for v1.12 (#10344) * generated assets and docs * remove 1.7 * update 1.12 * update plugin documentation under docs>tasks>extend-kubectl (#10259) * update plugin documentation under docs>tasks>extend-kubectl * Update kubectl-plugins.md
This commit is contained in:
Jim Angel
@@ -167,18 +167,18 @@ Here CPU utilization dropped to 0, and so HPA autoscaled the number of replicas
|
||||
## Autoscaling on multiple metrics and custom metrics
|
||||
|
||||
You can introduce additional metrics to use when autoscaling the `php-apache` Deployment
|
||||
by making use of the `autoscaling/v2beta1` API version.
|
||||
by making use of the `autoscaling/v2beta2` API version.
|
||||
|
||||
First, get the YAML of your HorizontalPodAutoscaler in the `autoscaling/v2beta1` form:
|
||||
First, get the YAML of your HorizontalPodAutoscaler in the `autoscaling/v2beta2` form:
|
||||
|
||||
```shell
|
||||
$ kubectl get hpa.v2beta1.autoscaling -o yaml > /tmp/hpa-v2.yaml
|
||||
$ kubectl get hpa.v2beta2.autoscaling -o yaml > /tmp/hpa-v2.yaml
|
||||
```
|
||||
|
||||
Open the `/tmp/hpa-v2.yaml` file in an editor, and you should see YAML which looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2beta1
|
||||
apiVersion: autoscaling/v2beta2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: php-apache
|
||||
@@ -194,7 +194,9 @@ spec:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
targetAverageUtilization: 50
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 50
|
||||
status:
|
||||
observedGeneration: 1
|
||||
lastScaleTime: <some-time>
|
||||
@@ -204,8 +206,9 @@ status:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
currentAverageUtilization: 0
|
||||
currentAverageValue: 0
|
||||
current:
|
||||
averageUtilization: 0
|
||||
averageValue: 0
|
||||
```
|
||||
|
||||
Notice that the `targetCPUUtilizationPercentage` field has been replaced with an array called `metrics`.
|
||||
@@ -215,8 +218,8 @@ the only other supported resource metric is memory. These resources do not chan
|
||||
to cluster, and should always be available, as long as the `metrics.k8s.io` API is available.
|
||||
|
||||
You can also specify resource metrics in terms of direct values, instead of as percentages of the
|
||||
requested value. To do so, use the `targetAverageValue` field instead of the `targetAverageUtilization`
|
||||
field.
|
||||
requested value, by using a `target` type of `AverageValue` instead of `AverageUtilization`, and
|
||||
setting the corresponding `target.averageValue` field instead of the `target.averageUtilization`.
|
||||
|
||||
There are two other types of metrics, both of which are considered *custom metrics*: pod metrics and
|
||||
object metrics. These metrics may have names which are cluster specific, and require a more
|
||||
@@ -224,31 +227,40 @@ advanced cluster monitoring setup.
|
||||
|
||||
The first of these alternative metric types is *pod metrics*. These metrics describe pods, and
|
||||
are averaged together across pods and compared with a target value to determine the replica count.
|
||||
They work much like resource metrics, except that they *only* have the `targetAverageValue` field.
|
||||
They work much like resource metrics, except that they *only* support a `target` type of `AverageValue`.
|
||||
|
||||
Pod metrics are specified using a metric block like this:
|
||||
|
||||
```yaml
|
||||
type: Pods
|
||||
pods:
|
||||
metricName: packets-per-second
|
||||
targetAverageValue: 1k
|
||||
metric:
|
||||
name: packets-per-second
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: 1k
|
||||
```
|
||||
|
||||
The second alternative metric type is *object metrics*. These metrics describe a different
|
||||
object in the same namespace, instead of describing pods. Note that the metrics are not
|
||||
fetched from the object -- they simply describe it. Object metrics do not involve averaging,
|
||||
and look like this:
|
||||
The second alternative metric type is *object metrics*. These metrics describe a different
|
||||
object in the same namespace, instead of describing pods. The metrics are not necessarily
|
||||
fetched from the object; they only describe it. Object metrics support `target` types of
|
||||
both `Value` and `AverageValue`. With `Value`, the target is compared directly to the returned
|
||||
metric from the API. With `AverageValue`, the value returned from the custom metrics API is divided
|
||||
by the number of pods before being compared to the target. The following example is the YAML
|
||||
representation of the `requests-per-second` metric.
|
||||
|
||||
```yaml
|
||||
type: Object
|
||||
object:
|
||||
metricName: requests-per-second
|
||||
target:
|
||||
metric:
|
||||
name: requests-per-second
|
||||
describedObject:
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Ingress
|
||||
name: main-route
|
||||
targetValue: 2k
|
||||
target:
|
||||
type: Value
|
||||
value: 2k
|
||||
```
|
||||
|
||||
If you provide multiple such metric blocks, the HorizontalPodAutoscaler will consider each metric in turn.
|
||||
@@ -275,19 +287,25 @@ spec:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
targetAverageUtilization: 50
|
||||
target:
|
||||
kind: AverageUtilization
|
||||
averageUtilization: 50
|
||||
- type: Pods
|
||||
pods:
|
||||
metricName: packets-per-second
|
||||
metric:
|
||||
name: packets-per-second
|
||||
targetAverageValue: 1k
|
||||
- type: Object
|
||||
object:
|
||||
metricName: requests-per-second
|
||||
target:
|
||||
metric:
|
||||
name: requests-per-second
|
||||
describedObject:
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Ingress
|
||||
name: main-route
|
||||
targetValue: 10k
|
||||
target:
|
||||
kind: Value
|
||||
value: 10k
|
||||
status:
|
||||
observedGeneration: 1
|
||||
lastScaleTime: <some-time>
|
||||
@@ -297,14 +315,47 @@ status:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
currentAverageUtilization: 0
|
||||
currentAverageValue: 0
|
||||
current:
|
||||
averageUtilization: 0
|
||||
averageValue: 0
|
||||
- type: Object
|
||||
object:
|
||||
metric:
|
||||
name: requests-per-second
|
||||
describedObject:
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Ingress
|
||||
name: main-route
|
||||
current:
|
||||
value: 10k
|
||||
```
|
||||
|
||||
Then, your HorizontalPodAutoscaler would attempt to ensure that each pod was consuming roughly
|
||||
50% of its requested CPU, serving 1000 packets per second, and that all pods behind the main-route
|
||||
Ingress were serving a total of 10000 requests per second.
|
||||
|
||||
### Autoscaling on more specific metrics
|
||||
|
||||
Many metrics pipelines allow you to describe metrics either by name or by a set of additional
|
||||
descriptors called _labels_. For all non-resource metric types (pod, object, and external,
|
||||
described below), you can specify an additional label selector which is passed to your metric
|
||||
pipeline. For instance, if you collect a metric `http_requests` with the `verb`
|
||||
label, you can specify the following metric block to scale only on GET requests:
|
||||
|
||||
```yaml
|
||||
type: Object
|
||||
object:
|
||||
metric:
|
||||
name: `http_requests`
|
||||
selector: `verb=GET`
|
||||
```
|
||||
|
||||
This selector uses the same syntax as the full Kubernetes label selectors. The monitoring pipeline
|
||||
determines how to collapse multiple series into a single value, if the name and selector
|
||||
match multiple series. The selector is additive, and cannot select metrics
|
||||
that describe objects that are **not** the target object (the target pods in the case of the `Pods`
|
||||
type, and the described object in the case of the `Object` type).
|
||||
|
||||
### Autoscaling on metrics not related to Kubernetes objects
|
||||
|
||||
Applications running on Kubernetes may need to autoscale based on metrics that don't have an obvious
|
||||
@@ -312,12 +363,14 @@ relationship to any object in the Kubernetes cluster, such as metrics describing
|
||||
no direct correlation to Kubernetes namespaces. In Kubernetes 1.10 and later, you can address this use case
|
||||
with *external metrics*.
|
||||
|
||||
Using external metrics requires a certain level of knowledge of your monitoring system, and it requires a cluster
|
||||
monitoring setup similar to one required for using custom metrics. With external metrics, you can autoscale
|
||||
based on any metric available in your monitoring system by providing a `metricName` field in your
|
||||
HorizontalPodAutoscaler manifest. Additionally you can use a `metricSelector` field to limit which
|
||||
metrics' time series you want to use for autoscaling. If multiple time series are matched by `metricSelector`,
|
||||
Using external metrics requires knowledge of your monitoring system; the setup is
|
||||
similar to that required when using custom metrics. External metrics allow you to autoscale your cluster
|
||||
based on any metric available in your monitoring system. Just provide a `metric` block with a
|
||||
`name` and `selector`, as above, and use the `External` metric type instead of `Object`.
|
||||
If multiple time series are matched by the `metricSelector`,
|
||||
the sum of their values is used by the HorizontalPodAutoscaler.
|
||||
External metrics support both the `Value` and `AverageValue` target types, which function exactly the same
|
||||
as when you use the `Object` type.
|
||||
|
||||
For example if your application processes tasks from a hosted queue service, you could add the following
|
||||
section to your HorizontalPodAutoscaler manifest to specify that you need one worker per 30 outstanding tasks.
|
||||
@@ -325,20 +378,21 @@ section to your HorizontalPodAutoscaler manifest to specify that you need one wo
|
||||
```yaml
|
||||
- type: External
|
||||
external:
|
||||
metricName: queue_messages_ready
|
||||
metricSelector:
|
||||
matchLabels:
|
||||
queue: worker_tasks
|
||||
targetAverageValue: 30
|
||||
metric:
|
||||
name: queue_messages_ready
|
||||
selector: "queue=worker_tasks"
|
||||
target:
|
||||
type: AverageValue
|
||||
averageValue: 30
|
||||
```
|
||||
|
||||
If your metric describes work or resources that can be divided between autoscaled pods the `targetAverageValue`
|
||||
field describes how much of that work each pod can handle. Instead of using the `targetAverageValue` field, you could use the
|
||||
`targetValue` to define a desired value of your external metric.
|
||||
When possible, it's preferrable to use the custom metric target types instead of external metrics, since it's
|
||||
easier for cluster administrators to secure the custom metrics API. The external metrics API potentially allows
|
||||
access to any metric, so cluster administrators should take care when exposing it.
|
||||
|
||||
## Appendix: Horizontal Pod Autoscaler Status Conditions
|
||||
|
||||
When using the `autoscaling/v2beta1` form of the HorizontalPodAutoscaler, you will be able to see
|
||||
When using the `autoscaling/v2beta2` form of the HorizontalPodAutoscaler, you will be able to see
|
||||
*status conditions* set by Kubernetes on the HorizontalPodAutoscaler. These status conditions indicate
|
||||
whether or not the HorizontalPodAutoscaler is able to scale, and whether or not it is currently restricted
|
||||
in any way.
|
||||
@@ -378,6 +432,16 @@ was capped by the maximum or minimum of the HorizontalPodAutoscaler. This is an
|
||||
you may wish to raise or lower the minimum or maximum replica count constraints on your
|
||||
HorizontalPodAutoscaler.
|
||||
|
||||
## Appendix: Quantities
|
||||
|
||||
All metrics in the HorizontalPodAutoscaler and metrics APIs are specified using
|
||||
a special whole-number notation known in Kubernetes as a *quantity*. For example,
|
||||
the quantity `10500m` would be written as `10.5` in decimal notation. The metrics APIs
|
||||
will return whole numbers without a suffix when possible, and will generally return
|
||||
quantities in milli-units otherwise. This means you might see your metric value fluctuate
|
||||
between `1` and `1500m`, or `1` and `1.5` when written in decimal notation. See the
|
||||
[glossary entry on quantities](/docs/reference/glossary/quantity.md) for more information.
|
||||
|
||||
## Appendix: Other possible scenarios
|
||||
|
||||
### Creating the autoscaler declaratively
|
||||
|
||||
@@ -55,15 +55,19 @@ or the custom metrics API (for all other metrics).
|
||||
the number of desired replicas.
|
||||
|
||||
Please note that if some of the pod's containers do not have the relevant resource request set,
|
||||
CPU utilization for the pod will not be defined and the autoscaler will not take any action
|
||||
for that metric. See the [autoscaling algorithm design document](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#autoscaling-algorithm) for further
|
||||
details about how the autoscaling algorithm works.
|
||||
CPU utilization for the pod will not be defined and the autoscaler will
|
||||
not take any action for that metric. See the [algorithm
|
||||
details](#algorithm-details) section below for more information about
|
||||
how the autoscaling algorithm works.
|
||||
|
||||
* For per-pod custom metrics, the controller functions similarly to per-pod resource metrics,
|
||||
except that it works with raw values, not utilization values.
|
||||
|
||||
* For object metrics, a single metric is fetched (which describes the object
|
||||
in question), and compared to the target value, to produce a ratio as above.
|
||||
* For object metrics and external metrics, a single metric is fetched, which describes
|
||||
the object in question. This metric is compared compared to the target
|
||||
value, to produce a ratio as above. In the `autoscaling/v2beta2` API
|
||||
version, this value can optionally be divided by the number of pods before the
|
||||
comparison is made.
|
||||
|
||||
The HorizontalPodAutoscaler normally fetches metrics from a series of aggregated APIs (`metrics.k8s.io`,
|
||||
`custom.metrics.k8s.io`, and `external.metrics.k8s.io`). The `metrics.k8s.io` API is usually provided by
|
||||
@@ -83,6 +87,85 @@ by using the scale sub-resource. Scale is an interface that allows you to dynami
|
||||
each of their current states. More details on scale sub-resource can be found
|
||||
[here](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#scale-subresource).
|
||||
|
||||
### Algorithm Details
|
||||
|
||||
From the most basic perspective, the Horizontal Pod Autoscaler controller
|
||||
operates on the ratio between desired metric value and current metric
|
||||
value:
|
||||
|
||||
```
|
||||
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
|
||||
```
|
||||
|
||||
For example, if the current metric value is `200m`, and the desired value
|
||||
is `100m`, the number of replicas will be doubled, since `200.0 / 100.0 ==
|
||||
2.0` If the the current value is instead `50m`, we'll halve the number of
|
||||
replicas, since `50.0 / 100.0 == 0.5`. We'll skip scaling if the ratio is
|
||||
sufficiently close to 1.0 (within a globally-configurable tolerance, from
|
||||
the `--horizontal-pod-autoscaler-tolerance` flag, which defaults to 0.1).
|
||||
|
||||
When a `targetAverageValue` or `targetAverageUtilization` is specified,
|
||||
the `currentMetricValue` is computed by taking the average of the given
|
||||
metric across all Pods in the HorizontalPodAutoscaler's scale target.
|
||||
Before checking the tolerance and deciding on the final values, we take
|
||||
pod readiness and missing metrics into consideration, however.
|
||||
|
||||
All Pods with a deletion timestamp set (i.e. Pods in the process of being
|
||||
shut down) and all failed Pods are discarded.
|
||||
|
||||
If a particular Pod is missing metrics, it is set aside for later; Pods
|
||||
with missing metrics will be used to adjust the final scaling amount.
|
||||
|
||||
When scaling on CPU, if any pod has yet to become ready (i.e. it's still
|
||||
initializing) *or* the most recent metric point for the pod was before it
|
||||
became ready, that pod is set aside as well.
|
||||
|
||||
Due to technical constraints, the HorizontalPodAutoscaler controller
|
||||
cannot exactly determine the first time a pod becomes ready when
|
||||
determinining whether to set aside certain CPU metrics. Instead, it
|
||||
considers a Pod "not yet ready" if it's unready and transitioned to
|
||||
unready within a short, configurable window of time since it started.
|
||||
This value is configured with the `--horizontal-pod-autoscaler-initial-readiness-delay` flag, and its default is 30
|
||||
seconds. Once a pod has become ready, it considers any transition to
|
||||
ready to be the first if it occurred within a longer, configurable time
|
||||
since it started. This value is configured with the `--horizontal-pod-autoscaler-cpu-initialization-period` flag, and its
|
||||
default is 5 minutes.
|
||||
|
||||
The `currentMetricValue / desiredMetricValue` base scale ratio is then
|
||||
calculated using the remaining pods not set aside or discarded from above.
|
||||
|
||||
If there were any missing metrics, we recompute the average more
|
||||
conservatively, assuming those pods were consuming 100% of the desired
|
||||
value in case of a scale down, and 0% in case of a scale up. This dampens
|
||||
the magnitude of any potential scale.
|
||||
|
||||
Futhermore, if any not-yet-ready pods were present, and we would have
|
||||
scaled up without factoring in missing metrics or not-yet-ready pods, we
|
||||
conservatively assume the non-yet-ready pods are consuming 0% of the
|
||||
desired metric, further dampening the magnitude of a scale up.
|
||||
|
||||
After factoring in the not-yet-ready pods and missing metrics, we
|
||||
recalculate the usage ratio. If the new ratio reverses the scale
|
||||
direction, or is within the tolerance, we skip scaling. Otherwise, we use
|
||||
the new ratio to scale.
|
||||
|
||||
Note that the *original* value for the average utilization is reported
|
||||
back via the HorizontalPodAutoscaler status, without factoring in the
|
||||
not-yet-ready pods or missing metrics, even when the new usage ratio is
|
||||
used.
|
||||
|
||||
If multiple metrics are specified in a HorizontalPodAutoscaler, this
|
||||
calculation is done for each metric, and then the largest of the desired
|
||||
replica counts is chosen. If any of those metrics cannot be converted
|
||||
into a desired replica count (e.g. due to an error fetching the metrics
|
||||
from the metrics APIs), scaling is skipped.
|
||||
|
||||
Finally, just before HPA scales the target, the scale reccomendation is recorded. The
|
||||
controller considers all reccomendations within a configurable window choosing the
|
||||
highest recommendation from within that window. This value can be configured using the `--horizontal-pod-autoscaler-downscale-stabilization-window` flag, which defaults to 5 minutes.
|
||||
This means that scaledowns will occur gradually, smothing out the impact of rapidly
|
||||
fluctuating metric values.
|
||||
|
||||
## API Object
|
||||
|
||||
The Horizontal Pod Autoscaler is an API resource in the Kubernetes `autoscaling` API group.
|
||||
@@ -90,7 +173,7 @@ The current stable version, which only includes support for CPU autoscaling,
|
||||
can be found in the `autoscaling/v1` API version.
|
||||
|
||||
The beta version, which includes support for scaling on memory and custom metrics,
|
||||
can be found in `autoscaling/v2beta1`. The new fields introduced in `autoscaling/v2beta1`
|
||||
can be found in `autoscaling/v2beta2`. The new fields introduced in `autoscaling/v2beta2`
|
||||
are preserved as annotations when working with `autoscaling/v1`.
|
||||
|
||||
More details about the API object can be found at
|
||||
@@ -131,16 +214,14 @@ dynamic nature of the metrics evaluated. This is sometimes referred to as *thras
|
||||
Starting from v1.6, a cluster operator can mitigate this problem by tuning
|
||||
the global HPA settings exposed as flags for the `kube-controller-manager` component:
|
||||
|
||||
Starting from v1.12, a new algorithmic update removes the need for the
|
||||
upscale delay.
|
||||
|
||||
- `--horizontal-pod-autoscaler-downscale-delay`: The value for this option is a
|
||||
duration that specifies how long the autoscaler has to wait before another
|
||||
downscale operation can be performed after the current one has completed.
|
||||
The default value is 5 minutes (`5m0s`).
|
||||
|
||||
- `--horizontal-pod-autoscaler-upscale-delay`: The value for this option is a
|
||||
duration that specifies how long the autoscaler has to wait before another
|
||||
upscale operation can be performed after the current one has completed.
|
||||
The default value is 3 minutes (`3m0s`).
|
||||
|
||||
{{< note >}}
|
||||
**Note**: When tuning these parameter values, a cluster operator should be aware of
|
||||
the possible consequences. If the delay (cooldown) value is set too long, there
|
||||
@@ -151,7 +232,7 @@ may keep thrashing as usual.
|
||||
|
||||
## Support for multiple metrics
|
||||
|
||||
Kubernetes 1.6 adds support for scaling based on multiple metrics. You can use the `autoscaling/v2beta1` API
|
||||
Kubernetes 1.6 adds support for scaling based on multiple metrics. You can use the `autoscaling/v2beta2` API
|
||||
version to specify multiple metrics for the Horizontal Pod Autoscaler to scale on. Then, the Horizontal Pod
|
||||
Autoscaler controller will evaluate each metric, and propose a new scale based on that metric. The largest of the
|
||||
proposed scales will be used as the new scale.
|
||||
@@ -164,7 +245,7 @@ custom metrics is still available, these metrics will not be available for use b
|
||||
annotations for specifying which custom metrics to scale on are no longer honored by the Horizontal Pod Autoscaler controller.
|
||||
|
||||
Kubernetes 1.6 adds support for making use of custom metrics in the Horizontal Pod Autoscaler.
|
||||
You can add custom metrics for the Horizontal Pod Autoscaler to use in the `autoscaling/v2beta1` API.
|
||||
You can add custom metrics for the Horizontal Pod Autoscaler to use in the `autoscaling/v2beta2` API.
|
||||
Kubernetes then queries the new custom metrics API to fetch the values of the appropriate custom metrics.
|
||||
|
||||
See [Support for metrics APIs](#support-for-metrics-APIs) for the requirements.
|
||||
|
||||
Reference in New Issue
Block a user