Merge branch 'kubernetes:master' into ingress-nginx-stabilization

This commit is contained in:
James Strong
2021-06-14 11:20:35 -04:00
266 changed files with 5575 additions and 5407 deletions
+3 -3
View File
@@ -43,12 +43,12 @@ Kubernetes is open source giving you the freedom to take advantage of on-premise
<button id="desktopShowVideoButton" onclick="kub.showVideo()">Watch Video</button>
<br>
<br>
<a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna20" button id="desktopKCButton">Attend KubeCon NA virtually on November 17-20, 2020</a>
<a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21" button id="desktopKCButton">Attend KubeCon North America on October 11-15, 2021</a>
<br>
<br>
<br>
<br>
<a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu21" button id="desktopKCButton">Attend KubeCon EU virtually on May 4 7, 2021</a>
<a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu21" button id="desktopKCButton">Revisit KubeCon EU 2021</a>
</div>
<div id="videoPlayer">
<iframe data-url="https://www.youtube.com/embed/H06qrNmGqyE?autoplay=1" frameborder="0" allowfullscreen></iframe>
@@ -58,4 +58,4 @@ Kubernetes is open source giving you the freedom to take advantage of on-premise
{{< blocks/kubernetes-features >}}
{{< blocks/case-studies >}}
{{< blocks/case-studies >}}
@@ -95,7 +95,7 @@ The core workloads API surface is stable, but its still software, and softwar
--Kenneth Owens, Software Engineer, Google
- [Download](http://get.k8s.io/) Kubernetes
- [Download](https://get.k8s.io/) Kubernetes
- Get involved with the Kubernetes project on [GitHub](https://github.com/kubernetes/kubernetes)
- Post questions (or answer questions) on [Stack Overflow](http://stackoverflow.com/questions/tagged/kubernetes)
- Connect with the community on [Slack](http://slack.k8s.io/)
@@ -120,7 +120,7 @@ Run the following command to have Docker run the application in a container and
```
docker run -p 5001:5000 hello-python
```
Now navigate to http://localhost:5001, and you should see the “Hello form Python!” message.
Now navigate to http://localhost:5001, and you should see the “Hello from Python!” message.
### More info
* [Get started with Docker](https://docs.docker.com/get-started/)
@@ -201,7 +201,7 @@ kubectl get pods
```
<img src="/images/blog/get-started-with-kubernetes-using-python/kubectl-get-pods.png" alt="Pod listing" />
Now navigate to http://localhost:6000, and you should see the “Hello form Python!” message.
Now navigate to http://localhost:6000, and you should see the “Hello from Python!” message.
Thats it! The application is now running in Kubernetes!
@@ -526,6 +526,6 @@ When you enable the API Priority and Fairness feature, the kube-apiserver serves
For background information on design details for API priority and fairness, see
the [enhancement proposal](https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190228-priority-and-fairness.md).
the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness).
You can make suggestions and feature requests via [SIG API Machinery](https://github.com/kubernetes/community/tree/master/sig-api-machinery)
or the feature's [slack channel](http://kubernetes.slack.com/messages/api-priority-and-fairness).
@@ -253,7 +253,7 @@ message AllocatableResourcesResponse {
`ContainerDevices` do expose the topology information declaring to which NUMA cells the device is affine.
The NUMA cells are identified using a opaque integer ID, which value is consistent to what device
plugins report [when they register themselves to the kubelet](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-integration-with-the-topology-manager).
plugins report [when they register themselves to the kubelet](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-integration-with-the-topology-manager).
The gRPC service is served over a unix socket at `/var/lib/kubelet/pod-resources/kubelet.sock`.
@@ -48,7 +48,7 @@ kubectl get pods --field-selector=status.phase!=Running,spec.restartPolicy=Alway
## Multiple resource types
You use field selectors across multiple resource types. This `kubectl` command selects all Statefulsets and Services that are not in the `default` namespace:
You can use field selectors across multiple resource types. This `kubectl` command selects all Statefulsets and Services that are not in the `default` namespace:
```shell
kubectl get statefulsets,services --all-namespaces --field-selector metadata.namespace!=default
@@ -464,12 +464,12 @@ allowed prefix, and a `readOnly` field indicating it must be mounted read-only.
For example:
```yaml
allowedHostPaths:
# This allows "/foo", "/foo/", "/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
# "/foo/../" is never valid.
- pathPrefix: "/foo"
readOnly: true # only allow read-only mounts
allowedHostPaths:
# This allows "/foo", "/foo/", "/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
# "/foo/../" is never valid.
- pathPrefix: "/foo"
readOnly: true # only allow read-only mounts
```
{{< warning >}}There are many ways a container with unrestricted access to the host
@@ -26,40 +26,41 @@ each resource to score nodes based on the request to capacity ratio. This
allows users to bin pack extended resources by using appropriate parameters
and improves the utilization of scarce resources in large clusters. The
behavior of the `RequestedToCapacityRatioResourceAllocation` priority function
can be controlled by a configuration option called
`requestedToCapacityRatioArguments`. This argument consists of two parameters
`shape` and `resources`. The `shape` parameter allows the user to tune the
function as least requested or most requested based on `utilization` and
`score` values. The `resources` parameter consists of `name` of the resource
to be considered during scoring and `weight` specify the weight of each
resource.
can be controlled by a configuration option called `RequestedToCapacityRatioArgs`.
This argument consists of two parameters `shape` and `resources`. The `shape`
parameter allows the user to tune the function as least requested or most
requested based on `utilization` and `score` values. The `resources` parameter
consists of `name` of the resource to be considered during scoring and `weight`
specify the weight of each resource.
Below is an example configuration that sets
`requestedToCapacityRatioArguments` to bin packing behavior for extended
resources `intel.com/foo` and `intel.com/bar`.
```yaml
apiVersion: v1
kind: Policy
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
profiles:
# ...
priorities:
# ...
- name: RequestedToCapacityRatioPriority
weight: 2
argument:
requestedToCapacityRatioArguments:
shape:
- utilization: 0
score: 0
- utilization: 100
score: 10
resources:
- name: intel.com/foo
weight: 3
- name: intel.com/bar
weight: 5
pluginConfig:
- name: RequestedToCapacityRatio
args:
shape:
- utilization: 0
score: 10
- utilization: 100
score: 0
resources:
- name: intel.com/foo
weight: 3
- name: intel.com/bar
weight: 5
```
Referencing the `KubeSchedulerConfiguration` file with the kube-scheduler
flag `--config=/path/to/config/file` will pass the configuration to the
scheduler.
**This feature is disabled by default**
### Tuning the Priority Function
@@ -10,7 +10,7 @@ weight: 40
<!-- overview -->
[_Node affinity_](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity),
[_Node affinity_](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
is a property of {{< glossary_tooltip text="Pods" term_id="pod" >}} that *attracts* them to
a set of {{< glossary_tooltip text="nodes" term_id="node" >}} (either as a preference or a
hard requirement). _Taints_ are the opposite -- they allow a node to repel a set of pods.
@@ -7,6 +7,7 @@ content_type: concept
weight: 20
---
<!-- overview -->
Kubernetes creates DNS records for services and pods. You can contact
services with consistent DNS names instead of IP addresses.
@@ -261,6 +262,8 @@ spec:
### Pod's DNS Config {#pod-dns-config}
{{< feature-state for_k8s_version="v1.14" state="stable" >}}
Pod's DNS Config allows users more control on the DNS settings for a Pod.
The `dnsConfig` field is optional and it can work with any `dnsPolicy` settings.
@@ -310,18 +313,6 @@ search default.svc.cluster-domain.example svc.cluster-domain.example cluster-dom
options ndots:5
```
### Feature availability
The availability of Pod DNS Config and DNS Policy "`None`" is shown as below.
| k8s version | Feature support |
| :---------: |:-----------:|
| 1.14 | Stable |
| 1.10 | Beta (on by default)|
| 1.9 | Alpha |
## {{% heading "whatsnext" %}}
@@ -212,9 +212,9 @@ This ensures that even pods that aren't selected by any other NetworkPolicy will
## SCTP support
{{< feature-state for_k8s_version="v1.19" state="beta" >}}
{{< feature-state for_k8s_version="v1.20" state="stable" >}}
As a beta feature, this is enabled by default. To disable SCTP at a cluster level, you (or your cluster administrator) will need to disable the `SCTPSupport` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the API server with `--feature-gates=SCTPSupport=false,…`.
As a stable feature, this is enabled by default. To disable SCTP at a cluster level, you (or your cluster administrator) will need to disable the `SCTPSupport` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the API server with `--feature-gates=SCTPSupport=false,…`.
When the feature gate is enabled, you can set the `protocol` field of a NetworkPolicy to `SCTP`.
{{< note >}}
@@ -13,7 +13,7 @@ weight: 45
_Topology Aware Hints_ enable topology aware routing by including suggestions
for how clients should consume endpoints. This approach adds metadata to enable
consumers of EndpointSlice and / or and Endpoints objects, so that traffic to
consumers of EndpointSlice and / or Endpoints objects, so that traffic to
those network endpoints can be routed closer to where it originated.
For example, you can route traffic within a locality to reduce
@@ -86,7 +86,7 @@ rolling out node software updates can cause voluntary disruptions. Also, some im
of cluster (node) autoscaling may cause voluntary disruptions to defragment and compact nodes.
Your cluster administrator or hosting provider should have documented what level of voluntary
disruptions, if any, to expect. Certain configuration options, such as
[using PriorityClasses](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/)
[using PriorityClasses](/docs/concepts/configuration/pod-priority-preemption/)
in your pod spec can also cause voluntary (and involuntary) disruptions.
+25
View File
@@ -0,0 +1,25 @@
---
title: Viewing Site Analytics
content_type: concept
weight: 100
card:
name: contribute
weight: 100
---
<!-- overview -->
This page contains information about the kubernetes.io analytics dashboard.
<!-- body -->
[View the dashboard](https://datastudio.google.com/reporting/fede2672-b2fd-402a-91d2-7473bdb10f04).
This dashboard is built using Google Data Studio and shows information collected on kubernetes.io using Google Analytics.
### Using the dashboard
By default, the dashboard shows all collected analytics for the past 30 days. Use the date selector to see data from a different date range. Other filtering options allow you to view data based on user location, the device used to access the site, the translation of the docs used, and more.
If you notice an issue with this dashboard, or would like to request any improvements, please [open an issue](https://github.com/kubernetes/website/issues/new/choose).
@@ -134,7 +134,6 @@ Go to `<k8s-base>` and run these scripts:
hack/update-generated-swagger-docs.sh
hack/update-openapi-spec.sh
hack/update-generated-protobuf.sh
hack/update-api-reference-docs.sh
```
Run `git status` to see what was generated.
@@ -143,8 +142,6 @@ Run `git status` to see what was generated.
On branch master
...
modified: api/openapi-spec/swagger.json
modified: api/swagger-spec/apps_v1.json
modified: docs/api-reference/apps/v1/definitions.html
modified: staging/src/k8s.io/api/apps/v1/generated.proto
modified: staging/src/k8s.io/api/apps/v1/types.go
modified: staging/src/k8s.io/api/apps/v1/types_swagger_doc_generated.go
+1 -1
View File
@@ -38,7 +38,7 @@ client libraries:
- [Kubernetes Python client library](https://github.com/kubernetes-client/python)
- [Kubernetes Java client library](https://github.com/kubernetes-client/java)
- [Kubernetes JavaScript client library](https://github.com/kubernetes-client/javascript)
- [Kubernetes Dotnet client library](https://github.com/kubernetes-client/csharp)
- [Kubernetes C# client library](https://github.com/kubernetes-client/csharp)
- [Kubernetes Haskell Client library](https://github.com/kubernetes-client/haskell)
## CLI
@@ -58,7 +58,7 @@ It acts synchronously to modify pods as they are created or updated. When this p
1. It ensures that the `ServiceAccount` referenced by the pod exists, and otherwise rejects it.
1. It adds a `volume` to the pod which contains a token for API access if neither the ServiceAccount `automountServiceAccountToken` nor the Pod's `automountServiceAccountToken` is set to `false`.
1. It adds a `volumeSource` to each container of the pod mounted at `/var/run/secrets/kubernetes.io/serviceaccount`, if the previous step has created a volume for ServiceAccount token.
1. If the pod does not contain any `ImagePullSecrets`, then `ImagePullSecrets` of the `ServiceAccount` are added to the pod.
1. If the pod does not contain any `imagePullSecrets`, then `imagePullSecrets` of the `ServiceAccount` are added to the pod.
#### Bound Service Account Token Volume
@@ -91,14 +91,14 @@ add the following projected volume instead of a Secret-based volume for the non-
This projected volume consists of three sources:
1. A ServiceAccountToken acquired from kube-apiserver via TokenRequest API. It will expire after 1 hour by default or when the pod is deleted. It is bound to the pod and has kube-apiserver as the audience.
1. A ConfigMap containing a CA bundle used for verifying connections to the kube-apiserver. This feature depends on the `RootCAConfigMap` feature gate being enabled, which publishes a "kube-root-ca.crt" ConfigMap to every namespace. `RootCAConfigMap` is enabled by default in 1.20, and always enabled in 1.21+.
1. A ConfigMap containing a CA bundle used for verifying connections to the kube-apiserver. This feature depends on the `RootCAConfigMap` feature gate, which publishes a "kube-root-ca.crt" ConfigMap to every namespace. `RootCAConfigMap` feature gate is graduated to GA in 1.21 and default to true. (This flag will be removed from --feature-gate arg in 1.22)
1. A DownwardAPI that references the namespace of the pod.
See more details about [projected volumes](/docs/tasks/configure-pod-container/configure-projected-volume-storage/).
You can manually migrate a secret-based service account volume to a projected volume when
You can manually migrate a Secret-based service account volume to a projected volume when
the `BoundServiceAccountTokenVolume` feature gate is not enabled by adding the above
projected volume to the pod spec. However, `RootCAConfigMap` needs to be enabled.
projected volume to the pod spec.
### Token Controller
@@ -844,7 +844,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
- `ValidateProxyRedirects`: This flag controls whether the API server should
validate that redirects are only followed to the same host. Only used if the
`StreamingProxyRedirects` flag is enabled.
- 'VolumeCapacityPriority`: Enable support for prioritizing nodes in different
- `VolumeCapacityPriority`: Enable support for prioritizing nodes in different
topologies based on available PV capacity.
- `VolumePVCDataSource`: Enable support for specifying an existing PVC as a DataSource.
- `VolumeScheduling`: Enable volume topology aware scheduling and make the
@@ -27,7 +27,7 @@ each Pod in the scheduling queue according to constraints and available
resources. The scheduler then ranks each valid Node and binds the Pod to a
suitable Node. Multiple different schedulers may be used within a cluster;
kube-scheduler is the reference implementation.
See [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/)
See [scheduling](/docs/concepts/scheduling-eviction/)
for more information about scheduling and the kube-scheduler component.
```
@@ -222,7 +222,9 @@ When a single IngressClass resource has this annotation set to `"true"`, new Ing
## kubernetes.io/ingress.class (deprecated)
{{< note >}} Starting in v1.18, this annotation is deprecated in favor of `spec.ingressClassName`. {{< /note >}}
{{< note >}}
Starting in v1.18, this annotation is deprecated in favor of `spec.ingressClassName`.
{{< /note >}}
## storageclass.kubernetes.io/is-default-class
@@ -230,7 +232,8 @@ Example: `storageclass.kubernetes.io/is-default-class=true`
Used on: StorageClass
When a single StorageClass resource has this annotation set to `"true"`, new Physical Volume Claim resource without a class specified will be assigned this default class.
When a single StorageClass resource has this annotation set to `"true"`, new PersistentVolumeClaim
resource without a class specified will be assigned this default class.
## alpha.kubernetes.io/provided-node-ip
@@ -250,7 +250,7 @@ only has one pending pods queue.
## {{% heading "whatsnext" %}}
* Read the [kube-scheduler reference](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/)
* Read the [kube-scheduler reference](/docs/reference/command-line-tools-reference/kube-scheduler/)
* Learn about [scheduling](/docs/concepts/scheduling-eviction/kube-scheduler/)
* Read the [kube-scheduler configuration (v1beta1)](/docs/reference/config-api/kube-scheduler-config.v1beta1/) reference
@@ -17,7 +17,7 @@ Generate keys and certificate signing requests
Generates keys and certificate signing requests (CSRs) for all the certificates required to run the control plane. This command also generates partial kubeconfig files with private key data in the "users &gt; user &gt; client-key-data" field, and for each kubeconfig file an accompanying ".csr" file is created.
This command is designed for use in [Kubeadm External CA Mode](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#external-ca-mode). It generates CSRs which you can then submit to your external certificate authority for signing.
This command is designed for use in [Kubeadm External CA Mode](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#external-ca-mode). It generates CSRs which you can then submit to your external certificate authority for signing.
The PEM encoded signed certificates should then be saved alongside the key files, using ".crt" as the file extension, or in the case of kubeconfig files, the PEM encoded signed certificate should be base64 encoded and added to the kubeconfig file in the "users &gt; user &gt; client-certificate-data" field.
@@ -49,7 +49,7 @@ Some resource types will have one or more sub-resources, represented as sub path
* Cluster-scoped subresource: `GET /apis/GROUP/VERSION/RESOURCETYPE/NAME/SUBRESOURCE`
* Namespace-scoped subresource: `GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME/SUBRESOURCE`
The verbs supported for each subresource will differ depending on the object - see the API documentation more information. It is not possible to access sub-resources across multiple resources - generally a new virtual resource type would be used if that becomes necessary.
The verbs supported for each subresource will differ depending on the object - see the API documentation for more information. It is not possible to access sub-resources across multiple resources - generally a new virtual resource type would be used if that becomes necessary.
## Efficient detection of changes
@@ -442,7 +442,7 @@ feature, see the section on
## Resource Versions
Resource versions are strings that identify the server's internal version of an object. Resource versions can be used by clients to determine when objects have changed, or to express data consistency requirements when getting, listing and watching resources. Resource versions must be treated as opaque by clients and passed unmodified back to the server. For example, clients must not assume resource versions are numeric, and may only compare two resource version for equality (i.e. must not compare resource versions for greater-than or less-than relationships).
Resource versions are strings that identify the server's internal version of an object. Resource versions can be used by clients to determine when objects have changed, or to express data consistency requirements when getting, listing and watching resources. Resource versions must be treated as opaque by clients and passed unmodified back to the server. For example, clients must not assume resource versions are numeric, and may only compare two resource versions for equality (i.e. must not compare resource versions for greater-than or less-than relationships).
### ResourceVersion in metadata
@@ -454,7 +454,7 @@ Clients find resource versions in resources, including the resources in watch ev
### The ResourceVersion Parameter
The get, list and watch operations support the `resourceVersion` parameter.
The get, list, and watch operations support the `resourceVersion` parameter.
The exact meaning of this parameter differs depending on the operation and the value of `resourceVersion`.
@@ -66,7 +66,6 @@ their authors, not the Kubernetes team.
| PHP | [github.com/maclof/kubernetes-client](https://github.com/maclof/kubernetes-client) |
| PHP | [github.com/travisghansen/kubernetes-client-php](https://github.com/travisghansen/kubernetes-client-php) |
| PHP | [github.com/renoki-co/php-k8s](https://github.com/renoki-co/php-k8s) |
| Python | [github.com/eldarion-gondor/pykube](https://github.com/eldarion-gondor/pykube) |
| Python | [github.com/fiaas/k8s](https://github.com/fiaas/k8s) |
| Python | [github.com/mnubo/kubernetes-py](https://github.com/mnubo/kubernetes-py) |
| Python | [github.com/tomplus/kubernetes_asyncio](https://github.com/tomplus/kubernetes_asyncio) |
@@ -74,7 +74,7 @@ The **policy/v1beta1** API version of PodDisruptionBudget will no longer be serv
PodSecurityPolicy in the **policy/v1beta1** API version will no longer be served in v1.25, and the PodSecurityPolicy admission controller will be removed.
PodSecurityPolicy replacements are still under discussion, but current use can be migrated to
[3rd-party admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) now.
[3rd-party admission webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers/) now.
#### RuntimeClass {#runtimeclass-v125}
@@ -245,7 +245,7 @@ field tags.
### Compatibility across topology changes
On rare occurences, a CRD or built-in type author may want to change the
On rare occurrences, a CRD or built-in type author may want to change the
specific topology of a field in their resource without incrementing its
version. Changing the topology of types, by upgrading the cluster or
updating the CRD, has different consequences when updating existing
@@ -253,7 +253,7 @@ objects. There are two categories of changes: when a field goes from
`map`/`set`/`granular` to `atomic` and the other way around.
When the `listType`, `mapType`, or `structType` changes from
`map`/`set`/`granular` to `atomic`, the whole list, map or struct of
`map`/`set`/`granular` to `atomic`, the whole list, map, or struct of
existing objects will end-up being owned by actors who owned an element
of these types. This means that any further change to these objects
would cause a conflict.
@@ -310,7 +310,7 @@ simplify the update logic of your controller. The main differences with a
read-modify-write and/or patch are the following:
* the applied object must contain all the fields that the controller cares about.
* there are no way to remove fields that haven't been applied by the controller
* there is no way to remove fields that haven't been applied by the controller
before (controller can still send a PATCH/UPDATE for these use-cases).
* the object doesn't have to be read beforehand, `resourceVersion` doesn't have
to be specified.
@@ -473,7 +473,7 @@ have an opinion about.
## Clearing ManagedFields
It is possible to strip all managedFields from an object by overwriting them
using `MergePatch`, `StrategicMergePatch`, `JSONPatch` or `Update`, so every
using `MergePatch`, `StrategicMergePatch`, `JSONPatch`, or `Update`, so every
non-apply operation. This can be done by overwriting the managedFields field
with an empty entry. Two examples are:
+29 -5
View File
@@ -3,11 +3,11 @@ reviewers:
- brendandburns
- erictune
- mikedanese
no_issue: true
title: Getting started
main_menu: true
weight: 20
content_type: concept
no_list: true
card:
name: setup
weight: 20
@@ -24,16 +24,40 @@ This section lists the different ways to set up and run Kubernetes.
When you install Kubernetes, choose an installation type based on: ease of maintenance, security,
control, available resources, and expertise required to operate and manage a cluster.
You can deploy a Kubernetes cluster on a local machine, cloud, on-prem datacenter, or choose a managed Kubernetes cluster. There are also custom solutions across a wide range of cloud providers, or bare metal environments.
You can [download Kubernetes](/releases/download/) to deploy a Kubernetes cluster
on a local machine, into the cloud, or for your own datacenter.
If you don't want to manage a Kubernetes cluster yourself, you could pick a managed service, including
[certified platforms](/docs/setup/production-environment/turnkey-solutions/).
There are also other standardized and custom solutions across a wide range of cloud and
bare metal environments.
<!-- body -->
## Learning environment
If you're learning Kubernetes, use the tools supported by the Kubernetes community, or tools in the ecosystem to set up a Kubernetes cluster on a local machine.
If you're learning Kubernetes, use the tools supported by the Kubernetes community,
or tools in the ecosystem to set up a Kubernetes cluster on a local machine.
See [Install tools](/docs/tasks/tools/).
## Production environment
When evaluating a solution for a production environment, consider which aspects of operating a Kubernetes cluster (or _abstractions_) you want to manage yourself or offload to a provider.
When evaluating a solution for a
[production environment](/docs/setup/production-environment/), consider which aspects of
operating a Kubernetes cluster (or _abstractions_) you want to manage yourself and which you
prefer to hand off to a provider.
[Kubernetes Partners](https://kubernetes.io/partners/#conformance) includes a list of [Certified Kubernetes](https://github.com/cncf/k8s-conformance/#certified-kubernetes) providers.
For a cluster you're managing yourself, the officially supported tool
for deploying Kubernetes is [kubeadm](/docs/setup/production-environment/tools/kubeadm/).
## {{% heading "whatsnext" %}}
- [Download Kubernetes](/releases/download/)
- Download and [install tools](/docs/tasks/tools/) including `kubectl`
- Select a [container runtime](/docs/setup/production-environment/container-runtimes/) for your new cluster
- Learn about [best practices](/docs/setup/best-practices/) for cluster setup
Kubernetes is designed for its {{< glossary_tooltip term_id="control-plane" text="control plane" >}} to
run on Linux. Within your cluster you can run applications on Linux or other operating systems, including
Windows.
- Learn to [set up clusters with Windows nodes](/docs/setup/production-environment/windows/)
@@ -12,7 +12,7 @@ or virtual machines) running Kubernetes agents, managed by the
Kubernetes {{< param "version" >}} supports clusters with up to 5000 nodes. More specifically,
Kubernetes is designed to accommodate configurations that meet *all* of the following criteria:
* No more than 100 pods per node
* No more than 110 pods per node
* No more than 5000 nodes
* No more than 150000 total pods
* No more than 300000 total containers
@@ -66,8 +66,8 @@ When creating a cluster, you can (using custom tooling):
* start and configure additional etcd instance
* configure the {{< glossary_tooltip term_id="kube-apiserver" text="API server" >}} to use it for storing events
See [Operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/) and
[Set up a High Availability etcd cluster with kubeadm](docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/)
See [Operating etcd clusters for Kubernetes](/docs/tasks/administer-cluster/configure-upgrade-etcd/) and
[Set up a High Availability etcd cluster with kubeadm](/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/)
for details on configuring and managing etcd for a large cluster.
## Addon resources
@@ -49,7 +49,7 @@ access cluster resources. You can use role-based access control
security mechanisms to make sure that users and workloads can get access to the
resources they need, while keeping workloads, and the cluster itself, secure.
You can set limits on the resources that users and workloads can access
by managing [policies](https://kubernetes.io/docs/concepts/policy/) and
by managing [policies](/docs/concepts/policy/) and
[container resources](/docs/concepts/configuration/manage-resources-containers/).
Before building a Kubernetes production environment on your own, consider
@@ -286,8 +286,8 @@ and the
deployment methods.
- Configure user management by determining your
[Authentication](/docs/reference/access-authn-authz/authentication/) and
[Authorization](docs/reference/access-authn-authz/authorization/) methods.
[Authorization](/docs/reference/access-authn-authz/authorization/) methods.
- Prepare for application workloads by setting up
[resource limits](docs/tasks/administer-cluster/manage-resources/),
[resource limits](/docs/tasks/administer-cluster/manage-resources/),
[DNS autoscaling](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/)
and [service accounts](/docs/reference/access-authn-authz/service-accounts-admin/).
@@ -23,7 +23,7 @@ of Containers for each.
- Fetch all Pods in all namespaces using `kubectl get pods --all-namespaces`
- Format the output to include only the list of Container image names
using `-o jsonpath={..image}`. This will recursively parse out the
using `-o jsonpath={.items[*].spec.containers[*].image}`. This will recursively parse out the
`image` field from the returned json.
- See the [jsonpath reference](/docs/reference/kubectl/jsonpath/)
for further information on how to use jsonpath.
@@ -33,7 +33,7 @@ of Containers for each.
- Use `uniq` to aggregate image counts
```shell
kubectl get pods --all-namespaces -o jsonpath="{..image}" |\
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c
@@ -80,7 +80,7 @@ To target only Pods matching a specific label, use the -l flag. The
following matches only Pods with labels matching `app=nginx`.
```shell
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=nginx
kubectl get pods --all-namespaces -o=jsonpath="{.items[*].spec.containers[*].image}" -l app=nginx
```
## List Container images filtering by Pod namespace
@@ -89,7 +89,7 @@ To target only pods in a specific namespace, use the namespace flag. The
following matches only Pods in the `kube-system` namespace.
```shell
kubectl get pods --namespace kube-system -o jsonpath="{..image}"
kubectl get pods --namespace kube-system -o jsonpath="{.items[*].spec.containers[*].image}"
```
## List Container images using a go-template instead of jsonpath
@@ -26,7 +26,7 @@ volume is automatically deleted when a user deletes the corresponding
PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume
contains precious data. In that case, it is more appropriate to use the "Retain"
policy. With the "Retain" policy, if a user deletes a PersistentVolumeClaim,
the corresponding PersistentVolume is not be deleted. Instead, it is moved to the
the corresponding PersistentVolume will not be deleted. Instead, it is moved to the
Released phase, where all of its data can be manually recovered.
## Changing the reclaim policy of a PersistentVolume
@@ -1,16 +1,17 @@
---
reviewers:
- jszczepkowski
title: Set up High-Availability Kubernetes Masters
title: Set up a High-Availability Control Plane
content_type: task
aliases: [ '/docs/tasks/administer-cluster/highly-available-master/' ]
---
<!-- overview -->
{{< feature-state for_k8s_version="v1.5" state="alpha" >}}
You can replicate Kubernetes masters in `kube-up` or `kube-down` scripts for Google Compute Engine.
This document describes how to use kube-up/down scripts to manage highly available (HA) masters and how HA masters are implemented for use with GCE.
You can replicate Kubernetes control plane nodes in `kube-up` or `kube-down` scripts for Google Compute Engine.
This document describes how to use kube-up/down scripts to manage a highly available (HA) control plane and how HA control planes are implemented for use with GCE.
@@ -28,17 +29,17 @@ This document describes how to use kube-up/down scripts to manage highly availab
To create a new HA-compatible cluster, you must set the following flags in your `kube-up` script:
* `MULTIZONE=true` - to prevent removal of master replicas kubelets from zones different than server's default zone.
Required if you want to run master replicas in different zones, which is recommended.
* `MULTIZONE=true` - to prevent removal of control plane kubelets from zones different than server's default zone.
Required if you want to run control plane nodes in different zones, which is recommended.
* `ENABLE_ETCD_QUORUM_READ=true` - to ensure that reads from all API servers will return most up-to-date data.
If true, reads will be directed to leader etcd replica.
Setting this value to true is optional: reads will be more reliable but will also be slower.
Optionally, you can specify a GCE zone where the first master replica is to be created.
Optionally, you can specify a GCE zone where the first control plane node is to be created.
Set the following flag:
* `KUBE_GCE_ZONE=zone` - zone where the first master replica will run.
* `KUBE_GCE_ZONE=zone` - zone where the first control plane node will run.
The following sample command sets up a HA-compatible cluster in the GCE zone europe-west1-b:
@@ -46,50 +47,52 @@ The following sample command sets up a HA-compatible cluster in the GCE zone eur
MULTIZONE=true KUBE_GCE_ZONE=europe-west1-b ENABLE_ETCD_QUORUM_READS=true ./cluster/kube-up.sh
```
Note that the commands above create a cluster with one master;
however, you can add new master replicas to the cluster with subsequent commands.
Note that the commands above create a cluster with one control plane node;
however, you can add new control plane nodes to the cluster with subsequent commands.
## Adding a new master replica
## Adding a new control plane node
After you have created an HA-compatible cluster, you can add master replicas to it.
You add master replicas by using a `kube-up` script with the following flags:
After you have created an HA-compatible cluster, you can add control plane nodes to it.
You add control plane nodes by using a `kube-up` script with the following flags:
* `KUBE_REPLICATE_EXISTING_MASTER=true` - to create a replica of an existing
master.
* `KUBE_REPLICATE_EXISTING_MASTER=true` - to create a replica of an existing control plane
node.
* `KUBE_GCE_ZONE=zone` - zone where the master replica will run.
Must be in the same region as other replicas' zones.
* `KUBE_GCE_ZONE=zone` - zone where the control plane node will run.
Must be in the same region as other control plane nodes' zones.
You don't need to set the `MULTIZONE` or `ENABLE_ETCD_QUORUM_READS` flags,
as those are inherited from when you started your HA-compatible cluster.
The following sample command replicates the master on an existing HA-compatible cluster:
The following sample command replicates the control plane node on an existing
HA-compatible cluster:
```shell
KUBE_GCE_ZONE=europe-west1-c KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
```
## Removing a master replica
## Removing a control plane node
You can remove a master replica from an HA cluster by using a `kube-down` script with the following flags:
You can remove a control plane node from an HA cluster by using a `kube-down` script with the following flags:
* `KUBE_DELETE_NODES=false` - to restrain deletion of kubelets.
* `KUBE_GCE_ZONE=zone` - the zone from where master replica will be removed.
* `KUBE_GCE_ZONE=zone` - the zone from where the control plane node will be removed.
* `KUBE_REPLICA_NAME=replica_name` - (optional) the name of master replica to remove.
If empty: any replica from the given zone will be removed.
* `KUBE_REPLICA_NAME=replica_name` - (optional) the name of control plane node to
remove. If empty: any replica from the given zone will be removed.
The following sample command removes a master replica from an existing HA cluster:
The following sample command removes a control plane node from an existing HA cluster:
```shell
KUBE_DELETE_NODES=false KUBE_GCE_ZONE=europe-west1-c ./cluster/kube-down.sh
```
## Handling master replica failures
## Handling control plane node failures
If one of the master replicas in your HA cluster fails,
the best practice is to remove the replica from your cluster and add a new replica in the same zone.
If one of the control plane nodes in your HA cluster fails,
the best practice is to remove the node from your cluster and add a new control plane
node in the same zone.
The following sample commands demonstrate this process:
1. Remove the broken replica:
@@ -98,26 +101,31 @@ The following sample commands demonstrate this process:
KUBE_DELETE_NODES=false KUBE_GCE_ZONE=replica_zone KUBE_REPLICA_NAME=replica_name ./cluster/kube-down.sh
```
<ol start="2"><li>Add a new replica in place of the old one:</li></ol>
<ol start="2"><li>Add a new node in place of the old one:</li></ol>
```shell
KUBE_GCE_ZONE=replica-zone KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
```
## Best practices for replicating masters for HA clusters
## Best practices for replicating control plane nodes for HA clusters
* Try to place master replicas in different zones. During a zone failure, all masters placed inside the zone will fail.
* Try to place control plane nodes in different zones. During a zone failure, all
control plane nodes placed inside the zone will fail.
To survive zone failure, also place nodes in multiple zones
(see [multiple-zones](/docs/setup/best-practices/multiple-zones/) for details).
* Do not use a cluster with two master replicas. Consensus on a two-replica cluster requires both replicas running when changing persistent state.
As a result, both replicas are needed and a failure of any replica turns cluster into majority failure state.
A two-replica cluster is thus inferior, in terms of HA, to a single replica cluster.
* Do not use a cluster with two control plane nodes. Consensus on a two-node
control plane requires both nodes running when changing persistent state.
As a result, both nodes are needed and a failure of any node turns the cluster
into majority failure state.
A two-node control plane is thus inferior, in terms of HA, to a cluster with
one control plane node.
* When you add a master replica, cluster state (etcd) is copied to a new instance.
* When you add a control plane node, cluster state (etcd) is copied to a new instance.
If the cluster is large, it may take a long time to duplicate its state.
This operation may be sped up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration)
(we are considering adding support for etcd data dir migration in future).
This operation may be sped up by migrating the etcd data directory, as described in
the [etcd administration guide](https://etcd.io/docs/v2.3/admin_guide/#member-migration)
(we are considering adding support for etcd data dir migration in the future).
@@ -129,7 +137,7 @@ This operation may be sped up by migrating etcd data directory, as described [he
### Overview
Each of master replicas will run the following components in the following mode:
Each of the control plane nodes will run the following components in the following mode:
* etcd instance: all instances will be clustered together using consensus;
@@ -143,9 +151,9 @@ In addition, there will be a load balancer in front of API servers that will rou
### Load balancing
When starting the second master replica, a load balancer containing the two replicas will be created
When starting the second control plane node, a load balancer containing the two replicas will be created
and the IP address of the first replica will be promoted to IP address of load balancer.
Similarly, after removal of the penultimate master replica, the load balancer will be removed and its IP address will be assigned to the last remaining replica.
Similarly, after removal of the penultimate control plane node, the load balancer will be removed and its IP address will be assigned to the last remaining replica.
Please note that creation and removal of load balancer are complex operations and it may take some time (~20 minutes) for them to propagate.
### Master service & kubelets
@@ -153,17 +161,17 @@ Please note that creation and removal of load balancer are complex operations an
Instead of trying to keep an up-to-date list of Kubernetes apiserver in the Kubernetes service,
the system directs all traffic to the external IP:
* in one master cluster the IP points to the single master,
* in case of a single node control plane, the IP points to the control plane node,
* in multi-master cluster the IP points to the load balancer in-front of the masters.
* in case of an HA control plane, the IP points to the load balancer in-front of the masters.
Similarly, the external IP will be used by kubelets to communicate with master.
Similarly, the external IP will be used by kubelets to communicate with the control plane.
### Master certificates
### Control plane node certificates
Kubernetes generates Master TLS certificates for the external public IP and local IP for each replica.
There are no certificates for the ephemeral public IP for replicas;
to access a replica via its ephemeral public IP, you must skip TLS verification.
Kubernetes generates TLS certificates for the external public IP and local IP for each control plane node.
There are no certificates for the ephemeral public IP for control plane nodes;
to access a control plane node via its ephemeral public IP, you must skip TLS verification.
### Clustering etcd
@@ -172,7 +180,7 @@ To make such deployment secure, communication between etcd instances is authoriz
### API server identity
{{< feature-state state="alpha" for_k8s_version="v1.20" >}}
{{< feature-state state="alpha" for_k8s_version="v1.20" >}}
The API Server Identity feature is controlled by a
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
@@ -188,7 +188,7 @@ To install a specific version of containerD specify the version with -ContainerD
```powershell
# Example
.\Install-Containerd.ps1 -ContainerDVersion v1.4.1
.\Install-Containerd.ps1 -ContainerDVersion 1.4.1
```
{{< /note >}}
@@ -239,7 +239,7 @@ The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet servin
certificates by requesting them from the `certificates.k8s.io` API. One known limitation
is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically
approved by the default signer in the kube-controller-manager -
[`kubernetes.io/kubelet-serving`](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers).
[`kubernetes.io/kubelet-serving`](/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers).
This will require action from the user or a third party controller.
These CSRs can be viewed using:
@@ -23,7 +23,7 @@ You need to have a Kubernetes cluster and the kubectl command-line tool must be
## Set the Username for a Pod
To specify the username with which to execute the Pod's container processes, include the `securityContext` field ([PodSecurityContext](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritycontext-v1-core) in the Pod specification, and within it, the `windowsOptions` ([WindowsSecurityContextOptions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core) field containing the `runAsUserName` field.
To specify the username with which to execute the Pod's container processes, include the `securityContext` field ([PodSecurityContext](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritycontext-v1-core)) in the Pod specification, and within it, the `windowsOptions` ([WindowsSecurityContextOptions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core)) field containing the `runAsUserName` field.
The Windows security context options that you specify for a Pod apply to all Containers and init Containers in the Pod.
@@ -63,7 +63,7 @@ ContainerUser
## Set the Username for a Container
To specify the username with which to execute a Container's processes, include the `securityContext` field ([SecurityContext](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#securitycontext-v1-core)) in the Container manifest, and within it, the `windowsOptions` ([WindowsSecurityContextOptions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core) field containing the `runAsUserName` field.
To specify the username with which to execute a Container's processes, include the `securityContext` field ([SecurityContext](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#securitycontext-v1-core)) in the Container manifest, and within it, the `windowsOptions` ([WindowsSecurityContextOptions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#windowssecuritycontextoptions-v1-core)) field containing the `runAsUserName` field.
The Windows security context options that you specify for a Container apply only to that individual Container, and they override the settings made at the Pod level.
@@ -167,8 +167,8 @@ The output is similar to this:
Name: build-robot-secret
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name=build-robot
kubernetes.io/service-account.uid=da68f9c6-9d26-11e7-b84e-002dc52800da
Annotations: kubernetes.io/service-account.name: build-robot
kubernetes.io/service-account.uid: da68f9c6-9d26-11e7-b84e-002dc52800da
Type: kubernetes.io/service-account-token
@@ -54,7 +54,7 @@ If you use a Docker credentials store, you won't see that `auth` entry but a `cr
## Create a Secret based on existing Docker credentials {#registry-secret-existing-credentials}
A Kubernetes cluster uses the Secret of `docker-registry` type to authenticate with
A Kubernetes cluster uses the Secret of `kubernetes.io/dockerconfigjson` type to authenticate with
a container registry to pull a private image.
If you already ran `docker login`, you can copy that credential into Kubernetes:
@@ -45,8 +45,12 @@ kubectl create namespace qos-example
For a Pod to be given a QoS class of Guaranteed:
* Every Container, including init containers, in the Pod must have a memory limit and a memory request, and they must be the same.
* Every Container, including init containers, in the Pod must have a CPU limit and a CPU request, and they must be the same.
* Every Container in the Pod must have a memory limit and a memory request.
* For every Container in the Pod, the memory limit must equal the memory request.
* Every Container in the Pod must have a CPU limit and a CPU request.
* For every Container in the Pod, the CPU limit must equal the CPU request.
These restrictions apply to init containers and app containers equally.
Here is the configuration file for a Pod that has one Container. The Container has a memory limit and a
memory request, both equal to 200 MiB. The Container has a CPU limit and a CPU request, both equal to 700 milliCPU:
@@ -272,5 +276,3 @@ kubectl delete namespace qos-example
@@ -41,7 +41,7 @@ the container starts.
kubectl apply -f https://k8s.io/examples/debug/termination.yaml
In the YAML file, in the `cmd` and `args` fields, you can see that the
In the YAML file, in the `command` and `args` fields, you can see that the
container sleeps for 10 seconds and then writes "Sleep expired" to
the `/dev/termination-log` file. After the container writes
the "Sleep expired" message, it terminates.
@@ -1,371 +0,0 @@
---
reviewers:
- piosz
- x13n
title: Logging Using Stackdriver
content_type: concept
---
<!-- overview -->
Before reading this page, it's highly recommended to familiarize yourself
with the [overview of logging in Kubernetes](/docs/concepts/cluster-administration/logging).
{{< note >}}
By default, Stackdriver logging collects only your container's standard output and
standard error streams. To collect any logs your application writes to a file (for example),
see the [sidecar approach](/docs/concepts/cluster-administration/logging#sidecar-container-with-a-logging-agent)
in the Kubernetes logging overview.
{{< /note >}}
<!-- body -->
## Deploying
To ingest logs, you must deploy the Stackdriver Logging agent to each node in your cluster.
The agent is a configured `fluentd` instance, where the configuration is stored in a `ConfigMap`
and the instances are managed using a Kubernetes `DaemonSet`. The actual deployment of the
`ConfigMap` and `DaemonSet` for your cluster depends on your individual cluster setup.
### Deploying to a new cluster
#### Google Kubernetes Engine
Stackdriver is the default logging solution for clusters deployed on Google Kubernetes Engine.
Stackdriver Logging is deployed to a new cluster by default unless you explicitly opt-out.
#### Other platforms
To deploy Stackdriver Logging on a *new* cluster that you're
creating using `kube-up.sh`, do the following:
1. Set the `KUBE_LOGGING_DESTINATION` environment variable to `gcp`.
1. **If not running on GCE**, include the `beta.kubernetes.io/fluentd-ds-ready=true`
in the `KUBE_NODE_LABELS` variable.
Once your cluster has started, each node should be running the Stackdriver Logging agent.
The `DaemonSet` and `ConfigMap` are configured as addons. If you're not using `kube-up.sh`,
consider starting a cluster without a pre-configured logging solution and then deploying
Stackdriver Logging agents to the running cluster.
{{< warning >}}
The Stackdriver logging daemon has known issues on platforms other
than Google Kubernetes Engine. Proceed at your own risk.
{{< /warning >}}
### Deploying to an existing cluster
1. Apply a label on each node, if not already present.
The Stackdriver Logging agent deployment uses node labels to determine to which nodes
it should be allocated. These labels were introduced to distinguish nodes with the
Kubernetes version 1.6 or higher. If the cluster was created with Stackdriver Logging
configured and node has version 1.5.X or lower, it will have fluentd as static pod. Node
cannot have more than one instance of fluentd, therefore only apply labels to the nodes
that don't have fluentd pod allocated already. You can ensure that your node is labelled
properly by running `kubectl describe` as follows:
```
kubectl describe node $NODE_NAME
```
The output should be similar to this:
```
Name: NODE_NAME
Role:
Labels: beta.kubernetes.io/fluentd-ds-ready=true
...
```
Ensure that the output contains the label `beta.kubernetes.io/fluentd-ds-ready=true`. If it
is not present, you can add it using the `kubectl label` command as follows:
```
kubectl label node $NODE_NAME beta.kubernetes.io/fluentd-ds-ready=true
```
{{< note >}}
If a node fails and has to be recreated, you must re-apply the label to
the recreated node. To make this easier, you can use Kubelet's command-line parameter
for applying node labels in your node startup script.
{{< /note >}}
1. Deploy a `ConfigMap` with the logging agent configuration by running the following command:
```
kubectl apply -f https://k8s.io/examples/debug/fluentd-gcp-configmap.yaml
```
The command creates the `ConfigMap` in the `default` namespace. You can download the file
manually and change it before creating the `ConfigMap` object.
1. Deploy the logging agent `DaemonSet` by running the following command:
```
kubectl apply -f https://k8s.io/examples/debug/fluentd-gcp-ds.yaml
```
You can download and edit this file before using it as well.
## Verifying your Logging Agent Deployment
After Stackdriver `DaemonSet` is deployed, you can discover logging agent deployment status
by running the following command:
```shell
kubectl get ds --all-namespaces
```
If you have 3 nodes in the cluster, the output should looks similar to this:
```
NAMESPACE NAME DESIRED CURRENT READY NODE-SELECTOR AGE
...
default fluentd-gcp-v2.0 3 3 3 beta.kubernetes.io/fluentd-ds-ready=true 5m
...
```
To understand how logging with Stackdriver works, consider the following
synthetic log generator pod specification [counter-pod.yaml](/examples/debug/counter-pod.yaml):
{{< codenew file="debug/counter-pod.yaml" >}}
This pod specification has one container that runs a bash script
that writes out the value of a counter and the datetime once per
second, and runs indefinitely. Let's create this pod in the default namespace.
```shell
kubectl apply -f https://k8s.io/examples/debug/counter-pod.yaml
```
You can observe the running pod:
```shell
kubectl get pods
```
```
NAME READY STATUS RESTARTS AGE
counter 1/1 Running 0 5m
```
For a short period of time you can observe the 'Pending' pod status, because the kubelet
has to download the container image first. When the pod status changes to `Running`
you can use the `kubectl logs` command to view the output of this counter pod.
```shell
kubectl logs counter
```
```
0: Mon Jan 1 00:00:00 UTC 2001
1: Mon Jan 1 00:00:01 UTC 2001
2: Mon Jan 1 00:00:02 UTC 2001
...
```
As described in the logging overview, this command fetches log entries
from the container log file. If the container is killed and then restarted by
Kubernetes, you can still access logs from the previous container. However,
if the pod is evicted from the node, log files are lost. Let's demonstrate this
by deleting the currently running counter container:
```shell
kubectl delete pod counter
```
```
pod "counter" deleted
```
and then recreating it:
```shell
kubectl create -f https://k8s.io/examples/debug/counter-pod.yaml
```
```
pod/counter created
```
After some time, you can access logs from the counter pod again:
```shell
kubectl logs counter
```
```
0: Mon Jan 1 00:01:00 UTC 2001
1: Mon Jan 1 00:01:01 UTC 2001
2: Mon Jan 1 00:01:02 UTC 2001
...
```
As expected, only recent log lines are present. However, for a real-world
application you will likely want to be able to access logs from all containers,
especially for the debug purposes. This is exactly when the previously enabled
Stackdriver Logging can help.
## Viewing logs
Stackdriver Logging agent attaches metadata to each log entry, for you to use later
in queries to select only the messages you're interested in: for example,
the messages from a particular pod.
The most important pieces of metadata are the resource type and log name.
The resource type of a container log is `container`, which is named
`GKE Containers` in the UI (even if the Kubernetes cluster is not on Google Kubernetes Engine).
The log name is the name of the container, so that if you have a pod with
two containers, named `container_1` and `container_2` in the spec, their logs
will have log names `container_1` and `container_2` respectively.
System components have resource type `compute`, which is named
`GCE VM Instance` in the interface. Log names for system components are fixed.
For a Google Kubernetes Engine node, every log entry from a system component has one of the following
log names:
* docker
* kubelet
* kube-proxy
You can learn more about viewing logs on [the dedicated Stackdriver page](https://cloud.google.com/logging/docs/view/logs_viewer).
One of the possible ways to view logs is using the
[`gcloud logging`](https://cloud.google.com/logging/docs/api/gcloud-logging)
command line interface from the [Google Cloud SDK](https://cloud.google.com/sdk/).
It uses Stackdriver Logging [filtering syntax](https://cloud.google.com/logging/docs/view/advanced_filters)
to query specific logs. For example, you can run the following command:
```none
gcloud beta logging read 'logName="projects/$YOUR_PROJECT_ID/logs/count"' --format json | jq '.[].textPayload'
```
```
...
"2: Mon Jan 1 00:01:02 UTC 2001\n"
"1: Mon Jan 1 00:01:01 UTC 2001\n"
"0: Mon Jan 1 00:01:00 UTC 2001\n"
...
"2: Mon Jan 1 00:00:02 UTC 2001\n"
"1: Mon Jan 1 00:00:01 UTC 2001\n"
"0: Mon Jan 1 00:00:00 UTC 2001\n"
```
As you can see, it outputs messages for the count container from both
the first and second runs, despite the fact that the kubelet already deleted
the logs for the first container.
### Exporting logs
You can export logs to [Google Cloud Storage](https://cloud.google.com/storage/)
or to [BigQuery](https://cloud.google.com/bigquery/) to run further
analysis. Stackdriver Logging offers the concept of sinks, where you can
specify the destination of log entries. More information is available on
the Stackdriver [Exporting Logs page](https://cloud.google.com/logging/docs/export/configure_export_v2).
## Configuring Stackdriver Logging Agents
Sometimes the default installation of Stackdriver Logging may not suit your needs, for example:
* You may want to add more resources because default performance doesn't suit your needs.
* You may want to introduce additional parsing to extract more metadata from your log messages,
like severity or source code reference.
* You may want to send logs not only to Stackdriver or send it to Stackdriver only partially.
In this case you need to be able to change the parameters of `DaemonSet` and `ConfigMap`.
### Prerequisites
If you're using GKE and Stackdriver Logging is enabled in your cluster, you
cannot change its configuration, because it's managed and supported by GKE.
However, you can disable the default integration and deploy your own.
{{< note >}}
You will have to support and maintain a newly deployed configuration
yourself: update the image and configuration, adjust the resources and so on.
{{< /note >}}
To disable the default logging integration, use the following command:
```
gcloud beta container clusters update --logging-service=none CLUSTER
```
You can find notes on how to then install Stackdriver Logging agents into
a running cluster in the [Deploying section](#deploying).
### Changing `DaemonSet` parameters
When you have the Stackdriver Logging `DaemonSet` in your cluster, you can modify the
`template` field in its spec. The DaemonSet controller manages the pods for you.
For example, assume you've installed the Stackdriver Logging as described above. Now you want to
change the memory limit to give fluentd more memory to safely process more logs.
Get the spec of `DaemonSet` running in your cluster:
```shell
kubectl get ds fluentd-gcp-v2.0 --namespace kube-system -o yaml > fluentd-gcp-ds.yaml
```
Then edit resource requirements in the spec file and update the `DaemonSet` object
in the apiserver using the following command:
```shell
kubectl replace -f fluentd-gcp-ds.yaml
```
After some time, Stackdriver Logging agent pods will be restarted with the new configuration.
### Changing fluentd parameters
Fluentd configuration is stored in the `ConfigMap` object. It is effectively a set of configuration
files that are merged together. You can learn about fluentd configuration on the
[official site](https://docs.fluentd.org).
Imagine you want to add a new parsing logic to the configuration, so that fluentd can understand
default Python logging format. An appropriate fluentd filter looks similar to this:
```
<filter reform.**>
type parser
format /^(?<severity>\w):(?<logger_name>\w):(?<log>.*)/
reserve_data true
suppress_parse_error_log true
key_name log
</filter>
```
Now you have to put it in the configuration and make Stackdriver Logging agents pick it up.
Get the current version of the Stackdriver Logging `ConfigMap` in your cluster
by running the following command:
```shell
kubectl get cm fluentd-gcp-config --namespace kube-system -o yaml > fluentd-gcp-configmap.yaml
```
Then in the value of the key `containers.input.conf` insert a new filter right after
the `source` section.
{{< note >}}
Order is important.
{{< /note >}}
Updating `ConfigMap` in the apiserver is more complicated than updating `DaemonSet`. It's better
to consider `ConfigMap` to be immutable. Then, in order to update the configuration, you should
create `ConfigMap` with a new name and then change `DaemonSet` to point to it
using [guide above](#changing-daemonset-parameters).
### Adding fluentd plugins
Fluentd is written in Ruby and allows to extend its capabilities using
[plugins](https://www.fluentd.org/plugins). If you want to use a plugin, which is not included
in the default Stackdriver Logging container image, you have to build a custom image. Imagine
you want to add Kafka sink for messages from a particular container for additional processing.
You can re-use the default [container image sources](https://git.k8s.io/contrib/fluentd/fluentd-gcp-image)
with minor changes:
* Change Makefile to point to your container repository, for example `PREFIX=gcr.io/<your-project-id>`.
* Add your dependency to the Gemfile, for example `gem 'fluent-plugin-kafka'`.
Then run `make build push` from this directory. After updating `DaemonSet` to pick up the
new image, you can use the plugin you installed in the fluentd configuration.
@@ -80,7 +80,7 @@ Removing an old version:
If this occurs, switch back to using `served:true` on the old version, migrate the
remaining clients to the new version and repeat this step.
1. Ensure the [upgrade of existing objects to the new stored version](#upgrade-existing-objects-to-a-new-stored-version) step has been completed.
1. Verify that the `stored` is set to `true` for the new version in the `spec.versions` list in the CustomResourceDefinition.
1. Verify that the `storage` is set to `true` for the new version in the `spec.versions` list in the CustomResourceDefinition.
1. Verify that the old version is no longer listed in the CustomResourceDefinition `status.storedVersions`.
1. Remove the old version from the CustomResourceDefinition `spec.versions` list.
1. Drop conversion support for the old version in conversion webhooks.
@@ -180,7 +180,7 @@ spec:
containers:
- name: app
image: my-app
volumeMount:
volumeMounts:
- name: config
mountPath: /config
volumes:
@@ -234,7 +234,7 @@ spec:
containers:
- image: my-app
name: app
volumeMount:
volumeMounts:
- mountPath: /config
name: config
volumes:
@@ -327,7 +327,7 @@ spec:
containers:
- name: app
image: my-app
volumeMount:
volumeMounts:
- name: password
mountPath: /secrets
volumes:
@@ -1,21 +0,0 @@
---
title: "gcloud kubectl install"
description: "How to install kubectl with gcloud snippet for inclusion in each OS-specific tab."
headless: true
---
You can install kubectl as part of the Google Cloud SDK.
1. Install the [Google Cloud SDK](https://cloud.google.com/sdk/).
1. Run the `kubectl` installation command:
```shell
gcloud components install kubectl
```
1. Test to ensure the version you installed is up-to-date:
```shell
kubectl version --client
```
@@ -22,7 +22,6 @@ The following methods exist for installing kubectl on Linux:
- [Install kubectl binary with curl on Linux](#install-kubectl-binary-with-curl-on-linux)
- [Install using native package management](#install-using-native-package-management)
- [Install using other package management](#install-using-other-package-management)
- [Install on Linux as part of the Google Cloud SDK](#install-on-linux-as-part-of-the-google-cloud-sdk)
### Install kubectl binary with curl on Linux
@@ -168,10 +167,6 @@ kubectl version --client
{{< /tabs >}}
### Install on Linux as part of the Google Cloud SDK
{{< include "included/install-kubectl-gcloud.md" >}}
## Verify kubectl configuration
{{< include "included/verify-kubectl.md" >}}
@@ -22,7 +22,6 @@ The following methods exist for installing kubectl on macOS:
- [Install kubectl binary with curl on macOS](#install-kubectl-binary-with-curl-on-macos)
- [Install with Homebrew on macOS](#install-with-homebrew-on-macos)
- [Install with Macports on macOS](#install-with-macports-on-macos)
- [Install on macOS as part of the Google Cloud SDK](#install-on-macos-as-part-of-the-google-cloud-sdk)
### Install kubectl binary with curl on macOS
@@ -31,6 +30,7 @@ The following methods exist for installing kubectl on macOS:
{{< tabs name="download_binary_macos" >}}
{{< tab name="Intel" codelang="bash" >}}
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x kubectl
{{< /tab >}}
{{< tab name="Apple Silicon" codelang="bash" >}}
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl"
@@ -148,11 +148,6 @@ If you are on macOS and using [Macports](https://macports.org/) package manager,
kubectl version --client
```
### Install on macOS as part of the Google Cloud SDK
{{< include "included/install-kubectl-gcloud.md" >}}
## Verify kubectl configuration
{{< include "included/verify-kubectl.md" >}}
@@ -21,7 +21,6 @@ The following methods exist for installing kubectl on Windows:
- [Install kubectl binary with curl on Windows](#install-kubectl-binary-with-curl-on-windows)
- [Install on Windows using Chocolatey or Scoop](#install-on-windows-using-chocolatey-or-scoop)
- [Install on Windows as part of the Google Cloud SDK](#install-on-windows-as-part-of-the-google-cloud-sdk)
### Install kubectl binary with curl on Windows
@@ -127,10 +126,6 @@ If you have installed Docker Desktop before, you may need to place your `PATH` e
Edit the config file with a text editor of your choice, such as Notepad.
{{< /note >}}
### Install on Windows as part of the Google Cloud SDK
{{< include "included/install-kubectl-gcloud.md" >}}
## Verify kubectl configuration
{{< include "included/verify-kubectl.md" >}}
@@ -147,4 +142,4 @@ Below are the procedures to set up autocompletion for Zsh, if you are running th
## {{% heading "whatsnext" %}}
{{< include "included/kubectl-whats-next.md" >}}
{{< include "included/kubectl-whats-next.md" >}}
+1 -1
View File
@@ -224,7 +224,7 @@ The minikube tool includes a set of built-in {{< glossary_tooltip text="addons"
The output is similar to:
```
metrics-server was successfully enabled
The 'metrics-server' addon is enabled
```
3. View the Pod and Service you created:
+55 -55
View File
@@ -10,10 +10,10 @@ For general information about Kubernetes release cycle, see the
## Cadence
Our typical patch release cadence is monthly. It is
Our typical patch release cadence is monthly. It is
commonly a bit faster (1 to 2 weeks) for the earliest patch releases
after a 1.X minor release. Critical bug fixes may cause a more
immediate release outside of the normal cadence. We also aim to not make
after a 1.X minor release. Critical bug fixes may cause a more
immediate release outside of the normal cadence. We also aim to not make
releases during major holiday periods.
## Contact
@@ -23,7 +23,7 @@ See the [Release Managers page][release-managers] for full contact details on th
Please give us a business day to respond - we may be in a different timezone!
In between releases the team is looking at incoming cherry pick
requests on a weekly basis. The team will get in touch with
requests on a weekly basis. The team will get in touch with
submitters via GitHub PR, SIG channels in Slack, and direct messages
in Slack and [email](mailto:release-managers-private@kubernetes.io)
if there are questions on the PR.
@@ -34,8 +34,8 @@ Please follow the [cherry pick process][cherry-picks].
Cherry picks must be merge-ready in GitHub with proper labels (e.g.,
`approved`, `lgtm`, `release-note`) and passing CI tests ahead of the
cherry pick deadline. This is typically two days before the target
release, but may be more. Earlier PR readiness is better, as we
cherry pick deadline. This is typically two days before the target
release, but may be more. Earlier PR readiness is better, as we
need time to get CI signal after merging your cherry picks ahead
of the actual release.
@@ -73,15 +73,15 @@ dates for simplicity (every month has it).
## Upcoming Monthly Releases
Timelines may vary with the severity of bug fixes, but for easier planning we
will target the following monthly release points. Unplanned, critical
will target the following monthly release points. Unplanned, critical
releases may also occur in between these.
| Monthly Patch Release | Target date |
| --- | --- |
| June 2021 | 2021-06-16 |
| July 2021 | 2021-07-14 |
| August 2021 | 2021-08-11 |
| September 2021 | 2021-09-15 |
| --------------------- | ----------- |
| June 2021 | 2021-06-16 |
| July 2021 | 2021-07-14 |
| August 2021 | 2021-08-11 |
| September 2021 | 2021-09-15 |
## Detailed Release History for Active Branches
@@ -92,7 +92,7 @@ releases may also occur in between these.
End of Life for **1.21** is **2022-06-28**
| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
|--- |--- |--- |
| ------------- | -------------------- | ----------- |
| 1.21.2 | 2021-06-12 | 2021-06-16 |
| 1.21.1 | 2021-05-07 | 2021-05-12 |
@@ -102,16 +102,16 @@ End of Life for **1.21** is **2022-06-28**
End of Life for **1.20** is **2022-02-28**
| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
|--- |--- |--- |
| 1.20.8 | 2021-06-12 | 2021-06-16 |
| 1.20.7 | 2021-05-07 | 2021-05-12 |
| 1.20.6 | 2021-04-09 | 2021-04-14 |
| 1.20.5 | 2021-03-12 | 2021-03-17 |
| 1.20.4 | 2021-02-12 | 2021-02-18 |
| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
| ------------- | ----------------------------------------------------------------------------------- | ----------- |
| 1.20.8 | 2021-06-12 | 2021-06-16 |
| 1.20.7 | 2021-05-07 | 2021-05-12 |
| 1.20.6 | 2021-04-09 | 2021-04-14 |
| 1.20.5 | 2021-03-12 | 2021-03-17 |
| 1.20.4 | 2021-02-12 | 2021-02-18 |
| 1.20.3 | [Conformance Tests Issue](https://groups.google.com/g/kubernetes-dev/c/oUpY9vWgzJo) | 2021-02-17 |
| 1.20.2 | 2021-01-08 | 2021-01-13 |
| 1.20.1 | [Tagging Issue](https://groups.google.com/g/kubernetes-dev/c/dNH2yknlCBA) | 2020-12-18 |
| 1.20.2 | 2021-01-08 | 2021-01-13 |
| 1.20.1 | [Tagging Issue](https://groups.google.com/g/kubernetes-dev/c/dNH2yknlCBA) | 2020-12-18 |
### 1.19
@@ -119,46 +119,46 @@ End of Life for **1.20** is **2022-02-28**
End of Life for **1.19** is **2021-10-28**
| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
|--- |--- |--- |
| 1.19.12 | 2021-06-12 | 2021-06-16 |
| 1.19.11 | 2021-05-07 | 2021-05-12 |
| 1.19.10 | 2021-04-09 | 2021-04-14 |
| 1.19.9 | 2021-03-12 | 2021-03-17 |
| 1.19.8 | 2021-02-12 | 2021-02-17 |
| 1.19.7 | 2021-01-08 | 2021-01-13 |
| PATCH RELEASE | CHERRY PICK DEADLINE | TARGET DATE |
| ------------- | ------------------------------------------------------------------------- | ----------- |
| 1.19.12 | 2021-06-12 | 2021-06-16 |
| 1.19.11 | 2021-05-07 | 2021-05-12 |
| 1.19.10 | 2021-04-09 | 2021-04-14 |
| 1.19.9 | 2021-03-12 | 2021-03-17 |
| 1.19.8 | 2021-02-12 | 2021-02-17 |
| 1.19.7 | 2021-01-08 | 2021-01-13 |
| 1.19.6 | [Tagging Issue](https://groups.google.com/g/kubernetes-dev/c/dNH2yknlCBA) | 2020-12-18 |
| 1.19.5 | 2020-12-04 | 2020-12-09 |
| 1.19.4 | 2020-11-06 | 2020-11-11 |
| 1.19.3 | 2020-10-09 | 2020-10-14 |
| 1.19.2 | 2020-09-11 | 2020-09-16 |
| 1.19.1 | 2020-09-04 | 2020-09-09 |
| 1.19.5 | 2020-12-04 | 2020-12-09 |
| 1.19.4 | 2020-11-06 | 2020-11-11 |
| 1.19.3 | 2020-10-09 | 2020-10-14 |
| 1.19.2 | 2020-09-11 | 2020-09-16 |
| 1.19.1 | 2020-09-04 | 2020-09-09 |
## Non-Active Branch History
These releases are no longer supported.
| Minor Version | Final Patch Release | EOL date |
| --- | --- | --- |
| 1.18 | 1.18.19 | 2021-05-12 |
| 1.17 | 1.17.17 | 2021-01-13 |
| 1.16 | 1.16.15 | 2020-09-02 |
| 1.15 | 1.15.12 | 2020-05-06 |
| 1.14 | 1.14.10 | 2019-12-11 |
| 1.13 | 1.13.12 | 2019-10-15 |
| 1.12 | 1.12.10 | 2019-07-08 |
| 1.11 | 1.11.10 | 2019-05-01 |
| 1.10 | 1.10.13 | 2019-02-13 |
| 1.9 | 1.9.11 | 2018-09-29 |
| 1.8 | 1.8.15 | 2018-07-12 |
| 1.7 | 1.7.16 | 2018-04-04 |
| 1.6 | 1.6.13 | 2017-11-23 |
| 1.5 | 1.5.8 | 2017-10-01 |
| 1.4 | 1.4.12 | 2017-04-21 |
| 1.3 | 1.3.10 | 2016-11-01 |
| 1.2 | 1.2.7 | 2016-10-23 |
| Minor Version | Final Patch Release | EOL date |
| ------------- | ------------------- | ---------- |
| 1.18 | 1.18.19 | 2021-05-12 |
| 1.17 | 1.17.17 | 2021-01-13 |
| 1.16 | 1.16.15 | 2020-09-02 |
| 1.15 | 1.15.12 | 2020-05-06 |
| 1.14 | 1.14.10 | 2019-12-11 |
| 1.13 | 1.13.12 | 2019-10-15 |
| 1.12 | 1.12.10 | 2019-07-08 |
| 1.11 | 1.11.10 | 2019-05-01 |
| 1.10 | 1.10.13 | 2019-02-13 |
| 1.9 | 1.9.11 | 2018-09-29 |
| 1.8 | 1.8.15 | 2018-07-12 |
| 1.7 | 1.7.16 | 2018-04-04 |
| 1.6 | 1.6.13 | 2017-11-23 |
| 1.5 | 1.5.8 | 2017-10-01 |
| 1.4 | 1.4.12 | 2017-04-21 |
| 1.3 | 1.3.10 | 2016-11-01 |
| 1.2 | 1.2.7 | 2016-10-23 |
[cherry-picks]: https://git.k8s.io/community/contributors/devel/sig-release/cherry-picks.md
[cherry-picks]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md
[release-managers]: /release-managers.md
[release process description]: /release.md
[yearly-support]: https://git.k8s.io/enhancements/keps/sig-release/1498-kubernetes-yearly-support-period/README.md
+2
View File
@@ -192,6 +192,8 @@ GitHub team: [@kubernetes/sig-release-leads](https://github.com/orgs/kubernetes/
### Technical Leads
- Adolfo García Veytia ([@puerco](https://github.com/puerco))
- Carlos Panato ([@cpanato](https://github.com/cpanato))
- Daniel Mangum ([@hasheddan](https://github.com/hasheddan))
- Jeremy Rickard ([@jeremyrickard](https://github.com/jeremyrickard))
+13 -12
View File
@@ -3,6 +3,7 @@ title: Kubernetes Release Cycle
type: docs
auto_generated: true
---
<!-- THIS CONTENT IS AUTO-GENERATED via ./scripts/releng/update-release-info.sh in k/website -->
{{< warning >}}
@@ -89,43 +90,43 @@ The general labeling process should be consistent across artifact types.
## Definitions
- *issue owners*: Creator, assignees, and user who moved the issue into a
- _issue owners_: Creator, assignees, and user who moved the issue into a
release milestone
- *Release Team*: Each Kubernetes release has a team doing project management
- _Release Team_: Each Kubernetes release has a team doing project management
tasks described [here][release-team].
The contact info for the team associated with any given release can be found
[here](https://git.k8s.io/sig-release/releases/).
- *Y days*: Refers to business days
- _Y days_: Refers to business days
- *enhancement*: see "[Is My Thing an Enhancement?](https://git.k8s.io/enhancements/README.md#is-my-thing-an-enhancement)"
- _enhancement_: see "[Is My Thing an Enhancement?](https://git.k8s.io/enhancements/README.md#is-my-thing-an-enhancement)"
- *[Enhancements Freeze][enhancements-freeze]*:
- _[Enhancements Freeze][enhancements-freeze]_:
the deadline by which [KEPs][keps] have to be completed in order for
enhancements to be part of the current release
- *[Exception Request][exceptions]*:
- _[Exception Request][exceptions]_:
The process of requesting an extension on the deadline for a particular
Enhancement
- *[Code Freeze][code-freeze]*:
- _[Code Freeze][code-freeze]_:
The period of ~4 weeks before the final release date, during which only
critical bug fixes are merged into the release.
- *[Pruning](https://git.k8s.io/sig-release/releases/release_phases.md#pruning)*:
- _[Pruning](https://git.k8s.io/sig-release/releases/release_phases.md#pruning)_:
The process of removing an Enhancement from a release milestone if it is not
fully implemented or is otherwise considered not stable.
- *release milestone*: semantic version string or
- _release milestone_: semantic version string or
[GitHub milestone](https://help.github.com/en/github/managing-your-work-on-github/associating-milestones-with-issues-and-pull-requests)
referring to a release MAJOR.MINOR `vX.Y` version.
See also
[release versioning](/contributors/design-proposals/release/versioning.md).
- *release branch*: Git branch `release-X.Y` created for the `vX.Y` milestone.
- _release branch_: Git branch `release-X.Y` created for the `vX.Y` milestone.
Created at the time of the `vX.Y-rc.0` release and maintained after the
release for approximately 12 months with `vX.Y.Z` patch releases.
@@ -160,7 +161,7 @@ conjunction with the Release Team's [Enhancements Lead](https://git.k8s.io/sig-r
After Enhancements Freeze, tracking milestones on PRs and issues is important.
Items within the milestone are used as a punchdown list to complete the
release. *On issues*, milestones must be applied correctly, via triage by the
release. _On issues_, milestones must be applied correctly, via triage by the
SIG, so that [Release Team][release-team] can track bugs and enhancements (any
enhancement-related issue needs a milestone).
@@ -354,7 +355,7 @@ issue kind labels must be set:
- `kind/feature`: New functionality.
- `kind/flake`: CI test case is showing intermittent failures.
[cherry-picks]: /contributors/devel/sig-release/cherry-picks.md
[cherry-picks]: /community/blob/master/contributors/devel/sig-release/cherry-picks.md
[code-freeze]: https://git.k8s.io/sig-release/releases/release_phases.md#code-freeze
[enhancements-freeze]: https://git.k8s.io/sig-release/releases/release_phases.md#enhancements-freeze
[exceptions]: https://git.k8s.io/sig-release/releases/release_phases.md#exceptions