Merge remote-tracking branch 'upstream/main' into dev-1.24

2022-01-17 09:46:19 -08:00
parent c8c474d07d c870be11cb
commit 015825f608
65 changed files with 5670 additions and 411 deletions
@@ -7,11 +7,16 @@ slug: are-you-ready-for-dockershim-removal

 **Author:** Sergey Kanzhelev, Google. With reviews from Davanum Srinivas, Elana Hashman, Noah Kantrowitz, Rey Lejano.

+{{% alert color="info" title="Poll closed" %}}
+This poll closed on January 7, 2022.
+{{% /alert %}}
+
 Last year we announced that Dockershim is being deprecated: [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/).
 Our current plan is to remove dockershim from the Kubernetes codebase soon.
 We are looking for feedback from you whether you are ready for dockershim
 removal and to ensure that you are ready when the time comes.
-**Please fill out this survey: https://forms.gle/svCJmhvTv78jGdSx8**.
+
+<del>Please fill out this survey: https://forms.gle/svCJmhvTv78jGdSx8</del>

 The dockershim component that enables Docker as a Kubernetes container runtime is
 being deprecated in favor of runtimes that directly use the [Container Runtime Interface](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)
@@ -25,7 +30,7 @@ are still not ready: [migrating telemetry and security agents](/docs/tasks/admin
 At this point, we believe that there is feature parity between Docker and the
 other runtimes. Many end-users have used our [migration guide](/docs/tasks/administer-cluster/migrating-from-dockershim/)
 and are running production workload using these different runtimes. The plan of
-record today is that dockershim will be removed in version 1.24, slated for 
+record today is that dockershim will be removed in version 1.24, slated for
 release around April of next year. For those developing or running alpha and
 beta versions, dockershim will be removed in December at the beginning of the
 1.24 release development cycle.
@@ -33,7 +38,7 @@ beta versions, dockershim will be removed in December at the beginning of the
 There is only one month left to give us feedback. We want you to tell us how
 ready you are.

-**We are collecting opinions through this survey: [https://forms.gle/svCJmhvTv78jGdSx8](https://forms.gle/svCJmhvTv78jGdSx8)**
+<del>We are collecting opinions through this survey: https://forms.gle/svCJmhvTv78jGdSx8</del>
 To better understand preparedness for the dockershim removal, our survey is
 asking the version of Kubernetes you are currently using, and an estimate of
 when you think you will adopt Kubernetes 1.24. All the aggregated information
@@ -0,0 +1,103 @@
+---
+layout: blog
+title: "Kubernetes is Moving on From Dockershim: Commitments and Next Steps"
+date: 2022-01-07
+slug: kubernetes-is-moving-on-from-dockershim
+---
+
+**Authors:** Sergey Kanzhelev (Google), Jim Angel (Google), Davanum Srinivas (VMware), Shannon Kularathna (Google), Chris Short (AWS), Dawn Chen (Google)
+
+Kubernetes is removing dockershim in the upcoming v1.24 release. We're excited
+to reaffirm our community values by supporting open source container runtimes,
+enabling a smaller kubelet, and increasing engineering velocity for teams using
+Kubernetes. If you [use Docker Engine as a container runtime](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/)
+for your Kubernetes cluster, get ready to migrate in 1.24! To check if you're
+affected, refer to [Check whether dockershim deprecation affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/).
+
+## Why we’re moving away from dockershim
+
+Docker was the first container runtime used by Kubernetes. This is one of the
+reasons why Docker is so familiar to many Kubernetes users and enthusiasts.
+Docker support was hardcoded into Kubernetes – a component the project refers to
+as dockershim.
+As containerization became an industry standard, the Kubernetes project added support
+for additional runtimes. This culminated in the implementation of the
+container runtime interface (CRI), letting system components (like the kubelet)
+talk to container runtimes in a standardized way. As a result, dockershim became
+an anomaly in the Kubernetes project. 
+Dependencies on Docker and dockershim have crept into various tools
+and projects in the CNCF ecosystem ecosystem, resulting in fragile code.
+
+By removing the
+dockershim CRI, we're embracing the first value of CNCF: "[Fast is better than
+slow](https://github.com/cncf/foundation/blob/master/charter.md#3-values)".
+Stay tuned for future communications on the topic!
+
+## Deprecation timeline
+
+We [formally announced](/blog/2020/12/08/kubernetes-1-20-release-announcement/) the dockershim deprecation in December 2020. Full removal is targeted
+in Kubernetes 1.24, in April 2022. This timeline
+aligns with our [deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior),
+which states that deprecated behaviors must function for at least 1 year
+after their announced deprecation.
+
+We'll support Kubernetes version 1.23, which includes
+dockershim, for another year in the Kubernetes project. For managed
+Kubernetes providers, vendor support is likely to last even longer, but this is
+dependent on the companies themselves. Regardless, we're confident all cluster operations will have
+time to migrate. If you have more questions about the dockershim removal, refer
+to the [Dockershim Deprecation FAQ](/dockershim).
+
+We asked you whether you feel prepared for the migration from dockershim in this
+survey: [Are you ready for Dockershim removal](/blog/2021/11/12/are-you-ready-for-dockershim-removal/).
+We had over 600 responses. To everybody who took time filling out the survey,
+thank you.
+
+The results show that we still have a lot of ground to cover to help you to
+migrate smoothly. Other container runtimes exist, and have been promoted
+extensively. However, many users told us they still rely on dockershim,
+and sometimes have dependencies that need to be re-worked. Some of these
+dependencies are outside of your control. Based on your feedback, here are some
+of the steps we are taking to help.
+
+## Our next steps
+
+Based on the feedback you provided:
+
+- CNCF and the 1.24 release team are committed to delivering documentation in
+  time for the 1.24 release. This includes more informative blog posts like this
+  one, updating existing code samples, tutorials, and tasks, and producing a
+  migration guide for cluster operators.
+- We are reaching out to the rest of the CNCF community to help prepare them for
+  this change.
+
+If you're part of a project with dependencies on dockershim, or if you're
+interested in helping with the migration effort, please join us! There's always
+room for more contributors, whether to our transition tools or to our
+documentation. To get started, say hello in the
+[#sig-node](https://kubernetes.slack.com/archives/C0BP8PW9G)
+channel on [Kubernetes Slack](https://slack.kubernetes.io/)!
+
+## Final thoughts
+
+As a project, we've already seen cluster operators increasingly adopt other
+container runtimes through 2021. 
+We believe there are no major blockers to migration. The steps we're taking to
+improve the migration experience will light the path more clearly for you.
+
+We understand that migration from dockershim is yet another action you may need to
+do to keep your Kubernetes infrastructure up to date. For most of you, this step
+will be straightforward and transparent. In some cases, you will encounter
+hiccups or issues. The community has discussed at length whether postponing the
+dockershim removal would be helpful. For example, we recently talked about it in
+the [SIG Node discussion on November 11th](https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#bookmark=id.r77y11bgzid)
+and in the [Kubernetes Steering committee meeting held on December 6th](https://docs.google.com/document/d/1qazwMIHGeF3iUh5xMJIJ6PDr-S3bNkT8tNLRkSiOkOU/edit#bookmark=id.m0ir406av7jx).
+We already [postponed](https://github.com/kubernetes/enhancements/pull/2481/) it
+once in 2021 because the adoption rate of other
+runtimes was lower than we wanted, which also gave us more time to identify
+potential blocking issues.
+
+At this point, we believe that the value that you (and Kubernetes) gain from
+dockershim removal makes up for the migration effort you'll have. Start planning
+now to avoid surprises. We'll have more updates and guides before Kubernetes
+1.24 is released.
@@ -0,0 +1,106 @@
+---
+layout: blog
+title: "Meet Our Contributors - APAC (India region)"
+date: 2022-01-10T12:00:00+0000
+slug: meet-our-contributors-india-ep-01
+canonicalUrl: https://kubernetes.dev/blog/2022/01/10/meet-our-contributors-india-ep-01/
+---
+
+**Authors & Interviewers:** [Anubhav Vardhan](https://github.com/anubha-v-ardhan), [Atharva Shinde](https://github.com/Atharva-Shinde), [Avinesh Tripathi](https://github.com/AvineshTripathi), [Debabrata Panigrahi](https://github.com/Debanitrkl), [Kunal Verma](https://github.com/verma-kunal), [Pranshu Srivastava](https://github.com/PranshuSrivastava), [Pritish Samal](https://github.com/CIPHERTron), [Purneswar Prasad](https://github.com/PurneswarPrasad), [Vedant Kakde](https://github.com/vedant-kakde)
+
+**Editor:** [Priyanka Saggu](https://psaggu.com)
+
+---
+
+Good day, everyone 👋
+
+Welcome to the first episode of the APAC edition of the "Meet Our Contributors" blog post series.
+
+
+In this post, we'll introduce you to five amazing folks from the India region who have been actively contributing to the upstream Kubernetes projects in a variety of ways, as well as being the leaders or maintainers of numerous community initiatives.
+
+💫 *Let's get started, so without further ado…* 
+
+
+## [Arsh Sharma](https://github.com/RinkiyaKeDad)
+
+Arsh is currently employed with Okteto as a Developer Experience engineer. As a new contributor, he realised that 1:1 mentorship opportunities were quite beneficial in getting him started with the upstream project.
+
+He is presently a CI Signal shadow on the Kubernetes 1.23 release team. He is also contributing to the SIG Testing and SIG Docs projects, as well as to the [cert-manager](https://github.com/cert-manager/infrastructure) tools development work that is being done under the aegis of SIG Architecture.
+
+To the newcomers, Arsh helps plan their early contributions sustainably.
+
+> _I would encourage folks to contribute in a way that's sustainable. What I mean by that
+> is that it's easy to be very enthusiastic early on and take up more stuff than one can
+> actually handle. This can often lead to burnout in later stages. It's much more sustainable
+> to work on things iteratively._
+
+## [Kunal Kushwaha](https://github.com/kunal-kushwaha)
+
+Kunal Kushwaha is a core member of the Kubernetes marketing council. He is also a CNCF ambassador and one of the founders of the [CNCF Students Program](https://community.cncf.io/cloud-native-students/).. He also served as a Communications role shadow during the 1.22 release cycle.
+
+At the end of his first year, Kunal began contributing to the [fabric8io kubernetes-client](https://github.com/fabric8io/kubernetes-client) project. He was then selected to work on the same project as part of Google Summer of Code. Kunal mentored people on the same project, first through Google Summer of Code then through Google Code-in. 
+
+As an open-source enthusiast, he believes that diverse participation in the community is beneficial since it introduces new perspectives and opinions and respect for one's peers. He has worked on various open-source projects, and his participation in communities has considerably assisted his development as a developer.
+
+
+> _I believe if you find yourself in a place where you do not know much about the
+> project, that's a good thing because now you can learn while contributing and the
+> community is there to help you. It has helped me a lot in gaining skills, meeting
+> people from around the world and also helping them. You can learn on the go,
+> you don't have to be an expert. Make sure to also check out no code contributions
+> because being a beginner is a skill and you can bring new perspectives to the
+> organisation._
+
+## [Madhav Jivarajani](https://github.com/MadhavJivrajani)
+
+
+Madhav Jivarajani works on the VMware Upstream Kubernetes stability team. He began contributing to the Kubernetes project in January 2021 and has since made significant contributions to several areas of work under SIG Architecture, SIG API Machinery, and SIG ContribEx (contributor experience).
+
+Among several significant contributions are his recent efforts toward the Archival of [design proposals](https://github.com/kubernetes/community/issues/6055), refactoring the ["groups" codebase](https://github.com/kubernetes/k8s.io/pull/2713) under k8s-infra repository to make it mockable and testable, and improving the functionality of the [GitHub k8s bot](https://github.com/kubernetes/test-infra/issues/23129).
+
+In addition to his technical efforts, Madhav oversees many projects aimed at assisting new contributors. He organises bi-weekly "KEP reading club" sessions to help newcomers understand the process of adding new features, deprecating old ones, and making other key changes to the upstream project. He has also worked on developing [Katacoda scenarios](https://github.com/kubernetes-sigs/contributor-katacoda) to assist new contributors to become acquainted with the process of contributing to k/k. In addition to his current efforts to meet with community members every week, he has organised several [new contributors workshops (NCW)](https://www.youtube.com/watch?v=FgsXbHBRYIc).
+
+> _I initially did not know much about Kubernetes. I joined because the community was
+> super friendly. But what made me stay was not just the people, but the project itself.
+> My solution to not feeling overwhelmed in the community was to gain as much context
+> and knowledge into the topics that I was interested in and were being discussed. And
+> as a result I continued to dig deeper into Kubernetes and the design of it.
+> I am a systems nut & thus Kubernetes was an absolute goldmine for me._
+
+
+## [Rajas Kakodkar](https://github.com/rajaskakodkar)
+
+Rajas Kakodkar currently works at VMware as a Member of Technical Staff. He has been engaged in many aspects of the upstream Kubernetes project since 2019.
+
+He is now a key contributor to the Testing special interest group. He is also active in the SIG Network community. Lately, Rajas has contributed significantly to the [NetworkPolicy++](https://docs.google.com/document/d/1AtWQy2fNa4qXRag9cCp5_HsefD7bxKe3ea2RPn8jnSs/) and [`kpng`](https://github.com/kubernetes-sigs/kpng) sub-projects.
+
+One of the first challenges he ran across was that he was in a different time zone than the upstream project's regular meeting hours. However, async interactions on community forums progressively corrected that problem.
+
+> _I enjoy contributing to Kubernetes not just because I get to work on
+> cutting edge tech but more importantly because I get to work with
+> awesome people and help in solving real world problems._
+
+## [Rajula Vineet Reddy](https://github.com/rajula96reddy)
+
+Rajula Vineet Reddy, a Junior Engineer at CERN, is a member of the Marketing Council team under SIG ContribEx . He also served as a release shadow for SIG Release during the 1.22 and 1.23 Kubernetes release cycles.
+
+He started looking at the Kubernetes project as part of a university project with the help of one of his professors. Over time, he spent a significant amount of time reading the project's documentation, Slack discussions, GitHub issues, and blogs, which helped him better grasp the Kubernetes project and piqued his interest in contributing upstream. One of his key contributions was his assistance with automation in the SIG ContribEx Upstream Marketing subproject.
+
+According to Rajula, attending project meetings and shadowing various project roles are vital for learning about the community.
+
+> _I find the community very helpful and it's always_
+> “you get back as much as you contribute”.
+> _The more involved you are, the more you will understand, get to learn and
+> contribute new things._
+> 
+> _The first step to_ “come forward and start” _is hard. But it's all gonna be
+> smooth after that. Just take that jump._
+
+---
+
+If you have any recommendations/suggestions for who we should interview next, please let us know in #sig-contribex. We're thrilled to have other folks assisting us in reaching out to even more wonderful individuals of the community. Your suggestions would be much appreciated.
+
+
+We'll see you all in the next one. Everyone, till then, have a happy contributing! 👋
+
@@ -0,0 +1,44 @@
+---
+layout: blog
+title: "Securing Admission Controllers"
+date: 2022-01-19
+slug: secure-your-admission-controllers-and-webhooks
+---
+
+**Author:** Rory McCune (Aqua Security)
+
+[Admission control](/docs/reference/access-authn-authz/admission-controllers/) is a key part of Kubernetes security, alongside authentication and authorization. Webhook admission controllers are extensively used to help improve the security of Kubernetes clusters in a variety of ways including restricting the privileges of workloads and ensuring that images deployed to the cluster meet organization’s security requirements.
+
+However, as with any additional component added to a cluster, security risks can present themselves. A security risk example is if the deployment and management of the admission controller are not handled correctly. To help admission controller users and designers manage these risks appropriately, the [security documentation](https://github.com/kubernetes/community/tree/master/sig-security#security-docs) subgroup of SIG Security has spent some time developing a [threat model for admission controllers](https://github.com/kubernetes/sig-security/tree/main/sig-security-docs/papers/admission-control). This threat model looks at likely risks which may arise from the incorrect use of admission controllers, which could allow security policies to be bypassed, or even allow an attacker to get unauthorised access to the cluster.
+
+From the threat model, we developed a set of security best practices that should be adopted to ensure that cluster operators can get the security benefits of admission controllers whilst avoiding any risks from using them.
+
+## Admission controllers and good practices for security
+
+From the threat model, a couple of themes emerged around how to ensure the security of admission controllers.
+
+### Secure webhook configuration
+
+It’s important to ensure that any security component in a cluster is well configured and admission controllers are no different here. There are a couple of security best practices to consider when using admission controllers
+
+* **Correctly configured TLS for all webhook traffic**. Communications between the API server and the admission controller webhook should be authenticated and encrypted to ensure that attackers who may be in a network position to view or modify this traffic cannot do so. To achieve this access the API server and webhook must be using certificates from a trusted certificate authority so that they can validate their mutual identities
+* **Only authenticated access allowed**. If an attacker can send an admission controller large numbers of requests, they may be able to overwhelm the service causing it to fail. Ensuring all access requires strong authentication should mitigate that risk.
+* **Admission controller fails closed**. This is a security practice that has a tradeoff, so whether a cluster operator wants to configure it will depend on the cluster’s threat model. If an admission controller fails closed, when the API server can’t get a response from it, all deployments will fail. This stops attackers bypassing the admission controller by disabling it, but, can disrupt the cluster’s operation. As clusters can have multiple webhooks, one approach to hit a middle ground might be to have critical controls on a fail closed setups and less critical controls allowed to fail open.
+* **Regular reviews of webhook configuration**. Configuration mistakes can lead to security issues, so it’s important that the admission controller webhook configuration is checked to make sure the settings are correct. This kind of review could be done automatically by an Infrastructure As Code scanner or manually by an administrator.
+
+
+### Secure cluster configuration for admission control
+
+In most cases, the admission controller webhook used by a cluster will be installed as a workload in the cluster. As a result, it’s important to ensure that Kubernetes' security features that could impact its operation are well configured.
+
+* **Restrict [RBAC](/docs/reference/access-authn-authz/rbac/) rights**. Any user who has rights which would allow them to modify the configuration of the webhook objects or the workload that the admission controller uses could disrupt its operation. So it’s important to make sure that only cluster administrators have those rights.
+* **Prevent privileged workloads**. One of the realities of container systems is that if a workload is given certain privileges, it will be possible to break out to the underlying cluster node and impact other containers on that node. Where admission controller services run in the cluster they’re protecting, it’s important to ensure that any requirement for privileged workloads is carefully reviewed and restricted as much as possible.
+* **Strictly control external system access**. As a security service in a cluster admission controller systems will have access to sensitive information like credentials. To reduce the risk of this information being sent outside the cluster, [network policies](/docs/concepts/services-networking/network-policies/) should be used to restrict the admission controller services access to external networks.
+* **Each cluster has a dedicated webhook**. Whilst it may be possible to have admission controller webhooks that serve multiple clusters, there is a risk when using that model that an attack on the webhook service would have a larger impact where it’s shared. Also where multiple clusters use an admission controller there will be increased complexity and access requirements, making it harder to secure.
+
+
+### Admission controller rules
+
+A key element of any admission controller used for Kubernetes security is the rulebase it uses. The rules need to be able to accurately meet their goals avoiding false positive and false negative results. 
+
+* **Regularly test and review rules**. Admission controller rules need to be tested to ensure their accuracy. They also need to be regularly reviewed as the Kubernetes API will change with each new version, and rules need to be assessed with each Kubernetes release to understand any changes that may be required to keep them up to date.
@@ -15,7 +15,7 @@ each Node in your cluster, so that the
 {{< glossary_tooltip text="kubelet" term_id="kubelet" >}} can launch
 {{< glossary_tooltip text="Pods" term_id="pod" >}} and their containers.

-{{< glossary_definition term_id="container-runtime-interface" length="all" >}}
+{{< glossary_definition prepend="The Container Runtime Interface (CRI) is" term_id="container-runtime-interface" length="all" >}}

 <!-- body -->

@@ -33,7 +33,7 @@ There are two main ways to have Nodes added to the {{< glossary_tooltip text="AP
 1. The kubelet on a node self-registers to the control plane
 2. You (or another human user) manually add a Node object

-After you create a Node object, or the kubelet on a node self-registers, the
+After you create a Node {{< glossary_tooltip text="object" term_id="object" >}}, or the kubelet on a node self-registers, the
 control plane checks whether the new Node object is valid. For example, if you
 try to create a Node from the following JSON manifest:

@@ -79,7 +79,7 @@ addressing, and it can be used in combination with other CNI plugins.

 ### CNI-Genie from Huawei

-[CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) is a CNI plugin that enables Kubernetes to [simultaneously have access to different implementations](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-cni-plugins/README.md#what-cni-genie-feature-1-multiple-cni-plugins-enables) of the [Kubernetes network model](/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model) in runtime. This includes any implementation that runs as a [CNI plugin](https://github.com/containernetworking/cni#3rd-party-plugins), such as [Flannel](https://github.com/coreos/flannel#flannel), [Calico](https://docs.projectcalico.org/), [Romana](https://romana.io), [Weave-net](https://www.weave.works/products/weave-net/).
+[CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) is a CNI plugin that enables Kubernetes to [simultaneously have access to different implementations](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-cni-plugins/README.md#what-cni-genie-feature-1-multiple-cni-plugins-enables) of the [Kubernetes network model](/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model) in runtime. This includes any implementation that runs as a [CNI plugin](https://github.com/containernetworking/cni#3rd-party-plugins), such as [Flannel](https://github.com/coreos/flannel#flannel), [Calico](https://docs.projectcalico.org/), [Weave-net](https://www.weave.works/products/weave-net/).

 CNI-Genie also supports [assigning multiple IP addresses to a pod](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-ips/README.md#feature-2-extension-cni-genie-multiple-ip-addresses-per-pod), each from a different CNI plugin.

@@ -104,10 +104,6 @@ network complexity required to deploy Kubernetes at scale within AWS.
 [Coil](https://github.com/cybozu-go/coil) is a CNI plugin designed for ease of integration, providing flexible egress networking.
 Coil operates with a low overhead compared to bare metal, and allows you to define arbitrary egress NAT gateways for external networks.

-### Contiv
-
-[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan,  classic l2, or Cisco-SDN/ACI) for various use cases.
-
 ### Contrail / Tungsten Fabric

 [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is a truly open, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with various orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide different isolation modes for virtual machines, containers/pods and bare metal workloads.
@@ -130,6 +126,10 @@ With this toolset DANM is able to provide multiple separated network interfaces,
 network that satisfies the Kubernetes requirements. Many
 people have reported success with Flannel and Kubernetes.

+### Hybridnet
+
+[Hybridnet](https://github.com/alibaba/hybridnet) is an open source CNI plugin designed for hybrid clouds which provides both overlay and underlay networking for containers in one or more clusters. Overlay and underlay containers can run on the same node and have cluster-wide bidirectional network connectivity.
+
 ### Jaguar

 [Jaguar](https://gitlab.com/sdnlab/jaguar) is an open source solution for Kubernetes's network based on OpenDaylight. Jaguar provides overlay network using vxlan and Jaguar CNIPlugin provides one IP address per pod.
@@ -1,7 +1,6 @@
 ---
-reviewers:
 title: Device Plugins
-description: Use the Kubernetes device plugin framework to implement plugins for GPUs, NICs, FPGAs, InfiniBand, and similar resources that require vendor-specific setup.
+description: Device plugins let you configure your cluster with support for devices or resources that require vendor-specific setup, such as GPUs, NICs, FPGAs, or non-volatile main memory.
 content_type: concept
 weight: 20
 ---
@@ -48,12 +47,14 @@ For example, after a device plugin registers `hardware-vendor.example/foo` with
 and reports two healthy devices on a node, the node status is updated
 to advertise that the node has 2 "Foo" devices installed and available.

-Then, users can request devices in a
-[Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core)
-specification as they request other types of resources, with the following limitations:
-
+Then, users can request devices as part of a Pod specification
+(see [`container`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container)).
+Requesting extended resources is similar to how you manage requests and limits for
+other resources, with the following differences:
 * Extended resources are only supported as integer resources and cannot be overcommitted.
-* Devices cannot be shared among Containers.
+* Devices cannot be shared between containers.
+
+### Example {#example-pod}

 Suppose a Kubernetes cluster is running a device plugin that advertises resource `hardware-vendor.example/foo`
 on certain nodes. Here is an example of a pod requesting this resource to run a demo workload:
@@ -174,7 +175,7 @@ a Kubernetes release with a newer device plugin API version, upgrade your device
 to support both versions before upgrading these nodes. Taking that approach will
 ensure the continuous functioning of the device allocations during the upgrade.

-## Monitoring Device Plugin Resources
+## Monitoring device plugin resources

 {{< feature-state for_k8s_version="v1.15" state="beta" >}}

@@ -310,7 +311,7 @@ DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a
 Support for the `PodResourcesLister service` requires `KubeletPodResources` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to be enabled.
 It is enabled by default starting with Kubernetes 1.15 and is v1 since Kubernetes 1.20.

-## Device Plugin integration with the Topology Manager
+## Device plugin integration with the Topology Manager

 {{< feature-state for_k8s_version="v1.18" state="beta" >}}

@@ -319,7 +320,7 @@ The Topology Manager is a Kubelet component that allows resources to be co-ordin

 ```gRPC
 message TopologyInfo {
-	repeated NUMANode nodes = 1;
+    repeated NUMANode nodes = 1;
 }

 message NUMANode {
@@ -338,6 +339,8 @@ pluginapi.Device{ID: "25102017", Health: pluginapi.Healthy, Topology:&pluginapi.

 ## Device plugin examples {#examples}

+{{% thirdparty-content %}}
+
 Here are some examples of device plugin implementations:

 * The [AMD GPU device plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin)
@@ -357,5 +360,5 @@ Here are some examples of device plugin implementations:

 * Learn about [scheduling GPU resources](/docs/tasks/manage-gpus/scheduling-gpus/) using device plugins
 * Learn about [advertising extended resources](/docs/tasks/administer-cluster/extended-resource-node/) on a node
-* Read about using [hardware acceleration for TLS ingress](https://kubernetes.io/blog/2019/04/24/hardware-accelerated-ssl/tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass/) with Kubernetes
 * Learn about the [Topology Manager](/docs/tasks/administer-cluster/topology-manager/)
+* Read about using [hardware acceleration for TLS ingress](/blog/2019/04/24/hardware-accelerated-ssl/tls-termination-in-ingress-controllers-using-kubernetes-device-plugins-and-runtimeclass/) with Kubernetes
@@ -42,9 +42,10 @@ on every resource object.
 | `app.kubernetes.io/managed-by`      | The tool being used to manage the operation of an application | `helm` | string |
 | `app.kubernetes.io/created-by`      | The controller/user who created this resource | `controller-manager` | string |

-To illustrate these labels in action, consider the following StatefulSet object:
+To illustrate these labels in action, consider the following {{< glossary_tooltip text="StatefulSet" term_id="statefulset" >}} object:

 ```yaml
+# This is an excerpt
 apiVersion: apps/v1
 kind: StatefulSet
 metadata:
@@ -106,7 +106,7 @@ description: "This priority class should be used for XYZ service pods only."

 {{< feature-state for_k8s_version="v1.19" state="beta" >}}

-Pods with `PreemptionPolicy: Never` will be placed in the scheduling queue
+Pods with `preemptionPolicy: Never` will be placed in the scheduling queue
 ahead of lower-priority pods,
 but they cannot preempt other pods.
 A non-preempting pod waiting to be scheduled will stay in the scheduling queue,
@@ -122,16 +122,16 @@ allowing other pods with lower priority to be scheduled before them.
 Non-preempting pods may still be preempted by other,
 high-priority pods.

-`PreemptionPolicy` defaults to `PreemptLowerPriority`,
+`preemptionPolicy` defaults to `PreemptLowerPriority`,
 which will allow pods of that PriorityClass to preempt lower-priority pods
 (as is existing default behavior).
-If `PreemptionPolicy` is set to `Never`,
+If `preemptionPolicy` is set to `Never`,
 pods in that PriorityClass will be non-preempting.

 An example use case is for data science workloads.
 A user may submit a job that they want to be prioritized above other workloads,
 but do not wish to discard existing work by preempting running pods.
-The high priority job with `PreemptionPolicy: Never` will be scheduled
+The high priority job with `preemptionPolicy: Never` will be scheduled
 ahead of other queued pods,
 as soon as sufficient cluster resources "naturally" become free.

@@ -434,7 +434,7 @@ provisioner: example.com/external-nfs
 parameters:
  server: nfs-server.example.com
  path: /share
-  readOnly: false
+  readOnly: "false"
 ```

 * `server`: Server is the hostname or IP address of the NFS server.
@@ -797,7 +797,7 @@ parameters:
  storagePool: sp1
  storageMode: ThinProvisioned
  secretRef: sio-secret
-  readOnly: false
+  readOnly: "false"
  fsType: xfs
 ```

@@ -615,7 +615,7 @@ spec:
 ```

 The new Job itself will have a different uid from `a8f3d00d-c6d2-11e5-9f87-42010af00002`.  Setting
-`manualSelector: true` tells the system to that you know what you are doing and to allow this
+`manualSelector: true` tells the system that you know what you are doing and to allow this
 mismatch.

 ### Job tracking with finalizers
@@ -229,7 +229,7 @@ when both the following statements apply:
 * All conditions specified in `readinessGates` are `True`.

 When a Pod's containers are Ready but at least one custom condition is missing or
-`False`, the kubelet sets the Pod's [condition](#pod-condition) to `ContainersReady`.
+`False`, the kubelet sets the Pod's [condition](#pod-conditions) to `ContainersReady`.

 ## Container probes

@@ -249,6 +249,7 @@ Home | [All heading and subheading URLs](/docs/home/)
 Setup | [All heading and subheading URLs](/docs/setup/)
 Tutorials | [Kubernetes Basics](/docs/tutorials/kubernetes-basics/), [Hello Minikube](/docs/tutorials/hello-minikube/)
 Site strings | [All site strings](#Site-strings-in-i18n) in a new localized TOML file
+Releases | [All heading and subheading URLs](/releases)

 Translated documents must reside in their own `content/**/` subdirectory, but otherwise follow the same URL path as the English source. For example, to prepare the [Kubernetes Basics](/docs/tutorials/kubernetes-basics/) tutorial for translation into German, create a subfolder under the `content/de/` folder and copy the English source:

@@ -147,7 +147,7 @@ separately for reviewer status in SIG Docs.
 To apply:

 1. Open a pull request that adds your GitHub user name to a section of the
-   [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS) file
+   [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS_ALIASES) file
   in the `kubernetes/website` repository.

   {{< note >}}
@@ -219,7 +219,7 @@ separately for approver status in SIG Docs.
 To apply:

 1. Open a pull request adding yourself to a section of the
-   [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS)
+   [OWNERS_ALIASES](https://github.com/kubernetes/website/blob/main/OWNERS_ALIASES)
   file in the `kubernetes/website` repository.

    {{< note >}}
@@ -53,7 +53,7 @@ different Kubernetes components.
 |---------|---------|-------|-------|-------|
 | `APIListChunking` | `false` | Alpha | 1.8 | 1.8 |
 | `APIListChunking` | `true` | Beta | 1.9 | |
-| `APIPriorityAndFairness` | `false` | Alpha | 1.17 | 1.19 |
+| `APIPriorityAndFairness` | `false` | Alpha | 1.18 | 1.19 |
 | `APIPriorityAndFairness` | `true` | Beta | 1.20 | |
 | `APIResponseCompression` | `false` | Alpha | 1.7 | 1.15 |
 | `APIResponseCompression` | `true` | Beta | 1.16 | |
@@ -167,8 +167,8 @@ different Kubernetes components.
 | `NodeSwap` | `false` | Alpha | 1.22 | |
 | `NonPreemptingPriority` | `false` | Alpha | 1.15 | 1.18 |
 | `NonPreemptingPriority` | `true` | Beta | 1.19 | |
-| `OpenAPIEnum` | `false` | Alpha | 1.23 | |
-| `OpenAPIv3` | `false` | Alpha | 1.23 | |
+| `OpenAPIEnums` | `false` | Alpha | 1.23 | |
+| `OpenAPIV3` | `false` | Alpha | 1.23 | |
 | `PodAndContainerStatsFromCRI` | `false` | Alpha | 1.23 | |
 | `PodAffinityNamespaceSelector` | `false` | Alpha | 1.21 | 1.21 |
 | `PodAffinityNamespaceSelector` | `true` | Beta | 1.22 | |
@@ -784,7 +784,8 @@ Each feature gate is designed for enabling/disabling a specific feature:
  [readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes).
 - `ExpandCSIVolumes`: Enable the expanding of CSI volumes.
 - `ExpandedDNSConfig`: Enable kubelet and kube-apiserver to allow more DNS
-  search paths and longer list of DNS search paths. See
+  search paths and longer list of DNS search paths. This feature requires container
+  runtime support(Containerd: v1.5.6 or higher, CRI-O: v1.22 or higher). See
  [Expanded DNS Configuration](/docs/concepts/services-networking/dns-pod-service/#expanded-dns-configuration).
 - `ExpandInUsePersistentVolumes`: Enable expanding in-use PVCs. See
  [Resizing an in-use PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim).
@@ -913,9 +914,9 @@ Each feature gate is designed for enabling/disabling a specific feature:
  Must be used with `KubeletConfiguration.failSwapOn` set to false.
  For more details, please see [swap memory](/docs/concepts/architecture/nodes/#swap-memory)
 - `NonPreemptingPriority`: Enable `preemptionPolicy` field for PriorityClass and Pod.
- `OpenAPIEnum`: Enables populating "enum" fields of OpenAPI schemas in the 
+- `OpenAPIEnums`: Enables populating "enum" fields of OpenAPI schemas in the 
  spec returned from the API server.
- `OpenAPIv3`: Enables the API server to publish OpenAPI v3.
+- `OpenAPIV3`: Enables the API server to publish OpenAPI v3.
 - `PVCProtection`: Enable the prevention of a PersistentVolumeClaim (PVC) from
  being deleted when it is still used by any Pod.
 - `PodDeletionCost`: Enable the [Pod Deletion Cost](/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost)
@@ -19,4 +19,4 @@ The Kubernetes Container Runtime Interface (CRI) defines the main
 [gRPC](https://grpc.io) protocol for the communication between the
 [cluster components](/docs/concepts/overview/components/#node-components)
 {{< glossary_tooltip text="kubelet" term_id="kubelet" >}} and
-{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}.
+{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}.
@@ -41,8 +41,6 @@ For control-plane nodes additional steps are performed:

 1. Adding new local etcd member.

-1. Adding this node to the ClusterStatus of the kubeadm cluster.
-
 ### Using join phases with kubeadm {#join-phases}

 Kubeadm allows you join a node to the cluster in phases using `kubeadm join phase`.
@@ -16,8 +16,7 @@ Performs a best effort revert of changes made by `kubeadm init` or `kubeadm join

 `kubeadm reset` is responsible for cleaning up a node local file system from files that were created using
 the `kubeadm init` or `kubeadm join` commands. For control-plane nodes `reset` also removes the local stacked
-etcd member of this node from the etcd cluster and also removes this node's information from the kubeadm
-`ClusterStatus` object. `ClusterStatus` is a kubeadm managed Kubernetes API object that holds a list of kube-apiserver endpoints.
+etcd member of this node from the etcd cluster.

 `kubeadm reset phase` can be used to execute the separate phases of the above workflow.
 To skip a list of phases you can use the `--skip-phases` flag, which works in a similar way to
@@ -16,19 +16,20 @@ or upgrades for such nodes. The long term plan is to empower the tool
 aspects.
 {{< /note >}}

-Kubeadm defaults to running a single member etcd cluster in a static pod managed
-by the kubelet on the control plane node. This is not a high availability setup
-as the etcd cluster contains only one member and cannot sustain any members
-becoming unavailable. This task walks through the process of creating a high
-availability etcd cluster of three members that can be used as an external etcd
-when using kubeadm to set up a kubernetes cluster.
+By default, kubeadm runs a local etcd instance on each control plane node.
+It is also possible to treat the etcd cluster as external and provision
+etcd instances on separate hosts. The differences between the two approaches are covered in the
+[Options for Highly Available topology][/docs/setup/production-environment/tools/kubeadm/ha-topology] page.
+This task walks through the process of creating a high availability external
+etcd cluster of three members that can be used by kubeadm during cluster creation.

 ## {{% heading "prerequisites" %}}

-* Three hosts that can talk to each other over ports 2379 and 2380. This
+* Three hosts that can talk to each other over TCP ports 2379 and 2380. This
  document assumes these default ports. However, they are configurable through
  the kubeadm config file.
-* Each host must [have docker, kubelet, and kubeadm installed](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/).
+* Each host must have systemd and a bash compatible shell installed.
+* Each host must [have a container runtime, kubelet, and kubeadm installed](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/).
 * Each host should have access to the Kubernetes container image registry (`k8s.gcr.io`) or list/pull the required etcd image using
 `kubeadm config images list/pull`. This guide will setup etcd instances as
 [static pods](/docs/tasks/configure-pod-container/static-pod/) managed by a kubelet.
@@ -48,6 +49,11 @@ the certificates described below; no other cryptographic tooling is required for
 this example.
 {{< /note >}}

+{{< note >}}
+The examples below use IPv4 addresses but you can also configure kubeadm, the kubelet and etcd
+to use IPv6 addresses. Dual-stack is supported by some Kubernetes options, but not by etcd. For more details
+on Kubernetes dual-stack support see [Dual-stack support with kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/).
+{{< /note >}}

 1. Configure the kubelet to be a service manager for etcd.

@@ -59,8 +65,9 @@ this example.
    cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
    [Service]
    ExecStart=
-    #  Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
-    ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
+    # Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
+    # Replace the value of "--container-runtime-endpoint" for a different container runtime if needed.
+    ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock
    Restart=always
    EOF

@@ -80,21 +87,34 @@ this example.
    member running on it using the following script.

    ```sh
-    # Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
+    # Update HOST0, HOST1 and HOST2 with the IPs of your hosts
    export HOST0=10.0.0.6
    export HOST1=10.0.0.7
    export HOST2=10.0.0.8

+    # Update NAME0, NAME1 and NAME2 with the hostnames of your hosts
+    export NAME0="infra0"
+    export NAME1="infra1"
+    export NAME2="infra2"
+
    # Create temp directories to store files that will end up on other hosts.
    mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

-    ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
-    NAMES=("infra0" "infra1" "infra2")
+    HOSTS=(${HOST0} ${HOST1} ${HOST2})
+    NAMES=(${NAME0} ${NAME1} ${NAME2})

-    for i in "${!ETCDHOSTS[@]}"; do
-    HOST=${ETCDHOSTS[$i]}
+    for i in "${!HOSTS[@]}"; do
+    HOST=${HOSTS[$i]}
    NAME=${NAMES[$i]}
    cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
+    ---
+    apiVersion: "kubeadm.k8s.io/v1beta3"
+    kind: InitConfiguration
+    nodeRegistration:
+        name: ${NAME}
+    localAPIEndpoint:
+        advertiseAddress: ${HOST}
+    ---
    apiVersion: "kubeadm.k8s.io/v1beta3"
    kind: ClusterConfiguration
    etcd:
@@ -104,7 +124,7 @@ this example.
            peerCertSANs:
            - "${HOST}"
            extraArgs:
-                initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380,${NAMES[2]}=https://${ETCDHOSTS[2]}:2380
+                initial-cluster: ${NAMES[0]}=https://${HOSTS[0]}:2380,${NAMES[1]}=https://${HOSTS[1]}:2380,${NAMES[2]}=https://${HOSTS[2]}:2380
                initial-cluster-state: new
                name: ${NAME}
                listen-peer-urls: https://${HOST}:2380
@@ -37,8 +37,11 @@ The upgrade workflow at high level is the following:

 ### Additional information

- [Draining nodes](/docs/tasks/administer-cluster/safely-drain-node/) before kubelet MINOR version
-  upgrades is required. In the case of control plane nodes, they could be running CoreDNS Pods or other critical workloads.
+- The instructions below outline when to drain each node during the upgrade process.
+If you are performing a **minor** version upgrade for any kubelet, you **must**
+first drain the node (or nodes) that you are upgrading. In the case of control plane nodes,
+they could be running CoreDNS Pods or other critical workloads. For more information see
+[Draining nodes](/docs/tasks/administer-cluster/safely-drain-node/).
 - All containers are restarted after upgrade, because the container spec hash value is changed.

 <!-- steps -->
@@ -52,7 +52,7 @@ For example: on COS images, Docker exposes its Unix domain socket at

 Here's a sample shell script to find Pods that have a mount directly mapping the
 Docker socket. This script outputs the namespace and name of the pod. You can
-remove the grep `/var/run/docker.sock` to review other mounts.
+remove the `grep '/var/run/docker.sock'` to review other mounts.

 ```bash
 kubectl get pods --all-namespaces \
@@ -26,19 +26,18 @@ Reload your shell and verify that bash-completion is correctly installed by typi

 ### Enable kubectl autocompletion

+#### Bash
+
 You now need to ensure that the kubectl completion script gets sourced in all your shell sessions. There are two ways in which you can do this:

- Source the completion script in your `~/.bashrc` file:
-
-   ```bash
-   echo 'source <(kubectl completion bash)' >>~/.bashrc
-   ```
-
- Add the completion script to the `/etc/bash_completion.d` directory:
-
-   ```bash
-   kubectl completion bash >/etc/bash_completion.d/kubectl
-   ```
+{{< tabs name="kubectl_bash_autocompletion" >}}
+{{{< tab name="User" codelang="bash" >}}
+echo 'source <(kubectl completion bash)' >>~/.bashrc
+{{< /tab >}}
+{{< tab name="System" codelang="bash" >}}
+kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
+{{< /tab >}}}
+{{< /tabs >}}

 If you have an alias for kubectl, you can extend shell completion to work with that alias:

@@ -47,12 +47,6 @@ Before walking through each tutorial, you may want to bookmark the

 * [Running ZooKeeper, A CP Distributed System](/docs/tutorials/stateful-application/zookeeper/)

-## Clusters
-
-* [AppArmor](/docs/tutorials/clusters/apparmor/)
-
-* [seccomp](/docs/tutorials/clusters/seccomp/)
-
 ## Services

 * [Using Source IP](/docs/tutorials/services/source-ip/)
@@ -61,7 +55,8 @@ Before walking through each tutorial, you may want to bookmark the

 * [Apply Pod Security Standards at Cluster level](/docs/tutorials/security/cluster-level-pss/)
 * [Apply Pod Security Standards at Namespace level](/docs/tutorials/security/ns-level-pss/)
-
+* [AppArmor](/docs/tutorials/security/apparmor/)
+* [seccomp](/docs/tutorials/security/seccomp/)
 ## {{% heading "whatsnext" %}}

 If you would like to write a tutorial, see
@@ -1,5 +0,0 @@
---
-title: "Clusters"
-weight: 60
---
-