* Update kubeadm-upgrade.md
`apt-mark hold` is still required when `apt-get` is used with `--allow-change-held-packages`.
`--allow-change-held-packages` unholds the package but it doesn't pin the new version.
* Add missing changes
* Update kubeadm-upgrade.md
Remove apt >1.1 examples.
* Transferred applicable modifications for 1.23 over to 1.22.
* kublet -> kubelet
* Update content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md
Co-authored-by: Mark Rossetti <marosset@microsoft.com>
Co-authored-by: Mark Rossetti <marosset@microsoft.com>
* Update docs to clarify the dns configmap format.
* Update content/en/docs/tasks/administer-cluster/nodelocaldns.md
Co-authored-by: Qiming Teng <tengqm@outlook.com>
Co-authored-by: Qiming Teng <tengqm@outlook.com>
* Advertise that we need to remove `spec.replicas` when a Horizontal Pod
Autoscaler is active to prevent unnecessary changes in Pod counts during
Deployment object changes
* Make note that a Deployment that has this value set behave awkwardly if a
Deployment is scaled manually outside of the Deployment object
Signed-off-by: John T Skarbek <jtslear@gmail.com>
* Update images.md
typo
* sync en docs
* [zh] translate images.md
[zh] translate interpretation of config.json images.md
* [zh] modify the wording in images.md
* [zh] remove origin english in images.md
* [zh] del extra # in images.md
* [zh] del origin english in images.md
* added line separation in kubectl page
added line separation in kubectl page - 2
added line separation in kubectl page - 3 added space at ends
final changes
added spaces at end
* removed Line-549 from end of the file
* added Line-548 at the end of file
In my humble view, `kubectl certificate approve my-svc.my-namespace` command may understand easily to follow up this procedure.
Update managing-tls-in-a-cluster.md
Change some text as Divya Mohan's guidance.
Update managing-tls-in-a-cluster.md
Change text as sftim's guidance.! Thank you so much!
* Fixes grammatical errors
* Removes period as recommended in review
* Update content/en/docs/setup/production-environment/tools/kubespray.md
Co-authored-by: Rey Lejano <rlejano@gmail.com>
Co-authored-by: Rey Lejano <rlejano@gmail.com>
Introduce concepts in an order that should suit people new to
Kubernetes, so that you don't encounter details like volume cloning
until you've seen more of the basic volume types (persistent,
ephemeral, projected).
This abandoned account was identified by a security researcher, and reported through the Kubernetes bug bounty program.
Don't we have a link checker that should identify cases like this? I'm guessing the github username was changed, so the link was redirecting (still valid), until the researcher recreated it. We should probably have the link checker flag redirects as well.
Add missing note to review Kubernetes Network Policy Ingress rules during debugging Kubernetes Service issues
Update document link
Co-authored-by: Qiming Teng <tengqm@outlook.com>
The 'kubr' gem has last been updated 7 years ago. It doesn't provide the required modern functionalities. The documentation consists of one code example and that's it. It's inclusion in this list only wastes time of Ruby on Rails developers because no sane person would chose it.
* Fix broken anchor link of #type-nodeport in service.md
* Update anchor link from #nodeport to #type-nodeport
Update anchor link from #nodeport to #type-nodeport in overview.md
* Update anchor link from #nodeport to #type-nodeport
Update anchor link from #nodeport to #type-nodeport in ingress.md
* Update anchor link from #nodeport to #type-nodeport
Update anchor link from #nodeport to #type-nodeport in troubleshooting-kubeadm.md
* Update anchor link from #nodeport to #type-nodeport
Update anchor link from #nodeport to #type-nodeport in connecting-frontend-backend.md
* Update anchor link from #nodeport to #type-nodeport
Update anchor link from #nodeport to #type-nodeport in source-ip.md
This change updates the information about the node controller in the
cloud controller managers. The node controller is not responsible for
creating new node objects, it updates them with information from the
provider infrastructure.
* kubelet-in-userns.md: fix typoes
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
* kubelet-in-userns.md: fix unexpected config paths
Referring to `/etc/containerd/config.toml` makes sense only when the user has
read/write permissions to `/etc/containerd/config.toml` in the current mount
namespace, which is not always assumed in the context of this documentation.
The same applies to `/etc/crio/crio.conf`, too.
Partially revert PR 30020.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
* kubelet-in-userns.md: add back the link to example port forwarder implementation
Add back the link to `k3s/pkg/rootlessports/controller.go` removed in PR 30020.
As stated in `{{ <note> }}`, the corresponding section is written for developers
of Kubernetes distros, not for end users.
So we should retain the implementation details here.
Partially revert PR 30020.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The doc version for brazilian portuguese scale-intro page seems to be generated by an AI powered engine, which is not quite right and/or specific as the original one.
This PR changes a few words and a bit of the structure of the phrase in the pt-br doc, using de original (en) version as source of information.
Boosting the contributor site up in Google rankings is a priority for the Kubernetes Upstream Marketing team: https://github.com/kubernetes/contributor-site/issues/202
Adding links here where appropriate:
- Contributor Cheatsheet
- kubernetes.dev itself
- contributor resources
We have seen a number of questions around the usage of "config print".
Clarify the usage of this command. Add a note that the output should
not be taken literally.
Link to the "customizing components" page instead of the "kubelet
integration" page. The former already links to the later.
Remove note about kube-dns which hasn't been supported for a while.
It is merely a pod placement constraint but the phrasing leads readers to the incorrect understanding that some actual CPU time is guaranteed/reserved. Re-phrase to make it clear this is nothing more than a request.
The "kubelet-finalize" step does not work on worker nodes,
because commonly they do not have the cluster CA key and all
"init" phases have a pre-step to check for the existence of
the CA key, designating the cluster as one that uses external CA
or not.
Changing this behavior is complicated in kubeadm, thus manually
instruct the user how to do edit the file and to restart the kubelet.
This is already what we do in:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration
Removing this content seems to be appropriate:
Content does not contain a link to a CNI.
Content is not required for k8s to function.
Content seems to be replicated in longer form on the Google Cloud docs site.
Open vSwitch is in no way required for Kubernetes to function. Kubernetes is not mentioned once on this project's website, nor are there any instructions for how you might use this in a k8s environment.
This entry is not necessary for Kubernetes to function. Entry provides a link to the main corporate website, not to any relevant information. No CNI. Acronym refers to product marketing, not an actual technology.
Update namespace definition to not use _Virtual Clusters_ as a means of explaining namespaces.
Motif: KubeCon NA 21 Talk on ***"Beyond Namespaces: Virtual Clusters are the Future of Multi-Tenancy - Lukas Gentele, Loft Labs"***
- more compact headers when announcements are showing
- when no announcement showing, top level docs sections
(eg Concepts, Tasks, Reference) have a heading in the top nav
- when announcement is showing, rely on the breadcrumb trail plus
page introduction text for each top level section
- switch from plain black docs header to match other sections
As per the content guide, Docker is (essential) third party software.
Alternatives exist but Docker is certainly the most common tool used for
this kind of task.
Anyway, mark that it's third party.
- Include more details about custom image repositories, such
as making sure that paths in custom repositories comply with
kubeadm execution.
- Move the section "without internet connection" above
the section about custom images.
- Link the "create-cluster-kubeadm.md" guide these new section
as a pre-req step for nodes.
Add example for querying SA permissions
Add missing example for querying the API authorization layer for checking the permissions of a Service Account
Add missing SA identifying prefix
Improve suggested text to align with current content
Co-authored-by: Sam Roth <2413031+sejr@users.noreply.github.com>
Improve suggested text to align with current content
Co-authored-by: Sam Roth <2413031+sejr@users.noreply.github.com>
* Adding diagrams to certificates page
* Cropped diagrams
* Changed diagrams to tree output
* Formatting fix
* Fixed text block markup and spacing
* Changed tree view of files to full-path view
* Changed order of two cert files
* Broke up links into separate sentences, per review comment
* More changes per review comments
As the cherry pick deadline for october '21 lands right during kubecon
week, SIG Release decided to move the patch releases one week later.
This PR modifies the dates in the schedule data file.
Signed-off-by: Adolfo García Veytia (Puerco) <adolfo.garcia@uservers.net>
When the (new) third party content disclaimer is the current target,
highlight it with a flash of yellow that fades to a yellow background,
rather than the usual gray.
Partial tidying to bring this page more in line with the Kubernetes
documentation style guide.
Co-authored-by: Shannon Kularathna <ax3shannonkularathna@gmail.com>
The content describing a declarative API in the custom controller
section of the custom resources doc was confusing:
> A declarative API allows you to declare or specify the desired state
of your resource **and tries to keep the current state of Kubernetes
objects in sync with the desired state**. The controller interprets the
structured data as a record of the user's desired state, and continually
maintains this state.
(emphasis added)
It is not the declarative API that tries to keep the current state of
the objects in sync with the desired state. It's the controller that
does that.
I've reworded this paragraph to hopefully clarify this.
Closes Issue #29348
Signed-off-by: Jay Pipes <jaypipes@gmail.com>
Docsy uses Bootstrap to restrict the width of text paragraphs when the
viewport is very wide. Also apply that to <figure> elements within blog
articles.
* Clarified scenarios that could lead to privilege escalation
Made it clearer that it's not just creating pods which enables the privilege escalation. It's all workloads, all reconfiguration of workloads, and conceptually the creation and reconfiguration of custom resources which create workloads.
* Allowing link to priv escalation heading if required
* Update content/en/docs/reference/access-authn-authz/authorization.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Adding further clarifications
* Retitled escalation section
* Apply suggestions from vjftw
Co-authored-by: VJ Patel <VJftw@users.noreply.github.com>
* Clarified CRDs and reduced duplication
* Updating caution based on Geoffrey's comments
* Updating controller comment and linking out to reference docs
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: VJ Patel <VJftw@users.noreply.github.com>
- Refactored ports and protocols info under docs/reference
- Updated the ports for kube-scheduler and kube-controller based on
current state
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Apstra AOS (now owned by Juniper) has no direct integration with Kubernetes.
No CNI, no operator, no CRDs, nothing.
Kubernetes is not natively supported in the product, there is no mention of any k8s construct in the product.
This entry should be removed because it does not follow https://github.com/kubernetes/website/issues/20232
The Service Catalog architecture changed from using api aggregation to CRDs, but the docs still refer to the older architecture using api aggregation.
Couple of changes here:
1. Change the sentence on how Service Catalog is implemented
2. Replace the example for usage of api aggregation from service-catalog to metrics-server. There are multiple implementations that can be linked to(keda, prometheus, datadog,...), but keeping the documentation neutral by pointing to kubernetes-sigs/metrics-server
References:
- Service Catalog [v0.3.0 release notes](https://github.com/kubernetes-sigs/service-catalog/releases/tag/v0.3.0):
> In release 0.3.0, we've focused on replacing the Aggregated API Server with the CustomResourceDefinitions (CRDs) and the Admission Webhook solution.
- Project [README](https://github.com/kubernetes-sigs/service-catalog/pull/2691/files)
> Service Catalog recently switched to a new CRDs-based architecture. The old API Server-based implementation is available on the v0.2 branch. We support this implementation by providing bug fixes until July 2020.
use ",_PILLAR__DNS__SERVER_" as old pattern instead of "_PILLAR__DNS__SERVER_" when use ipvs mode.
In English localization, the pattern is ",_PILLAR__DNS__SERVER_" but when in zh localization, the comma lost.
The interpretation between Docker and Kubernetes varies when comparing
its implementations. This allows different use cases and should be
documented accordingly.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Co-authored-by: Geoffrey Cline <gcline@amazon.com>
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
* Update English version of controlling-access.md as --insecure-port flag deprecated
* Update controlling-access.md as --insecure-port flag deprecated
* Update content/en/docs/concepts/security/controlling-access.md
Co-authored-by: Qiming Teng <tengqm@outlook.com>
Co-authored-by: Qiming Teng <tengqm@outlook.com>
* 1.22 feature blog for API server tracing
* Add initial draft of descriptive tracing portions
* demo
demo
* Apply suggestions from code review
Co-authored-by: Chris Negus <cnegus@redhat.com>
* address comments
* address comments and grammar
* Add more explanation to the Demo section; add conclusion
* Update content/en/blog/_posts/2021-08-06-api-server-tracing.md
Co-authored-by: Punya Biswal <punya@google.com>
* address comments
* address feedback
* update alt text
* Update content/en/blog/_posts/2021-08-06-api-server-tracing.md
Co-authored-by: Rey Lejano <rlejano@gmail.com>
* Rename 2021-08-06-api-server-tracing.md to 2021-09-03-api-server-tracing.md
* update alt text on first image
* better alt text.
Co-authored-by: David Ashpole <dashpole@google.com>
Co-authored-by: Chris Negus <cnegus@redhat.com>
Co-authored-by: Punya Biswal <punya@google.com>
Co-authored-by: Rey Lejano <rlejano@gmail.com>
I think adding the "for" word in the below sentence makes more sense.
"This document outlines the various components you need to have "for"
a complete and working Kubernetes cluster."
* Update CronJob page translation to Portuguese.
Update CronJob page to reflect the latest English version in the Brazilian
Portuguese translation.
Signed-off-by: Mauren Berti <mribeirobert@vmware.com>
* Incorporate feedback from pull request.
Signed-off-by: Mauren Berti <mribeirobert@vmware.com>
This is a note which helps users to understand the interaction between
privileged containers and seccomp profiles.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
In Kubernetes v1.19, the new Events API events.k8s.io was promoted to
v1. As such it now supersedes the original core Events API.
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
Update api-ref-generator submodule to 55bce68 to include changes
updating the recommended Events API from core to events.k8s.io in the
kubernetes-api doc.
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
Modernise the page by:
- rewording to follow the style guide
- adding some glossary tooltips
- linking to new-style API reference
- linking to Safely Drain a Node
plus general tweaks.
* Added announcements of KubeCon NA & China
* svg images added & required changes made
* Requested changes made
* svg images fixed
* small correction required
* added the to the message
* small correction
* Netlify build error fixed
The kubelet reference is not auto-generated. This PR is about fixing the
outdated information by manually comparing the reference against the
output from `kubelet --help`.
When attempting to keep the localized sites well synced to the English
upstream, some files were found MOVED. It is difficult to detect such
changes. This PR removes a file that were localized twice.
Given 'Aggregated APIs are subordinate API servers that sit behind the primary API server, which acts as a proxy', the comparison table indicates a requirement for the subordinate API servers to use Go, when it is not a requirement as long as the subordinate API server follows the expected contract
If we using string then k8s api is throwing validation error
``` (HorizontalPodAutoscaler.spec.metrics[0].object.metric.selector): invalid type for io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector: got "string", expected "map"; if you choose to ignore these errors, turn validation off with --validate=false ```
Current documentation simply suggest
"Remove v1beta1 from the CustomResourceDefinition
status.storedVersions field."
but this cannot be done just with kubectl and
it's not so intuitive.
Adding an example to make it more clear.
Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
- general cleanup
- update sample output
- use more tooltips
- avoid specifying specific cloud providers
The website repo doesn't maintain a definitive list of cloud providers that
pass Kubernetes conformance tests. It's certainly more than AWS and GCP as
the previous revision stated.
"Understand the basics" to "Understand Kubernetes". There is no place in the entire docs really to go "Understand the *non* basics". There is one section "Concepts" for better or worse. Don't give the impression there is something else somewhere else. And anyway, this section should aspire to be that cardinal. Also change name of weird button to "Learn" -> "View" to make it clear this is just a link to a section of the documentation.
fixing redirects
Fixing few redirects
changing few redirects and links
fixing redirect
Update content/en/blog/_posts/2016-08-00-Kubernetes-Namespaces-Use-Cases-Insights.md
Co-authored-by: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com>
Update content/en/blog/_posts/2016-12-00-Statefulset-Run-Scale-Stateful-Applications-In-Kubernetes.md
Co-authored-by: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com>
Update content/en/blog/_posts/2016-12-00-Statefulset-Run-Scale-Stateful-Applications-In-Kubernetes.md
Co-authored-by: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com>
Makes option 3 for "reclaiming" a released PV a bit clearer. Used to be:
> Manually delete the associated storage asset, or if you want to reuse the same storage asset, create a new PersistentVolume with the same storage asset definition.
But the 2nd part applies more to reclamation option 1 and is kind of contradictory with option 3 (which isn't really "reclaiming" anything AFAICT). So just move to it's own stand alone sentence.
k8s dashboard required https.
http does not currently have a corresponding endpoint.
So if you try to access it like this, you will get an error: "no endpoints available for service".
आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में कोई भी परिवर्तन करना चाहते हैं, और जब आप उन परिवर्तनों को हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं।
आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में परिवर्तन करने के बाद जब आप उनको हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं।
एक बार जब आपका pull request बन जाता है, तो एक कुबरनेट्स समीक्षक स्पष्ट, कार्रवाई योग्य प्रतिक्रिया प्रदान करने की जिम्मेदारी लेगा। pull request के मालिक के रूप में, **यह आपकी जिम्मेदारी है कि आप कुबरनेट्स समीक्षक द्वारा प्रदान की गई प्रतिक्रिया को संबोधित करने के लिए अपने pull request को संशोधित करें।**
@@ -23,9 +23,12 @@
## `README.md`'s स्थानीयकरण कुबरनेट्स प्रलेखन
आप पर हिंदी स्थानीयकरण के maintainers तक पहुँच सकते हैं:
आप हिंदी स्थानीयकरण के मैन्टेनरों तक पहुँच सकते हैं:
@@ -18,7 +18,7 @@ Aby móc skorzystać z tego repozytorium, musisz lokalnie zainstalować:
- [npm](https://www.npmjs.com/)
- [Go](https://golang.org/)
- [Hugo (Extended version)](https://gohugo.io/)
- Środowisko obsługi kontenerów, np. [Docker-a](https://www.docker.com/).
- Środowisko obsługi kontenerów, np. [Dockera](https://www.docker.com/).
Przed rozpoczęciem zainstaluj niezbędne zależności. Sklonuj repozytorium i przejdź do odpowiedniego katalogu:
@@ -43,7 +43,9 @@ make container-image
make container-serve
```
Aby obejrzeć zawartość serwisu otwórz w przeglądarce adres http://localhost:1313. Po każdej zmianie plików źródłowych, Hugo automatycznie aktualizuje stronę i odświeża jej widok w przeglądarce.
Jeśli widzisz błędy, prawdopodobnie kontener z Hugo nie dysponuje wystarczającymi zasobami. Aby rozwiązać ten problem, zwiększ ilość dostępnych zasobów CPU i pamięci dla Dockera na Twojej maszynie ([MacOSX](https://docs.docker.com/docker-for-mac/#resources) i [Windows](https://docs.docker.com/docker-for-windows/#resources)).
Aby obejrzeć zawartość serwisu, otwórz w przeglądarce adres http://localhost:1313. Po każdej zmianie plików źródłowych, Hugo automatycznie aktualizuje stronę i odświeża jej widok w przeglądarce.
## Jak uruchomić lokalną kopię strony przy pomocy Hugo?
Данный репозиторий содержит все необходимые файлы для сборки [сайта Kubernetes и документации](https://kubernetes.io/). Мы благодарим вас за желание внести свой вклад!
### [Kubernetes (K8s)]({{<relref"/docs/concepts/overview/what-is-kubernetes">}}) ist ein Open-Source-System zur Automatisierung der Bereitstellung, Skalierung und Verwaltung von containerisierten Anwendungen.
Es gruppiert Container, aus denen sich eine Anwendung zusammensetzt, in logische Einheiten, um die Verwaltung und Erkennung zu erleichtern. Kubernetes baut auf [15 Jahre Erfahrung in Bewältigung von Produktions-Workloads bei Google](http://queue.acm.org/detail.cfm?id=2898444), kombiniert mit Best-of-Breed-Ideen und Praktiken aus der Community.
Es gruppiert Container, aus denen sich eine Anwendung zusammensetzt, in logische Einheiten, um die Verwaltung und Erkennung zu erleichtern. Kubernetes baut auf [15 Jahre Erfahrung in Bewältigung von Produktions-Workloads bei Google](http://queue.acm.org/detail.cfm?id=2898444), kombiniert mit Best-of-Breed-Ideen und Praktiken aus der Community.
{{% /blocks/feature %}}
{{% blocks/feature image="scalable" %}}
@@ -42,12 +42,12 @@ Kubernetes ist Open Source und bietet Dir die Freiheit, die Infrastruktur vor Or
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu20"buttonid="desktopKCButton">Besuche die KubeCon - 13-16 August 2020 in Amsterdam</a>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21"buttonid="desktopKCButton">Besuche die KubeCon North America vom 11. bis 15. Oktober 2021</a>
<br>
<br>
<br>
<br>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna20"buttonid="desktopKCButton">Besuche die KubeCon - 17-20 November 2020 in Boston</a>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22"buttonid="desktopKCButton">Besuche die KubeCon Europe vom 17. bis 20. Mai 2022</a>
<h3>Die Gewissheit, dass Kubernetes überall und für alle gut funktioniert.</h3>
<p>Verbinden Sie sich mit der Kubernetes-Community in unserem <ahref="http://slack.k8s.io/">Slack Kanal</a>, <ahref="https://discuss.kubernetes.io/">Diskussionsforum</a>, oder beteiligen Sie sich an der <ahref="https://groups.google.com/g/kubernetes-dev"> Kubernetes-dev-Google-Gruppe</a>. Eine wöchentlichesCommunity-Meeting findet per Videokonferenz statt, um den Stand der Dinge zu diskutieren, folgen Sie
<ahref="https://github.com/kubernetes/community/blob/master/events/community-meeting.md">diesen Anweisungen</a> für Informationen wie Sie teilnehmen können.</p>
<p>Sie können Kubernetes auch auf der ganzen Welt über unsere
<ahref="https://www.meetup.com/topics/kubernetes/">Kubernetes Meetup Community</a> und der
<p>Haben Sie ein besonderes Interesse daran, wie Kubernetes mit einer anderen Technologie arbeitet? Werfen Sie einen Blick auf unsere kontinuierlich wachsende
<ahref="https://git.k8s.io/community/sig-list.md">Listen von SIGs</a>, von AWS und Openstack bis hin zu Big Data und Skalierbarkeit, es gibt einen Platz für Sie, an dem Sie mitwirken können, und Anweisungen zur Gründung einer neuen SIG finden, wenn Ihr besonderes Interesse (noch) nicht abgedeckt ist.
<p>Als Mitglied der Kubernetes-Community sind Sie herzlich eingeladen, an allen SIG-Treffen teilzunehmen, die Sie interessieren. Eine Registrierung ist nicht erforderlich.</p>
<divclass="intro">
<brclass="mobile">
<p>Die Kubernetes-Community - Nutzer, Mitwirkende und die Kultur, die wir gemeinsam aufgebaut haben - ist einer der Hauptgründe für den kometenhaften Aufstieg dieses Open-Source-Projekts. Unsere Kultur und unsere Werte wachsen und entwickeln sich mit dem Wachstum und der Veränderung des Projekts selbst. Wir alle arbeiten gemeinsam an der ständigen Verbesserung des Projekts und der Art und Weise, wie wir daran arbeiten.
<br><br>Wir sind die Leute, die Probleme und Pull-Requests einreichen, an SIG-Treffen (Special Interest Groups), Kubernetes-Treffen und der KubeCon teilnehmen, sich für die Einführung und Innovation von Kubernetes einsetzen, <code>kubectl get pods</code> ausführen und auf tausend andere wichtige Arten beitragen. Lies weiter, um zu erfahren, wie Du dich engagieren und Teil dieser faszinierenden Gemeinschaft werden kannst.</p>
<brclass="mobile">
</div>
</div>
<divclass="community__navbar">
<divclass="content">
<h3>Verhaltensregeln</h3>
<p>Die Kubernetes-Community schätzt Respekt und Inklusivität und setzt einen <ahref="code-of-conduct/">Verhaltenskodex</a>
in allen Interaktionen durch. Wenn Sie einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder Sitzung,
in Slack oder in einem anderen Kommunikationsmechanismus feststellen, wenden Sie sich
bitte an das <ahref="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct">Kubernetes Code of Conduct Committee</a><ahref="mailto:conduct@kubernetes.io">conduct@kubernetes.io</a>. Ihre Anonymität wird geschützt.
Die Kubernetes-Gemeinschaft legt Wert auf Respekt und Inklusivität und setzt bei allen Interaktionen einen Verhaltenskodex durch. Wenn Du einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder einem Treffen, in Slack oder in einem anderen Kommunikationsmechanismus bemerkst, wende dich an das Kubernetes Code of Conduct Committee unter <ahref="mailto:conduct@kubernetes.io"style="color:#0662EE;font-weight:300">conduct@kubernetes.io</a>. Alle Berichte werden vertraulich behandelt. Du kannst <ahref="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct"style="color:#0662EE;font-weight:300">hier</a> mehr über den Ausschuss erfahren.
<divstyle="margin-bottom:4%;font-weight:300;text-align:center;padding-left:10%;padding-right:10%">Wir sind auf YouTube, und zwar oft. Abonniere uns für eine Vielzahl von Themen.</div>
<iframewidth="100%"height="250"src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ"title="Wöchentliche Treffen der Gemeinschaft"frameborder="0"allow="autoplay; encrypted-media"allowfullscreen></iframe>
Wöchentliche Treffen der Gemeinschaft ansehen ▶
</div>
</a>
</div>
<divclass="video">
<iframewidth="100%"height="250"src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F"title="Vortrag eines Mitglieds der Gemeinschaft"frameborder="0"allow="autoplay; encrypted-media"allowfullscreen></iframe>
Vortrag eines Mitglieds der Gemeinschaft ansehen ▶
</div>
</a>
<aname="discuss"></a>
</div>
</div>
</div>
<divclass="resources">
<brclass="mobile"><brclass="mobile">
<brclass="tablet"><brclass="tablet">
<h1style="padding-top:1%">Diskussionen</h1>
<divstyle="font-weight:300;text-align:center">Wir reden gerne und viel. Triff uns auf einer dieser Plattformen und beteilige dich an den Diskussionen.</div>
<divclass="resourceboxtext"style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">Echtzeit-Ankündigungen von Blogeinträgen, Veranstaltungen, Neuigkeiten und Ideen
Mit mehr als 150 Treffen auf der ganzen Welt, Tendenz steigend, solltest du deine lokalen Kube-Leute finden. Wenn keins in der Nähe ist, nimm die Sache in die Hand und gründe dein eigenes.
<aclass="twitter-timeline"data-tweet-limit="1"href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">Tweets von kubernetesio</a><scriptasyncsrc="https://platform.twitter.com/widgets.js"charset="utf-8"></script>
<h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, dynamische Codebasis zu schaffen, die ein Spektrum von aufeinander abgestimmten Plattformen unterstützt.</h5>
<divclass="col-container">
<divclass="col-nav">
<center>
<h5>
<b>Kubernetes zertifizierte Service Provider</b>
</h5>
<br>Geprüfte Service Provider mit großer Erfahrung, die Unternehmen bei der erfolgreichen Einführung von Kubernetes unterstützen.
<h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, lebendige Codebasis zu schaffen, die ein Spektrum von ergänzenden Plattformen unterstützt.</h5>
<divclass="col-container">
<divclass="col-nav">
<center>
<h5>
<b>Kubernetes-zertifizierte Service-Anbieter</b>
</h5>
<br>Geprüfte Dienstleister mit umfassender Erfahrung bei der erfolgreichen Einführung von Kubernetes in Unternehmen.
bigheader: Kubernetes Schulungen und Zertifizierungen
abstract: Schulungsprogramme, Zertifizierungen und Partner.
layout: basic
cid: training
class: training
---
<sectionclass="call-to-action">
<divclass="main-section">
<divclass="call-to-action"id="cta-certification">
<divclass="cta-text">
<h2>Gestalte deine Cloud Native Karriere</h2>
<p>Kubernetes ist das Herzstück der Cloud Native-Bewegung. Mit den Schulungen und Zertifizierungen der Linux Foundation und unserer Schulungspartner kannst Du in deine Karriere investieren, Kubernetes lernen und deine Cloud Native-Projekte zum Erfolg führen.</p>
<b>Einführung in Cloud-Infrastruktur Technologien</b>
</h5>
<p>Lerne die Grundlagen für den Aufbau und die Verwaltung von Cloud-Technologien direkt von der Linux Foundation, dem Marktführer im Bereich Open Source.</p>
<p>Du hast nie Linux gelernt? Willst du eine Auffrischung? Erarbeite dir gute Linux-Kenntnisse über die grafische Oberfläche und die Kommandozeile der wichtigsten Linux-Distributionen.</p>
<p>Die Linux Foundation bietet Kurse für alle Aspekte der Entwicklung und des Betriebs von Kubernetes-Anwendungen an, die entweder von Lehrkräften geleitet werden oder zum Selbststudium geeignet sind.</p>
<b>Kubernetes and Cloud Native Associate (KCNA)</b>
</h5>
<p>Die Prüfung zum Kubernetes and Cloud Native Associate (KCNA) weist die grundlegenden Kenntnisse und Fähigkeiten eines Benutzers in Kubernetes und dem breiteren Cloud Native-Ökosystem nach.</p>
<p>Ein zertifizierter KCNA bestätigt konzeptionelles Wissen über das gesamte Cloud Native Ecosystem, mit besonderem Fokus auf Kubernetes.</p>
<p>Die Prüfung zum Certified Kubernetes Application Developer (Zertifizierter Kubernetes-Anwendungsentwickler) bescheinigt, dass Teilnehmer Cloud Native-Anwendungen für Kubernetes entwerfen, erstellen, konfigurieren und bereitstellen können.</p>
<p>Ein CKAD kann Anwendungsressourcen definieren und zentrale Elemente verwenden, um skalierbare Anwendungen und Tools in Kubernetes zu erstellen, zu überwachen und Fehler zu beheben.</p>
<p>Das Certified Kubernetes Administrator (CKA)-Programm garantiert, dass CKAs die Fähigkeiten, das Wissen und die Kompetenz besitzen, um die Aufgaben eines Kubernetes-Administrators zu erfüllen.</p>
<p>Ein zertifizierter Kubernetes-Administrator hat nachgewiesen, dass er in der Lage ist, grundlegende Installationen durchzuführen sowie Kubernetes-Cluster in einer Produktionsumgebung zu konfigurieren und zu verwalten.</p>
<p>Das Programm Certified Kubernetes Security Specialist (CKS) bietet die Gewissheit, dass der Zertifikatsinhaber mit einem breiten Spektrum an Best Practices vertraut ist und diese beherrscht. Die CKS-Zertifizierung umfasst Fähigkeiten zur Sicherung von Container-basierten Anwendungen und Kubernetes-Plattformen während der Erstellung, Bereitstellung und Laufzeit.</p>
<p><em>Kandidaten für den CKS müssen über eine aktuelle Zertifizierung als Certified Kubernetes Administrator (CKA) verfügen, um nachzuweisen, dass sie über ausreichende Kubernetes-Kenntnisse verfügen, bevor sie sich für den CKS anmelden.</em></p>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21"buttonid="desktopKCButton">Attend KubeCon North America on October 11-15, 2021</a>
<br>
<br>
<br>
<br>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22"buttonid="desktopKCButton">Attend KubeCon Europe on May 17-20, 2022</a>
<br>
<br>
<br>
<br>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21"buttonid="desktopKCButton">Attend KubeCon North America on October 24-28, 2022</a>
@@ -125,7 +125,7 @@ You may wish to, but you cannot create a hierarchy of namespaces. Namespaces can
Namespaces are easy to create and use but it’s also easy to deploy code inadvertently into the wrong namespace. Good DevOps hygiene suggests documenting and automating processes where possible and this will help. The other way to avoid using the wrong namespace is to set a [kubectl context](/docs/user-guide/kubectl/kubectl_config_set-context/).
Namespaces are easy to create and use but it’s also easy to deploy code inadvertently into the wrong namespace. Good DevOps hygiene suggests documenting and automating processes where possible and this will help. The other way to avoid using the wrong namespace is to set a [kubectl context](/docs/reference/generated/kubectl/kubectl-commands#-em-set-context-em-).
In Kubernetes 1.4, we introduced a new node performance analysis tool, called the _node performance dashboard_, to visualize and explore the behavior of the Kubelet in much richer details. This new feature will make it easy to understand and improve code performance for Kubelet developers, and lets cluster maintainer set configuration according to provided Service Level Objectives (SLOs).
@@ -37,7 +37,7 @@ If you run your storage application on high-end hardware or extra-large instance
[ZooKeeper](https://zookeeper.apache.org/doc/current/) is an interesting use case for StatefulSet for two reasons. First, it demonstrates that StatefulSet can be used to run a distributed, strongly consistent storage application on Kubernetes. Second, it's a prerequisite for running workloads like [Apache Hadoop](http://hadoop.apache.org/) and [Apache Kakfa](https://kafka.apache.org/) on Kubernetes. An [in-depth tutorial](/docs/tutorials/stateful-application/zookeeper/) on deploying a ZooKeeper ensemble on Kubernetes is available in the Kubernetes documentation, and we’ll outline a few of the key features below.
**Creating a ZooKeeper Ensemble**
Creating an ensemble is as simple as using [kubectl create](/docs/user-guide/kubectl/kubectl_create/) to generate the objects stored in the manifest.
Creating an ensemble is as simple as using [kubectl create](/docs/reference/generated/kubectl/kubectl-commands#create) to generate the objects stored in the manifest.
```
@@ -297,7 +297,7 @@ zk-0 0/1 Terminating 0 15m
You can use [kubectl apply](/docs/user-guide/kubectl/kubectl_apply/) to recreate the zk StatefulSet and redeploy the ensemble.
You can use [kubectl apply](/docs/reference/generated/kubectl/kubectl-commands#apply) to recreate the zk StatefulSet and redeploy the ensemble.
@@ -29,7 +29,7 @@ They join continuing members Christoph Blecker ([@cblecker](https://github.com/c
* Josh Berkus ([@jberkus](https://github.com/jberkus)), Red Hat
* Thanks to the Emeritus Steering Committee Members. Your prior service is appreciated by the community:
* Aaron Crickenberger ([@spiffxp](https://github.com/spiffxp)), Google
* and Lachlan Evenson([@lachie8e)](https://github.com/lachie8e)), Microsoft
* and Lachlan Evenson([@lachie83)](https://github.com/lachie83)), Microsoft
* And thank you to all the candidates who came forward to run for election. As [Jorge Castro put it](https://twitter.com/castrojo/status/1315718627639820288?s=20): we are spoiled with capable, kind, and selfless volunteers who put the needs of the project first.
To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=false option for `kubectl delete` to delete an object and orphan its children.
To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=orphan option for `kubectl delete` to delete an object and orphan its children.
In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=false, the parent is deleted but the child still exists:
In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=orphan, the parent is deleted but the child still exists:
@@ -54,7 +54,7 @@ An alpha feature for default seccomp profiles has been added to the kubelet, alo
A new alpha feature allows running the `kubeadm` control plane components as non-root users. This is a long requested security measure in `kubeadm`. To try it you must enable the `kubeadm` specific RootlessControlPlane feature gate. When you deploy a cluster using this alpha feature, your control plane runs with lower privileges.
For `kubeadm`, Kubernetes 1.22 also brings a new [v1beta3 configuration API](https://github.com/kubernetes/kubeadm/issues/1796). This iteration adds some long requested features and deprecates some existing ones. The v1beta3 version is now the preferred API version; the v1beta2 API also remains available and is not yet deprecated.
For `kubeadm`, Kubernetes 1.22 also brings a new [v1beta3 configuration API](/docs/reference/config-api/kubeadm-config.v1beta3/). This iteration adds some long requested features and deprecates some existing ones. The v1beta3 version is now the preferred API version; the v1beta2 API also remains available and is not yet deprecated.
## Major Changes
@@ -140,7 +140,7 @@ In the v1.22 release cycle, which ran for 15 weeks (April 26 to August 4), we sa
# Upcoming release webinar
Join members of the Kubernetes 1.22 release team on September 7, 2021 to learn about the major features of this release, as well as deprecations and removals to help plan for upgrades. For more information and registration, visit the [event page](https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-live-webinar-kubernetes-122-release/) on the CNCF Online Programs site.
Join members of the Kubernetes 1.22 release team on October 5, 2021 to learn about the major features of this release, as well as deprecations and removals to help plan for upgrades. For more information and registration, visit the [event page](https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-live-webinar-kubernetes-122-release/) on the CNCF Online Programs site.
title: "Kubernetes 1.22: Server Side Apply moves to GA"
date: 2021-08-06
slug: server-side-apply-ga
---
**Authors:** Jeffrey Ying, Google & Joe Betz, Google
Server-side Apply (SSA) has been promoted to GA in the Kubernetes v1.22 release. The GA milestone means you can depend on the feature and its API, without fear of future backwards-incompatible changes. GA features are protected by the Kubernetes [deprecation policy](/docs/reference/using-api/deprecation-policy/).
## What is Server-side Apply?
Server-side Apply helps users and controllers manage their resources through declarative configurations. Server-side Apply replaces the client side apply feature implemented by “kubectl apply” with a server-side implementation, permitting use by tools/clients other than kubectl. Server-side Apply is a new merging algorithm, as well as tracking of field ownership, running on the Kubernetes api-server. Server-side Apply enables new features like conflict detection, so the system knows when two actors are trying to edit the same field. Refer to the [Server-side Apply Documentation](/docs/reference/using-api/server-side-apply/) and [Beta 2 release announcement](https://kubernetes.io/blog/2020/04/01/kubernetes-1.18-feature-server-side-apply-beta-2/) for more information.
## What’s new since Beta?
Since the [Beta 2 release](https://kubernetes.io/blog/2020/04/01/kubernetes-1.18-feature-server-side-apply-beta-2/) subresources support has been added, and both client-go and Kubebuilder have added comprehensive support for Server-side Apply. This completes the Server-side Apply functionality required to make controller development practical.
### Support for subresources
Server-side Apply now fully supports subresources like `status` and `scale`. This is particularly important for [controllers](/docs/concepts/architecture/controller/), which are often responsible for writing to subresources.
## Server-side Apply support in client-go
Previously, Server-side Apply could only be called from the client-go typed client using the `Patch` function, with `PatchType` set to `ApplyPatchType`. Now, `Apply` functions are included in the client to allow for a more direct and typesafe way of calling Server-side Apply. Each `Apply` function takes an "apply configuration" type as an argument, which is a structured representation of an Apply request. For example:
Note in this example that `HorizontalPodAutoscaler` is imported from an "applyconfigurations" package. Each "apply configuration" type represents the same Kubernetes object kind as the corresponding go struct, but where all fields are pointers to make them optional, allowing apply requests to be accurately represented. For example, when the apply configuration in the above example is marshalled to YAML, it produces:
```yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: myHPA
namespace: myNamespace
spec:
minReplicas: 0
```
To understand why this is needed, the above YAML cannot be produced by the v1.HorizontalPodAutoscaler go struct. Take for example:
```go
hpa := v1.HorizontalPodAutoscaler{
TypeMeta: metav1.TypeMeta{
APIVersion: "autoscaling/v1",
Kind: "HorizontalPodAutoscaler",
},
ObjectMeta: ObjectMeta{
Namespace: ns,
Name: autoscalerName,
},
Spec: v1.HorizontalPodAutoscalerSpec{
MinReplicas: pointer.Int32Ptr(0),
},
}
```
The above code attempts to declare the same apply configuration as shown in the previous examples, but when marshalled to YAML, produces:
```yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v1
metadata
name: myHPA
namespace: myNamespace
creationTimestamp: null
spec:
scaleTargetRef:
kind: ""
name: ""
minReplicas: 0
maxReplicas: 0
```
Which, among other things, contains `spec.maxReplicas` set to `0`. This is almost certainly not what the caller intended (the intended apply configuration says nothing about the `maxReplicas` field), and could have serious consequences on a production system: it directs the autoscaler to downscale to zero pods. The problem here originates from the fact that the go structs contain required fields that are zero valued if not set explicitly. The go structs work as intended for create and update operations, but are fundamentally incompatible with apply, which is why we have introduced the generated "apply configuration" types.
The "apply configurations" also have convenience `With<FieldName>` functions that make it easier to build apply requests. This allows developers to set fields without having to deal with the fact that all the fields in the "apply configuration" types are pointers, and are inconvenient to set using go. For example `MinReplicas: &0` is not legal go code, so without the `With` functions, developers would work around this problem by using a library, e.g. `MinReplicas: pointer.Int32Ptr(0)`, but string enumerations like `corev1.Protocol` are still a problem since they cannot be supported by a general purpose library. In addition to the convenience, the `With` functions also isolate developers from the underlying representation, which makes it safer for the underlying representation to be changed to support additional features in the future.
## Using Server-side Apply in a controller
You can use the new support for Server-side Apply no matter how you implemented your controller. However, the new client-go support makes it easier to use Server-side Apply in controllers.
When authoring new controllers to use Server-side Apply, a good approach is to have the controller recreate the apply configuration for an object each time it reconciles that object. This ensures that the controller fully reconciles all the fields that it is responsible for. Controllers typically should unconditionally set all the fields they own by setting `Force: true` in the `ApplyOptions`. Controllers must also provide a `FieldManager` name that is unique to the reconciliation loop that apply is called from.
When upgrading existing controllers to use Server-side Apply the same approach often works well--migrate the controllers to recreate the apply configuration each time it reconciles any object. Unfortunately, the controller might have multiple code paths that update different parts of an object depending on various conditions. Migrating a controller like this to Server-side Apply can be risky because if the controller forgets to include any fields in an apply configuration that is included in a previous apply request, a field can be accidently deleted. To ease this type of migration, client-go apply support provides a way to replace any controller reconciliation code that performs a "read/modify-in-place/update" (or patch) workflow with a "extract/modify-in-place/apply" workflow. Here's an example of the new workflow:
For developers using Custom Resource Definitions (CRDs), the Kubebuilder apply support will provide the same capabilities. Documentation will be included in the Kubebuilder book when available.
## Server-side Apply and CustomResourceDefinitions
It is strongly recommended that all [Custom Resource Definitions](/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRDs) have a schema. CRDs without a schema are treated as unstructured data by Server-side Apply. Keys are treated as fields in a struct and lists are assumed to be atomic.
CRDs that specify a schema are able to specify additional annotations in the schema. Please refer to the documentation on the full list of available annotations.
New annotations since beta:
**Defaulting:** Values for fields that appliers do not express explicit interest in should be defaulted. This prevents an applier from unintentionally owning a defaulted field that might cause conflicts with other appliers. If unspecified, the default value is nil or the nil equivalent for the corresponding type.
- Usage: see the [CRD Defaulting](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#defaulting) documentation for more details.
- Golang: `+default=<value>`
- OpenAPI extension: `default: <value>`
Atomic for maps and structs:
**Maps:** By default maps are granular. A different manager is able to manage each map entry. They can also be configured to be atomic such that a single manager owns the entire map.
- Usage: Refer to [Merge Strategy](/docs/reference/using-api/server-side-apply/#merge-strategy) for a more detailed overview
**Structs:** By default structs are granular and a separate applier may own each field. For certain kinds of structs, atomicity may be desired. This is most commonly seen in small coordinate-like structs such as Field/Object/Namespace Selectors, Object References, RGB values, Endpoints (Protocol/Port pairs), etc.
- Usage: Refer to [Merge Strategy](/docs/reference/using-api/server-side-apply/#merge-strategy) for a more detailed overview
After Server Side Apply, the next focus for the API Expression working-group is around improving the expressiveness and size of the published Kubernetes API schema. To see the full list of items we are working on, please join our working group and refer to the work items document.
## How to get involved?
The working-group for apply is [wg-api-expression](https://github.com/kubernetes/community/tree/master/wg-api-expression). It is available on slack [#wg-api-expression](https://kubernetes.slack.com/archives/C0123CNN8F3), through the [mailing list](https://groups.google.com/g/kubernetes-wg-api-expression) and we also meet every other Tuesday at 9.30 PT on Zoom.
We would also like to use the opportunity to thank the hard work of all the contributors involved in making this promotion to GA possible:
title: 'New in Kubernetes v1.22: alpha support for using swap memory'
date: 2021-08-09
slug: run-nodes-with-swap-alpha
---
**Author:** Elana Hashman (Red Hat)
The 1.22 release introduced alpha support for configuring swap memory usage for
Kubernetes workloads on a per-node basis.
In prior releases, Kubernetes did not support the use of swap memory on Linux,
as it is difficult to provide guarantees and account for pod memory utilization
when swap is involved. As part of Kubernetes' earlier design, swap support was
considered out of scope, and a kubelet would by default fail to start if swap
was detected on a node.
However, there are a number of [use cases](https://github.com/kubernetes/enhancements/blob/9d127347773ad19894ca488ee04f1cd3af5774fc/keps/sig-node/2400-node-swap/README.md#user-stories)
that would benefit from Kubernetes nodes supporting swap, including improved
node stability, better support for applications with high memory overhead but
smaller working sets, the use of memory-constrained devices, and memory
flexibility.
Hence, over the past two releases, [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node#readme) has
been working to gather appropriate use cases and feedback, and propose a design
for adding swap support to nodes in a controlled, predictable manner so that
Kubernetes users can perform testing and provide data to continue building
cluster capabilities on top of swap. The alpha graduation of swap memory
support for nodes is our first milestone towards this goal!
## How does it work?
There are a number of possible ways that one could envision swap use on a node.
To keep the scope manageable for this initial implementation, when swap is
already provisioned and available on a node, [we have proposed](https://github.com/kubernetes/enhancements/blob/9d127347773ad19894ca488ee04f1cd3af5774fc/keps/sig-node/2400-node-swap/README.md#proposal)
the kubelet should be able to be configured such that:
- It can start with swap on.
- It will direct the Container Runtime Interface to allocate zero swap memory
to Kubernetes workloads by default.
- You can configure the kubelet to specify swap utilization for the entire
node.
Swap configuration on a node is exposed to a cluster admin via the
[`memorySwap` in the KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/).
As a cluster administrator, you can specify the node's behaviour in the
presence of swap memory by setting `memorySwap.swapBehavior`.
This is possible through the addition of a `memory_swap_limit_in_bytes` field
to the container runtime interface (CRI). The kubelet's config will control how
much swap memory the kubelet instructs the container runtime to allocate to
each container via the CRI. The container runtime will then write the swap
settings to the container level cgroup.
## How do I use it?
On a node where swap memory is already provisioned, Kubernetes use of swap on a
node can be enabled by enabling the `NodeSwap` feature gate on the kubelet, and
disabling the `failSwapOn` [configuration setting](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
or the `--fail-swap-on` command line flag.
You can also optionally configure `memorySwap.swapBehavior` in order to
specify how a node will use swap memory. For example,
```yaml
memorySwap:
swapBehavior: LimitedSwap
```
The available configuration options for `swapBehavior` are:
- `LimitedSwap` (default): Kubernetes workloads are limited in how much swap
they can use. Workloads on the node not managed by Kubernetes can still swap.
- `UnlimitedSwap`: Kubernetes workloads can use as much swap memory as they
request, up to the system limit.
If configuration for `memorySwap` is not specified and the feature gate is
enabled, by default the kubelet will apply the same behaviour as the
`LimitedSwap` setting.
The behaviour of the `LimitedSwap` setting depends if the node is running with
v1 or v2 of control groups (also known as "cgroups"):
- **cgroups v1:** Kubernetes workloads can use any combination of memory and
swap, up to the pod's memory limit, if set.
- **cgroups v2:** Kubernetes workloads cannot use swap memory.
### Caveats
Having swap available on a system reduces predictability. Swap's performance is
worse than regular memory, sometimes by many orders of magnitude, which can
cause unexpected performance regressions. Furthermore, swap changes a system's
behaviour under memory pressure, and applications cannot directly control what
portions of their memory usage are swapped out. Since enabling swap permits
greater memory usage for workloads in Kubernetes that cannot be predictably
accounted for, it also increases the risk of noisy neighbours and unexpected
packing configurations, as the scheduler cannot account for swap memory usage.
The performance of a node with swap memory enabled depends on the underlying
physical storage. When swap memory is in use, performance will be significantly
worse in an I/O operations per second (IOPS) constrained environment, such as a
cloud VM with I/O throttling, when compared to faster storage mediums like
solid-state drives or NVMe.
Hence, we do not recommend the use of swap for certain performance-constrained
workloads or environments. Cluster administrators and developers should
benchmark their nodes and applications before using swap in production
scenarios, and [we need your help](#how-do-i-get-involved) with that!
## Looking ahead
The Kubernetes 1.22 release introduces alpha support for swap memory on nodes,
and we will continue to work towards beta graduation in the 1.23 release. This
will include:
* Adding support for controlling swap consumption at the Pod level via cgroups.
* This will include the ability to set a system-reserved quantity of swap
from what kubelet detects on the host.
* Determining a set of metrics for node QoS in order to evaluate the
performance and stability of nodes with and without swap enabled.
* Collecting feedback from test user cases.
* We will consider introducing new configuration modes for swap, such as a
node-wide swap limit for workloads.
## How can I learn more?
You can review the current [documentation](https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory)
on the Kubernetes website.
For more information, and to assist with testing and provide feedback, please
see [KEP-2400](https://github.com/kubernetes/enhancements/issues/2400) and its
**Authors:** Mauricio Poppe (Google), Jing Xu (Google), and Deep Debroy (Apple)
*The stable version of CSI Proxy for Windows has been released alongside Kubernetes 1.22. CSI Proxy enables CSI Drivers running on Windows nodes to perform privileged storage operations.*
## Background
Container Storage Interface (CSI) for Kubernetes went GA in the Kubernetes 1.13 release. CSI has become the standard for exposing block and file storage to containerized workloads on Container Orchestration systems (COs) like Kubernetes. It enables third-party storage providers to write and deploy plugins without the need to alter the core Kubernetes codebase. Legacy in-tree drivers are deprecated and new storage features are introduced in CSI, therefore it is important to get CSI Drivers to work on Windows.
A CSI Driver in Kubernetes has two main components: a controller plugin which runs in the control plane and a node plugin which runs on every node.
- The controller plugin generally does not need direct access to the host and can perform all its operations through the Kubernetes API and external control plane services.
- The node plugin, however, requires direct access to the host for making block devices and/or file systems available to the Kubernetes kubelet. Due to the missing capability of running privileged operations from containers on Windows nodes [CSI Proxy was introduced as alpha in Kubernetes 1.18](https://kubernetes.io/blog/2020/04/03/kubernetes-1-18-feature-windows-csi-support-alpha/) as a way to enable containers to perform privileged storage operations. This enables containerized CSI Drivers to run on Windows nodes.
## What's CSI Proxy and how do CSI drivers interact with it?
When a workload that uses persistent volumes is scheduled, it'll go through a sequence of steps defined in the [CSI Spec](https://github.com/container-storage-interface/spec/blob/master/spec.md). First, the workload will be scheduled to run on a node. Then the controller component of a CSI Driver will attach the persistent volume to the node. Finally the node component of a CSI Driver will mount the persistent volume on the node.
The node component of a CSI Driver needs to run on Windows nodes to support Windows workloads. Various privileged operations like scanning of disk devices, mounting of file systems, etc. cannot be done from a containerized application running on Windows nodes yet ([Windows HostProcess containers](https://github.com/kubernetes/enhancements/issues/1981) introduced in Kubernetes 1.22 as alpha enable functionalities that require host access like the operations mentioned before). However, we can perform these operations through a binary (CSI Proxy) that's pre-installed on the Window nodes. CSI Proxy has a client-server architecture and allows CSI drivers to issue privileged storage operations through a gRPC interface exposed over named pipes created during the startup of CSI Proxy.
The CSI Proxy development team has worked closely with storage vendors, many of whom started integrating CSI Proxy into their CSI Drivers and provided feedback as early as CSI Proxy design proposal. This cooperation uncovered use cases where additional APIs were needed, found bugs, and identified areas for documentation improvement.
The CSI Proxy design [KEP](https://github.com/kubernetes/enhancements/pull/2737) has been updated to reflect the current CSI Proxy architecture. Additional [development documentation](https://github.com/kubernetes-csi/csi-proxy/blob/master/docs/DEVELOPMENT.md) is included for contributors interested in helping with new features or bug fixes.
Before we reached GA we wanted to make sure that our API is simple and consistent. We went through an extensive API review of the v1beta API groups where we made sure that the CSI Proxy API methods and messages are consistent with the naming conventions defined in the [CSI Spec](https://github.com/container-storage-interface/spec/blob/master/spec.md). As part of this effort we're graduating the [Disk](https://github.com/kubernetes-csi/csi-proxy/blob/master/docs/apis/disk_v1.md), [Filesystem](https://github.com/kubernetes-csi/csi-proxy/blob/master/docs/apis/filesystem_v1.md), [SMB](https://github.com/kubernetes-csi/csi-proxy/blob/master/docs/apis/smb_v1.md) and [Volume](https://github.com/kubernetes-csi/csi-proxy/blob/master/docs/apis/volume_v1.md) API groups to v1.
Additional Windows system APIs to get information from the Windows nodes and support to mount iSCSI targets in Windows nodes, are available as alpha APIs in the [System API](https://github.com/kubernetes-csi/csi-proxy/tree/v1.0.0/client/api/system/v1alpha1) and the [iSCSI API](https://github.com/kubernetes-csi/csi-proxy/tree/v1.0.0/client/api/iscsi/v1alpha2). These APIs will continue to be improved before we graduate them to v1.
CSI Proxy v1 is compatible with all the previous v1betaX releases. The GA `csi-proxy.exe` binary can handle requests from v1betaX clients thanks to the autogenerated conversion layer that transforms any versioned client request to a version-agnostic request that the server can process. Several [integration tests](https://github.com/kubernetes-csi/csi-proxy/tree/v1.0.0/integrationtests) were added for all the API versions of the API groups that are graduating to v1 to ensure that CSI Proxy is backwards compatible.
Version drift between CSI Proxy and the CSI Drivers that interact with it was also carefully considered. A [connection fallback mechanism](https://github.com/kubernetes-csi/csi-proxy/pull/124) has been provided for CSI Drivers to handle multiple versions of CSI Proxy for a smooth upgrade to v1. This allows CSI Drivers, like the GCE PD CSI Driver, [to recognize which version of the CSI Proxy binary is running](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/738) and handle multiple versions of the CSI Proxy binary deployed on the node.
CSI Proxy v1 is already being used by many CSI Drivers, including the [AWS EBS CSI Driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/966), [Azure Disk CSI Driver](https://github.com/kubernetes-sigs/azuredisk-csi-driver/pull/919), [GCE PD CSI Driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/pull/738), and [SMB CSI Driver](https://github.com/kubernetes-csi/csi-driver-smb/pull/319).
## Future plans
We're very excited for the future of CSI Proxy. With the upcoming [Windows HostProcess containers](https://github.com/kubernetes/enhancements/issues/1981), we are considering converting the CSI Proxy in to a library consumed by CSI Drivers in addition to the current client/server design. This will allow us to iterate faster on new features because the `csi-proxy.exe` binary will no longer be needed.
## How to get involved?
This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. Those interested in getting involved with the design and development of CSI Proxy, or any part of the Kubernetes Storage system, may join the Kubernetes Storage Special Interest Group (SIG). We’re rapidly growing and always welcome new contributors.
For those interested in more details about CSI support in Windows please reach out in the [#csi-windows](https://kubernetes.slack.com/messages/csi-windows) Kubernetes slack channel.
## Acknowledgments
CSI-Proxy received many contributions from members of the Kubernetes community. We thank all of the people that contributed to CSI Proxy with design reviews, bug reports, bug fixes, and for their continuous support in reaching this milestone:
The value for `--memory-manager-policy` is straightforward: `Static`. Deciding what to specify for `--reserved-memory` takes more thought. To configure it correctly, you should follow two main rules:
- The amount of reserved memory for the `memory` resource must be greater than zero.
- The amount of reserved memory for the resource type must be equal
to [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)
(`kube-reserved + system-reserved + eviction-hard`) for the resource.
You can read more about memory reservations in [Reserve Compute Resources for System Daemons](/docs/tasks/administer-cluster/reserve-compute-resources/).
The 1.22 release and promotion to beta brings along enhancements and fixes, but the Memory Manager still has several limitations.
### Single vs Cross NUMA node allocation
The NUMA node can not have both single and cross NUMA node allocations. When the container memory is pinned to two or more NUMA nodes, we can not know from which NUMA node the container will consume the memory.
1. The `container1` started on the NUMA node 0 and requests *5Gi* of the memory but currently is consuming only *3Gi* of the memory.
2. For container2 the memory request is 10Gi, and no single NUMA node can satisfy it.
3. The `container2` consumes *3.5Gi* of the memory from the NUMA node 0, but once the `container1` will require more memory, it will not have it, and the kernel will kill one of the containers with the *OOM* error.
To prevent such issues, the Memory Manager will fail the admission of the `container2` until the machine has two NUMA nodes without a single NUMA node allocation.
### Works only for Guaranteed pods
The Memory Manager can not guarantee memory allocation for Burstable pods,
also when the Burstable pod has specified equal memory limit and request.
Let's assume you have two Burstable pods: `pod1` has containers with
equal memory request and limits, and `pod2` has containers only with a
memory request set. You want to guarantee memory allocation for the `pod1`.
To the Linux kernel, processes in either pod have the same *OOM score*,
once the kernel finds that it does not have enough memory, it can kill
processes that belong to pod `pod1`.
### Memory fragmentation
The sequence of Pods and containers that start and stop can fragment the memory on NUMA nodes.
The alpha implementation of the Memory Manager does not have any mechanism to balance pods and defragment memory back.
## Future work for the Memory Manager
We do not want to stop with the current state of the Memory Manager and are looking to
make improvements, including in the following areas.
### Make the Memory Manager allocation algorithm smarter
The current algorithm ignores distances between NUMA nodes during the
calculation of the allocation. If same-node placement isn't available, we can still
provide better performance compared to the current implementation, by changing the
Memory Manager to prefer the closest NUMA nodes for cross-node allocation.
### Reduce the number of admission errors
The default Kubernetes scheduler is not aware of the node's NUMA topology, and it can be a reason for many admission errors during the pod start.
We're hoping to add a KEP (Kubernetes Enhancement Proposal) to cover improvements in this area.
Follow [Topology aware scheduler plugin in kube-scheduler](https://github.com/kubernetes/enhancements/issues/2044) to see how this idea progresses.
## Conclusion
With the promotion of the Memory Manager to beta in 1.22, we encourage everyone to give it a try and look forward to any feedback you may have. While there are still several limitations, we have a set of enhancements planned to address them and look forward to providing you with many new features in upcoming releases.
If you have ideas for additional enhancements or a desire for certain features, please let us know. The team is always open to suggestions to enhance and improve the Memory Manager.
We hope you have found this blog informative and helpful! Let us know if you have any questions or comments.
You can contact us via:
- The Kubernetes [#sig-node ](https://kubernetes.slack.com/messages/sig-node)
channel in Slack (visit https://slack.k8s.io/ for an invitation if you need one)
- The SIG Node mailing list, [kubernetes-sig-node@googlegroups.com](https://groups.google.com/g/kubernetes-sig-node)
title: 'Alpha in v1.22: Windows HostProcess Containers'
date: 2021-08-16
slug: windows-hostprocess-containers
---
**Authors:** Brandon Smith (Microsoft)
Kubernetes v1.22 introduced a new alpha feature for clusters that
include Windows nodes: HostProcess containers.
HostProcess containers aim to extend the Windows container model to enable a wider
range of Kubernetes cluster management scenarios. HostProcess containers run
directly on the host and maintain behavior and access similar to that of a regular
process. With HostProcess containers, users can package and distribute management
operations and functionalities that require host access while retaining versioning
and deployment methods provided by containers. This allows Windows containers to
be used for a variety of device plugin, storage, and networking management scenarios
in Kubernetes. With this comes the enablement of host network mode—allowing
HostProcess containers to be created within the host's network namespace instead of
their own. HostProcess containers can also be built on top of existing Windows server
2019 (or later) base images, managed through the Windows container runtime, and run
as any user that is available on or in the domain of the host machine.
Linux privileged containers are currently used for a variety of key scenarios in
Kubernetes, including kube-proxy (via kubeadm), storage, and networking scenarios.
Support for these scenarios in Windows previously required workarounds via proxies
or other implementations. Using HostProcess containers, cluster operators no longer
need to log onto and individually configure each Windows node for administrative
tasks and management of Windows services. Operators can now utilize the container
model to deploy management logic to as many clusters as needed with ease.
## How does it work?
Windows HostProcess containers are implemented with Windows _Job Objects_, a break from the
previous container model using server silos. Job objects are components of the Windows OS which offer the ability to
manage a group of processes as a group (a.k.a. _jobs_) and assign resource constraints to the
group as a whole. Job objects are specific to the Windows OS and are not associated with the Kubernetes [Job API](https://kubernetes.io/docs/concepts/workloads/controllers/job/). They have no process or file system isolation,
enabling the privileged payload to view and edit the host file system with the
correct permissions, among other host resources. The init process, and any processes
it launches or that are explicitly launched by the user, are all assigned to the
job object of that container. When the init process exits or is signaled to exit,
all the processes in the job will be signaled to exit, the job handle will be
closed and the storage will be unmounted.
HostProcess and Linux privileged containers enable similar scenarios but differ
greatly in their implementation (hence the naming difference). HostProcess containers
have their own pod security policies. Those used to configure Linux privileged
containers **do not** apply. Enabling privileged access to a Windows host is a
fundamentally different process than with Linux so the configuration and
capabilities of each differ significantly. Below is a diagram detailing the
overall architecture of Windows HostProcess containers:
**Authors:** Ravi Gudimetla (Red Hat), Maciej Szulik (Red Hat)
This blog describes the notion of Availability for `StatefulSet` workloads, and a new alpha feature in Kubernetes 1.22 which adds `minReadySeconds` configuration for `StatefulSets`.
## What problems does this solve?
Prior to Kubernetes 1.22 release, once a `StatefulSet``Pod` is in the `Ready` state it is considered `Available` to receive traffic. For some of the `StatefulSet` workloads, it may not be the case. For example, a workload like Prometheus with multiple instances of Alertmanager, it should be considered `Available` only when Alertmanager's state transfer is complete, not when the `Pod` is in `Ready` state. Since `minReadySeconds` adds buffer, the state transfer may be complete before the `Pod` becomes `Available`. While this is not a fool proof way of identifying if the state transfer is complete or not, it gives a way to the end user to express their intention of waiting for sometime before the `Pod` is considered `Available` and it is ready to serve requests.
Another case, where `minReadySeconds` helps is when using `LoadBalancer``Services` with cloud providers. Since `minReadySeconds` adds latency after a `Pod` is `Ready`, it provides buffer time to prevent killing pods in rotation before new pods show up. Imagine a load balancer in unhappy path taking 10-15s to propagate. If you have 2 replicas then, you'd kill the second replica only after the first one is up but in reality, first replica cannot be seen because it is not yet ready to serve requests.
So, in general, the notion of `Availability` in `StatefulSets` is pretty useful and this feature helps in solving the above problems. This is a feature that already exists for `Deployments` and `DaemonSets` and we now have them for `StatefulSets` too to give users consistent workload experience.
## How does it work?
The statefulSet controller watches for both `StatefulSets` and the `Pods` associated with them. When the feature gate associated with this feature is enabled, the statefulSet controller identifies how long a particular `Pod` associated with a `StatefulSet` has been in the `Running` state.
If this value is greater than or equal to the time specified by the end user in `.spec.minReadySeconds` field, the statefulSet controller updates a field called `availableReplicas` in the `StatefulSet`'s status subresource to include this `Pod`. The `status.availableReplicas` in `StatefulSet`'s status is an integer field which tracks the number of pods that are `Available`.
## How do I use it?
You are required to prepare the following things in order to try out the feature:
- Download and install a kubectl greater than v1.22.0 version
- Switch on the feature gate with the command line flag `--feature-gates=StatefulSetMinReadySeconds=true` on `kube-apiserver` and `kube-controller-manager`
After successfully starting `kube-apiserver` and `kube-controller-manager`, you will see `AvailableReplicas` in the status and `minReadySeconds` of spec (with a default value of 0).
Specify a value for `minReadySeconds` for any StatefulSet and you can check if `Pods` are available or not by checking `AvailableReplicas` field using:
`kubectl get statefulset/<name_of_the_statefulset> -o yaml`
## How can I learn more?
- Read the KEP: [minReadySeconds for StatefulSets](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2599-minreadyseconds-for-statefulsets#readme)
- Read the documentation: [Minimum ready seconds](/docs/concepts/workloads/controllers/statefulset/#minimum-ready-seconds) for StatefulSet
- Review the [API definition](/docs/reference/kubernetes-api/workload-resources/stateful-set-v1/) for StatefulSet
## How do I get involved?
Please reach out to us in the [#sig-apps](https://kubernetes.slack.com/archives/C18NZM5K9) channel on Slack (visit https://slack.k8s.io/ for an invitation if you need one), or on the SIG Apps mailing list: kubernetes-sig-apps@googlegroups.com
and to only supply a small controller wrapper around the library, and a pod image
capable of attaching to volumes and writing the appropriate data to the volume.
Individual populators can be extremely generic such that they work with every type
of PVC, or they can do vendor specific things to rapidly fill a volume with data
if the volume was provisioned by a specific CSI driver from the same vendor, for
example, by communicating directly with the storage for that volume.
## The future
As this feature is still in alpha, we expect to update the out of tree controllers
with more tests and documentation. The community plans to eventually re-implement
the populator library as a sidecar, for ease of operations.
We hope to see some official community-supported populators for some widely-shared
use cases. Also, we expect that volume populators will be used by backup vendors
as a way to "restore" backups to volumes, and possibly a standardized API to do
this will evolve.
## How can I learn more?
The enhancement proposal,
[Volume Populators](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators), includes lots of detail about the history and technical implementation
of this feature.
[Volume populators and data sources](/docs/concepts/storage/persistent-volumes/#volume-populators-and-data-sources), within the documentation topic about persistent volumes,
explains how to use this feature in your cluster.
Please get involved by joining the Kubernetes storage SIG to help us enhance this
feature. There are a lot of good ideas already and we'd be thrilled to have more!
title: 'Alpha in Kubernetes v1.22: API Server Tracing'
date: 2021-09-03
slug: api-server-tracing
---
**Authors:** David Ashpole (Google)
In distributed systems, it can be hard to figure out where problems are. You grep through one component's logs just to discover that the source of your problem is in another component. You search there only to discover that you need to enable debug logs to figure out what really went wrong... And it goes on. The more complex the path your request takes, the harder it is to answer questions about where it went. I've personally spent many hours doing this dance with a variety of Kubernetes components. Distributed tracing is a tool which is designed to help in these situations, and the Kubernetes API Server is, perhaps, the most important Kubernetes component to be able to debug. At Kubernetes' Sig Instrumentation, our mission is to make it easier to understand what's going on in your cluster, and we are happy to announce that distributed tracing in the Kubernetes API Server reached alpha in 1.22.
## What is Tracing?
Distributed tracing links together a bunch of super-detailed information from multiple different sources, and structures that telemetry into a single tree for that request. Unlike logging, which limits the quantity of data ingested by using log levels, tracing collects all of the details and uses sampling to collect only a small percentage of requests. This means that once you have a trace which demonstrates an issue, you should have all the information you need to root-cause the problem--no grepping for object UID required! My favorite aspect, though, is how useful the visualizations of traces are. Even if you don't understand the inner workings of the API Server, or don't have a clue what an etcd "Transaction" is, I'd wager you (yes, you!) could tell me roughly what the order of events was, and which components were involved in the request. If some step takes a long time, it is easy to tell where the problem is.
## Why OpenTelemetry?
It's important that Kubernetes works well for everyone, regardless of who manages your infrastructure, or which vendors you choose to integrate with. That is particularly true for Kubernetes' integrations with telemetry solutions. OpenTelemetry, being a CNCF project, shares these core values, and is creating exactly what we need in Kubernetes: A set of open standards for Tracing client library APIs and a standard trace format. By using OpenTelemetry, we can ensure users have the freedom to choose their backend, and ensure vendors have a level playing field. The timing couldn't be better: the OpenTelemetry golang API and SDK are very close to their 1.0 release, and will soon offer backwards-compatibility for these open standards.
## Why instrument the API Server?
The Kubernetes API Server is a great candidate for tracing for a few reasons:
* It follows the standard "RPC" model (serve a request by making requests to downstream components), which makes it easy to instrument.
* Users are latency-sensitive: If a request takes more than 10 seconds to complete, many clients will time-out.
* It has a complex service topology: A single request could require consulting a dozen webhooks, or involve multiple requests to etcd.
## Trying out APIServer Tracing with a webhook
### Enabling API Server Tracing
1. Enable the APIServerTracing [feature-gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/).
2. Set our configuration for tracing by pointing the `--tracing-config-file` flag on the kube-apiserver at our config file, which contains:
```yaml
apiVersion: apiserver.config.k8s.io/v1alpha1
kind: TracingConfiguration
# 1% sampling rate
samplingRatePerMillion: 10000
```
### Enabling Etcd Tracing
Add `--experimental-enable-distributed-tracing`, `--experimental-distributed-tracing-address=0.0.0.0:4317`, `--experimental-distributed-tracing-service-name=etcd` flags to etcd to enable tracing. Note that this traces every request, so it will probably generate a lot of traces if you enable it.
### Example Trace: List Nodes
I could've used any trace backend, but decided to use Jaeger, since it is one of the most popular open-source tracing projects. I deployed [the Jaeger All-in-one container](https://hub.docker.com/r/jaegertracing/all-in-one) in my cluster, deployed [the OpenTelemetry collector](https://github.com/open-telemetry/opentelemetry-collector) on my control-plane node ([example](https://github.com/dashpole/dashpole_demos/tree/master/otel/controlplane)), and captured traces like this one:
The teal lines are from the API Server, and includes it serving a request to `/api/v1/nodes`, and issuing a grpc `Range` RPC to ETCD. The yellow-ish line is from ETCD handling the `Range` RPC.
### Example Trace: Create Pod with Mutating Webhook
I instrumented the [example webhook](https://github.com/kubernetes-sigs/controller-runtime/tree/master/examples/builtins) with OpenTelemetry (I had to [patch](https://github.com/dashpole/controller-runtime/commit/85fdda7ba03dd2c22ef62c1a3dbdf5aa651f90da) controller-runtime, but it makes a neat demo), and routed traces to Jaeger as well. I collected traces like this one:
Compared with the previous trace, there are two new spans: A teal span from the API Server making a request to the admission webhook, and a brown span from the admission webhook serving the request. Even if you didn't instrument your webhook, you would still get the span from the API Server making the request to the webhook.
## Get involved!
As this is our first attempt at adding distributed tracing to a Kubernetes component, there is probably a lot we can improve! If my struggles resonated with you, or if you just want to try out the latest Kubernetes has to offer, please give the feature a try and open issues with any problem you encountered and ways you think the feature could be improved.
This is just the very beginning of what we can do with distributed tracing in Kubernetes. If there are other components you think would benefit from distributed tracing, or want to help bring API Server Tracing to GA, join sig-instrumentation at our [regular meetings](https://github.com/kubernetes/community/tree/master/sig-instrumentation#instrumentation-special-interest-group) and get involved!
title: "Introducing Single Pod Access Mode for PersistentVolumes"
date: 2021-09-13
slug: read-write-once-pod-access-mode-alpha
---
**Author:** Chris Henzie (Google)
Last month's release of Kubernetes v1.22 introduced a new ReadWriteOncePod access mode for [PersistentVolumes](/docs/concepts/storage/persistent-volumes/#persistent-volumes) and [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims).
With this alpha feature, Kubernetes allows you to restrict volume access to a single pod in the cluster.
## What are access modes and why are they important?
When using storage, there are different ways to model how that storage is consumed.
For example, a storage system like a network file share can have many users all reading and writing data simultaneously.
In other cases maybe everyone is allowed to read data but not write it.
For highly sensitive data, maybe only one user is allowed to read and write data but nobody else.
In the world of Kubernetes, [access modes](/docs/concepts/storage/persistent-volumes/#access-modes) are the way you can define how durable storage is consumed.
These access modes are a part of the spec for PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs).
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: shared-cache
spec:
accessModes:
- ReadWriteMany # Allow many nodes to access shared-cache simultaneously.
resources:
requests:
storage: 1Gi
```
Before v1.22, Kubernetes offered three access modes for PVs and PVCs:
- ReadWriteOnce – the volume can be mounted as read-write by a single node
- ReadOnlyMany – the volume can be mounted read-only by many nodes
- ReadWriteMany – the volume can be mounted as read-write by many nodes
These access modes are enforced by Kubernetes components like the `kube-controller-manager` and `kubelet` to ensure only certain pods are allowed to access a given PersistentVolume.
## What is this new access mode and how does it work?
Kubernetes v1.22 introduced a fourth access mode for PVs and PVCs, that you can use for CSI volumes:
- ReadWriteOncePod – the volume can be mounted as read-write by a single pod
If you create a pod with a PVC that uses the ReadWriteOncePod access mode, Kubernetes ensures that pod is the only pod across your whole cluster that can read that PVC or write to it.
If you create another pod that references the same PVC with this access mode, the pod will fail to start because the PVC is already in use by another pod.
For example:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 1s default-scheduler 0/1 nodes are available: 1 node has pod using PersistentVolumeClaim with the same name and ReadWriteOncePod access mode.
```
### How is this different than the ReadWriteOnce access mode?
The ReadWriteOnce access mode restricts volume access to a single *node*, which means it is possible for multiple pods on the same node to read from and write to the same volume.
This could potentially be a major problem for some applications, especially if they require at most one writer for data safety guarantees.
With ReadWriteOncePod these issues go away.
Set the access mode on your PVC, and Kubernetes guarantees that only a single pod has access.
## How do I use it?
The ReadWriteOncePod access mode is in alpha for Kubernetes v1.22 and is only supported for CSI volumes.
As a first step you need to enable the ReadWriteOncePod [feature gate](/docs/reference/command-line-tools-reference/feature-gates) for `kube-apiserver`, `kube-scheduler`, and `kubelet`.
You can enable the feature by setting command line arguments:
```
--feature-gates="...,ReadWriteOncePod=true"
```
You also need to update the following CSI sidecars to these versions or greater:
In order to use the ReadWriteOncePod access mode for your PVs and PVCs, you will need to create a new PVC with the access mode:
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: single-writer-only
spec:
accessModes:
- ReadWriteOncePod # Allow only a single pod to access single-writer-only.
resources:
requests:
storage: 1Gi
```
If your storage plugin supports [dynamic provisioning](/docs/concepts/storage/dynamic-provisioning/), new PersistentVolumes will be created with the ReadWriteOncePod access mode applied.
#### Migrating existing PersistentVolumes
If you have existing PersistentVolumes, they can be migrated to use ReadWriteOncePod.
In this example, we already have a "cat-pictures-pvc" PersistentVolumeClaim that is bound to a "cat-pictures-pv" PersistentVolume, and a "cat-pictures-writer" Deployment that uses this PersistentVolumeClaim.
As a first step, you need to edit your PersistentVolume's `spec.persistentVolumeReclaimPolicy` and set it to `Retain`.
This ensures your PersistentVolume will not be deleted when we delete the corresponding PersistentVolumeClaim:
Next you need to stop any workloads that are using the PersistentVolumeClaim bound to the PersistentVolume you want to migrate, and then delete the PersistentVolumeClaim.
Once that is done, you need to clear your PersistentVolume's `spec.claimRef.uid` to ensure PersistentVolumeClaims can bind to it upon recreation:
You can read [Configure a Pod to Use a PersistentVolume for Storage](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/) for more details on working with PersistentVolumes and PersistentVolumeClaims.
## What volume plugins support this?
The only volume plugins that support this are CSI drivers.
SIG Storage does not plan to support this for in-tree plugins because they are being deprecated as part of [CSI migration](/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/#what-is-the-timeline-status).
Support may be considered for beta for users that prefer to use the legacy in-tree volume APIs with CSI migration enabled.
## As a storage vendor, how do I add support for this access mode to my CSI driver?
The ReadWriteOncePod access mode will work out of the box without any required updates to CSI drivers, but [does require updates to CSI sidecars](#update-your-csi-sidecars).
With that being said, if you would like to stay up to date with the latest changes to the CSI specification (v1.5.0+), read on.
Two new access modes were introduced to the CSI specification in order to disambiguate the legacy [`SINGLE_NODE_WRITER`](https://github.com/container-storage-interface/spec/blob/v1.5.0/csi.proto#L418-L420) access mode.
They are [`SINGLE_NODE_SINGLE_WRITER` and `SINGLE_NODE_MULTI_WRITER`](https://github.com/container-storage-interface/spec/blob/v1.5.0/csi.proto#L437-L447).
In order to communicate to sidecars (like the [external-provisioner](https://github.com/kubernetes-csi/external-provisioner)) that your driver understands and accepts these two new CSI access modes, your driver will also need to advertise the `SINGLE_NODE_MULTI_WRITER` capability for the [controller service](https://github.com/container-storage-interface/spec/blob/v1.5.0/csi.proto#L1073-L1081) and [node service](https://github.com/container-storage-interface/spec/blob/v1.5.0/csi.proto#L1515-L1524).
If you'd like to read up on the motivation for these access modes and capability bits, you can also read the [CSI Specification Changes, Volume Capabilities](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/2485-read-write-once-pod-pv-access-mode/README.md#csi-specification-changes-volume-capabilities) section of KEP-2485 (ReadWriteOncePod PersistentVolume Access Mode).
### Update your CSI driver to use the new interface
As a first step you will need to update your driver's `container-storage-interface` dependency to v1.5.0+, which contains support for these new access modes and capabilities.
### Accept new CSI access modes
If your CSI driver contains logic for validating CSI access modes for requests , it may need updating.
If it currently accepts `SINGLE_NODE_WRITER`, it should be updated to also accept `SINGLE_NODE_SINGLE_WRITER` and `SINGLE_NODE_MULTI_WRITER`.
Using the [GCP PD CSI driver validation logic](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/v1.2.2/pkg/gce-pd-csi-driver/utils.go#L116-L130) as an example, here is how it can be extended:
case csi.VolumeCapability_AccessMode_SINGLE_NODE_READER_ONLY:
case csi.VolumeCapability_AccessMode_MULTI_NODE_READER_ONLY:
case csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER:
+ case csi.VolumeCapability_AccessMode_SINGLE_NODE_SINGLE_WRITER:
+ case csi.VolumeCapability_AccessMode_SINGLE_NODE_MULTI_WRITER:
default:
return fmt.Errorf("%v access mode is not supported for for PD", am.GetMode())
}
```
### Advertise new CSI controller and node service capabilities
Your CSI driver will also need to return the new `SINGLE_NODE_MULTI_WRITER` capability as part of the `ControllerGetCapabilities` and `NodeGetCapabilities` RPCs.
Using the [GCP PD CSI driver capability advertisement logic](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/v1.2.2/pkg/gce-pd-csi-driver/gce-pd-driver.go#L54-L77) as an example, here is how it can be extended:
The CSI spec outlines expected behavior for the `NodePublishVolume` RPC when called more than once for the same volume but with different arguments (like the target path).
Please refer to [the second table in the NodePublishVolume section of the CSI spec](https://github.com/container-storage-interface/spec/blob/v1.5.0/spec.md#nodepublishvolume) for more details on expected behavior when implementing in your driver.
### Update your CSI sidecars
When deploying your CSI drivers, you must update the following CSI sidecars to versions that depend on CSI spec v1.5.0+ and the Kubernetes v1.22 API.
As part of the beta graduation for this feature, SIG Storage plans to update the Kubenetes scheduler to support pod preemption in relation to ReadWriteOncePod storage.
This means if two pods request a PersistentVolumeClaim with ReadWriteOncePod, the pod with highest priority will gain access to the PersistentVolumeClaim and any pod with lower priority will be preempted from the node and be unable to access the PersistentVolumeClaim.
## How can I learn more?
Please see [KEP-2485](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/2485-read-write-once-pod-pv-access-mode/README.md) for more details on the ReadWriteOncePod access mode and motivations for CSI spec changes.
## How do I get involved?
The [Kubernetes #csi Slack channel](https://kubernetes.slack.com/messages/csi) and any of the [standard SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and the CSI teams.
Special thanks to the following people for their insightful reviews and design considerations:
* Abdullah Gharaibeh (ahg-g)
* Aldo Culquicondor (alculquicondor)
* Ben Swartzlander (bswartz)
* Deep Debroy (ddebroy)
* Hemant Kumar (gnufied)
* Humble Devassy Chirammal (humblec)
* James DeFelice (jdef)
* Jan Šafránek (jsafrane)
* Jing Xu (jingxu97)
* Jordan Liggitt (liggitt)
* Michelle Au (msau42)
* Saad Ali (saad-ali)
* Tim Hockin (thockin)
* Xing Yang (xing-yang)
If you’re interested in getting involved with the design and development of CSI or any part of the Kubernetes storage system, join the [Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG).
We’re rapidly growing and always welcome new contributors.
In Kubernetes, a _Node_ is a representation of a single machine in your cluster. [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) owns that very important Node component and supports various subprojects such as Kubelet, Container Runtime Interface (CRI) and more to support how the pods and host resources interact. In this blog, we have summarized our conversation with [Elana Hashman (EH)](https://twitter.com/ehashdn) & [Sergey Kanzhelev (SK)](https://twitter.com/SergeyKanzhelev), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved.
## A summary of our conversation
### Could you tell us a little about what SIG Node does?
SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIG's focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. This SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability.
### "SIG Node is a vertical SIG" could you explain a bit more?
EH: There are two kinds of SIGs: horizontal and vertical. Horizontal SIGs are concerned with a particular function of every component in Kubernetes: for example, SIG Security considers security aspects of every component in Kubernetes, or SIG Instrumentation looks at the logs, metrics, traces and events of every component in Kubernetes. Such SIGs don't tend to own a lot of code.
Vertical SIGs, on the other hand, own a single component, and are responsible for approving and merging patches to that code base. SIG Node owns the "Node" vertical, pertaining to the kubelet and its lifecycle. This includes the code for the kubelet itself, as well as the node controller, the container runtime interface, and related subprojects like the node problem detector.
### How did the CI subproject start? Is this specific to SIG Node and how does it help the SIG?
SK: The subproject started as a follow up after one of the releases was blocked by numerous test failures of critical tests. These tests haven’t started falling all at once, rather continuous lack of attention led to slow degradation of tests quality. SIG Node was always prioritizing quality and reliability, and forming of the subproject was a way to highlight this priority.
### As the 3rd largest SIG in terms of number of issues and PRs, how does your SIG juggle so much work?
EH: It helps to be organized. When I increased my contributions to the SIG in January of 2021, I found myself overwhelmed by the volume of pull requests and issues and wasn't sure where to start. We were already tracking test-related issues and pull requests on the CI subproject board, but that was missing a lot of our bugfixes and feature work. So I began putting together a triage board for the rest of our pull requests, which allowed me to sort each one by status and what actions to take, and documented its use for other contributors. We closed or merged over 500 issues and pull requests tracked by our two boards in each of the past two releases. The Kubernetes devstats showed that we have significantly increased our velocity as a result.
In June, we ran our first bug scrub event to work through the backlog of issues filed against SIG Node, ensuring they were properly categorized. We closed over 130 issues over the course of this 48 hour global event, but as of writing we still have 333 open issues.
### Why should new and existing contributors consider joining SIG Node?
SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you the foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster.
SIG Node maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support.
### What do you do to help new contributors get started?
EH: Getting started in SIG Node can be intimidating, since there is so much work to be done, our SIG meetings are very large, and it can be hard to find a place to start.
I always encourage new contributors to work on things that they have some investment in already. In SIG Node, that might mean volunteering to help fix a bug that you have personally been affected by, or helping to triage bugs you care about by priority.
To come up to speed on any open source code base, there are two strategies you can take: start by exploring a particular issue deeply, and follow that to expand the edges of your knowledge as needed, or briefly review as many issues and change requests as you possibly can to get a higher level picture of how the component works. Ultimately, you will need to do both if you want to become a Node reviewer or approver.
[Davanum Srinivas](https://twitter.com/dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG.
### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn?
SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads.
The other way of contribution is to help document SIG node features. This type of contribution requires a deep understanding of features, and ability to explain them in simple terms.
Finally, we are always looking for feedback on how best to run your workload. Come and explain specifics of it, and what features in SIG Node components may help to run it better.
### What are you getting positive feedback on, and what’s coming up next for SIG Node?
EH: Over the past year SIG Node has adopted some new processes to help manage our feature development and Kubernetes enhancement proposals, and other SIGs have looked to us for inspiration in managing large workloads. I hope that this is an area we can continue to provide leadership in and further iterate on.
We have a great balance of new features and deprecations in flight right now. Deprecations of unused or difficult to maintain features help us keep technical debt and maintenance load under control, and examples include the dockershim and DynamicKubeletConfiguration deprecations. New features will unlock additional functionality in end users' clusters, and include exciting features like support for cgroups v2, swap memory, graceful node shutdowns, and device management policies.
### Any closing thoughts/resources you’d like to share?
SK/EH: It takes time and effort to get to any open source community. SIG Node may overwhelm you at first with the number of participants, volume of work, and project scope. But it is totally worth it. Join our welcoming community! [SIG Node GitHub Repo](https://github.com/kubernetes/community/tree/master/sig-node) contains many useful resources including Slack, mailing list and other contact info.
## Wrap Up
SIG Node hosted a [KubeCon + CloudNativeCon Europe 2021 talk](https://www.youtube.com/watch?v=z5aY4e2RENA) with an intro and deep dive to their awesome SIG. Join the SIG's meetings to find out about the most recent research results, what the plans are for the forthcoming year, and how to get involved in the upstream Node team as a contributor!
{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png" alt="Screenshot of a terminal showing volume snapshot creation on GCP" title="GCP snapshot creation" >}}
GCP identifies the disk by its PVC name, so it’s direct mapping. In AWS, you need to
find volume by the CSIVolumeName AWS tag with PVC name value first that will be used for snapshot creation.
{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png" alt="Screenshot of AWS web console, showing EBS volume identification" title="Identify disk ID on AWS" >}}
Mark done Volume (volume-id) ```vol-00c7ecd873c6fb3ec``` and ether create EBS snapshot in AWS Console, or use ```aws cli```.
on August 3rd, 2021. The guidance details threats to Kubernetes environments
and provides secure configuration guidance to minimize risk.
The following sections of this blog correlate to the sections in the NSA/CISA guidance.
Any missing sections are skipped because of limited opportunities to add
anything new to the existing content.
_Note_: This blog post is not a substitute for reading the guide. Reading the published
guidance is recommended before proceeding as the following content is
complementary.
## Introduction and Threat Model
Note that the threats identified as important by the NSA/CISA, or the intended audience of this guidance, may be different from the threats that other enterprise users of Kubernetes consider important. This section
is still useful for organizations that care about data, resource theft and
service unavailability.
The guidance highlights the following three sources of compromises:
- Supply chain risks
- Malicious threat actors
- Insider threats (administrators, users, or cloud service providers)
The [threat model](https://en.wikipedia.org/wiki/Threat_model) tries to take a step back and review threats that not only
exist within the boundary of a Kubernetes cluster but also include the underlying
infrastructure and surrounding workloads that Kubernetes does not manage.
For example, when a workload outside the cluster shares the same physical
network, it has access to the kubelet and to control plane components: etcd, controller manager, scheduler and API
server. Therefore, the guidance recommends having network level isolation
separating Kubernetes clusters from other workloads that do not need connectivity
to Kubernetes control plane nodes. Specifically, scheduler, controller-manager,
etcd only need to be accessible to the API server. Any interactions with Kubernetes
from outside the cluster can happen by providing access to API server port.
List of ports and protocols for each of these components are
defined in [Ports and Protocols](/docs/reference/ports-and-protocols/)
within the Kubernetes documentation.
> Special note: kube-scheduler and kube-controller-manager uses different ports than the ones mentioned in the guidance
The [Threat modelling](https://cnsmap.netlify.app/threat-modelling) section
from the CNCF [Cloud Native Security Whitepaper + Map](https://github.com/cncf/tag-security/tree/main/security-whitepaper)
provides another perspective on approaching threat modelling Kubernetes, from a
cloud native lens.
## Kubernetes Pod security
Kubernetes by default does not guarantee strict workload isolation between pods
running in the same node in a cluster. However, the guidance provides several
techniques to enhance existing isolation and reduce the attack surface in case of a
compromise.
### "Non-root" containers and "rootless" container engines
Several best practices related to basic security principle of least privilege
i.e. provide only the permissions are needed; no more, no less, are worth a
second look.
The guide recommends setting non-root user at build time instead of relying on
setting `runAsUser` at runtime in your Pod spec. This is a good practice and provides
some level of defense in depth. For example, if the container image is built with user `10001`
and the Pod spec misses adding the `runAsuser` field in its `Deployment` object. In this
case there are certain edge cases that are worth exploring for awareness:
1. Pods can fail to start, if the user defined at build time is different from
the one defined in pod spec and some files are as a result inaccessible.
2. Pods can end up sharing User IDs unintentionally. This can be problematic
even if the User IDs are non-zero in a situation where a container escape to
host file system is possible. Once the attacker has access to the host file
system, they get access to all the file resources that are owned by other
unrelated pods that share the same UID.
3. Pods can end up sharing User IDs, with other node level processes not managed
by Kubernetes e.g. node level daemons for auditing, vulnerability scanning,
telemetry. The threat is similar to the one above where host file system
access can give attacker full access to these node level daemons without
needing to be root on the node.
However, none of these cases will have as severe an impact as a container
running as root being able to escape as a root user on the host, which can provide
an attacker with complete control of the worker node, further allowing lateral
movement to other worker or control plane nodes.
Kubernetes 1.22 introduced
an [alpha feature](/docs/tasks/administer-cluster/kubelet-in-userns/)
that specifically reduces the impact of such a control plane component running
as root user to a non-root user through user namespaces.
That ([alpha stage](/docs/reference/command-line-tools-reference/feature-gates/#feature-stages)) support for user namespaces / rootless mode is available with
The NSA/CISA Kubernetes Hardening Guidance highlights an often overlooked feature `readOnlyRootFileSystem`, with a
working example in [Appendix B](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=42). This example limits execution and tampering of
containers at runtime. Any read/write activity can then be limited to few
directories by using `tmpfs` volume mounts.
However, some applications that modify the container filesystem at runtime, like exploding a WAR or JAR file at container startup,
could face issues when enabling this feature. To avoid this issue, consider making minimal changes to the filesystem at runtime
when possible.
### Building secure container images
Kubernetes Hardening Guidance also recommends running a scanner at deploy time as an admission controller,
to prevent vulnerable or misconfigured pods from running in the cluster.
Theoretically, this sounds like a good approach but there are several caveats to
consider before this can be implemented in practice:
- Depending on network bandwidth, available resources and scanner of choice,
scanning for vulnerabilities for an image can take an indeterminate amount of
time. This could lead to slower or unpredictable pod start up times, which
could result in spikes of unavailability when apps are serving peak load.
- If the policy that allows or denies pod startup is made using incorrect or
incomplete data it could result in several false positive or false negative
outcomes like the following:
- inside a container image, the `openssl` package is detected as vulnerable. However,
the application is written in Golang and uses the Go `crypto` package for TLS. Therefore, this vulnerability
is not in the code execution path and as such has minimal impact if it
remains unfixed.
- A vulnerability is detected in the `openssl` package for a Debian base image.
However, the upstream Debian community considers this as a Minor impact
vulnerability and as a result does not release a patch fix for this
vulnerability. The owner of this image is now stuck with a vulnerability that
cannot be fixed and a cluster that does not allow the image to run because
of predefined policy that does not take into account whether the fix for a
vulnerability is available or not
- A Golang app is built on top of a [distroless](https://github.com/GoogleContainerTools/distroless)
image, but it is compiled with a Golang version that uses a vulnerable [standard library](https://pkg.go.dev/std).
The scanner has
no visibility into golang version but only on OS level packages. So it
allows the pod to run in the cluster in spite of the image containing an
app binary built on vulnerable golang.
To be clear, relying on vulnerability scanners is absolutely a good idea but
policy definitions should be flexible enough to allow:
- Creation of exception lists for images or vulnerabilities through labelling
- Overriding the severity with a risk score based on impact of a vulnerability
- Applying the same policies at build time to catch vulnerable images with
fixable vulnerabilities before they can be deployed into Kubernetes clusters
Special considerations like offline vulnerability database fetch, may also be
needed, if the clusters run in an air-gapped environment and the scanners
require internet access to update the vulnerability database.
### Pod Security Policies
Since Kubernetes v1.21, the [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/)
API and related features are [deprecated](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/),
but some of the guidance in this section will still apply for the next few years, until cluster operators
upgrade their clusters to newer Kubernetes versions.
The Kubernetes project is working on a replacement for PodSecurityPolicy.
Kubernetes v1.22 includes an alpha feature called [Pod Security Admission](/docs/concepts/security/pod-security-admission/)
that is intended to allow enforcing a minimum level of isolation between pods.
The built-in isolation levels for Pod Security Admission are derived
from [Pod Security Standards](/docs/concepts/security/pod-security-standards/), which is a superset of all the components mentioned in Table I [page 10](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=17) of
the guidance.
Information about migrating from PodSecurityPolicy to the Pod Security
Admission feature is available
in
[Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/).
One important behavior mentioned in the guidance that remains the same between
Pod Security Policy and its replacement is that enforcing either of them does
not affect pods that are already running. With both PodSecurityPolicy and Pod Security Admission,
the enforcement happens during the pod creation
stage.
### Hardening container engines
Some container workloads are less trusted than others but may need to run in the
same cluster. In those cases, running them on dedicated nodes that include
hardened container runtimes that provide stricter pod isolation boundaries can
act as a useful security control.
Kubernetes supports
an API called [RuntimeClass](/docs/concepts/containers/runtime-class/) that is
stable / GA (and, therefore, enabled by default) stage as of Kubernetes v1.20.
RuntimeClass allows you to ensure that Pods requiring strong isolation are scheduled onto
nodes that can offer it.
Some third-party projects that you can use in conjunction with RuntimeClass are:
only allows creation of services that don't require external IPs or whose
external IPs are within the range specified by the administrator.
> CVE-2020-8554 - Kubernetes API server in all versions allow an attacker
> who is able to create a ClusterIP service and set the `spec.externalIPs` field,
> to intercept traffic to that IP address. Additionally, an attacker who is able to
> patch the `status` (which is considered a privileged operation and should not
> typically be granted to users) of a LoadBalancer service can set the
> `status.loadBalancer.ingress.ip` to similar effect.
### Resource Policies
In addition to configuring ResourceQuotas and limits, consider restricting how many process
IDs (PIDs) a given Pod can use, and also to reserve some PIDs for node-level use to avoid
resource exhaustion. More details to apply these limits can be
found in [Process ID Limits And Reservations](/docs/concepts/policy/pid-limiting/).
### Control Plane Hardening
In the next section, the guide covers control plane hardening. It is worth
noting that
from [Kubernetes 1.20](https://github.com/kubernetes/kubernetes/issues/91506),
insecure port from API server, has been removed.
### Etcd
As a general rule, the etcd server should be configured to only trust
certificates assigned to the API server. It limits the attack surface and prevents a
malicious attacker from gaining access to the cluster. It might be beneficial to
use a separate CA for etcd, as it by default trusts all the certificates issued
by the root CA.
### Kubeconfig Files
In addition to specifying the token and certificates directly, `.kubeconfig`
supports dynamic retrieval of temporary tokens using auth provider plugins.
Beware of the possibility of malicious
shell [code execution](https://banzaicloud.com/blog/kubeconfig-security/) in a
`kubeconfig` file. Once attackers gain access to the cluster, they can steal ssh
keys/secrets or more.
### Secrets
Kubernetes [Secrets](/docs/concepts/configuration/secret/) is the native way of managing secrets as a Kubernetes
API object. However, in some scenarios such as a desire to have a single source of truth for all app secrets, irrespective of whether they run on Kubernetes or not, secrets can be managed loosely coupled with
Kubernetes and consumed by pods through side-cars or init-containers with minimal usage of Kubernetes Secrets API.
and [csi-secrets-store](https://github.com/kubernetes-sigs/secrets-store-csi-driver)
are some of these alternatives to Kubernetes Secrets
## Log Auditing
The NSA/CISA guidance stresses monitoring and alerting based on logs. The key points
include logging at the host level, application level, and on the cloud. When
running Kubernetes in production, it's important to understand who's
responsible, and who's accountable, for each layer of logging.
### Kubernetes API auditing
One area that deserves more focus is what exactly should alert or be logged. The
document outlines a sample policy in [Appendix L: Audit Policy](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=55) that logs all
RequestResponse's including metadata and request / response bodies. While helpful for a demo, it may not be practical for production.
Each organization needs to evaluate their
own threat model and build an audit policy that complements or helps troubleshooting incident response. Think
about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug-application-cluster/audit/#audit-policy).
It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point.
Audit logging configurations can also be tested with
kind following these [instructions](https://kind.sigs.k8s.io/docs/user/auditing).
### Streaming logs and auditing
Logging is important for threat and anomaly detection. As the document outlines,
it's a best practice to scan and alert on logs as close to real time as possible
and to protect logs from tampering if a compromise occurs. It's important to
reflect on the various levels of logging and identify the critical areas such as
API endpoints.
Kubernetes API audit logging can stream to a webhook and there's an example in [Appendix N: Webhook configuration](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=58). Using a webhook could be a method that
stores logs off cluster and/or centralizes all audit logs. Once logs are
centrally managed, look to enable alerting based on critical events. Also ensure
you understand what the baseline is for normal activities.
### Alert identification
While the guide stressed the importance of notifications, there is not a blanket
event list to alert from. The alerting requirements vary based on your own
requirements and threat model. Examples include the following events:
- Changes to the `securityContext` of a Pod
- Updates to admission controller configs
- Accessing certain files / URLs
### Additional logging resources
- [Seccomp Security Profiles and You: A Practical Guide - Duffie Cooley](https://www.youtube.com/watch?v=OPuu8wsu2Zc)
- [TGI Kubernetes 119: Gatekeeper and OPA](https://www.youtube.com/watch?v=ZJgaGJm9NJE)
- [Abusing The Lack of Kubernetes Auditing Policies](https://www.lacework.com/blog/hiding-in-plaintext-sight-abusing-the-lack-of-kubernetes-auditing-policies/)
- [Enable seccomp for all workloads with a new v1.22 alpha feature](https://kubernetes.io/blog/2021/08/25/seccomp-default/)
- [This Week in Cloud Native: Auditing / Pod Security](https://www.twitch.tv/videos/1147889860)
## Upgrading and Application Security practices
Kubernetes releases three times per year, so upgrade-related toil is a common problem for
people running production clusters. In addition to this, operators must
regularly upgrade the underlying node's operating system and running
applications. This is a best practice to ensure continued support and to reduce
the likelihood of bugs or vulnerabilities.
Kubernetes supports the three most recent stable releases. While each Kubernetes
release goes through a large number of tests before being published, some
teams aren't comfortable running the latest stable release until some time has
passed. No matter what version you're running, ensure that patch upgrades
happen frequently or automatically. More information can be found in
the [version skew](/releases/version-skew-policy/) policy
pages.
When thinking about how you'll manage node OS upgrades, consider ephemeral
nodes. Having the ability to destroy and add nodes allows your team to respond
quicker to node issues. In addition, having deployments that tolerate node
instability (and a culture that encourages frequent deployments) allows for
easier cluster upgrades.
Additionally, it's worth reiterating from the guidance that periodic
vulnerability scans and penetration tests can be performed on the various system
components to proactively look for insecure configurations and vulnerabilities.
### Finding release & security information
To find the most recent Kubernetes supported versions, refer to
[https://k8s.io/releases](https://k8s.io/releases), which includes minor versions. It's good to stay up to date with
your minor version patches.
If you're running a managed Kubernetes offering, look for their release
documentation and find their various security channels.
Subscribe to
the [Kubernetes Announce mailing list](https://groups.google.com/g/kubernetes-announce).
The Kubernetes Announce mailing list is searchable for terms such
as "[Security Advisories](https://groups.google.com/g/kubernetes-announce/search?q=%5BSecurity%20Advisory%5D)".
You can set up alerts and email notifications as long as you know what key
words to alert on.
## Conclusion
In summary, it is fantastic to see security practitioners sharing this
level of detailed guidance in public. This guidance further highlights
Kubernetes going mainstream and how securing Kubernetes clusters and the
application containers running on Kubernetes continues to need attention and focus of
practitioners. Only a few weeks after the guidance was published, an open source
tool [kubescape](https://github.com/armosec/kubescape) to validate cluster
against this guidance became available.
This tool can be a great starting point to check the current state of your
clusters, after which you can use the information in this blog post and in the guidance to assess
where improvements can be made.
Finally, it is worth reiterating that not all controls in this guidance will
make sense for all practitioners. The best way to know which controls matter is
to rely on the threat model of your own Kubernetes environment.
_A special shout out and thanks to Rory McCune (@raesene) for his inputs to this blog post_
title: "Introducing ClusterClass and Managed Topologies in Cluster API"
date: 2021-10-08
slug: capi-clusterclass-and-managed-topologies
---
**Author:** Fabrizio Pandini (VMware)
The [Cluster API community](https://cluster-api.sigs.k8s.io/) is happy to announce the implementation of *ClusterClass and Managed Topologies*, a new feature that will greatly simplify how you can provision, upgrade, and operate multiple Kubernetes clusters in a declarative way.
## A little bit of context…
Before getting into the details, let's take a step back and look at the history of Cluster API.
The [Cluster API project](https://github.com/kubernetes-sigs/cluster-api/) started three years ago, and the first releases focused on extensibility and implementing a declarative API that allows a seamless experience across infrastructure providers. This was a success with many cloud providers: AWS, Azure, Digital Ocean, GCP, Metal3, vSphere and still counting.
With extensibility addressed, the focus shifted to features, like automatic control plane and etcd management, health-based machine remediation, machine rollout strategies and more.
Fast forwarding to 2021, with lots of companies using Cluster API to manage fleets of Kubernetes clusters running workloads in production, the community focused its effort on stabilization of both code, APIs, documentation, and on extensive test signals which inform Kubernetes releases.
With solid foundations in place, and a vibrant and welcoming community that still continues to grow, it was time to plan another iteration on our UX for both new and advanced users.
Enter ClusterClass and Managed Topologies, tada!
## ClusterClass
As the name suggests, ClusterClass and managed topologies are built in two parts.
The idea behind ClusterClass is simple: define the shape of your cluster once, and reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away.
ClusterClass, at its heart, is a collection of Cluster and Machine templates. You can use it as a “stamp” that can be leveraged to create many clusters of a similar shape.
The possibilities are endless; you can get a default ClusterClass from the community, “off-the-shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios.
## Managed Topologies
Managed Topologies let you put the power of ClusterClass into action.
Given a ClusterClass, you can create many Clusters of a similar shape by providing a single resource, the Cluster.
Here is an example:
```yaml
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: my-amazing-cluster
namespace: bar
spec:
topology: # define a managed topology
class: my-amazing-cluster-class # use the ClusterClass mentioned earlier
version: v1.21.2
controlPlane:
replicas: 3
workers:
machineDeployments:
- class: type1-workers
name: big-pool-of-machines
replicas: 5
- class: type2-workers
name: small-pool-of-machines
replicas: 1
```
But there is more than simplified cluster creation. Now the Cluster acts as a single control point for your entire topology.
All the power of Cluster API, extensibility, lifecycle automation, stability, all the features required for managing an enterprise grade Kubernetes cluster on the infrastructure provider of your choice are now at your fingertips: you can create your Cluster, add new machines, upgrade to the next Kubernetes version, and all from a single place.
It is just as simple as it looks!
## What’s next
While the amazing Cluster API community is working hard to deliver the first version of ClusterClass and managed topologies later this year, we are already looking forward to what comes next for the project and its ecosystem.
There are a lot of great ideas and opportunities ahead!
We want to make managed topologies even more powerful and flexible, allowing users to dynamically change bits of a ClusterClass according to the specific needs of a Cluster; this will ensure the same simple and intuitive UX for solving complex problems like e.g. selecting machine image for a specific Kubernetes version and for a specific region of your infrastructure provider, or injecting proxy configurations in the entire Cluster, and so on.
Stay tuned for what comes next, and if you have any questions, comments or suggestions:
* Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T)
* Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents
* Join our [Zoom meeting](https://zoom.us/j/861487554), every Wednesday at 10:00 Pacific Time
* Check out the [ClusterClass tutorial](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-classes.html) in the Cluster API book.
Assuming the JSON output above is stored in `/tmp/out` ([jq](https://stedolan.github.io/jq/) is an *awesome* program!).
As this is an example we make it really simple for ourselves by using
a minor variation of the `kpng-json` backend above. Instead of just
printing, a program is called and the JSON output is passed as `stdin`
to that program. The backend can be tested stand-alone:
```
CALLOUT=jq kpng-callout
```
Where `jq` can be replaced with your own program or script. A script
may look like the example above. For more info and the complete
example please see [https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
## Summary
While [kpng](https://github.com/kubernetes-sigs/kpng) is in early
stage of development this post wants to show how you may build your
own specialized K8s proxiers in the future. The only thing your
applications need to do is to add the
`service.kubernetes.io/service-proxy-name` label in the Service
manifest.
It is a tedious process to get new features into the `kube-proxy` and
it is not unlikely that they will be rejected, so to write a
title: "Announcing the 2021 Steering Committee Election Results"
date: 2021-11-08
slug: steering-committee-results-2021
---
**Author**: Kaslin Fields
The [2021 Steering Committee Election](https://github.com/kubernetes/community/tree/master/events/elections/2021) is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2021. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.
This community body is significant since it oversees the governance of the entire Kubernetes project. With that great power comes great responsibility. You can learn more about the steering committee’s role in their [charter](https://github.com/kubernetes/steering/blob/master/charter.md).
## Results
Congratulations to the elected committee members whose two year terms begin immediately (listed in alphabetical order by GitHub handle):
* **Christoph Blecker ([@cblecker](https://github.com/cblecker)), Red Hat**
* **Stephen Augustus ([@justaugustus](https://github.com/justaugustus)), Cisco**
Special thanks to Arnaud Meukam ([@ameukam](https://github.com/ameukam)), k8s-infra liaison, who enabled our voting software on community-owned infrastructure.
Thanks to the Emeritus Steering Committee Members. Your prior service is appreciated by the community:
And thank you to all the candidates who came forward to run for election.
## Get Involved with the Steering Committee
This governing body, like all of Kubernetes, is open to all. You can follow along with Steering Committee [backlog items](https://github.com/kubernetes/steering/projects/1) and weigh in by filing an issue or creating a PR against their [repo](https://github.com/kubernetes/steering). They have an open meeting on [the first Monday at 9:30am PT of every month](https://github.com/kubernetes/steering) and regularly attend Meet Our Contributors. They can also be contacted at their public mailing list steering@kubernetes.io.
You can see what the Steering Committee meetings are all about by watching past meetings on the [YouTube Playlist](https://www.youtube.com/playlist?list=PL69nYSiGNLP1yP1B_nd9-drjoxp0Q14qM).
---
_This post was written by the [Upstream Marketing Working Group](https://github.com/kubernetes/community/tree/master/communication/marketing-team#contributor-marketing). If you want to write stories about the Kubernetes community, learn more about us._
The user/group ID related security settings in Pod's `securityContext` trigger a problem when users want to
deploy containers that use accelerator devices (via [Kubernetes Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)) on Linux. In this blog
post I talk about the problem and describe the work done so far to address it. It's not meant to be a long story about getting the [k/k issue](https://github.com/kubernetes/kubernetes/issues/92211) fixed.
Instead, this post aims to raise awareness of the issue and to highlight important device use-cases too. This is needed as Kubernetes works on new related features such as support for user namespaces.
## Why non-root containers can't use devices and why it matters
One of the key security principles for running containers in Kubernetes is the
principle of least privilege. The Pod/container `securityContext` specifies the config
options to set, e.g., Linux capabilities, MAC policies, and user/group ID values to achieve this.
Furthermore, the cluster admins are supported with tools like [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) (deprecated) or
[Pod Security Admission](/docs/concepts/security/pod-security-admission/) (alpha) to enforce the desired security settings for pods that are being deployed in
the cluster. These settings could, for instance, require that containers must be `runAsNonRoot` or
that they are forbidden from running with root's group ID in `runAsGroup` or `supplementalGroups`.
In Kubernetes, the kubelet builds the list of [`Device`](https://pkg.go.dev/k8s.io/cri-api@v0.22.1/pkg/apis/runtime/v1#Device) resources to be made available to a container
(based on inputs from the Device Plugins) and the list is included in the CreateContainer CRI message
sent to the CRI container runtime. Each `Device` contains little information: host/container device
paths and the desired devices cgroups permissions.
The [OCI Runtime Spec for Linux Container Configuration](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md)
expects that in addition to the devices cgroup fields, more detailed information about the devices
must be provided:
```yaml
{
"type": "<string>",
"path": "<string>",
"major": <int64>,
"minor": <int64>,
"fileMode": <uint32>,
"uid": <uint32>,
"gid": <uint32>
},
```
The CRI container runtimes (containerd, CRI-O) are responsible for obtaining this information
from the host for each `Device`. By default, the runtimes copy the host device's user and group IDs:
- `uid` (uint32, OPTIONAL) - id of device owner in the container namespace.
- `gid` (uint32, OPTIONAL) - id of device group in the container namespace.
Similarly, the runtimes prepare other mandatory `config.json` sections based on the CRI fields,
including the ones defined in `securityContext`: `runAsUser`/`runAsGroup`, which become part of the POSIX
platforms user structure via:
- `uid` (int, REQUIRED) specifies the user ID in the container namespace.
- `gid` (int, REQUIRED) specifies the group ID in the container namespace.
- `additionalGids` (array of ints, OPTIONAL) specifies additional group IDs in the container namespace to be added to the process.
However, the resulting `config.json` triggers a problem when trying to run containers with
both devices added and with non-root uid/gid set via `runAsUser`/`runAsGroup`: the container user process
has no permission to use the device even when its group id (gid, copied from host) was permissive to
non-root groups. This is because the container user does not belong to that host group (e.g., via `additionalGids`).
Being able to run applications that use devices as non-root user is normal and expected to work so that
the security principles can be met. Therefore, several alternatives were considered to get the gap filled with what the PodSec/CRI/OCI supports today.
## What was done to solve the issue?
You might have noticed from the problem definition that it would at least be possible to workaround
the problem by manually adding the device gid(s) to `supplementalGroups`, or in
the case of just one device, set `runAsGroup` to the device's group id. However, this is problematic because the device gid(s) may have
different values depending on the nodes' distro/version in the cluster. For example, with GPUs the following commands for different distros and versions return different gids:
Fedora 33:
```
$ ls -l /dev/dri/
total 0
drwxr-xr-x. 2 root root 80 19.10. 10:21 by-path
crw-rw----+ 1 root video 226, 0 19.10. 10:42 card0
Which number to choose in your `securityContext`? Also, what if the `runAsGroup`/`runAsUser` values cannot be hard-coded because
they are automatically assigned during pod admission time via external security policies?
Unlike volumes with `fsGroup`, the devices have no official notion of `deviceGroup`/`deviceUser` that the CRI runtimes (or kubelet)
would be able to use. We considered using container annotations set by the device plugins (e.g., `io.kubernetes.cri.hostDeviceSupplementalGroup/`) to get custom OCI `config.json` uid/gid values.
This would have required changes to all existing device plugins which was not ideal.
Instead, a solution that is *seamless* to end-users without getting the device plugin vendors involved was preferred. The selected approach was
to re-use `runAsUser` and `runAsGroup` values in `config.json` for devices:
```yaml
{
"type": "c",
"path": "/dev/foo",
"major": 123,
"minor": 4,
"fileMode": 438,
"uid": <runAsUser>,
"gid": <runAsGroup>
},
```
With `runc` OCI runtime (in non-rootless mode), the device is created (`mknod(2)`) in
the container namespace and the ownership is changed to `runAsUser`/`runAsGroup` using `chmod(2)`.
{{< note >}}
[Rootless mode](/docs/tasks/administer-cluster/kubelet-in-userns/) and devices is not supported.
{{</note>}}
Having the ownership updated in the container namespace is justified as the user process is the only one accessing the device. Only `runAsUser`/`runAsGroup`
are taken into account, and, e.g., the `USER` setting in the container is currently ignored.
While it is likely that the "faulty" deployments (i.e., non-root `securityContext` + devices) do not exist, to be absolutely sure no
deployments break, an opt-in config entry in both containerd and CRI-O to enable the new behavior was added. The following:
`device_ownership_from_security_context (bool)`
defaults to `false` and must be enabled to use the feature.
## See non-root containers using devices after the fix
To demonstrate the new behavior, let's use a Data Plane Development Kit (DPDK) application using hardware accelerators, Kubernetes CPU manager, and HugePages as an example. The cluster runs containerd with:
```toml
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
device_ownership_from_security_context = true
```
or CRI-O with:
```toml
[crio.runtime]
device_ownership_from_security_context = true
```
and the `Guaranteed` QoS Class Pod that runs DPDK's crypto-perf test utility with this YAML:
```yaml
...
metadata:
name: qat-dpdk
spec:
securityContext:
runAsUser: 1000
runAsGroup: 2000
fsGroup: 3000
containers:
- name: crypto-perf
image: intel/crypto-perf:devel
...
resources:
requests:
cpu: "3"
memory: "128Mi"
qat.intel.com/generic: '4'
hugepages-2Mi: "128Mi"
limits:
cpu: "3"
memory: "128Mi"
qat.intel.com/generic: '4'
hugepages-2Mi: "128Mi"
...
```
To verify the results, check the user and group ID that the container runs as:
```
$ kubectl exec -it qat-dpdk -c crypto-perf -- id
```
They are set to non-zero values as expected:
```
uid=1000 gid=2000 groups=2000,3000
```
Next, check the device node permissions (`qat.intel.com/generic` exposes `/dev/vfio/` devices) are accessible to `runAsUser`/`runAsGroup`:
`fsGroup` gives a `runAsUser` writable HugePages emptyDir mountpoint:
```
total 0
drwxrwsr-x 2 root 3000 0 Sep 7 10:55 .
drwxr-xr-x 7 root root 380 Sep 7 10:55 ..
```
## Help us test it and provide feedback!
The functionality described here is expected to help with cluster security and the configurability of device permissions. To allow
non-root containers to use devices requires cluster admins to opt-in to the functionality by setting
`device_ownership_from_security_context = true`. To make it a default setting, please test it and provide your feedback (via SIG-Node meetings or issues)!
The flag is available in CRI-O v1.22 release and queued for containerd v1.6.
More work is needed to get it *properly* supported. It is known to work with `runc` but it also needs to be made to function
with other OCI runtimes too, where applicable. For instance, Kata Containers supports device passthrough and allows it to make devices
available to containers in VM sandboxes too.
Moreover, the additional challenge comes with support of user names and devices. This problem is still [open](https://github.com/kubernetes/enhancements/pull/2101)
and requires more brainstorming.
Finally, it needs to be understood whether `runAsUser`/`runAsGroup` are enough or if device specific settings similar to `fsGroups` are needed in PodSpec/CRI v2.
## Thanks
My thanks goes to Mike Brown (IBM, containerd), Peter Hunt (Redhat, CRI-O), and Alexander Kanevskiy (Intel) for providing all the feedback and good conversations.
title: "Dockershim removal is coming. Are you ready?"
date: 2021-11-12
slug: are-you-ready-for-dockershim-removal
---
**Author:** Sergey Kanzhelev, Google. With reviews from Davanum Srinivas, Elana Hashman, Noah Kantrowitz, Rey Lejano.
Last year we announced that Dockershim is being deprecated: [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/).
Our current plan is to remove dockershim from the Kubernetes codebase soon.
We are looking for feedback from you whether you are ready for dockershim
removal and to ensure that you are ready when the time comes.
**Please fill out this survey: https://forms.gle/svCJmhvTv78jGdSx8**.
The dockershim component that enables Docker as a Kubernetes container runtime is
being deprecated in favor of runtimes that directly use the [Container Runtime Interface](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)
created for Kubernetes. Many Kubernetes users have migrated to
other container runtimes without problems. However we see that dockershim is
still very popular. You may see some public numbers in recent [Container Report](https://www.datadoghq.com/container-report/#8) from DataDog.
Some Kubernetes hosting vendors just recently enabled other runtimes support
(especially for Windows nodes). And we know that many third party tools vendors
are still not ready: [migrating telemetry and security agents](/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents/#telemetry-and-security-agent-vendors).
At this point, we believe that there is feature parity between Docker and the
other runtimes. Many end-users have used our [migration guide](/docs/tasks/administer-cluster/migrating-from-dockershim/)
and are running production workload using these different runtimes. The plan of
record today is that dockershim will be removed in version 1.24, slated for
release around April of next year. For those developing or running alpha and
beta versions, dockershim will be removed in December at the beginning of the
1.24 release development cycle.
There is only one month left to give us feedback. We want you to tell us how
ready you are.
**We are collecting opinions through this survey: [https://forms.gle/svCJmhvTv78jGdSx8](https://forms.gle/svCJmhvTv78jGdSx8)**
To better understand preparedness for the dockershim removal, our survey is
asking the version of Kubernetes you are currently using, and an estimate of
when you think you will adopt Kubernetes 1.24. All the aggregated information
on dockershim removal readiness will be published.
Free form comments will be reviewed by SIG Node leadership. If you want to
discuss any details of migrating from dockershim, report bugs or adoption
blockers, you can use one of the SIG Node contact options any time:
Kubernetes is a mature project. This deprecation is another
step in the effort to get away from permanent beta features and providing more
stability and compatibility guarantees. With the migration from dockershim you
will get more flexibility and choice of container runtime features as well as
less dependencies of your apps on specific underlying technology. Please take
time to review the [dockershim migration documentation](/docs/tasks/administer-cluster/migrating-from-dockershim/)
and consult your Kubernetes hosting vendor (if you have one) what container runtime options are available for you.
Read up [container runtime documentation with instructions on how to use containerd and CRI-O](/docs/setup/production-environment/container-runtimes/#container-runtimes)
to help prepare you when you're ready to upgrade to 1.24. CRI-O, containerd, and
Docker with [Mirantis cri-dockerd](https://github.com/Mirantis/cri-dockerd) are
not the only container runtime options, we encourage you to explore the [CNCF landscape on container runtimes](https://landscape.cncf.io/card-mode?category=container-runtime&grouping=category)
Kubernetes v1.22, released in August 2021, introduced a new alpha feature that improves how Linux nodes implement memory resource requests and limits.
In prior releases, Kubernetes did not support memory quality guarantees.
For example, if you set container resources as follows:
```
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
containers:
- name: nginx
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "64Mi"
cpu: "500m"
```
`spec.containers[].resources.requests`(e.g. cpu, memory) is designed for scheduling. When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled Containers is less than the capacity of the node.
`spec.containers[].resources.limits` is passed to the container runtime when the kubelet starts a container. CPU is considered a "compressible" resource. If your app starts hitting your CPU limits, Kubernetes starts throttling your container, giving your app potentially worse performance. However, it won’t be terminated. That is what "compressible" means.
In cgroup v1, and prior to this feature, the container runtime never took into account and effectively ignored spec.containers[].resources.requests["memory"]. This is unlike CPU, in which the container runtime consider both requests and limits. Furthermore, memory actually can't be compressed in cgroup v1. Because there is no way to throttle memory usage, if a container goes past its memory limit it will be terminated by the kernel with an OOM (Out of Memory) kill.
Fortunately, cgroup v2 brings a new design and implementation to achieve full protection on memory. The new feature relies on cgroups v2 which most current operating system releases for Linux already provide. With this experimental feature, [quality-of-service for pods and containers](/docs/tasks/configure-pod-container/quality-service-pod/) extends to cover not just CPU time but memory as well.
## How does it work?
Memory QoS uses the memory controller of cgroup v2 to guarantee memory resources in Kubernetes. Memory requests and limits of containers in pod are used to set specific interfaces `memory.min` and `memory.high` provided by the memory controller. When `memory.min` is set to memory requests, memory resources are reserved and never reclaimed by the kernel; this is how Memory QoS ensures the availability of memory for Kubernetes pods. And if memory limits are set in the container, this means that the system needs to limit container memory usage, Memory QoS uses `memory.high` to throttle workload approaching it's memory limit, ensuring that the system is not overwhelmed by instantaneous memory allocation.
The following table details the specific functions of these two parameters and how they correspond to Kubernetes container resources.
<table>
<tr>
<th style="text-align:center">File</th>
<th style="text-align:center">Description</th>
</tr>
<tr>
<td>memory.min</td>
<td><code>memory.min</code> specifies a minimum amount of memory the cgroup must always retain, i.e., memory that can never be reclaimed by the system. If the cgroup's memory usage reaches this low limit and can’t be increased, the system OOM killer will be invoked.
<br>
<br>
<i>We map it to the container's memory request</i>
</td>
</tr>
<tr>
<td>memory.high</td>
<td><code>memory.high</code> is the memory usage throttle limit. This is the main mechanism to control a cgroup's memory use. If a cgroup's memory use goes over the high boundary specified here, the cgroup’s processes are throttled and put under heavy reclaim pressure. The default is max, meaning there is no limit.
<br>
<br>
<i>We use a formula to calculate <code>memory.high</code>, depending on container's memory limit or node allocatable memory (if container's memory limit is empty) and a throttling factor. Please refer to the KEP for more details on the formula.</i>
</td>
</tr>
</table>
When container memory requests are made, kubelet passes `memory.min` to the back-end CRI runtime (possibly containerd, cri-o) via the `Unified` field in CRI during container creation. The `memory.min` in container level cgroup will be set to:
<sub>i: the i<sup>th</sup> container in one pod</sub>
Since the `memory.min` interface requires that the ancestor cgroup directories are all set, the pod and node cgroup directories need to be set correctly.
`memory.min` in pod level cgroup:
<sub>i: the i<sup>th</sup> container in one pod</sub>
`memory.min` in node level cgroup:
<sub>i: the i<sup>th</sup> pod in one node, j: the j<sup>th</sup> container in one pod</sub>
Kubelet will manage the cgroup hierarchy of the pod level and node level cgroups directly using runc libcontainer library, while container cgroup limits are managed by the container runtime.
For memory limits, in addition to the original way of limiting memory usage, Memory QoS adds an additional feature of throttling memory allocation. A throttling factor is introduced as a multiplier (default is 0.8). If the result of multiplying memory limits by the factor is greater than memory requests, kubelet will set `memory.high` to the value and use `Unified` via CRI. And if the container does not specify memory limits, kubelet will use node allocatable memory instead. The `memory.high` in container level cgroup is set to:
<sub>i: the i<sup>th</sup> container in one pod</sub>
This can can help improve stability when pod memory usage increases, ensuring that memory is throttled as it approaches the memory limit.
## How do I use it?
Here are the prerequisites for enabling Memory QoS on your Linux node, some of these are related to [Kubernetes support for cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2254-cgroup-v2).
1. Kubernetes since v1.22
2. [runc](https://github.com/opencontainers/runc) since v1.0.0-rc93; [containerd](https://containerd.io/) since 1.4; [cri-o](https://cri-o.io/) since 1.20
3. Linux kernel minimum version: 4.15, recommended version: 5.2+
4. Linux image with cgroupv2 enabled or enabling cgroupv2 unified_cgroup_hierarchy manually
OCI runtimes such as runc and crun already support cgroups v2 [`Unified`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#unified), and Kubernetes CRI has also made the desired changes to support passing [`Unified`](https://github.com/kubernetes/kubernetes/pull/102578). However, CRI Runtime support is required as well. Memory QoS in Alpha phase is designed to support containerd and cri-o. Related PR [Feature: containerd-cri support LinuxContainerResources.Unified #5627](https://github.com/containerd/containerd/pull/5627) has been merged and will be released in containerd 1.6. CRI-O [implement kube alpha features for 1.22 #5207](https://github.com/cri-o/cri-o/pull/5207) is still in WIP.
With those prerequisites met, you can enable the memory QoS feature gate (see [Set kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/)).
## How can I learn more?
You can find more details as follows:
- [Support Memory QoS with cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos/#readme)
title: "Contribution, containers and cricket: the Kubernetes 1.22 release interview"
date: 2021-12-01
---
**Author**: Craig Box (Google)
The Kubernetes release train rolls on, and we look ahead to the release of 1.23 next week. [As is our tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the process that brought us the previous version.
The release team for 1.22 was led by [Savitha Raghunathan](https://twitter.com/coffeeartgirl), who was, at the time, a Senior Platform Engineer at MathWorks. [I spoke to Savitha](https://kubernetespodcast.com/episode/157-kubernetes-1.22/) on the [Kubernetes Podcast from Google](https://kubernetespodcast.com/), the weekly<super>*</super> show covering the Kubernetes and Cloud Native ecosystem.
Our release conversations shine a light on the team that puts together each Kubernetes release. Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/) so you catch the story of 1.23.
And in case you're interested in why the show has been on a hiatus the last few weeks, all will be revealed in the next episode!
*This transcript has been lightly edited and condensed for clarity.*
---
**CRAIG BOX: Welcome to the show, Savitha.**
SAVITHA RAGHUNATHAN: Hey, Craig. Thanks for having me on the show. How are you today?
**CRAIG BOX: I'm very well, thank you. I've interviewed a lot of people on the show, and you're actually the first person who's asked that of me.**
SAVITHA RAGHUNATHAN: I'm glad. It's something that I always do. I just want to make sure the other person is good and happy.
**CRAIG BOX: That's very kind of you. Thank you for kicking off on a wonderful foot there. I want to ask first of all — you grew up in Chennai. My association with Chennai is the [Super Kings cricket team](https://en.wikipedia.org/wiki/Chennai_Super_Kings). Was cricket part of your upbringing?**
SAVITHA RAGHUNATHAN: Yeah. Actually, a lot. My mom loves watching cricket. I have a younger brother, and when we were growing up, we used to play cricket on the terrace. Everyone surrounding me, my best friends — and even now, my partner — loves watching cricket, too. Cricket is a part of my life.
I stopped watching it a while ago, but I still enjoy a good game.
**CRAIG BOX: It's probably a bit harder in the US. Everything's in a different time zone. I find, with my cricket team being on the other side of the world, that it's a lot easier when they're playing near me, as opposed to trying to keep up with what they're doing when they're playing at 3:00 in the morning.**
SAVITHA RAGHUNATHAN: That is actually one of the things that made me lose touch with cricket. I'm going to give you a piece of interesting information. I never supported Chennai Super Kings. I always supported [Royal Challengers of Bangalore](https://en.wikipedia.org/wiki/Royal_Challengers_Bangalore).
I once went to the stadium, and it was a match between the Chennai Super Kings and the RCB. I was the only one who was cheering whenever the RCB hit a 6, or when they were scoring. I got the stares of thousands of people looking at me. I'm like, "what are you doing?" My friends are like, "you're going to get us killed! Just stop screaming!"
**CRAIG BOX: I hear you. As a New Zealander in the UK, there are a lot of international cricket matches I've been to where I am one of the few people dressed in the full beige kit. But I have to ask, why an affiliation with a different team?**
SAVITHA RAGHUNATHAN: I'm not sure. When the IPL came out, I really liked Virat Kohli. He was playing for RCB at that time, and I think pretty much that's it.
**CRAIG BOX: Well, what I know about the Chennai Super Kings is that their coach is New Zealand's finest batsmen and [air conditioning salesman](https://www.youtube.com/watch?v=vSZAaUCAclw), [Stephen Fleming](https://en.wikipedia.org/wiki/Stephen_Fleming).**
SAVITHA RAGHUNATHAN: Oh, really?
**CRAIG BOX: Yeah, he's a dead ringer for the guy who played the [yellow Wiggle](https://s1.reutersmedia.net/resources/r/?m=02&d=20061130&t=2&i=153531&w=&fh=545px&fw=&ll=&pl=&sq=&r=153531) back in the day.**
SAVITHA RAGHUNATHAN: Oh, interesting. I remember the name, but I cannot put the picture and the name together. I stopped watching cricket once I moved to the States. Then, all my focus was on studies and extracurriculars. I have always been an introvert. The campus — it was a new thing for me — they had international festivals.
And every week, they'd have some kind of new thing going on, so I'd go check them out. I wouldn't participate, but I did go out and check them out. That was a big feat for me around that time because a lot of people — and still, even now, a lot of people — they kind of scare me. I don't know how to make a conversation with everyone.
I'll just go and say, "hi, how are you? OK, I'm good. I'm just going to move on". And I'll just go to the next person. And after two hours, I'm out of that place.
**CRAIG BOX: Perhaps a pleasant side effect of the last 12 months — a lot fewer gatherings of people.**
SAVITHA RAGHUNATHAN: Could be that, but I'm so excited about KubeCon. But when I think about it, I'm like "oh my God. There's going to be a lot of people. What am I going to do? I'm going to meet all my friends over there".
Sometimes I have social anxiety like, what's going to happen?
**CRAIG BOX: What's going to happen is you're going to ask them how they are at the beginning, and they're immediately going to be set at ease.**
SAVITHA RAGHUNATHAN: *laughs* I hope so.
**CRAIG BOX: Let's talk a little bit, then, about your transition from India to the US. You did your undergraduate degree in computer science at the SSN College of Engineering. How did you end up at Arizona State?**
SAVITHA RAGHUNATHAN: I always wanted to pursue higher studies when I was in India, and I didn't have the opportunity immediately. Once I graduated from my school there, I went and I worked for a couple of years. My aim was always to get out of there and come here, do my graduate studies.
Eventually, I want to do a PhD. I have an idea of what I want to do. I always wanted to keep studying. If there's an option that I could just keep studying and not do work or anything of that sort, I'd just pick that other one — I'll just keep studying.
But unfortunately, you need money and other things to live and sustain in this world. So I'm like, OK, I'll take a break from studies, and I will work for a while.
**CRAIG BOX: The road to success is littered with dreams of PhDs. I have a lot of friends who thought that that was the path they were going to take, and they've had a beautiful career and probably aren't going to go back to study. Did you use the [Matlab](https://en.wikipedia.org/wiki/MATLAB) software at all while you were going through your schooling?**
SAVITHA RAGHUNATHAN: No, unfortunately. That is a question that everyone asks. I have not used Matlab. I haven't used it even now. I don't use it for work. I didn't have any necessity for my school work. I didn't have anything to do with Matlab. I never analysed, or did data processing, or anything, with Matlab. So unfortunately, no.
Everyone asks me like, you're working at [MathWorks](https://en.wikipedia.org/wiki/MathWorks). Have you used Matlab? I'm like, no.
**CRAIG BOX: Fair enough. Nor have I. But it's been around since the late 1970s, so I imagine there are a lot of people who will have come across it at some point. Do you work with a lot of people who have been working on it that whole time?**
SAVITHA RAGHUNATHAN: Kind of. Not all the time, but I get to meet some folks who work on the product itself. Most of my interactions are with the infrastructure team and platform engineering teams at MathWorks. One other interesting fact is that when I joined the company — MathWorks has an extensive internal curriculum for training and learning, which I really love. They have an "Intro to Matlab" course, and that's on my bucket of things to do.
It was like 500 years ago. I added it, and I never got to it. I'm like, OK, maybe this year at least I want to get to it and I want to learn something new. My partner used Matlab extensively. He misses it right now at his current employer. And he's like, "you have the entire licence! You have access to the entire suite and you haven't used it?" I'm like, "no!"
**CRAIG BOX: Well, I have bad news for the idea of you doing a PhD, I'm sorry.**
SAVITHA RAGHUNATHAN: Another thing is that none of my family knew about the company MathWorks and Matlab. The only person who knew was my younger brother. He was so proud. He was like, "oh my God".
When he was 12 years old, he started getting involved in robotics and all that stuff. That's how he got introduced to Matlab. He goes absolutely bananas for the swag. So all the t-shirts, all the hoodies — any swag that I get from MathWorks goes to him, without saying.
Over the five, six years, the things that I've got — there was only one sweatshirt that I kept for myself. Everything else I've just given to him. And he cherishes it. He's the only one in my family who knew about Matlab and MathWorks.
Now, everyone knows, because I'm working there. They were initially like, I don't even know that company name. Is it like Amazon? I'm like, no, we make software that can send people to the moon. And we also make software that can do amazing robotic surgeries and even make a car drive on its own. That's something that I take immense pride in.
I know I don't directly work on the product, but I'm enabling the people who are creating the product. I'm really, really proud of that.
**CRAIG BOX: I think Jeff Bezos is working on at least two out of three of those disciplines that you mentioned before, so it's maybe a little bit like Amazon. One thing I've always thought about Matlab is that, because it's called Matlab, it solves that whole problem where [Americans call it math, and the rest of the world call it maths](https://www.grammar.com/math_vs._maths). Why do Americans think there's only one math?**
SAVITHA RAGHUNATHAN: Definitely. I had trouble — growing up in India, it's always British English. And I had so much trouble when I moved here. So many things changed.
One of the things is maths. I always got used to writing maths, physics, and everything.
**CRAIG BOX: They don't call it "physic" in the US, do they?**
SAVITHA RAGHUNATHAN: No, no, they don't. Luckily, they don't. That still stays "physics". But math — I had trouble. It's maths. Even when you do the full abbreviations like mathematics and you are still calling it math, I'm like, mm.
**CRAIG BOX: They can do the computer science abbreviation thing and call it math-7-S or whatever the number of letters is.**
SAVITHA RAGHUNATHAN: Just like Kubernetes. K-8-s.
**CRAIG BOX: Your path to Kubernetes is through MathWorks. They started out as a company making software which was distributed in a physical sense — boxed copies, if you will. I understand now there is a cloud version. Can I assume that that is where the two worlds intersect?**
SAVITHA RAGHUNATHAN: Kind of. I have interaction with the team that supports Matlab on the cloud, but I don't get to work with them on a day-to-day basis. They use Docker containers, and they are building the platform using Kubernetes. So yeah, a little bit of that.
**CRAIG BOX: So what exactly is the platform that you are engineering day to day?**
SAVITHA RAGHUNATHAN: Providing Kubernetes as a platform, obviously — that goes without saying — to some of the internal development teams. In the future we might expand it to more teams within the company. That is a focus area right now, so that's what we are doing. In the process, we might even get to work with the people who are deploying Matlab on the cloud, which is exciting.
**CRAIG BOX: Now, your path to contribution to Kubernetes, you've said before, was through [fixing a 404 error on the Kubernetes.io website](https://github.com/kubernetes/website/pull/15588). Do you remember what the page was?**
SAVITHA RAGHUNATHAN: I do. I was going to something for work, and I came across this changelog. In Kubernetes there's a nice page — once you got to the release page, there would be a long list of changelogs.
One of the things that I fixed was, the person who worked on the feature had changed their GitHub handle, and that wasn't reflected on this page. So that was my first. I got curious and clicked on the links. One of the links was the handle, and that went to a 404. And I was like "Yeah, I'll just fix that. They have done all the hard work. They can get the credit that's due".
It was easy. It wasn't overwhelming for me to pick it up as my first issue. Before that I logged on around Kubernetes for about six to eight months without doing anything because it was just a lot.
**CRAIG BOX: One of the other things that you said about your initial contribution is that you had to learn how to use Git. As a very powerful tool, I find Git is a high barrier to entry for even contributing code to a project. When you want to contribute a blog post or documentation or a fix like you did before, I find it almost impossible to think how a new user would come along and do that. What was your process? Do you think that there's anything we can do to make that barrier lower for new contributors?**
SAVITHA RAGHUNATHAN: Of course. There are more and more tutorials available these days. There is a new contributor workshop. They actually have a [GitHub workflow section](https://www.kubernetes.dev/docs/guide/github-workflow/), [how to do a pull request](https://www.kubernetes.dev/docs/guide/pull-requests/) and stuff like that. I know a couple of folks from SIG Docs that are working on which Git commands that you need, or how to get to writing something small and getting it committed. But more tutorials or more links to intro to Git would definitely help.
The thing is also, someone like a documentation writer — they don't actually want to know the entirety of Git. Honestly, it's an ocean. I don't know how to do it. Most of the time, I still ask for help even though I work with Git on a day to day basis. There are several articles and a lot of help is available already within the community. Maybe we could just add a couple more to [kubernetes.dev](https://kubernetes.dev/). That is an amazing site for all the new contributors and existing contributors who want to build code, who want to write documentation.
We could just add a tutorial there like, "hey, don't know Git, you are new to Git? You just need to know these main things".
**CRAIG BOX: I find it a shame, to be honest, that people need to use Git for that, by comparison to Wikipedia where you can come along, and even though it might be written in Markdown or something like it, it seems like the barrier is a lot lower. Similar to you, I always have to look up anything more complicated than the five or six Git commands that I use on a day to day basis. Even to do simple things, I basically just go and follow a recipe which I find on the internet.**
SAVITHA RAGHUNATHAN: This is how I got introduced to one of the amazing mentors in Kubernetes. Everyone knows him by his handle, Dims. It was my second PR to the Kubernetes website, and I made a mistake. I destroyed the Git history. I could not push my reviews and comments — I addressed them. I couldn't push them back.
My immediate thought was to delete it and recreate, do another pull request. But then I was like, "what happens to others who have already put effort into reviewing them?" I asked for help, and Dims was there.
I would say I just got lucky he was there. And he was like, "OK, let me walk you through". We did troubleshooting through Slack messages. I copied and pasted all the errors. Every single command that he said, I copied and pasted. And then he was like, "OK, run this one. Try this one. And do this one".
Finally, I got it fixed. So you know what I did? I went and I stored the command history somewhere local for the next time when I run into this problem. Luckily, I haven't. But I find the contributors so helpful. They are busy. They have a lot of things to do, but they take moments to stop and help someone who's new.
That is also another part of the reason why I stay — I want to contribute more. It's mainly the community. It's the Kubernetes community. I know you asked me about Git, and I just took the conversation to the Kubernetes community. That's how my brain works.
**CRAIG BOX: A lot of people in the community do that and think that's fantastic, obviously, people like Dims who are just floating around on Slack and seem to have endless time. I don't know how they do it.**
SAVITHA RAGHUNATHAN: I really want to know the secret for endless time. If I only had 48 hours in a day. I would sleep for 16 hours, and I would use the rest of the time for doing the things that I want.
**CRAIG BOX: If I had a chance to sleep up to 48 hours a day, I think it'd be a lot more than 16.**
**Now, one of the areas that you've been contributing to Kubernetes is in the release team. In 1.18, you were a shadow for the docs role. You led that role in 1.19. And you were a release lead shadow for versions 1,20 and 1.21 before finally leading this release, 1.22, which we will talk about soon.**
**How did you get involved? And how did you decide which roles to take as you went through that process?**
SAVITHA RAGHUNATHAN: That is a topic I love to talk about. This was fresh when I started learning about Kubernetes and using Kubernetes at work. And I got so much help from the community, I got interested in contributing back.
At the first KubeCon that I attended in 2018, in Seattle, they had a speed mentoring session. Now they call it "pod mentoring". I went to the session, and said, "hey, I want to contribute. I don't know where to start". And I got a lot of information on how to get started.
One of the places was SIG Release and the release team. I came back and diligently attended all the SIG Release meetings for four to six months. And in between, I applied to the Kubernetes release team — 1.14 and 1.15. I didn't get through. So I took a little bit of a break, and I focused on doing some documentation work. Then I applied for 1.18.
Since I was already working on some kinds of — not like full fledged "documentation" documentation, I still don't write. I eventually want to write something really nice and full fledged documentation like other awesome folks.
**CRAIG BOX: You'll need a lot more than 48 hours in your day to do that.**
SAVITHA RAGHUNATHAN: *laughing* That's how I applied for the docs role, because I know a little bit about the website. I've done a few pull requests and commits. That's how I got started. I applied for that one role, and I got selected for the 1.18 team. That's how my journey just took off.
And the next release, I was leading the documentation team. And as everyone knows, the pandemic hit. It was one of the longest releases. I could lean back on the community. I would just wait for the release team meetings.
It was my way of coping with the pandemic. It took my mind off. It was actually more than a release team, they were people. They were all people first, and we took care of each other. So it felt good.
And then, I became a release lead shadow for 1.20 and 1.21 because I wanted to know more. I wanted to learn more. I wasn't ready. I still don't feel ready, but I have led 1.22. So if I could do it, anyone could do it.
**CRAIG BOX: How much of this work is day job?**
SAVITHA RAGHUNATHAN: I am lucky to be blessed with an awesome team. I do most of my work after work, but there have been times where I have to take meetings and attend to immediate urgent stuff. During the time of exception requests and stuff like that, I take a little bit of time from my work.
My team has been wonderful: they support me in all possible ways, and the management as well. Other than the meetings, I don't do much of the work during the day job. It just takes my focus and attention away too much, and I end up having to spend a lot of time sitting in front of the computer, which I don't like.
Before the pandemic I had a good work life balance. I'd just go to work at 7:00, 7:30, and I'd be back by 4 o'clock. I never touched my laptop ever again. I left all work behind when I came home. So right now, I'm still learning how to get through.
I try to limit the amount of open source work that I do during work time. The release lead shadow and the release lead job — they require a lot of time, effort. So on average, I'd be spending two to three hours post work time on the release activities.
**CRAIG BOX: Before the pandemic, everyone was worried that if we let people work from home, they wouldn't work enough. I think the opposite has actually happened, is that now we're worried that if we let people work from home, they will just get on the computer in the morning and you'll have to pry it out of their hands at midnight.**
SAVITHA RAGHUNATHAN: Yeah, I think the productivity has increased at least twofold, I would say, for everyone, once they started working from home.
**CRAIG BOX: But at the expense of work-life balance, though, because as you say, when you're sitting in the same chair in front of, perhaps, the same computer doing your MathWorks work and then your open source work, they kind of can blur into one perhaps?**
SAVITHA RAGHUNATHAN: That is a challenge. I face it every day. But so many others are also facing it. I implemented a few little tricks to help me. When I used to come back home from work, the first thing I would do is remove my watch. That was an indication that OK, I'm done.
That's the thing that I still do. I just remove my watch, and I just keep it right where my workstation is. And I just close the door so that I never look back. Even going past the room, I don't get a glimpse of my work office. I start implementing tiny little things like that to avoid burnout.
I think I'm still facing a little bit of burnout. I don't know if I have fully recovered from it. I constantly feel like I need a vacation. And I could just take a vacation for like a month or two. If it's possible, I will just do it.
**CRAIG BOX: I do hope that travel opens up for everyone as an opportunity because I know that, for a lot of people, it's not so much they've been working from home but they've been living at work. The idea of taking vacation effectively means, well, I've been stuck in the same place, if I've been under a lockdown. It's hard to justify that. It will be good as things improve worldwide for us to be able to start focusing more on mental health and perhaps getting away from the "everything room," as I sometimes call it.**
SAVITHA RAGHUNATHAN: I'm totally looking forward to it. I hope that travel opens up and I could go home and I could meet my siblings and my aunt and my parents.
**CRAIG BOX: Catch a cricket match?**
SAVITHA RAGHUNATHAN: Yeah. Probably yes, if I have company and if there is anything interesting happening around the time. I don't mind going back to the Chepauk Stadium and catching a match or two.
**CRAIG BOX: Let's turn now to the recently released [Kubernetes 1.22](https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/). Congratulations on the launch.**
SAVITHA RAGHUNATHAN: Thank you.
**CRAIG BOX: Each launch comes with a theme and a mascot or a logo. What is the theme for this release?**
SAVITHA RAGHUNATHAN: The theme for the release is reaching new peaks. I am fascinated with a lot of space travel and chasing stars, the Milky Way. The best place to do that is over the top of a mountain. So that is the release logo, basically. It's a mountain — Mount Rainier. On top of that, there is a Kubernetes flag, and it's overlooking the Milky Way.
It's also symbolic that with every release, that we are achieving something new, bigger, and better, and we are making the release awesome. So I just wanted to incorporate that into the team as to say, we are achieving new things with every release. That's the "reaching new peaks" theme.
**CRAIG BOX: The last couple of releases have both been incrementally larger — as a result, perhaps, of the fact there are now only three releases per year rather than four. There were also changes to the process, where the work has been driven a lot more by the SIGs than by the release team having to go and ask the SIGs what was going on. What can you say about the size and scope of the 1.22 release?**
SAVITHA RAGHUNATHAN: The 1.22 release is the largest release to date. We have 56 enhancements if I'm not wrong, and we have a good amount of features that's graduated as stable. You can now say that Kubernetes as a project has become more mature because you see new features coming in. At the same time, you see the features that weren't used getting deprecated — we have like three deprecations in this release.
Aside from that fact, we also have a big team that's supporting one of the longest releases. This is the first official release cycle after the cadence KEP got approved. Officially, we are at four months, even though 1.19 was six months, and 1.21 was like 3 and 1/2 months, I think, this is the first one after the official KEP approval.
**CRAIG BOX: What changes did you make to the process knowing that you had that extra month?**
SAVITHA RAGHUNATHAN: One of the things the community had asked for is more time for development. We tried to incorporate that in the release schedule. We had about six weeks between the enhancements freeze and the code freeze. That's one.
It might not be visible to everyone, but one of the things that I wanted to make sure of was the health of the team — since it was a long, long release, we had time to plan out, and not have everyone work during the weekends or during their evenings or time off. That actually helped everyone keep their sanity, and also in making good progress and delivering good results at the end of the release. That's one of the process improvements that I'd call out.
We got better by making a post during the exception request process. Everyone works around the world. People from the UK start a little earlier than the people in the US East Coast. The West Coast starts three hours later than the East Coast. We used to make a post every Friday evening saying "hey, we actually received this many requests. We have addressed a number of them. We are waiting on a couple, or whatever. All the release team members are done for the day. We will see you around on Monday. Have a good weekend." Something like that.
We set the expectations from the community as well. We understand things are really important and urgent, but we are done. This gave everyone their time back. They don't have to worry over the weekend thinking like, hey, what's happening? What's happening in the release? They could spend time with their family, or they could do whatever they want to do, like go on a hike, or just sit and watch TV.
There have been weekends that I just did that. I just binge-watched a series. That's what I did.
**CRAIG BOX: Any recommendations?**
SAVITHA RAGHUNATHAN: I'm a big fan of Marvel, so I have watched the new [Loki](https://en.wikipedia.org/wiki/Loki_(TV_series)), which I really love. Loki is one of my favourite characters in Marvel. And I also liked [WandaVision](https://en.wikipedia.org/wiki/WandaVision). That was good, too.
**CRAIG BOX: I've not seen Loki yet, but I've heard it described as the best series of Doctor Who in the last few years.**
SAVITHA RAGHUNATHAN: Really?
**CRAIG BOX: There must be an element of time-travelling in there if that's how people are describing it.**
SAVITHA RAGHUNATHAN: You should really go and watch it whenever you have time. It's really amazing. I might go back and watch it again because I might have missed bits and pieces. That always happens in Marvel movies and the episodes; you need to watch them a couple of times to catch, "oh, this is how they relate".
**CRAIG BOX: Yes, the mark of good media that you want to immediately go back and watch it again once you've seen it.**
**Let's look now at some of the new features in Kubernetes 1.22. A couple of things that have graduated to general availability — server-side apply, external credential providers, a couple of new security features — the replacement for pod security policy has been announced, and seccomp is now available by default.**
**Do you have any favourite features in 1.22 that you'd like to discuss?**
SAVITHA RAGHUNATHAN: I have a lot of them. All my favourite features are related to security. OK, one of them is not security, but a major theme of my favourite KEPs is security. I'll start with the [default seccomp](https://github.com/kubernetes/enhancements/issues/2413). I think it will help make clusters secure by default, and may assist in preventing more vulnerabilities, which means less headaches for the cluster administrators.
This is close to my heart because the base of the MathWorks platform is provisioning Kubernetes clusters. Knowing that they are secure by default will definitely provide me with some good sleep. And also, I'm paranoid about security most of the time. I'm super interested in making everything secure. It might get in the way of making the users of the platform angry because it's not usable in any way.
My next one is [rootless Kubelet](https://github.com/kubernetes/enhancements/issues/2033). That feature's going to enable the cluster admin, the platform developers to deploy Kubernetes components to run in a user namespace. And I think that is also a great addition.
Like you mention, the most awaited drop in for the PSP replacement is here. It's [pod admission control](https://github.com/kubernetes/enhancements/issues/2579). It lets cluster admins apply the pod security standards. And I think it's just not related to the cluster admins. I might have to go back and check on that. Anyone can probably use it — the developers and the admins alike.
It also supports various modes, which is most welcome. There are times where you don't want to just cut the users off because they are trying to do something which is not securely correct. You just want to warn them, hey, this is what you are doing. This might just cause a security issue later, so you might want to correct it. But you just don't want to cut them off from using the platform, or them trying to attempt to do something — deploy their workload and get their day-to-day job done. That is something that I really like, that it also supports a warning mechanism.
Another one which is not security is [node swap support](https://github.com/kubernetes/enhancements/issues/2400). Kubernetes didn't have support for swap before, but it is taken into consideration now. This is an alpha feature. With this, you can take advantage of the swap, which is provisioned on the Linux VMs.
Some of the workloads — when they are deployed, they might need a lot of swap for the start-up — example, like Node and Java applications, which I just took out of their KEP user stories. So if anyone's interested, they can go and look in the KEP. That's useful. And it also increases the node stability and whatnot. So I think it's going to be beneficial for a lot of folks.
We know how Java and containers work. I think it has gotten better, but five years ago, it was so hard to get a Java application to fit in a small container. It always needed a lot of memory, swap, and everything to start up and run. I think this will help the users and help the admins and keep the cost low, and it will tie into so many other things as well. I'm excited about that feature.
Another feature that I want to just call out — I don't use Windows that much, but I just want to give a shout out to the folks who are doing an amazing job bringing all the Kubernetes features to Windows as well, to give a seamless experience.
One of the things is [Windows privileged containers](https://github.com/kubernetes/enhancements/issues/1981). I think it went alpha this release. And that is a wonderful addition, if you ask me. It can take advantage of whatever that's happening on the Linux side. And they can also port it over and see, OK, I can now run Windows containers in a privileged mode.
So whatever they are trying to achieve, they can do it. So that's a noteworthy mention. I need to give a shout out for the folks who work and make things happen in the Windows ecosystem as well.
**CRAIG BOX: One of the things that's great about the release process is the continuity between groups and teams. There's always an emeritus advisor who was a lead from a previous release. One thing that I always ask when I do these interviews is, what is the advice that you give to the next person? When [we talked to Nabarun for the 1.21 interview](https://kubernetespodcast.com/episode/146-kubernetes-1.21/), he said that his advice to you would be "do, delegate, and defer". Figure out what you can do, figure out what you can ask other people to do, and figure out what doesn't need to be done. Were you able to take that advice on board?**
SAVITHA RAGHUNATHAN: Yeah, you won't believe it. [I have it right here stuck to my monitor.](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/3)
**CRAIG BOX: Next to your Git cheat sheet?**
SAVITHA RAGHUNATHAN: *laughs* Absolutely. I just have it stuck there. I just took a look at it.
**CRAIG BOX: Someone that you will have been able to delegate and defer to is Rey Lejano from Rancher Labs and SUSE, who is the release lead to be for 1.23.**
SAVITHA RAGHUNATHAN: I want to tell Rey to beware of the team's mental health. Schedule in such a way that it avoids burnout. Check in, and make sure that everyone is doing good. If they need some kind of help, create a safe space where they can actually ask for help, if they want to step back, if they need someone to cover.
I think that is most important. The releases are successful based on the thousands and thousands of contributors. But when it comes to a release team, you need to have a healthy team where people feel they are in a good place and they just want to make good contributions, which means they want to be heard. That's one thing that I want to tell Rey.
Also collaborate and learn from each other. I constantly learn. I think the team was 39 folks, including me. Every day I learned something or the other, even starting from how to interact.
Sometimes I have learned more leadership skills from my release lead shadows. They are awesome, and they are mature. I constantly learn from them, and I admire them a lot.
It also helps to have good, strong individuals in the team who can step up and help when needed. For example, unfortunately, we lost one of our teammates after the start of the release cycle. That was tragic. His name was [Peeyush Gupta](https://github.com/cncf/memorials/blob/main/peeyush-gupta.md). He was an awesome and wonderful human — very warm.
I didn't get more of a chance to interact with him. I had exchanged a few Slack messages, but I got his warm personality. I just want to take a couple of seconds to remember him. He was awesome.
After we lost him, we had this strong person from the team step up and lead the communications, who had never been a part of the release team before at all. He was a shadow for the first time. His name is Jesse Butler. So he stepped up, and he just took it away. He ran the comms show for 1.22.
That's what the community is about. You take care of team members, and the team will take care of you. So that's one other thing that I want to let Rey know, and maybe whoever — I think it's applicable overall.
**CRAIG BOX: There's a link to a [family education fund for Peeyush Gupta](https://milaap.org/fundraisers/support-peeyush-gupta-family-education), which you can find in the show notes.**
**Five releases in a row now you've been a member of the release team. Will you be putting your feet up now for 1.23?**
SAVITHA RAGHUNATHAN: I am going to take a break for a while. In the future, I want to be contributing, if not the release team, the SIG Release and the release management effort. But right now, I have been there for five releases. And I feel like, OK, I just need a little bit of fresh air.
And also the pandemic and the burnout has caught up, so I'm going to take a break from certain contributions. You will see me in the future. I will be around, but I might not be actively participating in the release team activities. I will be around the community. Anyone can reach out to me. They all know my Slack, so they can just reach out to me via Slack or Twitter.
**CRAIG BOX: Yes, your Twitter handle is CoffeeArtGirl. Does that mean that you'll be spending some time working on your lattes?**
SAVITHA RAGHUNATHAN: I am very bad at making lattes. The coffee art means that I used to [make art with coffee](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/1). You get instant coffee powder and just mix it with water. You get the colours, very beautiful brown colours. I used to make art using that.
And I love coffee. So I just combined all the words together. And I had to come up with it in a span of one hour or so because I was joining this 'meet our contributors' panel. And Paris asked me, "do you have a Twitter handle?" I was planning to create one, but I didn't have the time.
I'm like, well, let me just think what I could just come up with real quick. So I just came up with that. So that's the story behind my Twitter handle. Everyone's interested in it. You are not the first person you have asked me or mentioned about it. So many others are like, why coffee art?
**CRAIG BOX: And you are also interested in art with perhaps other materials?**
SAVITHA RAGHUNATHAN: Yes. My interests keep changing. I used to do pebble art. It's just collecting pebbles from wherever I go, and I used to paint on them. I used to use watercolour, but I want to come back to watercolour sometime.
My recent interests are coloured pencils, which came back. When I was very young, I used to do a lot of coloured pencils. And then I switched to watercolours and oil painting. So I just go around in circles.
One of the hobbies that I picked up during a pandemic is crochet. I made a scarf for Mother's Day. My mum and my dad were here last year. They got stuck because of the pandemic, and they couldn't go back home. So they stayed with me for 10 months. That is the jackpot that I had, that I got to spend so much time with my parents after I moved to the US.
**CRAIG BOX: And they got rewarded with a scarf.**
SAVITHA RAGHUNATHAN: Yeah.
**CRAIG BOX: One to share between them.**
SAVITHA RAGHUNATHAN: I started making a blanket for my dad. And it became so heavy, I might have to just pick up some lighter yarn. I still don't know the differences between different kinds of yarns, but I'm getting better.
I started out because I wanted to make these little toys. They call them [amigurumi](https://en.wikipedia.org/wiki/Amigurumi) in the crochet world. I wanted to make them. That's why I started out. I'm trying. I made [a little cat](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/2) which doesn't look like a cat, but it is a cat. I have to tell everyone that it's a cat so that they don't mock me later, but.
**CRAIG BOX: It's an artistic interpretation of a cat.**
SAVITHA RAGHUNATHAN: It definitely is!
---
_[Savitha Raghunathan](https://twitter.com/coffeeartgirl), now a Senior Software Engineer at Red Hat, served as the Kubernetes 1.22 release team lead._
_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._
@@ -72,7 +72,8 @@ The name of a Node object must be a valid
The [name](/docs/concepts/overview/working-with-objects/names#names) identifies a Node. Two Nodes
cannot have the same name at the same time. Kubernetes also assumes that a resource with the same
name is the same object. In case of a Node, it is implicitly assumed that an instance using the
same name will have the same state (e.g. network settings, root disk contents). This may lead to
same name will have the same state (e.g. network settings, root disk contents)
and attributes like node labels. This may lead to
inconsistencies if an instance was modified without changing its name. If the Node needs to be
replaced or updated significantly, the existing Node object needs to be removed from API server
first and re-added after the update.
@@ -98,6 +99,21 @@ When the [Node authorization mode](/docs/reference/access-authn-authz/node/) and
[NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) are enabled,
kubelets are only authorized to create/modify their own Node resource.
{{< note >}}
As mentioned in the [Node name uniqueness](#node-name-uniqueness) section,
when Node configuration needs to be updated, it is a good practice to re-register
the node with the API server. For example, if the kubelet being restarted with
the new set of `--node-labels`, but the same Node name is used, the change will
not take an effect, as labels are being set on the Node registration.
Pods already scheduled on the Node may misbehave or cause issues if the Node
configuration will be changed on kubelet restart. For example, already running
Pod may be tainted against the new labels assigned to the Node, while other
Pods, that are incompatible with that Pod will be scheduled based on this new
label. Node re-registration ensures all Pods will be drained and properly
re-scheduled.
{{< /note >}}
### Manual Node administration
You can create and modify Node objects using
@@ -122,6 +138,9 @@ To mark a Node unschedulable, run:
kubectl cordon $NODENAME
```
See [Safely Drain a Node](/docs/tasks/administer-cluster/safely-drain-node/)
for more details.
{{< note >}}
Pods that are part of a {{< glossary_tooltip term_id="daemonset" >}} tolerate
being run on an unschedulable Node. DaemonSets typically provide node-local services
@@ -162,8 +181,8 @@ The `conditions` field describes the status of all `Running` nodes. Examples of
| Node Condition | Description |
|----------------------|-------------|
| `Ready` | `True` if the node is healthy and ready to accept pods, `False` if the node is not healthy and is not accepting pods, and `Unknown` if the node controller has not heard from the node in the last `node-monitor-grace-period` (default is 40 seconds) |
| `DiskPressure` | `True` if pressure exists on the disk size--that is, if the disk capacity is low; otherwise `False` |
| `MemoryPressure` | `True` if pressure exists on the node memory--that is, if the node memory is low; otherwise `False` |
| `DiskPressure` | `True` if pressure exists on the disk size—that is, if the disk capacity is low; otherwise `False` |
| `MemoryPressure` | `True` if pressure exists on the node memory—that is, if the node memory is low; otherwise `False` |
| `PIDPressure` | `True` if pressure exists on the processes—that is, if there are too many processes on the node; otherwise `False` |
| `NetworkUnavailable` | `True` if the network for the node is not correctly configured, otherwise `False` |
{{< /table >}}
@@ -174,7 +193,8 @@ If you use command-line tools to print details of a cordoned Node, the Condition
cordoned nodes are marked Unschedulable in their spec.
{{< /note >}}
The node condition is represented as a JSON object. For example, the following structure describes a healthy node:
In the Kubernetes API, a node's condition is represented as part of the `.status`
of the Node resource. For example, the following JSON structure describes a healthy node:
```json
"conditions": [
@@ -189,7 +209,17 @@ The node condition is represented as a JSON object. For example, the following s
]
```
If the Status of the Ready condition remains `Unknown` or `False` for longer than the `pod-eviction-timeout` (an argument passed to the {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}), then all the Pods on the node are scheduled for deletion by the node controller. The default eviction timeout duration is **five minutes**. In some cases when the node is unreachable, the API server is unable to communicate with the kubelet on the node. The decision to delete the pods cannot be communicated to the kubelet until communication with the API server is re-established. In the meantime, the pods that are scheduled for deletion may continue to run on the partitioned node.
If the `status` of the Ready condition remains `Unknown` or `False` for longer
than the `pod-eviction-timeout` (an argument passed to the
for all Pods assigned to that node. The default eviction timeout duration is
**five minutes**.
In some cases when the node is unreachable, the API server is unable to communicate
with the kubelet on the node. The decision to delete the pods cannot be communicated to
the kubelet until communication with the API server is re-established. In the meantime,
the pods that are scheduled for deletion may continue to run on the partitioned node.
The node controller does not force delete pods until it is confirmed that they have stopped
running in the cluster. You can see the pods that might be running on an unreachable node as
@@ -199,10 +229,12 @@ may need to delete the node object by hand. Deleting the node object from Kubern
all the Pod objects running on the node to be deleted from the API server and frees up their
names.
The node lifecycle controller automatically creates
[taints](/docs/concepts/scheduling-eviction/taint-and-toleration/) that represent conditions.
When problems occur on nodes, the Kubernetes control plane automatically creates
[taints](/docs/concepts/scheduling-eviction/taint-and-toleration/) that match the conditions
affecting the node.
The scheduler takes the Node's taints into consideration when assigning a Pod to a Node.
Pods can also have tolerations which let them tolerate a Node's taints.
Pods can also have {{< glossary_tooltip text="tolerations" term_id="toleration" >}} that let
them run on a Node even though it has a specific taint.
See [Taint Nodes by Condition](/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-nodes-by-condition)
for more details.
@@ -222,10 +254,43 @@ on a Node.
### Info
Describes general information about the node, such as kernel version, Kubernetes version (kubelet and kube-proxy version), Docker version (if used), and OS name.
This information is gathered by Kubelet from the node.
Describes general information about the node, such as kernel version, Kubernetes
version (kubelet and kube-proxy version), container runtime details, and which
operating system the node uses.
The kubelet gathers this information from the node and publishes it into
the Kubernetes API.
### Node controller
## Heartbeats
Heartbeats, sent by Kubernetes nodes, help your cluster determine the
availability of each node, and to take action when failures are detected.
Lease is a lightweight resource, which improves the performance
of the node heartbeats as the cluster scales.
The kubelet is responsible for creating and updating the `NodeStatus` and
a Lease object.
- The kubelet updates the `NodeStatus` either when there is change in status
or if there has been no update for a configured interval. The default interval
for `NodeStatus` updates is 5 minutes, which is much longer than the 40 second default
timeout for unreachable nodes.
- The kubelet creates and then updates its Lease object every 10 seconds
(the default update interval). Lease updates occur independently from the
`NodeStatus` updates. If the Lease update fails, the kubelet retries with
exponential backoff starting at 200 milliseconds and capped at 7 seconds.
#### Reliability
### Rate limits on eviction
In most cases, the node controller limits the eviction rate to
`--node-eviction-rate` (default 0.1) per second, meaning it won't evict pods
@@ -281,7 +325,7 @@ from more than 1 node per 10 seconds.
The node eviction behavior changes when a node in a given availability zone
becomes unhealthy. The node controller checks what percentage of nodes in the zone
are unhealthy (NodeReady condition is ConditionUnknown or ConditionFalse) at
are unhealthy (NodeReady condition is `ConditionUnknown` or `ConditionFalse`) at
the same time:
- If the fraction of unhealthy nodes is at least `--unhealthy-zone-threshold`
(default 0.55), then the eviction rate is reduced.
@@ -293,15 +337,17 @@ the same time:
The reason these policies are implemented per availability zone is because one
availability zone might become partitioned from the master while the others remain
connected. If your cluster does not span multiple cloud provider availability zones,
then there is only one availability zone (i.e. the whole cluster).
then the eviction mechanism does not take per-zone unavailability into account.
A key reason for spreading your nodes across availability zones is so that the
workload can be shifted to healthy zones when one entire zone goes down.
Therefore, if all nodes in a zone are unhealthy, then the node controller evicts at
the normal rate of `--node-eviction-rate`. The corner case is when all zones are
completely unhealthy (i.e. there are no healthy nodes in the cluster). In such a
case, the node controller assumes that there is some problem with master
connectivity and stops all evictions until some connectivity is restored.
completely unhealthy (none of the nodes in the cluster are healthy). In such a
case, the node controller assumes that there is some problem with connectivity
between the control plane and the nodes, and doesn't perform any evictions.
(If there has been an outage and some nodes reappear, the node controller does
evict pods from the remaining nodes that are unhealthy or unreachable).
The node controller is also responsible for evicting pods running on nodes with
`NoExecute` taints, unless those pods tolerate that taint.
@@ -309,7 +355,7 @@ The node controller also adds {{< glossary_tooltip text="taints" term_id="taint"
corresponding to node problems like node unreachable or not ready. This means
that the scheduler won't place Pods onto unhealthy nodes.
### Nodecapacity
## Resource capacity tracking {#node-capacity}
Node objects track information about the Node's resource capacity: for example, the amount
of memory available and the number of CPUs.
@@ -356,7 +402,7 @@ Graceful node shutdown is controlled with the `GracefulNodeShutdown`
enabled by default in 1.21.
Note that by default, both configuration options described below,
`ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are set to zero,
`shutdownGracePeriod` and `shutdownGracePeriodCriticalPods` are set to zero,
thus not activating Graceful node shutdown functionality.
To activate the feature, the two kubelet config settings should be configured appropriately and set to non-zero values.
@@ -366,13 +412,13 @@ During a graceful shutdown, kubelet terminates pods in two phases:
2. Terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) running on the node.
Graceful node shutdown feature is configured with two [`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/) options:
* `ShutdownGracePeriod`:
* `shutdownGracePeriod`:
* Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
* `ShutdownGracePeriodCriticalPods`:
* Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `ShutdownGracePeriod`.
* `shutdownGracePeriodCriticalPods`:
* Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `shutdownGracePeriod`.
For example, if `ShutdownGracePeriod=30s`, and
`ShutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by
For example, if `shutdownGracePeriod=30s`, and
`shutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by
30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved
for gracefully terminating normal pods, and the last 10 seconds would be
reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
@@ -405,6 +451,11 @@ the kubelet, and the `--fail-swap-on` command line flag or `failSwapOn`
@@ -25,7 +25,7 @@ This page lists some of the available add-ons and links to their respective inst
* [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is an open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads.
* [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) is an overlay network provider that can be used with Kubernetes.
* [Knitter](https://github.com/ZTE/Knitter/) is a plugin to support multiple network interfaces in a Kubernetes pod.
* [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
* Multus is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
* [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) is a networking provider for Kubernetes based on [OVN (Open Virtual Network)](https://github.com/ovn-org/ovn/), a virtual networking implementation that came out of the Open vSwitch (OVS) project. OVN-Kubernetes provides an overlay based networking implementation for Kubernetes, including an OVS based implementation of load balancing and network policy.
* [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) is OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC), Multiple OVN overlay networking, dynamic subnet creation, dynamic creation of virtual networks, VLAN Provider network, Direct provider network and pluggable with other Multi-network plugins, ideal for edge based cloud native workloads in Multi-cluster networking
* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.
@@ -45,6 +45,11 @@ This page lists some of the available add-ons and links to their respective inst
## Infrastructure
* [KubeVirt](https://kubevirt.io/user-guide/#/installation/installation) is an add-on to run virtual machines on Kubernetes. Usually run on bare-metal clusters.
* The
[node problem detector](https://github.com/kubernetes/node-problem-detector)
runs on Linux nodes and reports system issues as either
[Events](/docs/reference/kubernetes-api/cluster-resources/event-v1/) or
@@ -160,7 +160,7 @@ If you're interested in learning more about `kubectl`, go ahead and read [kubect
The examples we've used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another.
For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels:
For instance, different applications would use different values for the `app` label, but a multi-tier application, such as the [guestbook example](https://github.com/kubernetes/examples/tree/master/guestbook/), would additionally need to distinguish each tier. The frontend could carry the following labels:
```yaml
labels:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Kubernetes Prow Robot
