Merge remote-tracking branch 'upstream/main' into dev-1.23

2021-10-26 14:13:06 -04:00
parent 0af4dd9687 2cf8506197
commit 4e755b2f2c
403 changed files with 11265 additions and 7503 deletions
@@ -3,6 +3,9 @@
 # When modifying this file, consider the security implications of
 # allowing listed reviewers / approvals to modify or remove any
 # configured GitHub Actions.
+#
+options:
+  no_parent_owners: true

 reviewers:
 - sig-docs-leads
@@ -33,7 +33,7 @@ exhaustive, and do not form part of our licenses.
     material not subject to the license. This includes other CC-
     licensed material, or material used under an exception or
     limitation to copyright. More considerations for licensors:
-	wiki.creativecommons.org/Considerations_for_licensors
+    wiki.creativecommons.org/Considerations_for_licensors

     Considerations for the public: By using one of our public
     licenses, a licensor grants the public permission to use the
@@ -48,9 +48,9 @@ exhaustive, and do not form part of our licenses.
     rights in the material. A licensor may make special requests,
     such as asking that all changes be marked or described.
     Although not required by our licenses, you are encouraged to
-     respect those requests where reasonable. More_considerations
-     for the public: 
-	wiki.creativecommons.org/Considerations_for_licensees
+     respect those requests where reasonable. More considerations
+     for the public:
+    wiki.creativecommons.org/Considerations_for_licensees

 =======================================================================

@@ -378,7 +378,7 @@ Section 8 -- Interpretation.
 Creative Commons is not a party to its public
 licenses. Notwithstanding, Creative Commons may elect to apply one of
 its public licenses to material it publishes and in those instances
-will be considered the "Licensor." The text of the Creative Commons
+will be considered the “Licensor.” The text of the Creative Commons
 public licenses is dedicated to the public domain under the CC0 Public
 Domain Dedication. Except for the limited purpose of indicating that
 material is shared under a Creative Commons public license or as
@@ -393,3 +393,4 @@ the avoidance of doubt, this paragraph does not form part of the
 public licenses.

 Creative Commons may be contacted at creativecommons.org.
+
@@ -18,6 +18,7 @@ aliases:
    - annajung
    - bradtopol
    - celestehorgan
+    - divya-mohan0209
    - jimangel
    - jlbutler
    - kbhawkey
@@ -31,6 +32,7 @@ aliases:
    - bradtopol
    - celestehorgan
    - daminisatya
+    - divya-mohan0209
    - jimangel
    - kbhawkey
    - onlydole
@@ -79,6 +81,7 @@ aliases:
    - anubha-v-ardhan
    - divya-mohan0209
    - mittalyashu
+    - verma-kunal
  sig-docs-id-owners: # Admins for Indonesian content
    - ariscahyadi
    - danninov
@@ -133,6 +136,7 @@ aliases:
    - yoonian
    - ysyukr
  sig-docs-leads: # Website chairs and tech leads
+    - divya-mohan0209
    - jimangel
    - kbhawkey
    - onlydole
@@ -7,7 +7,7 @@

 ## डॉक्स में योगदान देना

-आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में कोई भी परिवर्तन करना चाहते हैं, और जब आप उन परिवर्तनों को हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं।
+आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में परिवर्तन करने के बाद जब आप उनको हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं।

 एक बार जब आपका pull request बन जाता है, तो एक कुबरनेट्स समीक्षक स्पष्ट, कार्रवाई योग्य प्रतिक्रिया प्रदान करने की जिम्मेदारी लेगा। pull request के मालिक के रूप में, **यह आपकी जिम्मेदारी है कि आप कुबरनेट्स समीक्षक द्वारा प्रदान की गई प्रतिक्रिया को संबोधित करने के लिए अपने pull request को संशोधित करें।**

@@ -23,9 +23,12 @@

 ## `README.md`'s स्थानीयकरण कुबरनेट्स प्रलेखन

-आप पर हिंदी स्थानीयकरण के maintainers तक पहुँच सकते हैं:
+आप हिंदी स्थानीयकरण के मैन्टेनरों तक पहुँच सकते हैं:

+* Anubhav Vardhan ([Slack](https://kubernetes.slack.com/archives/D0261C0A3R8), [Twitter](https://twitter.com/anubha_v_ardhan), [GitHub](https://github.com/anubha-v-ardhan))
+* Divya Mohan ([Slack](https://kubernetes.slack.com/archives/D027R7BE804), [Twitter](https://twitter.com/Divya_Mohan02), [GitHub](https://github.com/divya-mohan0209))
 * Yashu Mittal ([Twitter](https://twitter.com/mittalyashu77), [GitHub](https://github.com/mittalyashu))
+
 * [Slack channel](https://kubernetes.slack.com/messages/kubernetes-docs-hi)

 ## स्थानीय रूप से डॉकर का उपयोग करके साइट चलाना
@@ -10,5 +10,6 @@
 # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE
 # INSTRUCTIONS AT https://kubernetes.io/security/

+divya-mohan0209
 jimangel
-sftim
+sftim
@@ -338,11 +338,35 @@ main {
    border-left-color: #d9534f;
  }

+
  h1:first-of-type + blockquote.callout {
    margin-top: 1.5em;
  }
 }

+// Special color for third party content disclaimers
+.alert.third-party-content { border-left-color: #222 };
+
+// Highlight disclaimer when targeted as a fragment
+
+#third-party-content-disclaimer {
+  color: #000;
+  background: #f8f9fa;
+  transition: all 0.5s ease;
+}
+
+@keyframes disclaimer-highlight {
+  from { background: #f8f922; color: #000; }
+  50% { background: #f8f944; color: #000; }
+  to { background: #f8f9cb; color: #000; }
+}
+
+#third-party-content-disclaimer:target {
+  color: #000;
+  animation: disclaimer-highlight 1.25s ease;
+  background: #f8f9cb;
+}
+
 .deprecation-warning {
    padding: 20px;
    margin: 20px 0;
@@ -354,6 +378,12 @@ body.td-home .deprecation-warning, body.td-blog .deprecation-warning, body.td-do
    border-radius: 3px;
 }

+
+.td-documentation .td-content > .highlight {
+  max-width: initial;
+  width: 100%;
+}
+
 body.td-home #deprecation-warning {
    max-width: 1000px;
    margin-top: 2.5rem;
@@ -635,6 +665,13 @@ body.td-documentation {
  }
 }

+// Match Docsy-imposed max width on text body
+@media (min-width: 1200px) {
+  body.td-blog main .td-content > figure {
+    max-width: 80%;
+  }
+}
+
 .td-content {
  table code {
    background-color: inherit !important;
@@ -642,3 +679,45 @@ body.td-documentation {
    font-size: inherit !important;
  }
 }
+
+/* Force size constraints on figures */
+figure {
+  &.diagram-small img {
+    max-height: clamp(20mm,12em,80vh);
+    margin-left: auto;
+    margin-right: auto;
+    display: block;
+  }
+  &.diagram-medium img {
+    max-height: clamp(25mm,20em,80vh);
+    margin-left: auto;
+    margin-right: auto;
+    display: block;
+  }
+  &.diagram-large img {
+    max-width: clamp(0vw, 95vw, 100%);
+    max-height: calc(80vh - 8rem);
+  }
+}
+
+@media only screen and (min-width: 768px) {
+  figure {
+    &.diagram-small, &.diagram-medium {
+      max-width: 80%;
+    }
+    &.diagram-large {
+      max-width: 100%;
+      width: 100%;
+    }
+    &.diagram-small img {
+      max-width: clamp(30rem, 45ch, 100mm);
+    }
+    &.diagram-medium img {
+      max-width: clamp(50rem, 20ch, 160mm);
+    }
+    &.diagram-large img {
+      max-width: clamp(25vw, 95vw, 100%);
+      max-height: calc(100vh - 10rem);
+    }
+  }
+}
@@ -8,7 +8,7 @@ options:
  substitution_option: ALLOW_LOOSE
 steps:
    # It's fine to bump the tag to a recent version, as needed
-  - name: "gcr.io/k8s-testimages/gcb-docker-gcloud:v20190906-745fed4"
+  - name: "gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20210917-12df099d55"
    entrypoint: make
    env:
      - DOCKER_CLI_EXPERIMENTAL=enabled
@@ -123,6 +123,7 @@ id = "UA-00000000-0"
 [params]
 copyright_k8s = "The Kubernetes Authors"
 copyright_linux =  "Copyright © 2020 The Linux Foundation ®."
+
 # privacy_policy = "https://policies.google.com/privacy"

 # First one is picked as the Twitter card image if not set on page.
@@ -42,12 +42,12 @@ Kubernetes ist Open Source und bietet Dir die Freiheit, die Infrastruktur vor Or
        <button id="desktopShowVideoButton" onclick="kub.showVideo()">Video ansehen</button>
        <br>
        <br>
-        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu20" button id="desktopKCButton">Besuche die KubeCon - 13-16 August 2020 in Amsterdam</a>
+        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21" button id="desktopKCButton">Besuche die KubeCon North America vom 11. bis 15. Oktober 2021</a>
        <br>
        <br>
        <br>
        <br>
-        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna20" button id="desktopKCButton">Besuche die KubeCon - 17-20 November 2020 in Boston</a>
+        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22" button id="desktopKCButton">Besuche die KubeCon Europe vom 17. bis 20. Mai 2022</a>
 </div>
 <div id="videoPlayer">
    <iframe data-url="https://www.youtube.com/embed/H06qrNmGqyE?autoplay=1" frameborder="0" allowfullscreen></iframe>
@@ -4,58 +4,253 @@ layout: basic
 cid: community
 ---

-<section id="mainContent">
-    <main>
-        <div class="content">
-            <h3>Die Gewissheit, dass Kubernetes überall und für alle gut funktioniert.</h3>
-            <p>Verbinden Sie sich mit der Kubernetes-Community in unserem <a href="http://slack.k8s.io/">Slack Kanal</a>, <a href="https://discuss.kubernetes.io/">Diskussionsforum</a>, oder beteiligen Sie sich an der <a href="https://groups.google.com/g/kubernetes-dev"> Kubernetes-dev-Google-Gruppe</a>. Eine wöchentliches Community-Meeting findet per Videokonferenz statt, um den Stand der Dinge zu diskutieren, folgen Sie
-		<a href="https://github.com/kubernetes/community/blob/master/events/community-meeting.md">diesen Anweisungen</a> für Informationen wie Sie teilnehmen können.</p>
-            <p>Sie können Kubernetes auch auf der ganzen Welt über unsere
-                <a href="https://www.meetup.com/topics/kubernetes/">Kubernetes Meetup Community</a> und der
-                <a href="https://www.meetup.com/Kubernetes-Cloud-Native-Online-Meetup/">Kubernetes Cloud Native Meetup Community</a> beitreten.</p>
-          </div>
-        <div class="content">
-            <h3>Special Interest Groups (SIGs)</h3>
-            <p>Haben Sie ein besonderes Interesse daran, wie Kubernetes mit einer anderen Technologie arbeitet?  Werfen Sie einen Blick auf unsere kontinuierlich wachsende
-                <a href="https://git.k8s.io/community/sig-list.md">Listen von SIGs</a>, von AWS und Openstack bis hin zu Big Data und Skalierbarkeit, es gibt einen Platz für Sie, an dem Sie mitwirken können, und Anweisungen zur Gründung einer neuen SIG finden, wenn Ihr besonderes Interesse (noch) nicht abgedeckt ist.
-            </p>
+<div class="newcommunitywrapper">
+<div class="banner1">
+  <img src="/images/community/kubernetes-community-final-02.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;padding-left:0px" class="desktop">
+  <img src="/images/community/kubernetes-community-02-mobile.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;padding-left:0px" class="mobile">
+</div>

-            <p>Als Mitglied der Kubernetes-Community sind Sie herzlich eingeladen, an allen SIG-Treffen teilzunehmen, die Sie interessieren. Eine Registrierung ist nicht erforderlich.</p>
+<div class="intro">
+<br class="mobile">
+<p>Die Kubernetes-Community - Nutzer, Mitwirkende und die Kultur, die wir gemeinsam aufgebaut haben - ist einer der Hauptgründe für den kometenhaften Aufstieg dieses Open-Source-Projekts. Unsere Kultur und unsere Werte wachsen und entwickeln sich mit dem Wachstum und der Veränderung des Projekts selbst. Wir alle arbeiten gemeinsam an der ständigen Verbesserung des Projekts und der Art und Weise, wie wir daran arbeiten.
+<br><br>Wir sind die Leute, die Probleme und Pull-Requests einreichen, an SIG-Treffen (Special Interest Groups), Kubernetes-Treffen und der KubeCon teilnehmen, sich für die Einführung und Innovation von Kubernetes einsetzen, <code>kubectl get pods</code> ausführen und auf tausend andere wichtige Arten beitragen. Lies weiter, um zu erfahren, wie Du dich engagieren und Teil dieser faszinierenden Gemeinschaft werden kannst.</p>
+<br class="mobile">
+</div>

-        </div>
+<div class="community__navbar">

-        <div class="content">
-          <h3>Verhaltensregeln</h3>
-          <p>Die Kubernetes-Community schätzt Respekt und Inklusivität und setzt einen <a href="code-of-conduct/">Verhaltenskodex</a>
-            in allen Interaktionen durch. Wenn Sie einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder Sitzung,
-            in Slack oder in einem anderen Kommunikationsmechanismus feststellen, wenden Sie sich
-            bitte an das <a href="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct">Kubernetes Code of Conduct Committee</a> <a href="mailto:conduct@kubernetes.io">conduct@kubernetes.io</a>. Ihre Anonymität wird geschützt.
-          </p>
-         </div>
-    </main>
-</section>
+<a href="#values">Gemeinschaftswerte</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#conduct">Verhaltenskodex </a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#videos">Videos</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#discuss">Diskussionen</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#events">Veranstaltungen und meetups</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#news">Neuigkeiten</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="/releases">Releases</a>

-<section id="talkToUs">
-    <main>
-        <h3>Talk to Us!</h3>
-        <h4>Wir würden uns freuen, von Ihnen zu hören, wie Sie Kubernetes verwenden<br>und was wir tun können, um es besser zu machen.</h4>
-        <div id="bigSocial">
-            <div>
-                <a href="https://twitter.com/kubernetesio">@kubernetesio</a>
-                <p>Erhalten Sie die neuesten Nachrichten und Updates.</p>
-            </div>
-            <div>
-                <a href="https://github.com/kubernetes/kubernetes">Github Project</a>
-                <p>Informieren Sie sich über das Projekt und erwägen Sie, einen Beitrag zu leisten.</p>
-            </div>
-            <div>
-                <a href="http://slack.k8s.io/">#kubernetes-users</a>
-                <p>Unser Slack-Kanal ist der beste Weg, um unsere Ingenieure zu kontaktieren und Ihre Ideen mit ihnen zu teilen.</p>
-            </div>
-            <div>
-                <a href="http://stackoverflow.com/questions/tagged/kubernetes">Stack Overflow</a>
-                <p>Unser Benutzerforum ist ein großartiger Ort, um Community-Support zu erhalten.</p>
-            </div>
-        </div>
-    </main>
-</section>
+</div>
+<br class="mobile"><br class="mobile">
+<div class="imagecols">
+<br class="mobile">
+  <div class="imagecol">
+    <img src="/images/community/kubernetes-community-final-03.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%" class="desktop">
+  </div>
+
+  <div class="imagecol">
+    <img src="/images/community/kubernetes-community-final-04.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%" class="desktop">
+  </div>
+
+  <div class="imagecol" style="margin-right:0% important">
+    <img src="/images/community/kubernetes-community-final-05.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;margin-right:0% important" class="desktop">
+  </div>
+  <img src="/images/community/kubernetes-community-04-mobile.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;margin-bottom:3%" class="mobile">
+<a name="values"></a>
+</div>
+
+<div><a name="values"></a></div>
+<div class="conduct">
+<div class="conducttext">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<div class="conducttextnobutton" style="margin-bottom:2%"><h1>Gemeinschaftswerte</h1>
+Die Werte der Kubernetes-Community sind der Grundstein für den anhaltenden Erfolg des Projekts.<br>
+Diese Prinzipien leiten jeden Aspekt des Kubernetes-Projekts.
+<br>
+<a href="/community/values/">
+<br class="mobile"><br class="mobile">
+<span class="fullbutton">
+  MEHR ERFAHREN
+</span>
+</a>
+</div><a name="conduct"></a>
+</div>
+</div>
+
+
+
+<div class="conduct">
+<div class="conducttext">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<div class="conducttextnobutton" style="margin-bottom:2%"><h1>Verhaltenskodex</h1>
+Die Kubernetes-Gemeinschaft legt Wert auf Respekt und Inklusivität und setzt bei allen Interaktionen einen Verhaltenskodex durch. Wenn Du einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder einem Treffen, in Slack oder in einem anderen Kommunikationsmechanismus bemerkst, wende dich an das Kubernetes Code of Conduct Committee unter <a href="mailto:conduct@kubernetes.io" style="color:#0662EE;font-weight:300">conduct@kubernetes.io</a>. Alle Berichte werden vertraulich behandelt. Du kannst&nbsp;<a href="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct" style="color:#0662EE;font-weight:300">hier</a> mehr über den Ausschuss erfahren.
+<br>
+<a href="https://kubernetes.io/de/community/code-of-conduct/">
+<br class="mobile"><br class="mobile">
+
+<span class="fullbutton">
+MEHR ERFAHREN
+</span>
+</a>
+</div><a name="videos"></a>
+</div>
+</div>
+
+
+
+<div class="videos">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<h1 style="margin-top:0px">Videos</h1>
+
+<div style="margin-bottom:4%;font-weight:300;text-align:center;padding-left:10%;padding-right:10%">Wir sind auf YouTube, und zwar oft. Abonniere uns für eine Vielzahl von&nbsp;Themen.</div>
+
+<div class="videocontainer">
+
+<div class="video">
+
+  <iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg" title="Monatliche Bürozeiten" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
+
+<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg">
+<div class="videocta">
+Monatliche Bürozeiten ansehen&nbsp;&#9654;</div>
+</a>
+</div>
+
+<div class="video">
+  <iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ" title="Wöchentliche Treffen der Gemeinschaft" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
+<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ">
+<div class="videocta">
+Wöchentliche Treffen der Gemeinschaft ansehen&nbsp;&#9654;
+</div>
+</a>
+</div>
+
+<div class="video">
+
+<iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F" title="Vortrag eines Mitglieds der Gemeinschaft" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
+
+<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F">
+<div class="videocta">
+Vortrag eines Mitglieds der Gemeinschaft ansehen&nbsp;&#9654;
+</div>
+
+</a>
+<a name="discuss"></a>
+</div>
+</div>
+</div>
+
+
+<div class="resources">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<h1 style="padding-top:1%">Diskussionen</h1>
+
+<div style="font-weight:300;text-align:center">Wir reden gerne und viel. Triff uns auf einer dieser Plattformen und beteilige dich an den Diskussionen.</div>
+
+<div class="resourcecontainer">
+
+<div class="resourcebox">
+<img src="/images/community/discuss.png" alt=Forum" style="width:80%;padding-bottom:2%">
+<a href="https://discuss.kubernetes.io/" style="color:#0662EE;display:block;margin-top:1%">
+forum&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">
+Themenbezogene technische Diskussionen, die eine Brücke zu Docs, StackOverflow und vielem mehr schlagen.
+</div>
+</div>
+
+<div class="resourcebox">
+<img src="/images/community/twitter.png" alt="Twitter" style="width:80%;padding-bottom:2%">
+<a href="https://twitter.com/kubernetesio" style="color:#0662EE;display:block;margin-top:1%">
+twitter&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">Echtzeit-Ankündigungen von Blogeinträgen, Veranstaltungen, Neuigkeiten und Ideen
+</div>
+</div>
+
+<div class="resourcebox">
+<img src="/images/community/github.png" alt="GitHub" style="width:80%;padding-bottom:2%">
+<a href="https://github.com/kubernetes/kubernetes" style="color:#0662EE;display:block;margin-top:1%">
+github&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">
+Die gesamte Projekt- und Problemverfolgung und natürlich der Code
+</div>
+</div>
+
+<div class="resourcebox">
+<img src="/images/community/stack.png" alt="Stack Overflow" style="width:80%;padding-bottom:2%">
+<a href="https://stackoverflow.com/search?q=kubernetes" style="color:#0662EE;display:block;margin-top:1%">
+stack overflow&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">
+  Technische Problemlösung für jeden Anwendungsfall
+  <a name="events"></a>
+</div>
+</div>
+
+<!--
+<div class="resourcebox">
+
+<img src="/images/community/slack.png" style="width:80%">
+
+slack&nbsp;&#9654;
+
+<div class="resourceboxtext" style="font-size:11px;text-transform:none !important;font-weight:200;line-height:1.4em;color:#333333;margin-top:4%">
+With 170+ channels, you'll find one that fits your needs.
+</div>
+
+</div>-->
+
+</div>
+</div>
+<div class="events">
+  <br class="mobile"><br class="mobile">
+  <br class="tablet"><br class="tablet">
+<div class="eventcontainer">
+ <h1 style="color:white !important">Bevorstehende Veranstaltungen</h1>
+    {{< upcoming-events >}}
+</div>
+</div>
+
+<div class="meetups">
+<div class="meetupcol">
+<div class="meetuptext">
+<h1 style="text-align:left">Globale Gemeinschaft</h1>
+Mit mehr als 150 Treffen auf der ganzen Welt, Tendenz steigend, solltest du deine lokalen Kube-Leute finden. Wenn keins in der Nähe ist, nimm die Sache in die Hand und gründe dein eigenes.
+</div>
+<a href="https://www.meetup.com/topics/kubernetes/">
+<div class="button">
+EIN MEETUP FINDEN
+</div>
+</a>
+<a name="news"></a>
+</div>
+</div>
+
+
+<!--
+<div class="contributor">
+<div class="contributortext">
+<br>
+<h1 style="text-align:left">
+New Contributors Site
+ </h1>
+Text about new contributors site.
+
+<br><br>
+
+<div class="button">
+VISIT SITE
+</div>
+</div>
+</div>
+
+-->
+
+<div class="news">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<h1 style="margin-bottom:2%">Aktuelle Neuigkeiten</h1>
+
+<br>
+<div class="twittercol1">
+<a class="twitter-timeline" data-tweet-limit="1" href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">Tweets von kubernetesio</a> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
+</div>
+
+<br>
+<br><br><br><br>
+</div>
+
+</div>
@@ -0,0 +1,201 @@
+/*  SECTIONS  */
+.section {
+	clear: both;
+	padding: 0px;
+	margin-bottom: 2em;
+}
+
+.kcsp_section {
+	clear: both;
+	padding: 0px;
+	margin-bottom: 2em;
+}
+
+/*  COLUMN SETUP  */
+.col {
+	display: block;
+	float:left;
+	margin: 1% 0 1% 1.6%;
+	background-color: #f9f9f9;
+}
+.col:first-child { margin-left: 0; }
+
+
+/*  GROUPING  */
+.group:before,
+.group:after {
+	content:"";
+	display:table;
+}
+.group:after {
+	clear:both;
+}
+.group {
+    zoom:1; /* For IE 6/7 */
+}
+
+/*  GRID OF THREE  */
+.span_3_of_3 {
+	width: 35%;
+	background-color: #f9f9f9;
+	padding: 20px;
+}
+.span_2_of_3 {
+	width: 35%;
+	background-color: #f9f9f9;
+	padding: 20px;
+}
+.span_1_of_3 {
+	width: 35%;
+	background-color: #f9f9f9;
+	padding: 20px;
+}
+
+.col-container {
+    display: table; /* Make the container element behave like a table */
+    width: 100%; /* Set full-width to expand the whole page */
+		padding-bottom: 30px;
+}
+
+.col-nav {
+    display: table-cell; /* Make elements inside the container behave like table cells */
+		width: 18%;
+		background-color: #f9f9f9;
+		padding: 20px;
+		border: 5px solid white;
+}
+
+/*  GO FULL WIDTH AT LESS THAN 480 PIXELS */
+
+@media only screen and (max-width: 480px) {
+	.col { margin: 1% 0 1% 0%;}
+	.span_3_of_3, .span_2_of_3, .span_1_of_3 { width: 100%; }
+}
+
+@media only screen and (max-width: 650px) {
+    .col-nav {
+        display: block;
+        width: 100%;
+    }
+}
+
+.button{
+  max-width: 100%;
+	box-sizing: border-box;
+	margin: 0;
+	display: inline-block;
+	border-radius: 6px;
+	padding: 0 20px;
+	line-height: 40px;
+	color: #ffffff;
+	font-size: 16px;
+	background-color: #3371e3;
+	text-decoration: none;
+ }
+
+h5 {
+	font-size: 16px;
+	line-height: 1.5em;
+	margin-bottom: 2em;
+}
+
+#usersGrid a {
+	display: inline-block;
+	background-color: #f9f9f9;
+}
+
+#ktpContainer, #distContainer, #kcspContainer, #isvContainer, #servContainer {
+	position: relative;
+	width: 100%;
+	display: flex;
+	justify-content: space-between;
+	flex-wrap: wrap;
+}
+
+#isvContainer {
+	margin-bottom: 80px;
+}
+
+#kcspContainer {
+	margin-bottom: 80px;
+}
+
+#distContainer {
+	margin-bottom: 80px;
+}
+
+#ktpContainer {
+	margin-bottom: 80px;
+}
+
+.partner-box {
+	position: relative;
+	width: 47%;
+	max-width: 48%;
+	min-width: 48%;
+	margin-bottom: 20px;
+	padding: 20px;
+	flex: 1;
+	display: flex;
+	justify-content: left;
+	align-items: flex-start;
+}
+
+.partner-box img {
+	background-color: #f9f9f9;
+}
+
+.partner-box > div {
+	margin-left: 30px;
+}
+
+.partner-box a {
+	color: #3576E3;
+}
+
+@media screen and (max-width: 1024px) {
+	.partner-box {
+		flex-direction: column;
+		justify-content: flex-start;
+	}
+
+	.partner-box > div {
+		margin: 20px 0 0;
+	}
+}
+
+@media screen and (max-width: 568px) {
+	#ktpContainer, #distContainter, #kcspContainer, #isvContainer, #servContainer {
+		justify-content: center;
+	}
+
+	.partner-box {
+		flex-direction: column;
+		justify-content: flex-start;
+		width: 100%;
+		max-width: 100%;
+		min-width: 100%;
+	}
+
+	.partner-box > div {
+		margin: 20px 0 0;
+	}
+}
+
+@media screen and (max-width: 568px) {
+	#ktpContainer, #distContainer, #kcspContainer, #isvContainer, #servContainer {
+		justify-content: center;
+	}
+
+	.partner-box {
+		flex-direction: column;
+		justify-content: flex-start;
+		width: 100%;
+		max-width: 100%;
+		min-width: 100%;
+	}
+
+	.partner-box > div {
+		margin: 20px 0 0;
+	}
+}
@@ -1,91 +1,53 @@
 ---
 title: Partner
 bigheader: Kubernetes Partner
-abstract: Entwicklung des Kubernetes-Ökosystems.
+abstract: Erweiterung des Kubernetes-Ökosystems.
 class: gridPage
 cid: partners
 ---

 <section id="users">
-    <main>
-        <h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, dynamische Codebasis zu schaffen, die ein Spektrum von aufeinander abgestimmten Plattformen unterstützt.</h5>
-        <div class="col-container">
-          <div class="col-nav">
-            <center>
-              <h5>
-                <b>Kubernetes zertifizierte Service Provider</b>
-              </h5>
-              <br>Geprüfte Service Provider mit großer Erfahrung, die Unternehmen bei der erfolgreichen Einführung von Kubernetes unterstützen.
-              <br><br><br>
-              <button id="kcsp" class="button" onClick="updateSrc(this.id)">KCSP-Partner anzeigen</button>
-              <br><br>Interessiert daran, ein <a href="https://www.cncf.io/certification/kcsp/">KCSP</a> zu werden?
-            </center>
-          </div>
-          <div class="col-nav">
-            <center>
-              <h5>
-                <b>Kubernetes-Distributionen, gehostete Plattformen und zertifizierte Installateure</b>
-              </h5>Software-Konformität stellt sicher, dass die Kubernetes-Versionen aller Hersteller die erforderlichen APIs unterstützen.
-              <br><br><br>
-              <button id="conformance" class="button" onClick="updateSrc(this.id)">Zertifizierte Partner anzeigen</button>
-              <br><br>Interessiert daran, <a href="https://www.cncf.io/certification/software-conformance/">Kubernetes zertifiziert</a> zu werden?
-            </center>
-          </div>
-          <div class="col-nav">
-            <center>
-              <h5><b>Kubernetes Training Partner</b></h5>
-              <br>Geprüfte Schulungsanbieter, die über umfassende Erfahrung in Cloud Native Technologietrainings verfügen.
-              <br><br><br><br>
-              <button id="ktp" class="button" onClick="updateSrc(this.id)">KTP Partner anzeigen</button>
-              <br><br>Interessiert daran, ein <a href="https://www.cncf.io/certification/training/">KTP</a> zu werden?
-            </center>
-          </div>
-        </div>
-<script src="https://code.jquery.com/jquery-3.3.1.min.js" integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
-<script type="text/javascript">
-
-			var defaultLink = "https://landscape.cncf.io/category=kubernetes-certified-service-provider&format=card-mode&grouping=category&embed=yes";
-			var firstLink = "https://landscape.cncf.io/category=certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer&format=card-mode&grouping=category&embed=yes";
-      var secondLink = "https://landscape.cncf.io/category=kubernetes-training-partner&format=card-mode&grouping=category&embed=yes";
-
-      function updateSrc(buttonId) {
-      	if (buttonId == "kcsp") {
-        		$("#landscape").attr("src",defaultLink);
-            window.location.hash = "#kcsp";
-        }
-        if (buttonId == "conformance") {
-         		$("#landscape").attr("src",firstLink);
-            window.location.hash = "#conformance";
-        }
-        if (buttonId == "ktp") {
-        		$("#landscape").attr("src",secondLink);
-            window.location.hash = "#ktp";
-        }
-      }
-
-      // Automatically load the correct iframe based on the URL fragment
-      document.addEventListener('DOMContentLoaded', function() {
-        var showContent = "kcsp";
-        if (window.location.hash) {
-          console.log('hash is:', window.location.hash.substring(1));
-          showContent = window.location.hash.substring(1);
-        }
-        updateSrc(showContent);
-      });
-</script>
-<body>
-    <div id="frameHolder">
-      <iframe id="landscape" frameBorder="0" scrolling="no" style="width: 1px; min-width: 100%" src=""></iframe>
-      <script src="https://landscape.cncf.io/iframeResizer.js"></script>
+    <h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, lebendige Codebasis zu schaffen, die ein Spektrum von ergänzenden Plattformen unterstützt.</h5>
+    <div class="col-container">
+      <div class="col-nav">
+        <center>
+          <h5>
+            <b>Kubernetes-zertifizierte Service-Anbieter</b>
+          </h5>
+          <br>Geprüfte Dienstleister mit umfassender Erfahrung bei der erfolgreichen Einführung von Kubernetes in Unternehmen.
+          <br><br><br>
+          <button class="button landscape-trigger landscape-default" data-landscape-types="kubernetes-certified-service-provider" id="kcsp">KCSP Partner anzeigen</button>
+          <br><br>Interessiert daran, ein
+          <a href="https://www.cncf.io/certification/kcsp/">KCSP</a> zu werden?
+        </center>
+      </div>
+      <div class="col-nav">
+        <center>
+          <h5>
+            <b>Zertifizierte Kubernetes-Distributionen, gehostete Plattformen und Installationssysteme</b>
+          </h5>Die Softwarekonformität stellt sicher, dass die Kubernetes-Version eines jeden Anbieters die erforderlichen APIs unterstützt.
+          <br><br><br>
+          <button class="button landscape-trigger" data-landscape-types="certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer" id="conformance">Konforme Partner anzeigen</button>
+          <br><br>Interessiert daran,
+          <a href="https://www.cncf.io/certification/software-conformance/">Kubernetes Zertifiziert</a> zu werden?
+        </center>
+      </div>
+      <div class="col-nav">
+        <center>
+          <h5>
+            <b>Kubernetes Schulungspartner</b>
+          </h5>
+          <br>Geprüfte Schulungsanbieter mit umfassender Erfahrung in der Weiterbildung im Bereich Cloud Native Technology.
+          <br><br><br>
+          <button class="button landscape-trigger" data-landscape-types="kubernetes-training-partner" id="ktp">KTP Partner anzeigen</button>
+          <br><br>Interessiert daran, ein
+          <a href="https://www.cncf.io/certification/training/">KTP</a> zu werden?
+        </center>
+      </div>
    </div>
-</body>
-    </main>
+    {{< cncf-landscape helpers=true >}}
 </section>

 <style>
    {{< include "partner-style.css" >}}
-</style>
-
-<script>
-    {{< include "partner-script.js" >}}
-</script>
+</style>
@@ -144,10 +144,9 @@ ways to address it.
 - Specifically add an iptables rule to drop the packets that are marked as
  *INVALID*, so it won’t reach to client pod and cause harm.

-The fix is drafted (https://github.com/kubernetes/kubernetes/pull/74840), but
-unfortunately it didn’t catch the v1.14 release window. However, for the users
-that are affected by this bug, there is a way to mitigate the problem by applying
-the following rule in your cluster.
+The [fix](https://github.com/kubernetes/kubernetes/pull/74840) is available in v1.15+.
+However, for the users that are affected by this bug, there is a way to mitigate the
+problem by applying the following rule in your cluster.

 ```yaml
 apiVersion: extensions/v1beta1
@@ -28,7 +28,7 @@ as cgroups v2 and user namespaces are being implemented in these newer CRI
 runtimes. Removing support for the dockershim will allow further development in
 those areas.

-[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1985-remove-dockershim
+[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim

 ### Can I still use Docker in Kubernetes 1.20?

@@ -42,9 +42,11 @@ startup if using Docker as the runtime.

 Given the impact of this change, we are using an extended deprecation timeline.
 It will not be removed before Kubernetes 1.22, meaning the earliest release without
-dockershim would be 1.23 in late 2021. We will be working closely with vendors
-and other ecosystem groups to ensure a smooth transition and will evaluate things
-as the situation evolves.
+dockershim would be 1.23 in late 2021. 
+_Update_: removal of dockershim is scheduled for Kubernetes v1.24, see 
+[Dockershim Removal Kubernetes Enhancement Proposal][drkep].
+We will be working closely with vendors and other ecosystem groups to ensure a smooth transition and will evaluate 
+things as the situation evolves.


 ### Can I still use dockershim after it is removed from Kubernetes?
@@ -0,0 +1,72 @@
+---
+layout: blog
+title: "Spotlight on SIG Node"
+date: 2021-09-27
+slug: sig-node-spotlight-2021
+---
+
+**Author:** Dewan Ahmed, Red Hat
+
+## Introduction
+
+In Kubernetes, a _Node_ is a representation of a single machine in your cluster. [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) owns that very important Node component and supports various subprojects such as Kubelet, Container Runtime Interface (CRI) and more to support how the pods and host resources interact. In this blog, we have summarized our conversation with [Elana Hashman (EH)](https://twitter.com/ehashdn) & [Sergey Kanzhelev (SK)](https://twitter.com/SergeyKanzhelev), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved.
+
+## A summary of our conversation
+
+### Could you tell us a little about what SIG Node does?
+
+SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIG's focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. This SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability.
+
+### "SIG Node is a vertical SIG" could you explain a bit more?
+
+EH: There are two kinds of SIGs: horizontal and vertical. Horizontal SIGs are concerned with a particular function of every component in Kubernetes: for example, SIG Security considers security aspects of every component in Kubernetes, or SIG Instrumentation looks at the logs, metrics, traces and events of every component in Kubernetes. Such SIGs don't tend to own a lot of code.
+
+Vertical SIGs, on the other hand, own a single component, and are responsible for approving and merging patches to that code base. SIG Node owns the "Node" vertical, pertaining to the kubelet and its lifecycle. This includes the code for the kubelet itself, as well as the node controller, the container runtime interface, and related subprojects like the node problem detector. 
+
+### How did the CI subproject start? Is this specific to SIG Node and how does it help the SIG?
+
+SK: The subproject started as a follow up after one of the releases was blocked by numerous test failures of critical tests. These tests haven’t started falling all at once, rather continuous lack of attention led to slow degradation of tests quality. SIG Node was always prioritizing quality and reliability, and forming of the subproject was a way to highlight this priority.
+
+### As the 3rd largest SIG in terms of number of issues and PRs, how does your SIG juggle so much work?
+
+EH: It helps to be organized. When I increased my contributions to the SIG in January of 2021, I found myself overwhelmed by the volume of pull requests and issues and wasn't sure where to start. We were already tracking test-related issues and pull requests on the CI subproject board, but that was missing a lot of our bugfixes and feature work. So I began putting together a triage board for the rest of our pull requests, which allowed me to sort each one by status and what actions to take, and documented its use for other contributors. We closed or merged over 500 issues and pull requests tracked by our two boards in each of the past two releases. The Kubernetes devstats showed that we have significantly increased our velocity as a result.
+
+In June, we ran our first bug scrub event to work through the backlog of issues filed against SIG Node, ensuring they were properly categorized. We closed over 130 issues over the course of this 48 hour global event, but as of writing we still have 333 open issues. 
+
+### Why should new and existing contributors consider joining SIG Node?
+
+SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you the foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster.
+
+SIG Node maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support. 
+
+### What do you do to help new contributors get started?
+
+EH: Getting started in SIG Node can be intimidating, since there is so much work to be done, our SIG meetings are very large, and it can be hard to find a place to start.
+
+I always encourage new contributors to work on things that they have some investment in already. In SIG Node, that might mean volunteering to help fix a bug that you have personally been affected by, or helping to triage bugs you care about by priority.
+
+To come up to speed on any open source code base, there are two strategies you can take: start by exploring a particular issue deeply, and follow that to expand the edges of your knowledge as needed, or briefly review as many issues and change requests as you possibly can to get a higher level picture of how the component works. Ultimately, you will need to do both if you want to become a Node reviewer or approver.
+
+[Davanum Srinivas](https://twitter.com/dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. 
+
+### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn?
+
+SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads.
+
+The other way of contribution is to help document SIG node features. This type of contribution requires a deep understanding of features, and ability to explain them in simple terms.
+
+Finally, we are always looking for feedback on how best to run your workload. Come and  explain specifics of it, and what features in SIG Node components may help to run it better. 
+
+### What are you getting positive feedback on, and what’s coming up next for SIG Node?
+
+EH: Over the past year SIG Node has adopted some new processes to help manage our feature development and Kubernetes enhancement proposals, and other SIGs have looked to us for inspiration in managing large workloads. I hope that this is an area we can continue to provide leadership in and further iterate on.
+
+We have a great balance of new features and deprecations in flight right now. Deprecations of unused or difficult to maintain features help us keep technical debt and maintenance load under control, and examples include the dockershim and DynamicKubeletConfiguration deprecations. New features will unlock additional functionality in end users' clusters, and include exciting features like support for cgroups v2, swap memory, graceful node shutdowns, and device management policies.
+
+### Any closing thoughts/resources you’d like to share?
+
+SK/EH: It takes time and effort to get to any open source community. SIG Node may overwhelm you at first with the number of participants, volume of work, and project scope. But it is totally worth it. Join our welcoming community! [SIG Node GitHub Repo](https://github.com/kubernetes/community/tree/master/sig-node) contains many useful resources including Slack, mailing list and other contact info. 
+
+## Wrap Up
+
+SIG Node hosted a [KubeCon + CloudNativeCon Europe 2021 talk](https://www.youtube.com/watch?v=z5aY4e2RENA) with an intro and deep dive to their awesome SIG. Join the SIG's meetings to find out about the most recent research results, what the plans are for the forthcoming year, and how to get involved in the upstream Node team as a contributor!
@@ -0,0 +1,243 @@
+---
+layout: blog
+title: "How to Handle Data Duplication in Data-Heavy Kubernetes Environments"
+date: 2021-09-29
+slug: how-to-handle-data-duplication-in-data-heavy-kubernetes-environments 
+---
+
+**Authors:**
+Augustinas Stirbis (CAST AI)
+
+## Why Duplicate Data?
+
+It’s convenient to create a copy of your application with a copy of its state for each team. 
+For example, you might want a separate database copy to test some significant schema changes 
+or develop other disruptive operations like bulk insert/delete/update...
+
+**Duplicating data takes a lot of time.** That’s because you need first to download 
+all the data from a source block storage provider to compute and then send 
+it back to a storage provider again. There’s a lot of network traffic and CPU/RAM used in this process.
+Hardware acceleration by offloading certain expensive operations to dedicated hardware is 
+**always a huge performance boost**. It reduces the time required to complete an operation by orders 
+of magnitude.
+
+## Volume Snapshots to the rescue
+
+Kubernetes introduced [VolumeSnapshots](/docs/concepts/storage/volume-snapshots/) as alpha in 1.12,
+beta in 1.17, and the Generally Available version in 1.20. 
+VolumeSnapshots use specialized APIs from storage providers to duplicate volume of data.
+
+Since data is already in the same storage device (array of devices), duplicating data is usually 
+a metadata operation for storage providers with local snapshots (majority of on-premise storage providers).
+All you need to do is point a new disk to an immutable snapshot and only 
+save deltas (or let it do a full-disk copy). As an operation that is inside the storage back-end,
+it’s much quicker and usually doesn’t involve sending traffic over the network.
+Public Clouds storage providers under the hood work a bit differently. They save snapshots
+to Object Storage and then copy back from Object storage to Block storage when "duplicating" disk.
+Technically there is a lot of Compute and network resources spent on Cloud providers side,
+but from Kubernetes user perspective VolumeSnapshots work the same way whether is it local or
+remote snapshot storage provider and no Compute and Network resources are involved in this operation.
+
+## Sounds like we have our solution, right?
+
+Actually, VolumeSnapshots are namespaced, and Kubernetes protects namespaced data from 
+being shared between tenants (Namespaces). This Kubernetes limitation is a conscious design 
+decision so that a Pod running in a different namespace can’t mount another application’s
+[PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) (PVC).
+
+One way around it would be to create multiple volumes with duplicate data in one namespace.
+However, you could easily reference the wrong copy.
+
+So the idea is to separate teams/initiatives by namespaces to avoid that and generally 
+limit access to the production namespace.
+
+## Solution? Creating a Golden Snapshot externally
+
+Another way around this design limitation is to create Snapshot externally (not through Kubernetes).
+This is also called pre-provisioning a snapshot manually. Next, I will import it 
+as a multi-tenant golden snapshot that can be used for many namespaces. Below illustration will be 
+for AWS EBS (Elastic Block Storage) and GCE PD (Persistent Disk) services.
+
+### High-level plan for preparing the Golden Snapshot
+
+1. Identify Disk (EBS/Persistent Disk) that you want to clone with data in the cloud provider
+2. Make a Disk Snapshot (in cloud provider console)
+3. Get Disk Snapshot ID
+
+### High-level plan for cloning data for each team
+
+1. Create Namespace “sandbox01”
+2. Import Disk Snapshot (ID) as VolumeSnapshotContent to Kubernetes
+3. Create VolumeSnapshot in the Namespace "sandbox01" mapped to VolumeSnapshotContent
+4. Create the PersistentVolumeClaim from VolumeSnapshot
+5. Install Deployment or StatefulSet with PVC
+
+## Step 1: Identify Disk
+
+First, you need to identify your golden source. In my case, it’s a PostgreSQL database
+on PersistentVolumeClaim “postgres-pv-claim” in the “production” namespace.
+
+```terminal
+kubectl -n <namespace> get pvc <pvc-name> -o jsonpath='{.spec.volumeName}'
+```
+
+The output will look similar to:
+```
+pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d90
+```
+
+## Step 2: Prepare your golden source
+
+You need to do this once or every time you want to refresh your golden data.
+
+### Make a Disk Snapshot
+
+Go to AWS EC2 or GCP Compute Engine console and search for an EBS volume
+(on AWS) or Persistent Disk (on GCP), that has a label matching the last output.
+In this case I saw: `pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d9`.
+
+Click on Create snapshot and give it a name. You can do it in Console manually,
+in AWS CloudShell / Google Cloud Shell, or in the terminal. To create a snapshot in the
+terminal you must have the AWS CLI tool (`aws`) or Google's CLI (`gcloud`)
+installed and configured.
+
+Here’s the command to create snapshot on GCP:
+
+```terminal
+gcloud compute disks snapshot <cloud-disk-id> --project=<gcp-project-id> --snapshot-names=<set-new-snapshot-name> --zone=<availability-zone> --storage-location=<region>
+```
+{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png" alt="Screenshot of a terminal showing volume snapshot creation on GCP" title="GCP snapshot creation" >}}
+
+
+GCP identifies the disk by its PVC name, so it’s direct mapping. In AWS, you need to 
+find volume by the CSIVolumeName AWS tag with PVC name value first that will be used for snapshot creation.
+
+{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png" alt="Screenshot of AWS web console, showing EBS volume identification" title="Identify disk ID on AWS" >}}
+
+Mark done Volume (volume-id) ```vol-00c7ecd873c6fb3ec``` and ether create EBS snapshot in AWS Console, or use ```aws cli```.
+
+```terminal
+aws ec2 create-snapshot --volume-id '<volume-id>' --description '<set-new-snapshot-name>' --tag-specifications 'ResourceType=snapshot'
+```
+
+## Step 3: Get your Disk Snapshot ID
+
+In AWS, the command above will output something similar to:
+```terminal
+"SnapshotId": "snap-09ed24a70bc19bbe4"
+```
+
+If you’re using the GCP cloud, you can get the snapshot ID from the gcloud command by querying for the snapshot’s given name:
+
+```terminal
+gcloud compute snapshots --project=<gcp-project-id> describe <new-snapshot-name> | grep id:
+```
+You should get similar output to:
+```
+id: 6645363163809389170
+```
+
+## Step 4: Create a development environment for each team
+
+Now I have my Golden Snapshot, which is immutable data. Each team will get a copy 
+of this data, and team members can modify it as they see fit, given that a new EBS/persistent 
+disk will be created for each team.
+
+Below I will define a manifest for each namespace. To save time, you can replace
+the namespace name (such as changing “sandbox01” → “sandbox42”) using tools
+such as `sed` or `yq`, with Kubernetes-aware templating tools like
+[Kustomize](/docs/tasks/manage-kubernetes-objects/kustomization/),
+or using variable substitution in a CI/CD pipeline.
+
+Here's an example manifest:
+
+```yaml
+---
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshotContent
+metadata:
+ name: postgresql-orders-db-sandbox01
+ namespace: sandbox01
+spec:
+ deletionPolicy: Retain
+ driver: pd.csi.storage.gke.io
+ source:
+   snapshotHandle: 'gcp/projects/staging-eu-castai-vt5hy2/global/snapshots/6645363163809389170'
+ volumeSnapshotRef:
+   kind: VolumeSnapshot
+   name: postgresql-orders-db-snap
+   namespace: sandbox01
+---
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshot
+metadata:
+ name: postgresql-orders-db-snap
+ namespace: sandbox01
+spec:
+ source:
+   volumeSnapshotContentName: postgresql-orders-db-sandbox01
+```
+
+In Kubernetes, VolumeSnapshotContent (VSC) objects are not namespaced.
+However, I need a separate VSC for each different namespace to use, so the
+`metadata.name` of each VSC must also be different. To make that straightfoward,
+I used the target namespace as part of the name.
+
+Now it’s time to replace the driver field with the CSI (Container Storage Interface) driver
+installed in your K8s cluster. Major cloud providers have CSI driver for block storage that
+support VolumeSnapshots but quite often CSI drivers are not installed by default, consult
+with your Kubernetes provider. 
+
+That manifest above defines a VSC that works on GCP.
+On AWS, driver and SnashotHandle values might look like:
+
+```YAML
+  driver: ebs.csi.aws.com
+  source:
+    snapshotHandle: "snap-07ff83d328c981c98"
+```
+
+At this point, I need to use the *Retain* policy, so that the CSI driver doesn’t try to
+delete my manually created EBS disk snapshot.
+
+For GCP, you will have to build this string by hand - add a full project ID and snapshot ID.
+For AWS, it’s just a plain snapshot ID.
+
+VSC also requires specifying which VolumeSnapshot (VS) will use it, so VSC and VS are
+referencing each other.
+
+Now I can create PersistentVolumeClaim from VS above. It’s important to set this first:
+
+
+```yaml
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: postgres-pv-claim
+ namespace: sandbox01
+spec:
+ dataSource:
+   kind: VolumeSnapshot
+   name: postgresql-orders-db-snap
+   apiGroup: snapshot.storage.k8s.io
+ accessModes:
+   - ReadWriteOnce
+ resources:
+   requests:
+     storage: 21Gi
+```
+
+If default StorageClass has [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) policy,
+then the actual Cloud Disk will be created from the Golden Snapshot only when some Pod bounds that PVC.
+
+Now I assign that PVC to my Pod (in my case, it’s Postgresql) as I would with any other PVC.
+
+```terminal
+kubectl -n <namespace> get volumesnapshotContent,volumesnapshot,pvc,pod
+```
+
+Both VS and VSC should be *READYTOUSE* true, PVC bound, and the Pod (from Deployment or StatefulSet) running.
+
+**To keep on using data from my Golden Snapshot, I just need to repeat this for the
+next namespace and voilà! No need to waste time and compute resources on the duplication process.**
@@ -0,0 +1,417 @@
+---
+layout: blog
+title: A Closer Look at NSA/CISA Kubernetes Hardening Guidance
+date: 2021-10-05
+slug: nsa-cisa-kubernetes-hardening-guidance
+---
+
+**Authors:** Jim Angel (Google), Pushkar Joglekar (VMware), and Savitha
+Raghunathan (Red Hat)
+
+{{% alert title="Disclaimer" %}}
+The open source tools listed in this article are to serve as examples only 
+and are in no way a direct recommendation from the Kubernetes community or authors.
+{{% /alert %}}
+
+## Background
+
+USA's National Security Agency (NSA) and the Cybersecurity and Infrastructure
+Security Agency (CISA)
+released, "[Kubernetes Hardening Guidance](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF)"
+on August 3rd, 2021. The guidance details threats to Kubernetes environments
+and provides secure configuration guidance to minimize risk.
+
+The following sections of this blog correlate to the sections in the NSA/CISA guidance. 
+Any missing sections are skipped because of limited opportunities to add 
+anything new to the existing content. 
+
+_Note_: This blog post is not a substitute for reading the guide. Reading the published 
+guidance is recommended before proceeding as the following content is 
+complementary.
+
+## Introduction and Threat Model
+
+Note that the threats identified as important by the NSA/CISA, or the intended audience of this guidance, may be different from the threats that other enterprise users of Kubernetes consider important. This section 
+is still useful for organizations that care about data, resource theft and 
+service unavailability.
+
+The guidance highlights the following three sources of compromises:
+
+- Supply chain risks
+- Malicious threat actors
+- Insider threats (administrators, users, or cloud service providers)
+
+The [threat model](https://en.wikipedia.org/wiki/Threat_model) tries to take a step back and review threats that not only
+exist within the boundary of a Kubernetes cluster but also include the underlying
+infrastructure and surrounding workloads that Kubernetes does not manage.
+
+For example, when a workload outside the cluster shares the same physical
+network, it has access to the kubelet and to control plane components: etcd, controller manager, scheduler and API
+server. Therefore, the guidance recommends having network level isolation
+separating Kubernetes clusters from other workloads that do not need connectivity
+to Kubernetes control plane nodes. Specifically, scheduler, controller-manager,
+etcd only need to be accessible to the API server. Any interactions with Kubernetes
+from outside the cluster can happen by providing access to API server port.
+
+List of ports and protocols for each of these components are
+defined in [Ports and Protocols](/docs/reference/ports-and-protocols/)
+within the Kubernetes documentation.
+
+> Special note: kube-scheduler and kube-controller-manager uses different ports than the ones mentioned in the guidance
+
+The [Threat modelling](https://cnsmap.netlify.app/threat-modelling) section
+from the CNCF [Cloud Native Security Whitepaper + Map](https://github.com/cncf/tag-security/tree/main/security-whitepaper)
+provides another perspective on approaching threat modelling Kubernetes, from a
+cloud native lens.
+
+## Kubernetes Pod security
+
+Kubernetes by default does not guarantee strict workload isolation between pods
+running in the same node in a cluster. However, the guidance provides several 
+techniques to enhance existing isolation and reduce the attack surface in case of a 
+compromise.
+
+### "Non-root" containers and "rootless" container engines
+
+Several best practices related to basic security principle of least privilege
+i.e. provide only the permissions are needed; no more, no less, are worth a
+second look.
+
+The guide recommends setting non-root user at build time instead of relying on
+setting `runAsUser` at runtime in your Pod spec. This is a good practice and provides
+some level of defense in depth. For example, if the container image is built with user `10001`
+and the Pod spec misses adding the `runAsuser` field in its `Deployment` object. In this
+case there are certain edge cases that are worth exploring for awareness:
+
+1. Pods can fail to start, if the user defined at build time is different from
+   the one defined in pod spec and some files are as a result inaccessible.
+2. Pods can end up sharing User IDs unintentionally. This can be problematic
+   even if the User IDs are non-zero in a situation where a container escape to
+   host file system is possible. Once the attacker has access to the host file
+   system, they get access to all the file resources that are owned by other
+   unrelated pods that share the same UID.
+3. Pods can end up sharing User IDs, with other node level processes not managed
+   by Kubernetes e.g. node level daemons for auditing, vulnerability scanning,
+   telemetry. The threat is similar to the one above where host file system
+   access can give attacker full access to these node level daemons without
+   needing to be root on the node.
+
+However, none of these cases will have as severe an impact as a container
+running as root being able to escape as a root user on the host, which can provide 
+an attacker with complete control of the worker node, further allowing lateral
+movement to other worker or control plane nodes.
+
+Kubernetes 1.22 introduced
+an [alpha feature](/docs/tasks/administer-cluster/kubelet-in-userns/)
+that specifically reduces the impact of such a control plane component running
+as root user to a non-root user through user namespaces.
+
+That ([alpha stage](/docs/reference/command-line-tools-reference/feature-gates/#feature-stages)) support for user namespaces / rootless mode is available with
+the following container runtimes:
+
+- [Docker Engine](https://docs.docker.com/engine/security/rootless/)
+- [Podman](https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics)
+
+Some distributions support running in rootless mode, like the following:
+
+- [kind](https://kind.sigs.k8s.io/docs/user/rootless/)
+- [k3s](https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental)
+- [Usernetes](https://github.com/rootless-containers/usernetes)
+
+### Immutable container filesystems
+
+The NSA/CISA Kubernetes Hardening Guidance highlights an often overlooked feature `readOnlyRootFileSystem`, with a
+working example in [Appendix B](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=42). This example limits execution and tampering of
+containers at runtime. Any read/write activity can then be limited to few
+directories by using `tmpfs` volume mounts.
+
+However, some applications that modify the container filesystem at runtime, like exploding a WAR or JAR file at container startup, 
+could face issues when enabling this feature. To avoid this issue, consider making minimal changes to the filesystem at runtime 
+when possible.
+
+### Building secure container images
+
+Kubernetes Hardening Guidance also recommends running a scanner at deploy time as an admission controller,
+to prevent vulnerable or misconfigured pods from running in the cluster.
+Theoretically, this sounds like a good approach but there are several caveats to
+consider before this can be implemented in practice:
+
+- Depending on network bandwidth, available resources and scanner of choice,
+  scanning for vulnerabilities for an image can take an indeterminate amount of
+  time. This could lead to slower or unpredictable pod start up times, which
+  could result in spikes of unavailability when apps are serving peak load.
+- If the policy that allows or denies pod startup is made using incorrect or
+  incomplete data it could result in several false positive or false negative
+  outcomes like the following:
+  - inside a container image, the `openssl` package is detected as vulnerable. However,
+    the application is written in Golang and uses the Go `crypto` package for TLS. Therefore, this vulnerability
+    is not in the code execution path and as such has minimal impact if it
+    remains unfixed.
+  - A vulnerability is detected in the `openssl` package for a Debian base image.
+    However, the upstream Debian community considers this as a Minor impact
+    vulnerability and as a result does not release a patch fix for this
+    vulnerability. The owner of this image is now stuck with a vulnerability that
+    cannot be fixed and a cluster that does not allow the image to run because
+    of predefined policy that does not take into account whether the fix for a
+    vulnerability is available or not
+  - A Golang app is built on top of a [distroless](https://github.com/GoogleContainerTools/distroless)
+    image, but it is compiled with a Golang version that uses a vulnerable [standard library](https://pkg.go.dev/std).
+    The scanner has
+    no visibility into golang version but only on OS level packages. So it 
+    allows the pod to run in the cluster in spite of the image containing an 
+    app binary built on vulnerable golang.
+
+To be clear, relying on vulnerability scanners is absolutely a good idea but
+policy definitions should be flexible enough to allow:
+
+- Creation of exception lists for images or vulnerabilities through labelling
+- Overriding the severity with a risk score based on impact of a vulnerability
+- Applying the same policies at build time to catch vulnerable images with
+  fixable vulnerabilities before they can be deployed into Kubernetes clusters
+
+Special considerations like offline vulnerability database fetch, may also be
+needed, if the clusters run in an air-gapped environment and the scanners
+require internet access to update the vulnerability database.
+
+### Pod Security Policies
+
+Since Kubernetes v1.21, the [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) 
+API and related features are [deprecated](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/),
+but some of the guidance in this section will still apply for the next few years, until cluster operators
+upgrade their clusters to newer Kubernetes versions.
+
+The Kubernetes project is working on a replacement for PodSecurityPolicy.
+Kubernetes v1.22 includes an alpha feature called [Pod Security Admission](/docs/concepts/security/pod-security-admission/)
+that is intended to allow enforcing a minimum level of isolation between pods.
+
+The built-in isolation levels for Pod Security Admission are derived
+from [Pod Security Standards](/docs/concepts/security/pod-security-standards/), which is a superset of all the components mentioned in Table I [page 10](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=17) of
+the guidance.
+
+Information about migrating from PodSecurityPolicy to the Pod Security
+Admission feature is available
+in
+[Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/).
+
+One important behavior mentioned in the guidance that remains the same between
+Pod Security Policy and its replacement is that enforcing either of them does
+not affect pods that are already running. With both PodSecurityPolicy and Pod Security Admission,
+the enforcement happens during the pod creation
+stage.
+
+### Hardening container engines
+
+Some container workloads are less trusted than others but may need to run in the
+same cluster. In those cases, running them on dedicated nodes that include
+hardened container runtimes that provide stricter pod isolation boundaries can
+act as a useful security control.
+
+Kubernetes supports
+an API called [RuntimeClass](/docs/concepts/containers/runtime-class/) that is
+stable / GA (and, therefore, enabled by default) stage as of Kubernetes v1.20.
+RuntimeClass allows you to ensure that Pods requiring strong isolation are scheduled onto
+nodes that can offer it.
+
+Some third-party projects that you can use in conjunction with RuntimeClass are:
+
+- [kata containers](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-runtime-class-for-kata-containers)
+- [gvisor](https://gvisor.dev/docs/user_guide/containerd/quick_start/)
+
+As discussed here and in the guidance, many features and tooling exist in and around
+Kubernetes that can enhance the isolation boundaries between
+pods. Based on relevant threats and risk posture, you should pick and choose
+between them, instead of trying to apply all the recommendations. Having said that, cluster
+level isolation i.e. running workloads in dedicated clusters, remains the strictest workload 
+isolation mechanism, in spite of improvements mentioned earlier here and in the guide.
+
+## Network Separation and Hardening
+
+Kubernetes Networking can be tricky and this section focuses on how to secure
+and harden the relevant configurations. The guide identifies the following as key
+takeaways: 
+- Using NetworkPolicies to create isolation between resources,
+- Securing the control plane
+- Encrypting traffic and sensitive data
+
+### Network Policies
+
+Network policies can be created with the help of network plugins. In order to
+make the creation and visualization easier for users, Cilium supports
+a [web GUI tool](https://editor.cilium.io). That web GUI lets you create Kubernetes
+NetworkPolicies (a generic API that nevertheless requires a compatible CNI plugin),
+and / or Cilium network policies (CiliumClusterwideNetworkPolicy and CiliumNetworkPolicy,
+which only work in clusters that use the Cilium CNI plugin).
+You can use these APIs to restrict network traffic between pods, and therefore minimize the
+attack vector.
+
+Another scenario that is worth exploring is the usage of external IPs. Some
+services, when misconfigured, can create random external IPs. An attacker can take
+advantage of this misconfiguration and easily intercept traffic. This vulnerability
+has been reported
+in [CVE-2020-8554](https://www.cvedetails.com/cve/CVE-2020-8554/).
+Using [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook)
+can mitigate this vulnerability by preventing the services from using random
+external IPs. [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook) 
+only allows creation of services that don't require external IPs or whose 
+external IPs are within the range specified by the administrator.
+
+> CVE-2020-8554 - Kubernetes API server in all versions allow an attacker
+> who is able to create a ClusterIP service and set the `spec.externalIPs` field,
+> to intercept traffic to that IP address. Additionally, an attacker who is able to
+> patch the `status` (which is considered a privileged operation and should not
+> typically be granted to users) of a LoadBalancer service can set the
+> `status.loadBalancer.ingress.ip` to similar effect.
+
+### Resource Policies
+
+In addition to configuring ResourceQuotas and limits, consider restricting how many process
+IDs (PIDs) a given Pod can use, and also to reserve some PIDs for node-level use to avoid
+resource exhaustion. More details to apply these limits can be
+found in [Process ID Limits And Reservations](/docs/concepts/policy/pid-limiting/).
+
+### Control Plane Hardening
+
+In the next section, the guide covers control plane hardening. It is worth
+noting that
+from [Kubernetes 1.20](https://github.com/kubernetes/kubernetes/issues/91506),
+insecure port from API server, has been removed.
+
+### Etcd
+
+As a general rule, the etcd server should be configured to only trust
+certificates assigned to the API server. It limits the attack surface and prevents a
+malicious attacker from gaining access to the cluster. It might be beneficial to
+use a separate CA for etcd, as it by default trusts all the certificates issued
+by the root CA.
+
+### Kubeconfig Files
+
+In addition to specifying the token and certificates directly, `.kubeconfig`
+supports dynamic retrieval of temporary tokens using auth provider plugins.
+Beware of the possibility of malicious
+shell [code execution](https://banzaicloud.com/blog/kubeconfig-security/) in a
+`kubeconfig` file. Once attackers gain access to the cluster, they can steal ssh
+keys/secrets or more.
+
+### Secrets
+Kubernetes [Secrets](/docs/concepts/configuration/secret/) is the native way of managing secrets as a Kubernetes
+API object. However, in some scenarios such as a desire to have a single source of truth for all app secrets, irrespective of whether they run on Kubernetes or not, secrets can be managed loosely coupled with 
+Kubernetes and consumed by pods through side-cars or init-containers with minimal usage of Kubernetes Secrets API. 
+
+[External secrets providers](https://github.com/external-secrets/kubernetes-external-secrets)
+and [csi-secrets-store](https://github.com/kubernetes-sigs/secrets-store-csi-driver)
+are some of these alternatives to Kubernetes Secrets
+
+## Log Auditing
+
+The NSA/CISA guidance stresses monitoring and alerting based on logs. The key points
+include logging at the host level, application level, and on the cloud. When
+running Kubernetes in production, it's important to understand who's
+responsible, and who's accountable, for each layer of logging.
+
+### Kubernetes API auditing
+
+One area that deserves more focus is what exactly should alert or be logged. The
+document outlines a sample policy in [Appendix L: Audit Policy](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=55) that logs all
+RequestResponse's including metadata and request / response bodies. While helpful for a demo, it may not be practical for production.
+
+Each organization needs to evaluate their
+own threat model and build an audit policy that complements or helps troubleshooting incident response. Think
+about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug-application-cluster/audit/#audit-policy).
+It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point. 
+
+Audit logging configurations can also be tested with
+kind following these [instructions](https://kind.sigs.k8s.io/docs/user/auditing).
+
+### Streaming logs and auditing
+
+Logging is important for threat and anomaly detection. As the document outlines,
+it's a best practice to scan and alert on logs as close to real time as possible
+and to protect logs from tampering if a compromise occurs. It's important to
+reflect on the various levels of logging and identify the critical areas such as
+API endpoints.
+
+Kubernetes API audit logging can stream to a webhook and there's an example in [Appendix N: Webhook configuration](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=58). Using a webhook could be a method that
+stores logs off cluster and/or centralizes all audit logs. Once logs are
+centrally managed, look to enable alerting based on critical events. Also ensure
+you understand what the baseline is for normal activities.
+
+### Alert identification
+
+While the guide stressed the importance of notifications, there is not a blanket
+event list to alert from. The alerting requirements vary based on your own
+requirements and threat model. Examples include the following events:
+
+- Changes to the `securityContext` of a Pod
+- Updates to admission controller configs
+- Accessing certain files / URLs
+
+### Additional logging resources
+
+- [Seccomp Security Profiles and You: A Practical Guide - Duffie Cooley](https://www.youtube.com/watch?v=OPuu8wsu2Zc)
+- [TGI Kubernetes 119: Gatekeeper and OPA](https://www.youtube.com/watch?v=ZJgaGJm9NJE)
+- [Abusing The Lack of Kubernetes Auditing Policies](https://www.lacework.com/blog/hiding-in-plaintext-sight-abusing-the-lack-of-kubernetes-auditing-policies/)
+- [Enable seccomp for all workloads with a new v1.22 alpha feature](https://kubernetes.io/blog/2021/08/25/seccomp-default/)
+- [This Week in Cloud Native: Auditing / Pod Security](https://www.twitch.tv/videos/1147889860)
+
+## Upgrading and Application Security practices
+
+Kubernetes releases three times per year, so upgrade-related toil is a common problem for
+people running production clusters. In addition to this, operators must
+regularly upgrade the underlying node's operating system and running
+applications. This is a best practice to ensure continued support and to reduce
+the likelihood of bugs or vulnerabilities.
+
+Kubernetes supports the three most recent stable releases. While each Kubernetes
+release goes through a large number of tests before being published, some
+teams aren't comfortable running the latest stable release until some time has
+passed. No matter what version you're running, ensure that patch upgrades
+happen frequently or automatically. More information can be found in
+the [version skew](/releases/version-skew-policy/) policy
+pages.
+
+When thinking about how you'll manage node OS upgrades, consider ephemeral
+nodes. Having the ability to destroy and add nodes allows your team to respond
+quicker to node issues. In addition, having deployments that tolerate node
+instability (and a culture that encourages frequent deployments) allows for
+easier cluster upgrades.
+
+Additionally, it's worth reiterating from the guidance that periodic 
+vulnerability scans and penetration tests can be performed on the various system
+components to proactively look for insecure configurations and vulnerabilities.
+
+### Finding release & security information
+
+To find the most recent Kubernetes supported versions, refer to
+[https://k8s.io/releases](https://k8s.io/releases), which includes minor versions. It's good to stay up to date with
+your minor version patches.
+
+If you're running a managed Kubernetes offering, look for their release
+documentation and find their various security channels.
+
+Subscribe to
+the [Kubernetes Announce mailing list](https://groups.google.com/g/kubernetes-announce).
+The Kubernetes Announce mailing list is searchable for terms such
+as "[Security Advisories](https://groups.google.com/g/kubernetes-announce/search?q=%5BSecurity%20Advisory%5D)".
+You can set up alerts and email notifications as long as you know what key
+words to alert on.
+
+## Conclusion
+
+In summary, it is fantastic to see security practitioners sharing this
+level of detailed guidance in public. This guidance further highlights
+Kubernetes going mainstream and how securing Kubernetes clusters and the
+application containers running on Kubernetes continues to need attention and focus of
+practitioners. Only a few weeks after the guidance was published, an open source
+tool [kubescape](https://github.com/armosec/kubescape) to validate cluster
+against this guidance became available.
+
+This tool can be a great starting point to check the current state of your
+clusters, after which you can use the information in this blog post and in the guidance to assess 
+where improvements can be made.
+
+Finally, it is worth reiterating that not all controls in this guidance will
+make sense for all practitioners. The best way to know which controls matter is
+to rely on the threat model of your own Kubernetes environment.
+
+_A special shout out and thanks to Rory McCune (@raesene) for his inputs to this blog post_
@@ -0,0 +1,141 @@
+---
+layout: blog
+title: "Introducing ClusterClass and Managed Topologies in Cluster API"
+date: 2021-10-08
+slug: capi-clusterclass-and-managed-topologies
+---
+
+**Author:** Fabrizio Pandini (VMware)
+
+The [Cluster API community](https://cluster-api.sigs.k8s.io/) is happy to announce the implementation of *ClusterClass and Managed Topologies*, a new feature that will greatly simplify how you can provision, upgrade, and operate multiple Kubernetes clusters in a declarative way.
+
+## A little bit of context…
+
+Before getting into the details, let's take a step back and look at the history of Cluster API.
+
+The [Cluster API project](https://github.com/kubernetes-sigs/cluster-api/) started three years ago, and the first releases focused on extensibility and implementing a declarative API that allows a seamless experience across infrastructure providers. This was a success with many cloud providers: AWS, Azure, Digital Ocean, GCP, Metal3, vSphere and still counting.
+
+With extensibility addressed, the focus shifted to features, like automatic control plane and etcd management, health-based machine remediation, machine rollout strategies and more.
+
+Fast forwarding to 2021, with lots of companies using Cluster API to manage fleets of Kubernetes clusters running workloads in production, the community focused its effort on stabilization of both code, APIs, documentation, and on extensive test signals which inform Kubernetes releases.
+
+With solid foundations in place, and a vibrant and welcoming community that still continues to grow, it was time to plan another iteration on our UX for both new and advanced users.
+
+Enter ClusterClass and Managed Topologies, tada!
+
+## ClusterClass
+
+As the name suggests, ClusterClass and managed topologies are built in two parts.
+
+The idea behind ClusterClass is simple: define the shape of your cluster once, and reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away.
+
+![Defining a ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/clusterclass.svg)
+
+ClusterClass, at its heart, is a collection of Cluster and Machine templates. You can use it as a “stamp” that can be leveraged to create many clusters of a similar shape.
+
+```yaml
+---
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: ClusterClass
+metadata:
+  name: my-amazing-cluster-class
+spec:
+  controlPlane:
+    ref:
+      apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+      kind: KubeadmControlPlaneTemplate
+      name: high-availability-control-plane
+    machineInfrastructure:
+      ref:
+        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+        kind: DockerMachineTemplate
+        name: control-plane-machine
+  workers:
+    machineDeployments:
+      - class: type1-workers
+        template:
+          bootstrap:
+            ref:
+              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+              kind: KubeadmConfigTemplate
+              name: type1-bootstrap
+          infrastructure:
+            ref:
+              apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+              kind: DockerMachineTemplate
+              name: type1-machine
+      - class: type2-workers
+        template:
+          bootstrap:
+            ref:
+              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+              kind: KubeadmConfigTemplate
+              name: type2-bootstrap
+          infrastructure:
+            ref:
+              kind: DockerMachineTemplate
+              apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+              name: type2-machine
+  infrastructure:
+    ref:
+      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+      kind: DockerClusterTemplate
+      name: cluster-infrastructure
+
+```
+
+The possibilities are endless; you can get a default ClusterClass from the community, “off-the-shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios.
+
+## Managed Topologies
+
+Managed Topologies let you put the power of ClusterClass into action.
+
+Given a ClusterClass, you can create many Clusters of a similar shape by providing a single resource, the Cluster.
+
+![Create a Cluster with ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/create-cluster.svg)
+
+Here is an example:
+
+```yaml
+---
+apiVersion: cluster.x-k8s.io/v1beta1
+ kind: Cluster
+ metadata:
+   name: my-amazing-cluster
+   namespace: bar
+ spec:
+   topology: # define a managed topology
+     class: my-amazing-cluster-class # use the ClusterClass mentioned earlier
+     version: v1.21.2
+     controlPlane:
+       replicas: 3
+     workers:
+       machineDeployments:
+       - class: type1-workers
+         name: big-pool-of-machines
+         replicas: 5
+       - class: type2-workers
+         name: small-pool-of-machines
+         replicas: 1
+```
+
+But there is more than simplified cluster creation. Now the Cluster acts as a single control point for your entire topology.
+
+All the power of Cluster API, extensibility, lifecycle automation, stability, all the features required for managing an enterprise grade Kubernetes cluster on the infrastructure provider of your choice are now at your fingertips: you can create your Cluster, add new machines, upgrade to the next Kubernetes version, and all from a single place.
+
+It is just as simple as it looks!
+
+## What’s next
+
+While the amazing Cluster API community is working hard to deliver the first version of ClusterClass and managed topologies later this year, we are already looking forward to what comes next for the project and its ecosystem.
+
+There are a lot of great ideas and opportunities ahead!
+
+We want to make managed topologies even more powerful and flexible, allowing users to dynamically change bits of a ClusterClass according to the specific needs of a Cluster; this will ensure the same simple and intuitive UX for solving complex problems like e.g. selecting machine image for a specific Kubernetes version and for a specific region of your infrastructure provider, or injecting proxy configurations in the entire Cluster, and so on.
+
+Stay tuned for what comes next, and if you have any questions, comments or suggestions:
+
+* Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T)
+* Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents
+* Join our [Zoom meeting](https://zoom.us/j/861487554), every Wednesday at 10:00 Pacific Time
+* Check out the [ClusterClass tutorial](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-classes.html) in the Cluster API book.
@@ -0,0 +1,241 @@
+---
+layout: blog
+title: "Use KPNG to Write Specialized kube-proxiers"
+date: 2021-10-18
+slug: use-kpng-to-write-specialized-kube-proxiers
+---
+
+**Author**: Lars Ekman (Ericsson)
+
+The post will show you how to create a specialized service kube-proxy
+style network proxier using Kubernetes Proxy NG
+[kpng](https://github.com/kubernetes-sigs/kpng) without interfering
+with the existing kube-proxy. The kpng project aims at renewing the
+the default Kubernetes Service implementation, the "kube-proxy". An
+important feature of kpng is that it can be used as a library to
+create proxiers outside K8s. While this is useful for CNI-plugins that
+replaces the kube-proxy it also opens the possibility for anyone to
+create a proxier for a special purpose.
+
+
+## Define a service that uses a specialized proxier
+
+```
+apiVersion: v1
+kind: Service
+metadata:
+  name: kpng-example
+  labels:
+    service.kubernetes.io/service-proxy-name: kpng-example
+spec:
+  clusterIP: None
+  ipFamilyPolicy: RequireDualStack
+  externalIPs:
+  - 10.0.0.55
+  - 1000::55
+  selector:
+    app: kpng-alpine
+  ports:
+  - port: 6000
+```
+
+If the `service.kubernetes.io/service-proxy-name` label is defined the
+`kube-proxy` will ignore the service. A custom controller can watch
+services with the label set to it's own name, "kpng-example" in
+this example, and setup specialized load-balancing.
+
+The `service.kubernetes.io/service-proxy-name` label is [not
+new](https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioservice-proxy-name),
+but so far is has been quite hard to write a specialized proxier.
+
+The common use for a specialized proxier is assumed to be handling
+external traffic for some use-case not supported by K8s. In that
+case `ClusterIP` is not needed, so we use a "headless" service in this
+example.
+
+
+## Specialized proxier using kpng
+
+A [kpng](https://github.com/kubernetes-sigs/kpng) based proxier
+consists of the `kpng` controller handling all the K8s api related
+functions, and a "backend" implementing the load-balancing. The
+backend can be linked with the `kpng` controller binary or be a
+separate program communicating with the controller using gRPC.
+
+```
+kpng kube --service-proxy-name=kpng-example to-api
+```
+
+This starts the `kpng` controller and tell it to watch only services
+with the "kpng-example" service proxy name. The "to-api" parameter
+will open a gRPC server for backends.
+
+You can test this yourself outside your cluster. Please see the example
+below.
+
+Now we start a backend that simply prints the updates from the
+controller.
+
+```
+$ kubectl apply -f kpng-example.yaml
+$ kpng-json | jq     # (this is the backend)
+{
+  "Service": {
+    "Namespace": "default",
+    "Name": "kpng-example",
+    "Type": "ClusterIP",
+    "IPs": {
+      "ClusterIPs": {},
+      "ExternalIPs": {
+        "V4": [
+          "10.0.0.55"
+        ],
+        "V6": [
+          "1000::55"
+        ]
+      },
+      "Headless": true
+    },
+    "Ports": [
+      {
+        "Protocol": 1,
+        "Port": 6000,
+        "TargetPort": 6000
+      }
+    ]
+  },
+  "Endpoints": [
+    {
+      "IPs": {
+        "V6": [
+          "1100::202"
+        ]
+      },
+      "Local": true
+    },
+    {
+      "IPs": {
+        "V4": [
+          "11.0.2.2"
+        ]
+      },
+      "Local": true
+    },
+    {
+      "IPs": {
+        "V4": [
+          "11.0.1.2"
+        ]
+      }
+    },
+    {
+      "IPs": {
+        "V6": [
+          "1100::102"
+        ]
+      }
+    }
+  ]
+}
+```
+
+A real backend would use some mechanism to load-balance traffic from
+the external IPs to the endpoints.
+
+
+
+## Writing a backend
+
+The `kpng-json` backend looks like this:
+
+```go
+package main
+import (
+        "os"
+        "encoding/json"
+        "sigs.k8s.io/kpng/client"
+)
+func main() {
+        client.Run(jsonPrint)
+}
+func jsonPrint(items []*client.ServiceEndpoints) {
+        enc := json.NewEncoder(os.Stdout)
+        for _, item := range items {
+                _ = enc.Encode(item)
+        }
+}
+```
+
+(yes, that is the entire program)
+
+A real backend would of course be much more complex, but this
+illustrates how `kpng` let you focus on load-balancing.
+
+You can have several backends connected to a `kpng` controller, so
+during development or debug it can be useful to let something like the
+`kpng-json` backend run in parallel with your real backend.
+
+
+## Example
+
+
+The complete example can be found [here](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
+
+As an example we implement an "all-ip" backend. It direct all traffic
+for the externalIPs to a local endpoint, regardless of ports and upper
+layer protocols. There is a
+[KEP](https://github.com/kubernetes/enhancements/pull/2611) for this
+function and this example is a much simplified version.
+
+To direct all traffic from an external address to a local POD [only
+one iptables rule is
+needed](https://github.com/kubernetes/enhancements/pull/2611#issuecomment-895061013),
+for instance;
+
+```
+ip6tables -t nat -A PREROUTING -d 1000::55/128 -j DNAT --to-destination 1100::202
+```
+
+As you can see the addresses are in the call to the backend and all it
+have to do is:
+
+* Extract the addresses with `Local: true`
+* Setup iptables rules for the `ExternalIPs`
+
+A script doing that may look like:
+
+```
+xip=$(cat /tmp/out | jq -r .Service.IPs.ExternalIPs.V6[0])
+podip=$(cat /tmp/out | jq -r '.Endpoints[]|select(.Local == true)|select(.IPs.V6 != null)|.IPs.V6[0]')
+ip6tables -t nat -A PREROUTING -d $xip/128 -j DNAT --to-destination $podip
+```
+
+Assuming the JSON output above is stored in `/tmp/out` ([jq](https://stedolan.github.io/jq/) is an *awesome* program!).
+
+
+As this is an example we make it really simple for ourselves by using
+a minor variation of the `kpng-json` backend above. Instead of just
+printing, a program is called and the JSON output is passed as `stdin`
+to that program. The backend can be tested stand-alone:
+
+```
+CALLOUT=jq kpng-callout
+```
+
+Where `jq` can be replaced with your own program or script. A script
+may look like the example above.  For more info and the complete
+example please see [https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
+
+
+## Summary
+
+While [kpng](https://github.com/kubernetes-sigs/kpng) is in early
+stage of development this post wants to show how you may build your
+own specialized K8s proxiers in the future. The only thing your
+applications need to do is to add the
+`service.kubernetes.io/service-proxy-name` label in the Service
+manifest.
+
+It is a tedious process to get new features into the `kube-proxy` and
+it is not unlikely that they will be rejected, so to write a
+specialized proxier may be the only option.
@@ -145,7 +145,7 @@ Coil operates with a low overhead compared to bare metal, and allows you to defi

 ### Contiv

-[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan,  classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](https://contiv.io) is all open sourced.
+[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan,  classic l2, or Cisco-SDN/ACI) for various use cases.

 ### Contrail / Tungsten Fabric

@@ -248,7 +248,7 @@ Lars Kellogg-Stedman.

 [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.

-Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes.
+Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/cni.dev/blob/main/content/plugins/v0.9/meta/flannel.md), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes.

 ### OVN4NFV-K8s-Plugin (OVN based CNI controller & plugin)

@@ -260,12 +260,6 @@ Multus supports all [reference plugins](https://github.com/containernetworking/p

 [NSX-T Container Plug-in (NCP)](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) provides integration between NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.

-### Nuage Networks VCS (Virtualized Cloud Services)
-
-[Nuage](https://www.nuagenetworks.net) provides a highly scalable policy-based Software-Defined Networking (SDN) platform. Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards.
-
-The Nuage platform uses overlays to provide seamless policy-based networking between Kubernetes Pods and non-Kubernetes environments (VMs and bare metal servers). Nuage's policy abstraction model is designed with applications in mind and makes it easy to declare fine-grained policies for applications.The platform's real-time analytics engine enables visibility and security monitoring for Kubernetes applications.
-
 ### OpenVSwitch

 [OpenVSwitch](https://www.openvswitch.org/) is a somewhat more mature but also
@@ -212,7 +212,8 @@ to create a Docker registry Secret, you can do:
 kubectl create secret docker-registry secret-tiger-docker \
  --docker-username=tiger \
  --docker-password=pass113 \
-  --docker-email=tiger@acme.com
+  --docker-email=tiger@acme.com \
+  --docker-server=my-registry.example:5000
 ```

 This command creates a Secret of type `kubernetes.io/dockerconfigjson`.
@@ -222,15 +223,21 @@ on the fly:

 ```json
 {
-  "auths": {
-    "https://index.docker.io/v1/": {
-      "username": "tiger",
-      "password": "pass113",
-      "email": "tiger@acme.com",
-      "auth": "dGlnZXI6cGFzczExMw=="
-    }
-  }
+    "apiVersion": "v1",
+    "data": {
+        ".dockerconfigjson": "eyJhdXRocyI6eyJteS1yZWdpc3RyeTo1MDAwIjp7InVzZXJuYW1lIjoidGlnZXIiLCJwYXNzd29yZCI6InBhc3MxMTMiLCJlbWFpbCI6InRpZ2VyQGFjbWUuY29tIiwiYXV0aCI6ImRHbG5aWEk2Y0dGemN6RXhNdz09In19fQ=="
+    },
+    "kind": "Secret",
+    "metadata": {
+        "creationTimestamp": "2021-07-01T07:30:59Z",
+        "name": "secret-tiger-docker",
+        "namespace": "default",
+        "resourceVersion": "566718",
+        "uid": "e15c1d7b-9071-4100-8681-f3a7a2ce89ca"
+    },
+    "type": "kubernetes.io/dockerconfigjson"
 }
+
 ```

 ### Basic authentication Secret
@@ -108,7 +108,7 @@ When you (or a controller) submit a new Pod to the API server, your cluster sets
  `:latest`, `imagePullPolicy` is automatically set to `Always`;
 - if you omit the `imagePullPolicy` field, and you don't specify the tag for the
  container image, `imagePullPolicy` is automatically set to `Always`;
- if you omit the `imagePullPolicy` field, and you don't specify the tag for the
+- if you omit the `imagePullPolicy` field, and you specify the tag for the
  container image that isn't `:latest`, the `imagePullPolicy` is automatically set to
  `IfNotPresent`.

@@ -35,11 +35,11 @@ On their own, custom resources let you store and retrieve structured data.
 When you combine a custom resource with a *custom controller*, custom resources
 provide a true _declarative API_.

-A [declarative API](/docs/concepts/overview/kubernetes-api/)
-allows you to _declare_ or specify the desired state of your resource and tries to
-keep the current state of Kubernetes objects in sync with the desired state.
-The controller interprets the structured data as a record of the user's
-desired state, and continually maintains this state.
+The Kubernetes [declarative API](/docs/concepts/overview/kubernetes-api/)
+enforces a separation of responsibilities. You declare the desired state of
+your resource. The Kubernetes controller keeps the current state of Kubernetes
+objects in sync with your declared desired state. This is in contrast to an
+imperative API, where you *instruct* a server what to do.

 You can deploy and update a custom controller on a running cluster, independently
 of the cluster's lifecycle. Custom controllers can work with any kind of resource,
@@ -19,11 +19,7 @@ When you deploy Kubernetes, you get a cluster.
 This document outlines the various components you need to have for
 a complete and working Kubernetes cluster.

-Here's the diagram of a Kubernetes cluster with all the components tied together.
-
-![Components of Kubernetes](/images/docs/components-of-kubernetes.svg)
-
-
+{{< figure src="/images/docs/components-of-kubernetes.svg" alt="Components of Kubernetes" caption="The components of a Kubernetes cluster" class="diagram-large" >}}

 <!-- body -->
 ## Control Plane Components
@@ -44,7 +44,8 @@ and the controller deletes the volume.

 ## Owner references, labels, and finalizers {#owners-labels-finalizers}

-Like {{<glossary_tooltip text="labels" term_id="label">}}, [owner references](/concepts/overview/working-with-objects/owners-dependents/)
+Like {{<glossary_tooltip text="labels" term_id="label">}},
+[owner references](/docs/concepts/overview/working-with-objects/owners-dependents/)
 describe the relationships between objects in Kubernetes, but are used for a
 different purpose. When a
 {{<glossary_tooltip text="controller" term_id="controller">}} manages objects
@@ -10,8 +10,7 @@ weight: 30

 <!-- overview -->

-Kubernetes supports multiple virtual clusters backed by the same physical cluster.
-These virtual clusters are called namespaces.
+In Kubernetes, _namespaces_ provides a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects _(e.g. Deployments, Services, etc)_ and not for cluster-wide objects _(e.g. StorageClass, Nodes, PersistentVolumes, etc)_.

 <!-- body -->

@@ -52,7 +52,7 @@ equivalent to "Predicate" and "Scoring" is equivalent to "Priority function".
 One plugin may register at multiple extension points to perform more complex or
 stateful tasks.

-{{< figure src="/images/docs/scheduling-framework-extensions.png" title="scheduling framework extension points" >}}
+{{< figure src="/images/docs/scheduling-framework-extensions.png" title="scheduling framework extension points" class="diagram-large">}}

 ### QueueSort {#queue-sort}

@@ -29,7 +29,7 @@ computing approach to security, which is widely regarded as a best practice for
 software systems.
 {{< /note >}}

-{{< figure src="/images/docs/4c.png" title="The 4C's of Cloud Native Security" >}}
+{{< figure src="/images/docs/4c.png" title="The 4C's of Cloud Native Security" class="diagram-large" >}}

 Each layer of the Cloud Native security model builds upon the next outermost layer.
 The Code layer benefits from strong base (Cloud, Cluster, Container) security layers.
@@ -60,6 +60,7 @@ Amazon Web Services | https://aws.amazon.com/security/ |
 Google Cloud Platform | https://cloud.google.com/security/ |
 IBM Cloud | https://www.ibm.com/cloud/security |
 Microsoft Azure | https://docs.microsoft.com/en-us/azure/security/azure-security |
+Oracle Cloud Infrastructure | https://www.oracle.com/security/ |
 VMWare VSphere | https://www.vmware.com/security/hardening-guides.html |

 {{< /table >}}
@@ -217,13 +217,13 @@ following pod-specific DNS policies. These policies are specified in the

 - "`Default`": The Pod inherits the name resolution configuration from the node
  that the pods run on.
-  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node)
+  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers)
  for more details.
 - "`ClusterFirst`": Any DNS query that does not match the configured cluster
  domain suffix, such as "`www.kubernetes.io`", is forwarded to the upstream
  nameserver inherited from the node. Cluster administrators may have extra
  stub-domain and upstream DNS servers configured.
-  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#effects-on-pods)
+  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers)
  for details on how DNS queries are handled in those cases.
 - "`ClusterFirstWithHostNet`": For Pods running with hostNetwork, you should
  explicitly set its DNS policy "`ClusterFirstWithHostNet`".
@@ -261,7 +261,7 @@ The following restrictions apply when using this field:
 at a cluster level, you (or your cluster administrator) need to disable the 
 `NetworkPolicyEndPort` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) 
 for the API server with `--feature-gates=NetworkPolicyEndPort=false,…`.
-* The `endPort` field must be equal than or greater to the `port` field.
+* The `endPort` field must be equal to or greater than the `port` field.
 * `endPort` can only be defined if `port` is also defined.
 * Both ports must be numeric.

@@ -183,6 +183,13 @@ Accessing a Service without a selector works the same as if it had a selector.
 In the example above, traffic is routed to the single endpoint defined in
 the YAML: `192.0.2.42:9376` (TCP).

+{{< note >}}
+The Kubernetes API server does not allow proxying to endpoints that are not mapped to 
+pods. Actions such as `kubectl proxy <service-name>` where the service has no 
+selector will fail due to this constraint. This prevents the Kubernetes API server 
+from being used as a proxy to endpoints the caller may not be authorized to access. 
+{{< /note >}}
+
 An ExternalName Service is a special case of Service that does not have
 selectors and uses DNS names instead. For more information, see the
 [ExternalName](#externalname) section later in this document.
@@ -414,7 +421,7 @@ endpoints, the kube-proxy does not forward any traffic for the relevant Service.
 {{< feature-state for_k8s_version="v1.22" state="alpha" >}}
 If you enable the `ProxyTerminatingEndpoints`
 [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
-`ProxyTerminatingEndpoints` for the kube-proxy, the kube-proxy checks if the node
+for the kube-proxy, the kube-proxy checks if the node
 has local endpoints and whether or not all the local endpoints are marked as terminating.
 If there are local endpoints and **all** of those are terminating, then the kube-proxy ignores
 any external traffic policy of `Local`. Instead, whilst the node-local endpoints remain as all
@@ -1065,6 +1072,9 @@ in those modified security groups.

 {{< /note >}}

+Further documentation on annotations for Elastic IPs and other common use-cases may be found
+in the [AWS Load Balancer Controller documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/).
+
 #### Other CLB annotations on Tencent Kubernetes Engine (TKE)

 There are other annotations for managing Cloud Load Balancers on TKE as shown below.
@@ -1121,7 +1131,7 @@ spec:
 ```

 {{< note >}}
-ExternalName accepts an IPv4 address string, but as a DNS names comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName
+ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName
 is intended to specify a canonical DNS name. To hardcode an IP address, consider using
 [headless Services](#headless-services).
 {{< /note >}}
@@ -146,7 +146,7 @@ features:
  parameters.
 - Typical operations on volumes are supported assuming that the driver
  supports them, including
-  ([snapshotting](/docs/concepts/storage/volume-snapshots/),
+  [snapshotting](/docs/concepts/storage/volume-snapshots/),
  [cloning](/docs/concepts/storage/volume-pvc-datasource/),
  [resizing](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims),
  and [storage capacity tracking](/docs/concepts/storage/storage-capacity/).
@@ -131,7 +131,9 @@ The `Retain` reclaim policy allows for manual reclamation of the resource. When

 1. Delete the PersistentVolume. The associated storage asset in external infrastructure (such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume) still exists after the PV is deleted.
 1. Manually clean up the data on the associated storage asset accordingly.
-1. Manually delete the associated storage asset, or if you want to reuse the same storage asset, create a new PersistentVolume with the storage asset definition.
+1. Manually delete the associated storage asset.
+
+If you want to reuse the same storage asset, create a new PersistentVolume with the same storage asset definition.

 #### Delete

@@ -412,11 +414,22 @@ A PersistentVolume can be mounted on a host in any way supported by the resource

 The access modes are:

-* ReadWriteOnce -- the volume can be mounted as read-write by a single node
-* ReadOnlyMany -- the volume can be mounted read-only by many nodes
-* ReadWriteMany -- the volume can be mounted as read-write by many nodes
-* ReadWriteOncePod -- the volume can be mounted as read-write by a single Pod.
-  This is only supported for CSI volumes and Kubernetes version 1.22+.
+`ReadWriteOnce` 
+: the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node.
+
+`ReadOnlyMany`
+: the volume can be mounted as read-only by many nodes.
+
+`ReadWriteMany`
+: the volume can be mounted as read-write by many nodes.
+
+ `ReadWriteOncePod`
+: the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+.
+
+
+
+The blog article [Introducing Single Pod Access Mode for PersistentVolumes](/blog/2021/09/13/read-write-once-pod-access-mode-alpha/) covers this in more detail.
+ 

 In the CLI, the access modes are abbreviated to:

@@ -136,11 +136,16 @@ and set this flag to `false`. For example:

 ## {{% heading "whatsnext" %}}

-[Cron expression format](https://en.wikipedia.org/wiki/Cron)
-documents the format of CronJob `schedule` fields.
-
-For instructions on creating and working with cron jobs, and for an example of CronJob
-manifest, see [Running automated tasks with cron jobs](/docs/tasks/job/automated-tasks-with-cron-jobs).
-
-For instructions to clean up failed or completed jobs automatically, see 
-[Clean up Jobs automatically](/docs/concepts/workloads/controllers/job/#clean-up-finished-jobs-automatically)
+* Learn about [Pods](/docs/concepts/workloads/pods/) and
+  [Jobs](/docs/concepts/workloads/controllers/job/), two concepts
+  that CronJobs rely upon.
+* Read about the [format](https://pkg.go.dev/github.com/robfig/cron/v3#hdr-CRON_Expression_Format)
+  of CronJob `.spec.schedule` fields.
+* For instructions on creating and working with CronJobs, and for an example
+  of a CronJob manifest,
+  see [Running automated tasks with CronJobs](/docs/tasks/job/automated-tasks-with-cron-jobs/).
+* For instructions to clean up failed or completed jobs automatically,
+  see [Clean up Jobs automatically](/docs/concepts/workloads/controllers/job/#clean-up-finished-jobs-automatically)
+* `CronJob` is part of the Kubernetes REST API.
+  Read the {{< api-reference page="workload-resources/cron-job-v1" >}}
+  object definition to understand the API for Kubernetes cron jobs.
@@ -235,3 +235,18 @@ all or certain hosts, if the DaemonSet provides node-level functionality that al
 For example, [network plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) often include a component that runs as a DaemonSet. The DaemonSet component makes sure that the node where it's running has working cluster networking.


+## {{% heading "whatsnext" %}}
+
+* Learn about [Pods](/docs/concepts/workloads/pods).
+  * Learn about [static Pods](#static-pods), which are useful for running Kubernetes
+    {{< glossary_tooltip text="control plane" term_id="control-plane" >}} components.
+* Find out how to use DaemonSets
+  * [Perform a rolling update on a DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/)
+  * [Perform a rollback on a DaemonSet](/docs/tasks/manage-daemon/rollback-daemon-set/)
+    (for example, if a roll out didn't work how you expected).
+* Understand [how Kubernetes assigns Pods to Nodes](/docs/concepts/scheduling-eviction/assign-pod-node/).
+* Learn about [device plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) and
+  [add ons](/docs/concepts/cluster-administration/addons/), which often run as DaemonSets.
+* `DaemonSet` is a top-level resource in the Kubernetes REST API.
+  Read the {{< api-reference page="workload-resources/daemon-set-v1" >}}
+  object definition to understand the API for daemon sets.
@@ -75,11 +75,6 @@ Follow the steps given below to create the above Deployment:
   kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml
   ```

-  {{< note >}}
-  You can specify the `--record` flag to write the command executed in the resource annotation `kubernetes.io/change-cause`.
-  The recorded change is useful for future introspection. For example, to see the commands executed in each Deployment revision.
-  {{< /note >}}
-

 2. Run `kubectl get deployments` to check if the Deployment was created.

@@ -169,13 +164,13 @@ Follow the steps given below to update your Deployment:
 1. Let's update the nginx Pods to use the `nginx:1.16.1` image instead of the `nginx:1.14.2` image.

    ```shell
-    kubectl --record deployment.apps/nginx-deployment set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1
+    kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1
    ```

    or use the following command:

    ```shell
-    kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1 --record
+    kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
    ```
  
    The output is similar to:
@@ -187,7 +182,7 @@ Follow the steps given below to update your Deployment:
    Alternatively, you can `edit` the Deployment and change `.spec.template.spec.containers[0].image` from `nginx:1.14.2` to `nginx:1.16.1`:

    ```shell
-    kubectl edit deployment.v1.apps/nginx-deployment
+    kubectl edit deployment/nginx-deployment
    ```

    The output is similar to:
@@ -370,7 +365,7 @@ rolled back.
 * Suppose that you made a typo while updating the Deployment, by putting the image name as `nginx:1.161` instead of `nginx:1.16.1`:

    ```shell
-    kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true
+    kubectl set image deployment/nginx-deployment nginx=nginx:1.161 
    ```

    The output is similar to this:
@@ -479,26 +474,25 @@ Follow the steps given below to check the rollout history:

 1. First, check the revisions of this Deployment:
    ```shell
-    kubectl rollout history deployment.v1.apps/nginx-deployment
+    kubectl rollout history deployment/nginx-deployment
    ```
    The output is similar to this:
    ```
    deployments "nginx-deployment"
    REVISION    CHANGE-CAUSE
-    1           kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml --record=true
-    2           kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true
-    3           kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.161 --record=true
+    1           kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml
+    2           kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
+    3           kubectl set image deployment/nginx-deployment nginx=nginx:1.161
    ```

    `CHANGE-CAUSE` is copied from the Deployment annotation `kubernetes.io/change-cause` to its revisions upon creation. You can specify the`CHANGE-CAUSE` message by:

-    * Annotating the Deployment with `kubectl annotate deployment.v1.apps/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"`
-    * Append the `--record` flag to save the `kubectl` command that is making changes to the resource.
+    * Annotating the Deployment with `kubectl annotate deployment/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"`
    * Manually editing the manifest of the resource.

 2. To see the details of each revision, run:
    ```shell
-    kubectl rollout history deployment.v1.apps/nginx-deployment --revision=2
+    kubectl rollout history deployment/nginx-deployment --revision=2
    ```

    The output is similar to this:
@@ -506,7 +500,7 @@ Follow the steps given below to check the rollout history:
    deployments "nginx-deployment" revision 2
      Labels:       app=nginx
              pod-template-hash=1159050644
-      Annotations:  kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true
+      Annotations:  kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
      Containers:
       nginx:
        Image:      nginx:1.16.1
@@ -523,7 +517,7 @@ Follow the steps given below to rollback the Deployment from the current version

 1. Now you've decided to undo the current rollout and rollback to the previous revision:
    ```shell
-    kubectl rollout undo deployment.v1.apps/nginx-deployment
+    kubectl rollout undo deployment/nginx-deployment
    ```

    The output is similar to this:
@@ -533,7 +527,7 @@ Follow the steps given below to rollback the Deployment from the current version
    Alternatively, you can rollback to a specific revision by specifying it with `--to-revision`:

    ```shell
-    kubectl rollout undo deployment.v1.apps/nginx-deployment --to-revision=2
+    kubectl rollout undo deployment/nginx-deployment --to-revision=2
    ```

    The output is similar to this:
@@ -567,7 +561,7 @@ Follow the steps given below to rollback the Deployment from the current version
    CreationTimestamp:      Sun, 02 Sep 2018 18:17:55 -0500
    Labels:                 app=nginx
    Annotations:            deployment.kubernetes.io/revision=4
-                            kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1 --record=true
+                            kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
    Selector:               app=nginx
    Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
    StrategyType:           RollingUpdate
@@ -610,7 +604,7 @@ Follow the steps given below to rollback the Deployment from the current version
 You can scale a Deployment by using the following command:

 ```shell
-kubectl scale deployment.v1.apps/nginx-deployment --replicas=10
+kubectl scale deployment/nginx-deployment --replicas=10
 ```
 The output is similar to this:
 ```
@@ -622,7 +616,7 @@ in your cluster, you can setup an autoscaler for your Deployment and choose the
 Pods you want to run based on the CPU utilization of your existing Pods.

 ```shell
-kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80
+kubectl autoscale deployment/nginx-deployment --min=10 --max=15 --cpu-percent=80
 ```
 The output is similar to this:
 ```
@@ -651,7 +645,7 @@ For example, you are running a Deployment with 10 replicas, [maxSurge](#max-surg

 * You update to a new image which happens to be unresolvable from inside the cluster.
    ```shell
-    kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:sometag
+    kubectl set image deployment/nginx-deployment nginx=nginx:sometag
    ```

    The output is similar to this:
@@ -730,7 +724,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess

 * Pause by running the following command:
    ```shell
-    kubectl rollout pause deployment.v1.apps/nginx-deployment
+    kubectl rollout pause deployment/nginx-deployment
    ```

    The output is similar to this:
@@ -740,7 +734,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess

 * Then update the image of the Deployment:
    ```shell
-    kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.16.1
+    kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
    ```

    The output is similar to this:
@@ -750,7 +744,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess

 * Notice that no new rollout started:
    ```shell
-    kubectl rollout history deployment.v1.apps/nginx-deployment
+    kubectl rollout history deployment/nginx-deployment
    ```

    The output is similar to this:
@@ -772,7 +766,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess

 * You can make as many updates as you wish, for example, update the resources that will be used:
    ```shell
-    kubectl set resources deployment.v1.apps/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi
+    kubectl set resources deployment/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi
    ```

    The output is similar to this:
@@ -785,7 +779,7 @@ apply multiple fixes in between pausing and resuming without triggering unnecess

 * Eventually, resume the Deployment and observe a new ReplicaSet coming up with all the new updates:
    ```shell
-    kubectl rollout resume deployment.v1.apps/nginx-deployment
+    kubectl rollout resume deployment/nginx-deployment
    ```

    The output is similar to this:
@@ -895,7 +889,7 @@ The following `kubectl` command sets the spec with `progressDeadlineSeconds` to
 lack of progress for a Deployment after 10 minutes:

 ```shell
-kubectl patch deployment.v1.apps/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}'
+kubectl patch deployment/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}'
 ```
 The output is similar to this:
 ```
@@ -1056,7 +1050,7 @@ A Deployment also needs a [`.spec` section](https://git.k8s.io/community/contrib

 ### Pod Template

-The `.spec.template` and `.spec.selector` are the only required field of the `.spec`.
+The `.spec.template` and `.spec.selector` are the only required fields of the `.spec`.

 The `.spec.template` is a [Pod template](/docs/concepts/workloads/pods/#pod-templates). It has exactly the same schema as a {{< glossary_tooltip text="Pod" term_id="pod" >}}, except it is nested and does not have an `apiVersion` or `kind`.

@@ -1176,4 +1170,12 @@ a paused Deployment and one that is not paused, is that any changes into the Pod
 Deployment will not trigger new rollouts as long as it is paused. A Deployment is not paused by default when
 it is created.

+## {{% heading "whatsnext" %}}

+* Learn about [Pods](/docs/concepts/workloads/pods).
+* [Run a Stateless Application Using a Deployment](/docs/tasks/run-application/run-stateless-application-deployment/).
+* `Deployment` is a top-level resource in the Kubernetes REST API.
+  Read the {{< api-reference page="workload-resources/deployment-v1" >}}
+  object definition to understand the API for deployments.
+* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how
+  you can use it to manage application availability during disruptions.
@@ -25,6 +25,9 @@ due to a node hardware failure or a node reboot).

 You can also use a Job to run multiple Pods in parallel.

+If you want to run a Job (either a single task, or several in parallel) on a schedule,
+see [CronJob](/docs/concepts/workloads/controllers/cron-jobs/).
+
 <!-- body -->

 ## Running an example Job
@@ -204,6 +207,7 @@ Jobs with _fixed completion count_ - that is, jobs that have non null
    {{< glossary_tooltip term_id="Service" >}}, Pods within the Job can use
    the deterministic hostnames to address each other via DNS.
  - From the containarized task, in the environment variable `JOB_COMPLETION_INDEX`.
+  
  The Job is considered complete when there is one successfully completed Pod
  for each index. For more information about how to use this mode, see
  [Indexed Job for Parallel Processing with Static Work Assignment](/docs/tasks/job/indexed-parallel-processing-static/).
@@ -638,6 +642,19 @@ driver, and then cleans up.
 An advantage of this approach is that the overall process gets the completion guarantee of a Job
 object, but maintains complete control over what Pods are created and how work is assigned to them.

-## Cron Jobs {#cron-jobs}
+## {{% heading "whatsnext" %}}

-You can use a [`CronJob`](/docs/concepts/workloads/controllers/cron-jobs/) to create a Job that will run at specified times/dates, similar to the Unix tool `cron`.
+* Learn about [Pods](/docs/concepts/workloads/pods).
+* Read about different ways of running Jobs:
+   * [Coarse Parallel Processing Using a Work Queue](/docs/tasks/job/coarse-parallel-processing-work-queue/)
+   * [Fine Parallel Processing Using a Work Queue](/docs/tasks/job/fine-parallel-processing-work-queue/)
+   * Use an [indexed Job for parallel processing with static work assignment](/docs/tasks/job/indexed-parallel-processing-static/) (beta)
+   * Create multiple Jobs based on a template: [Parallel Processing using Expansions](/docs/tasks/job/parallel-processing-expansion/)
+* Follow the links within [Clean up finished jobs automatically](#clean-up-finished-jobs-automatically)
+  to learn more about how your cluster can clean up completed and / or failed tasks.
+* `Job` is part of the Kubernetes REST API.
+  Read the {{< api-reference page="workload-resources/job-v1" >}}
+  object definition to understand the API for jobs.
+* Read about [`CronJob`](/docs/concepts/workloads/controllers/cron-jobs/), which you
+  can use to define a series of Jobs that will run based on a schedule, similar to
+  the Unix tool `cron`.
@@ -410,3 +410,14 @@ selector requirements as described in the [labels user guide](/docs/concepts/ove
 As such, ReplicaSets are preferred over ReplicationControllers


+## {{% heading "whatsnext" %}}
+
+* Learn about [Pods](/docs/concepts/workloads/pods).
+* Learn about [Deployments](/docs/concepts/workloads/controllers/deployment/).
+* [Run a Stateless Application Using a Deployment](/docs/tasks/run-application/run-stateless-application-deployment/),
+  which relies on ReplicaSets to work.
+* `ReplicaSet` is a top-level resource in the Kubernetes REST API.
+  Read the {{< api-reference page="workload-resources/replica-set-v1" >}}
+  object definition to understand the API for replica sets.
+* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how
+  you can use it to manage application availability during disruptions.
@@ -248,7 +248,7 @@ The ReplicationController ensures that the desired number of pods matches its la

 The ReplicationController is forever constrained to this narrow responsibility. It itself will not perform readiness nor liveness probes. Rather than performing auto-scaling, it is intended to be controlled by an external auto-scaler (as discussed in [#492](https://issue.k8s.io/492)), which would change its `replicas` field. We will not add scheduling policies (for example, [spreading](https://issue.k8s.io/367#issuecomment-48428019)) to the ReplicationController. Nor should it verify that the pods controlled match the currently specified template, as that would obstruct auto-sizing and other automated processes. Similarly, completion deadlines, ordering dependencies, configuration expansion, and other features belong elsewhere. We even plan to factor out the mechanism for bulk pod creation ([#170](https://issue.k8s.io/170)).

-The ReplicationController is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run, scale) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](https://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html) managing ReplicationControllers, auto-scalers, services, scheduling policies, canaries, etc.
+The ReplicationController is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run, scale) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](https://netflixtechblog.com/asgard-web-based-cloud-management-and-deployment-2c9fc4e4d3a1) managing ReplicationControllers, auto-scalers, services, scheduling policies, canaries, etc.

 ## API Object

@@ -284,6 +284,11 @@ machine-level function, such as machine monitoring or machine logging.  These po
 to a machine lifetime: the pod needs to be running on the machine before other pods start, and are
 safe to terminate when the machine is otherwise ready to be rebooted/shutdown.

-## For more information
+## {{% heading "whatsnext" %}}

-Read [Run Stateless Application Deployment](/docs/tasks/run-application/run-stateless-application-deployment/).
+* Learn about [Pods](/docs/concepts/workloads/pods).
+* Learn about [Deployment](/docs/concepts/workloads/controllers/deployment/), the replacement
+  for ReplicationController.
+* `ReplicationController` is part of the Kubernetes REST API.
+  Read the {{< api-reference page="workload-resources/replication-controller-v1" >}}
+  object definition to understand the API for replication controllers.
@@ -297,7 +297,18 @@ Please note that this field only works if you enable the `StatefulSetMinReadySec

 ## {{% heading "whatsnext" %}}

+* Learn about [Pods](/docs/concepts/workloads/pods).
+* Find out how to use StatefulSets
+  * Follow an example of [deploying a stateful application](/docs/tutorials/stateful-application/basic-stateful-set/).
+  * Follow an example of [deploying Cassandra with Stateful Sets](/docs/tutorials/stateful-application/cassandra/).
+  * Follow an example of [running a replicated stateful application](/docs/tasks/run-application/run-replicated-stateful-application/).
+  * Learn how to [scale a StatefulSet](/docs/tasks/run-application/scale-stateful-set/).
+  * Learn what's involved when you [delete a StatefulSet](/docs/tasks/run-application/delete-stateful-set/).
+  * Learn how to [configure a Pod to use a volume for storage](/docs/tasks/configure-pod-container/configure-volume-storage/).
+  * Learn how to [configure a Pod to use a PersistentVolume for storage](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/).
+* `StatefulSet` is a top-level resource in the Kubernetes REST API.
+  Read the {{< api-reference page="workload-resources/stateful-set-v1" >}}
+  object definition to understand the API for stateful sets.
+* Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how
+  you can use it to manage application availability during disruptions.

-* Follow an example of [deploying a stateful application](/docs/tutorials/stateful-application/basic-stateful-set/).
-* Follow an example of [deploying Cassandra with Stateful Sets](/docs/tutorials/stateful-application/cassandra/).
-* Follow an example of [running a replicated stateful application](/docs/tasks/run-application/run-replicated-stateful-application/).
@@ -48,6 +48,21 @@ with shared namespaces and shared filesystem volumes.

 ## Using Pods

+The following is an example of a Pod which consists of a container running the image `nginx:1.14.2`.
+
+{{< codenew file="pods/simple-pod.yaml" >}}
+
+To create the Pod shown above, run the following command:
+```shell
+kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml
+```
+
+Pods are generally not created directly and are created using workload resources.
+See [Working with Pods](#working-with-pods) for more information on how Pods are used
+with workload resources.
+
+### Workload resources for managing pods
+
 Usually you don't need to create Pods directly, even singleton Pods. Instead, create them using workload resources such as {{< glossary_tooltip text="Deployment"
 term_id="deployment" >}} or {{< glossary_tooltip text="Job" term_id="job" >}}.
 If your Pods need to track state, consider the
@@ -97,7 +112,7 @@ For example, you might have a container that
 acts as a web server for files in a shared volume, and a separate "sidecar" container
 that updates those files from a remote source, as in the following diagram:

-{{< figure src="/images/docs/pod.svg" alt="example pod diagram" width="50%" >}}
+{{< figure src="/images/docs/pod.svg" alt="Pod creation diagram" class="diagram-medium" >}}

 Some Pods have {{< glossary_tooltip text="init containers" term_id="init-container" >}} as well as {{< glossary_tooltip text="app containers" term_id="app-container" >}}. Init containers run and complete before the app containers are started.

@@ -309,9 +324,9 @@ in the Pod Lifecycle documentation.
 * Read about [Pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/).
 * Read about [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) and how you can use it to manage application availability during disruptions.
 * Pod is a top-level resource in the Kubernetes REST API.
-  The [Pod](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core)
+  The {{< api-reference page="workload-resources/pod-v1" >}}
  object definition describes the object in detail.
-* [The Distributed System Toolkit: Patterns for Composite Containers](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns) explains common layouts for Pods with more than one container.
+* [The Distributed System Toolkit: Patterns for Composite Containers](/blog/2015/06/the-distributed-system-toolkit-patterns/) explains common layouts for Pods with more than one container.

 To understand the context for why Kubernetes wraps a common Pod API in other resources (such as {{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} or {{< glossary_tooltip text="Deployments" term_id="deployment" >}}), you can read about the prior art, including:

@@ -55,7 +55,7 @@ exists. If that Pod is deleted for any reason, and even if an identical replacem
 is created, the related thing (a volume, in this example) is also destroyed and
 created anew.

-{{< figure src="/images/docs/pod.svg" title="Pod diagram" width="50%" >}}
+{{< figure src="/images/docs/pod.svg" title="Pod diagram" class="diagram-medium" >}}

 *A multi-container Pod that contains a file puller and a
 web server that uses a persistent volume for shared storage between the containers.*
@@ -234,7 +234,7 @@ To overcome this situation, you can either increase the `maxSkew` or modify one

 The scheduler will skip the non-matching nodes from the skew calculations if the incoming Pod has `spec.nodeSelector` or `spec.affinity.nodeAffinity` defined.

-    Suppose you have a 5-node cluster ranging from zoneA to zoneC:
+Suppose you have a 5-node cluster ranging from zoneA to zoneC:

    {{<mermaid>}}
    graph BT
@@ -268,7 +268,7 @@ The scheduler will skip the non-matching nodes from the skew calculations if the
    class zoneC cluster;
    {{< /mermaid >}}

-    and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed onto "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected.
+and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed onto "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected.

    {{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}}

@@ -29,6 +29,8 @@ Kubernetes documentation contributors:
 - Translate the documentation
 - Manage and publish the documentation parts of the Kubernetes release cycle

+
+
 <!-- body -->

 ## Getting started
@@ -44,18 +46,98 @@ to work effectively in the Kubernetes community.
 To get involved with documentation:

 1. Sign the CNCF [Contributor License Agreement](https://github.com/kubernetes/community/blob/master/CLA.md).
-1. Familiarize yourself with the [documentation repository](https://github.com/kubernetes/website)
+2. Familiarize yourself with the [documentation repository](https://github.com/kubernetes/website)
   and the website's [static site generator](https://gohugo.io).
-1. Make sure you understand the basic processes for
+3. Make sure you understand the basic processes for
   [opening a pull request](/docs/contribute/new-content/open-a-pr/) and
   [reviewing changes](/docs/contribute/review/reviewing-prs/).

+<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure -->
+<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it -->
+
+{{< mermaid >}}
+flowchart TB
+subgraph third[Open PR]
+direction TB
+U[ ] -.-
+Q[Improve content] --- N[Create content]
+N --- O[Translate docs]
+O --- P[Manage/publish docs parts<br>of K8s release cycle]
+
+end
+
+subgraph second[Review]
+direction TB
+   T[ ] -.-
+   D[Look over the<br>K8s/website<br>repository] --- E[Check out the<br>Hugo static site<br>generator]
+   E --- F[Understand basic<br>GitHub commands]
+   F --- G[Review open PR<br>and change review <br>processes]
+end
+
+subgraph first[Sign up]
+    direction TB
+    S[ ] -.-
+    B[Sign the CNCF<br>Contributor<br>License Agreement] --- C[Join sig-docs<br>Slack channel] 
+    C --- V[Join kubernetes-sig-docs<br>mailing list]
+    V --- M[Attend weekly<br>sig-docs calls<br>or slack meetings]
+end
+
+A([fa:fa-user New<br>Contributor]) --> first
+A --> second
+A --> third
+A --> H[Ask Questions!!!]
+
+
+classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
+classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
+classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
+class A,B,C,D,E,F,G,H,M,Q,N,O,P,V grey
+class S,T,U spacewhite
+class first,second,third white
+{{</ mermaid >}}
+***Figure - Getting started for a new contributor***
+
+The figure above outlines a roadmap for new contributors. You can follow some or all of the steps for `Sign up` and `Review`. Now you are ready to open PRs that achieve your contribution objectives with some listed under `Open PR`. Again, questions are always welcome!
+
 Some tasks require more trust and more access in the Kubernetes organization.
 See [Participating in SIG Docs](/docs/contribute/participate/) for more details about
 roles and permissions.

 ## Your first contribution

+You can prepare for your first contribution by reviewing several steps beforehand. The figure below outlines the steps and the details follow. 
+
+<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure -->
+<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it -->
+
+{{< mermaid >}}
+flowchart LR
+    subgraph second[First Contribution]
+    direction TB
+    S[ ] -.-
+    G[Review PRs from other<br>K8s members] -->
+    A[Check K8s/website<br>issues list for<br>good first PRs] --> B[Open a PR!!]
+    end
+    subgraph first[Suggested Prep]
+    direction TB
+       T[ ] -.-
+       D[Read contribution overview] -->E[Read K8s content<br>and style guides]
+       E --> F[Learn about Hugo page<br>content types<br>and shortcodes]
+    end
+    
+
+    first ----> second
+     
+
+classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
+classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
+classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
+class A,B,D,E,F,G grey
+class S,T spacewhite
+class first,second white
+{{</ mermaid >}}
+***Figure - Preparation for your first contribution***
+
 - Read the [Contribution overview](/docs/contribute/new-content/overview/) to
  learn about the different ways you can contribute.
 - Check [`kubernetes/website` issues list](https://github.com/kubernetes/website/issues/)
@@ -92,10 +174,12 @@ SIG Docs communicates with different methods:
  introduce yourself!
 - [Join the `kubernetes-sig-docs` mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-docs),
  where broader discussions take place and official decisions are recorded.
- Join the [weekly SIG Docs video meeting](https://github.com/kubernetes/community/tree/master/sig-docs). Meetings are always announced on `#sig-docs` and added to the [Kubernetes community meetings calendar](https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com&ctz=America/Los_Angeles). You'll need to download the [Zoom client](https://zoom.us/download) or dial in using a phone.
+- Join the [SIG Docs video meeting](https://github.com/kubernetes/community/tree/master/sig-docs) held every two weeks. Meetings are always announced on `#sig-docs` and added to the [Kubernetes community meetings calendar](https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com&ctz=America/Los_Angeles). You'll need to download the [Zoom client](https://zoom.us/download) or dial in using a phone.
+- Join the SIG Docs async Slack standup meeting on those weeks when the in-person Zoom video meeting does not take place. Meetings are always announced on `#sig-docs`. You can contribute to any one of the threads up to 24 hours after meeting announcement. 

 ## Other ways to contribute

 - Visit the [Kubernetes community site](/community/). Participate on Twitter or Stack Overflow, learn about local Kubernetes meetups and events, and more.
 - Read the [contributor cheatsheet](https://github.com/kubernetes/community/tree/master/contributors/guide/contributor-cheatsheet) to get involved with Kubernetes feature development.
 - Submit a [blog post or case study](/docs/contribute/new-content/blogs-case-studies/).
+
@@ -346,7 +346,7 @@ Teams must merge localized content into the same branch from which the content w

 For example:
 - a localization branch sourced from `main` must be merged into `main`.
- a localization branch sourced from `release-{{ skew "prevMinorVersion" }}` must be merged into `release-{{ skew "prevMinorVersion" }}`.
+- a localization branch sourced from `release-{{% skew "prevMinorVersion" %}}` must be merged into `release-{{% skew "prevMinorVersion" %}}`.

 {{< note >}}
 If your localization branch was created from `main` branch but it is not merged into `main` before new release branch `{{< release-branch >}}` created, merge it into both `main` and new release branch `{{< release-branch >}}`. To merge your localization branch into new release branch `{{< release-branch >}}`, you need to switch upstream branch of your localization branch to `{{< release-branch >}}`.
@@ -361,5 +361,4 @@ For more information about working from forks or directly from the repository, s
 ## Upstream contributions

 SIG Docs welcomes upstream contributions and corrections to the English source.
-
-
+                                                                                                                                      
@@ -1,4 +1,118 @@
 ---
 title: Contributing new content
+content_type: concept
+main_menu: true
 weight: 20
 ---
+
+
+
+<!-- overview -->
+
+This section contains information you should know before contributing new
+content. 
+<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure -->
+<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it -->
+
+{{< mermaid >}}
+flowchart LR 
+    subgraph second[Before you begin]
+    direction TB
+    S[ ] -.-
+    A[Sign the CNCF CLA] --> B[Choose Git branch]
+    B --> C[One language per PR]
+    C --> F[Check out<br>contributor tools]
+    end
+    subgraph first[Contributing Basics]
+    direction TB
+       T[ ] -.-
+       D[Write docs in markdown<br>and build site with Hugo] --- E[source in GitHub]
+       E --- G[_'/content/../docs'_ folder contains docs<br>for multiple languages]
+       G --- H[Review Hugo page content<br>types and shortcodes]
+    end
+    
+
+    first ----> second
+     
+
+classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
+classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
+classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
+class A,B,C,D,E,F,G,H grey
+class S,T spacewhite
+class first,second white
+{{</ mermaid >}}
+
+***Figure - Contributing new content preparation***
+
+The figure above depicts the information you should know
+prior to submitting new content. The information details follow.
+
+
+
+<!-- body -->
+
+## Contributing basics
+
+- Write Kubernetes documentation in Markdown and build the Kubernetes site
+  using [Hugo](https://gohugo.io/).
+- The source is in [GitHub](https://github.com/kubernetes/website). You can find
+  Kubernetes documentation at `/content/en/docs/`. Some of the reference
+  documentation is automatically generated from scripts in
+  the `update-imported-docs/` directory.
+- [Page content types](/docs/contribute/style/page-content-types/) describe the
+  presentation of documentation content in Hugo.
+- In addition to the standard Hugo shortcodes, we use a number of
+  [custom Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/) in our
+  documentation to control the presentation of content.
+- Documentation source is available in multiple languages in `/content/`. Each
+  language has its own folder with a two-letter code determined by the
+  [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php)
+  . For example, English documentation source is stored in `/content/en/docs/`.
+- For more information about contributing to documentation in multiple languages
+  or starting a new translation,
+  see [localization](/docs/contribute/localization).
+
+## Before you begin {#before-you-begin}
+
+### Sign the CNCF CLA {#sign-the-cla}
+
+All Kubernetes contributors **must** read
+the [Contributor guide](https://github.com/kubernetes/community/blob/master/contributors/guide/README.md)
+and [sign the Contributor License Agreement (CLA)](https://github.com/kubernetes/community/blob/master/CLA.md)
+.
+
+Pull requests from contributors who haven't signed the CLA fail the automated
+tests. The name and email you provide must match those found in
+your `git config`, and your git name and email must match those used for the
+CNCF CLA.
+
+### Choose which Git branch to use
+
+When opening a pull request, you need to know in advance which branch to base
+your work on.
+
+Scenario | Branch
+:---------|:------------
+Existing or new English language content for the current release | `main`
+Content for a feature change release | The branch which corresponds to the major and minor version the feature change is in, using the pattern `dev-<version>`. For example, if a feature changes in the `v{{< skew nextMinorVersion >}}` release, then add documentation changes to the ``dev-{{< skew nextMinorVersion >}}`` branch.
+Content in other languages (localizations) | Use the localization's convention. See the [Localization branching strategy](/docs/contribute/localization/#branching-strategy) for more information.
+
+If you're still not sure which branch to choose, ask in `#sig-docs` on Slack.
+
+{{< note >}} If you already submitted your pull request and you know that the
+base branch was wrong, you (and only you, the submitter) can change it. {{<
+/note >}}
+
+### Languages per PR
+
+Limit pull requests to one language per PR. If you need to make an identical
+change to the same code sample in multiple languages, open a separate PR for
+each language.
+
+## Tools for contributors
+
+The [doc contributors tools](https://github.com/kubernetes/website/tree/main/content/en/docs/doc-contributor-tools)
+directory in the `kubernetes/website` repository contains tools to help your
+contribution journey go more smoothly.
+
@@ -28,7 +28,40 @@ If your changes are large, read [Work from a local fork](#fork-the-repo) to lear
 ## Changes using GitHub

 If you're less experienced with git workflows, here's an easier method of
-opening a pull request.
+opening a pull request. The figure below outlines the steps and the details follow.
+
+<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure -->
+<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it -->
+
+{{< mermaid >}}
+flowchart LR
+A([fa:fa-user New<br>Contributor]) --- id1[(K8s/Website<br>GitHub)]
+subgraph tasks[Changes using GitHub]
+direction TB
+    0[ ] -.-
+    1[1. Edit this page] --> 2[2. Use GitHub markdown<br>editor to make changes]
+    2 --> 3[3. fill in Propose file change]
+
+end
+subgraph tasks2[ ]
+direction TB
+4[4. select Propose file change] --> 5[5. select Create pull request] --> 6[6. fill in Open a pull request]
+6 --> 7[7. select Create pull request] 
+end
+
+id1 --> tasks --> tasks2
+
+classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
+classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
+classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff;
+classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
+class A,1,2,3,4,5,6,7 grey
+class 0 spacewhite
+class tasks,tasks2 white
+class id1 k8s
+{{</ mermaid >}}
+
+***Figure - Steps for opening a PR using GitHub***

 1.  On the page where you see the issue, select the pencil icon at the top right.
    You can also scroll to the bottom of the page and select **Edit this page**.
@@ -89,6 +122,37 @@ work from a local fork.

 Make sure you have [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed on your computer. You can also use a git UI application.

+The figure below shows the steps to follow when you work from a local fork. The details for each step follow.
+
+<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure -->
+<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it -->
+
+{{< mermaid >}}
+flowchart LR
+1[Fork the K8s/website<br>repository] --> 2[Create local clone<br>and set upstream]
+subgraph changes[Your changes]
+direction TB
+S[ ] -.-
+3[Create a branch<br>example: my_new_branch] --> 3a[Make changes using<br>text editor] --> 4["Preview your changes<br>locally using Hugo<br>(localhost:1313)<br>or build container image"]
+end
+subgraph changes2[Commit / Push]
+direction TB
+T[ ] -.-
+5[Commit your changes] --> 6[Push commit to<br>origin/my_new_branch]
+end
+
+2 --> changes --> changes2
+
+classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
+classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
+classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff;
+classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
+class 1,2,3,3a,4,5,6 grey
+class S,T spacewhite
+class changes,changes2 white
+{{</ mermaid >}}
+***Figure - Working from a local fork to make your changes***
+
 ### Fork the kubernetes/website repository

 1. Navigate to the [`kubernetes/website`](https://github.com/kubernetes/website/) repository.
@@ -289,6 +353,34 @@ Alternately, install and use the `hugo` command on your computer:

 ### Open a pull request from your fork to kubernetes/website {#open-a-pr}

+The figure below shows the steps to open a PR from your fork to the K8s/website. The details follow.
+<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure -->
+<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it -->
+
+{{< mermaid >}}
+flowchart LR
+subgraph first[ ]
+direction TB
+1[1. Go to K8s/website repository] --> 2[2. Select New Pull Request]
+2 --> 3[3. Select compare across forks]
+3 --> 4[4. Select your fork from<br>head repository drop-down menu]
+end
+subgraph second [ ]
+direction TB
+5[5. Select your branch from<br>the compare drop-down menu] --> 6[6. Select Create Pull Request]
+6 --> 7[7. Add a description<br>to your PR]
+7 --> 8[8. Select Create pull request]
+end
+
+first --> second
+
+classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
+classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
+class 1,2,3,4,5,6,7,8 grey
+class first,second white
+{{</ mermaid >}}
+***Figure - Steps to open a PR from your fork to the K8s/website***
+
 1. In a web browser, go to the [`kubernetes/website`](https://github.com/kubernetes/website/) repository.
 2. Select **New Pull Request**.
 3. Select **compare across forks**.
@@ -303,7 +395,7 @@ Alternately, install and use the `hugo` command on your computer:

 8. Select the **Create pull request** button.

-  Congratulations! Your pull request is available in [Pull requests](https://github.com/kubernetes/website/pulls).
+Congratulations! Your pull request is available in [Pull requests](https://github.com/kubernetes/website/pulls).


 After opening a PR, GitHub runs automated tests and tries to deploy a preview using [Netlify](https://www.netlify.com/).
@@ -414,7 +506,6 @@ If another contributor commits changes to the same file in another PR, it can cr

    The pull request no longer shows any conflicts.

-
 ### Squashing commits

 {{< note >}}
@@ -500,11 +591,8 @@ Most repositories use issue and PR templates. Have a look through some open
 issues and PRs to get a feel for that team's processes. Make sure to fill out
 the templates with as much detail as possible when you file issues or PRs.

-
-
 ## {{% heading "whatsnext" %}}


 - Read [Reviewing](/docs/contribute/review/reviewing-prs) to learn more about the review process.

-
@@ -1,65 +0,0 @@
---
-title: Contributing new content overview
-linktitle: Overview
-content_type: concept
-main_menu: true
-weight: 5
---
-
-<!-- overview -->
-
-This section contains information you should know before contributing new content.
-
-
-
-
-<!-- body -->
-
-## Contributing basics
-
- Write Kubernetes documentation in Markdown and build the Kubernetes site using [Hugo](https://gohugo.io/).
- The source is in [GitHub](https://github.com/kubernetes/website). You can find Kubernetes documentation at `/content/en/docs/`. Some of the reference documentation is automatically generated from scripts in the `update-imported-docs/` directory.
- [Page content types](/docs/contribute/style/page-content-types/) describe the presentation of documentation content in Hugo.
- In addition to the standard Hugo shortcodes, we use a number of
-  [custom Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/) in our documentation to control the presentation of content.
- Documentation source is available in multiple languages in `/content/`. Each
-  language has its own folder with a two-letter code determined by the
-  [ISO 639-1 standard](https://www.loc.gov/standards/iso639-2/php/code_list.php). For
-  example, English documentation source is stored in `/content/en/docs/`.
- For more information about contributing to documentation in multiple languages or starting a new translation, see [localization](/docs/contribute/localization).
-
-## Before you begin {#before-you-begin}
-
-### Sign the CNCF CLA {#sign-the-cla}
-
-All Kubernetes contributors **must** read the [Contributor guide](https://github.com/kubernetes/community/blob/master/contributors/guide/README.md) and [sign the Contributor License Agreement (CLA)](https://github.com/kubernetes/community/blob/master/CLA.md).
-
-Pull requests from contributors who haven't signed the CLA fail the automated tests. The name and email you provide must match those found in your `git config`, and your git name and email must match those used for the CNCF CLA.
-
-### Choose which Git branch to use
-
-When opening a pull request, you need to know in advance which branch to base your work on.
-
-Scenario | Branch
-:---------|:------------
-Existing or new English language content for the current release | `main`
-Content for a feature change release | The branch which corresponds to the major and minor version the feature change is in, using the pattern `dev-<version>`. For example, if a feature changes in the `v{{< skew nextMinorVersion >}}` release, then add documentation changes to the ``dev-{{< skew nextMinorVersion >}}`` branch.
-Content in other languages (localizations) | Use the localization's convention. See the [Localization branching strategy](/docs/contribute/localization/#branching-strategy) for more information.
-
-
-If you're still not sure which branch to choose, ask in `#sig-docs` on Slack.
-
-{{< note >}}
-If you already submitted your pull request and you know that the base branch
-was wrong, you (and only you, the submitter) can change it.
-{{< /note >}}
-
-### Languages per PR
-
-Limit pull requests to one language per PR. If you need to make an identical change to the same code sample in multiple languages, open a separate PR for each language.
-
-## Tools for contributors
-
-The [doc contributors tools](https://github.com/kubernetes/website/tree/main/content/en/docs/doc-contributor-tools) directory in the `kubernetes/website` repository contains tools to help your contribution journey go more smoothly.
-
-
@@ -36,7 +36,38 @@ Before you start a review:

 ## Review process

-In general, review pull requests for content and style in English.
+In general, review pull requests for content and style in English. The figure below outlines the steps for the review process. The details for each step follow.
+
+<!-- See https://github.com/kubernetes/website/issues/28808 for live-editor URL to this figure -->
+<!-- You can also cut/paste the mermaid code into the live editor at https://mermaid-js.github.io/mermaid-live-editor to play around with it -->
+
+{{< mermaid >}}
+flowchart LR
+    subgraph fourth[Start review]
+    direction TB
+    S[ ] -.-
+    M[add comments] --> N[review changes]
+    N --> O[new contributors should<br>choose Comment]
+    end
+    subgraph third[Select PR]
+    direction TB
+    T[ ] -.-
+    J[read description<br>and comments]--> K[preview changes in<br>Netlify preview build]
+    end
+ 
+  A[Review open PR list]--> B[Filter open PRs<br>by label]
+  B --> third --> fourth
+     
+
+classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
+classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
+classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
+class A,B,J,K,M,N,O grey
+class S,T spacewhite
+class third,fourth white
+{{</ mermaid >}}
+
+***Figure - Review process steps***

 1.  Go to
    [https://github.com/kubernetes/website/pulls](https://github.com/kubernetes/website/pulls).
@@ -134,6 +134,21 @@ The output is similar to this:
 no
 ```

+Similarly, to check whether a Service Account named `dev-sa` in Namespace `dev`
+can list Pods in the Namespace `target`:
+
+```bash
+kubectl auth can-i list pods \
+	--namespace target \
+	--as system:serviceaccount:dev:dev-sa
+```
+
+The output is similar to this:
+
+```
+yes
+```
+
 `SelfSubjectAccessReview` is part of the `authorization.k8s.io` API group, which
 exposes the API server authorization to external services. Other resources in
 this group include:
@@ -192,22 +207,33 @@ The following flags can be used:
 You can choose more than one authorization module. Modules are checked in order
 so an earlier module has higher priority to allow or deny a request.

-## Privilege escalation via pod creation
+## Privilege escalation via workload creation or edits {#privilege-escalation-via-pod-creation}

-Users who have the ability to create pods in a namespace can potentially
-escalate their privileges within that namespace.  They can create pods that
-access their privileges within that namespace. They can create pods that access
-secrets the user cannot themselves read, or that run under a service account
-with different/greater permissions.
+Users who can create/edit pods in a namespace, either directly or through a [controller](/docs/concepts/architecture/controller/)
+such as an operator, could escalate their privileges in that namespace.

 {{< caution >}}
-System administrators, use care when granting access to pod creation. A user
-granted permission to create pods (or controllers that create pods) in the
-namespace can: read all secrets in the namespace; read all config maps in the
-namespace; and impersonate any service account in the namespace and take any
-action the account could take. This applies regardless of authorization mode.
+System administrators, use care when granting access to create or edit workloads.
+Details of how these can be misused are documented in [escalation paths](/docs/reference/access-authn-authz/authorization/#escalation-paths)
 {{< /caution >}}

+### Escalation paths {#escalation-paths}
+- Mounting arbitrary secrets in that namespace
+  - Can be used to access secrets meant for other workloads
+  - Can be used to obtain a more privileged service account's service account token
+- Using arbitrary Service Accounts in that namespace
+  - Can perform Kubernetes API actions as another workload (impersonation)
+  - Can perform any privileged actions that Service Account has
+- Mounting configmaps meant for other workloads in that namespace
+  - Can be used to obtain information meant for other workloads, such as DB host names.
+- Mounting volumes meant for other workloads in that namespace
+  - Can be used to obtain information meant for other workloads, and change it.
+
+{{< caution >}}
+System administrators should be cautious when deploying CRDs that
+change the above areas. These may open privilege escalations paths.
+This should be considered when deciding on your RBAC controls.
+{{< /caution >}}

 ## {{% heading "whatsnext" %}}

@@ -67,7 +67,7 @@ See the [webhook request](#request) section for details on the data sent to webh
 See the [webhook response](#response) section for the data expected from webhooks.

 The example admission webhook server leaves the `ClientAuth` field
-[empty](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/config.go#L47-L48),
+[empty](https://github.com/kubernetes/kubernetes/blob/v1.22.0/test/images/agnhost/webhook/config.go#L38-L39),
 which defaults to `NoClientCert`. This means that the webhook server does not
 authenticate the identity of the clients, supposedly apiservers. If you need
 mutual TLS or other ways to authenticate the clients, see
@@ -79,7 +79,7 @@ The webhook server in the e2e test is deployed in the Kubernetes cluster, via
 the [deployment API](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#deployment-v1-apps).
 The test also creates a [service](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core)
 as the front-end of the webhook server. See
-[code](https://github.com/kubernetes/kubernetes/blob/v1.15.0/test/e2e/apimachinery/webhook.go#L301).
+[code](https://github.com/kubernetes/kubernetes/blob/v1.22.0/test/e2e/apimachinery/webhook.go#L748).

 You may also deploy your webhooks outside of the cluster. You will need to update
 your webhook configurations accordingly.
@@ -125,7 +125,6 @@ different Kubernetes components.
 | `HPAScaleToZero` | `false` | Alpha | 1.16 | |
 | `IndexedJob` | `false` | Alpha | 1.21 | 1.21 |
 | `IndexedJob` | `true` | Beta | 1.22 | |
-| `JobTrackingWithFinalizers` | `false` | Alpha | 1.22 | |
 | `IngressClassNamespacedParams` | `false` | Alpha | 1.21 | 1.21 |
 | `IngressClassNamespacedParams` | `true` | Beta | 1.22 | |
 | `InTreePluginAWSUnregister` | `false` | Alpha | 1.21 | |
@@ -138,13 +137,13 @@ different Kubernetes components.
 | `IPv6DualStack` | `true` | Beta | 1.21 | |
 | `JobTrackingWithFinalizers` | `false` | Alpha | 1.22 | |
 | `KubeletCredentialProviders` | `false` | Alpha | 1.20 | |
+| `KubeletInUserNamespace` | `false` | Alpha | 1.22 | |
+| `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | |
 | `LocalStorageCapacityIsolation` | `false` | Alpha | 1.7 | 1.9 |
 | `LocalStorageCapacityIsolation` | `true` | Beta | 1.10 | |
 | `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | |
 | `LogarithmicScaleDown` | `false` | Alpha | 1.21 | 1.21 |
 | `LogarithmicScaleDown` | `true` | Beta | 1.22 | |
-| `KubeletInUserNamespace` | `false` | Alpha | 1.22 | |
-| `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | |
 | `MemoryManager` | `false` | Alpha | 1.21 | 1.21 |
 | `MemoryManager` | `true` | Beta | 1.22 | |
 | `MemoryQoS` | `false` | Alpha | 1.22 | |
@@ -287,9 +286,6 @@ different Kubernetes components.
 | `DynamicKubeletConfig` | `false` | Deprecated | 1.22 | - |
 | `DynamicProvisioningScheduling` | `false` | Alpha | 1.11 | 1.11 |
 | `DynamicProvisioningScheduling` | - | Deprecated| 1.12 | - |
-| `DynamicKubeletConfig` | `false` | Alpha | 1.4 | 1.10 |
-| `DynamicKubeletConfig` | `true` | Beta | 1.11 | 1.21 |
-| `DynamicKubeletConfig` | `false` | Deprecated | 1.22 | - |
 | `DynamicVolumeProvisioning` | `true` | Alpha | 1.3 | 1.7 |
 | `DynamicVolumeProvisioning` | `true` | GA | 1.8 | - |
 | `EnableAggregatedDiscoveryTimeout` | `true` | Deprecated | 1.16 | - |
@@ -51,7 +51,7 @@ kubelet [flags]
 <td colspan="2">--address string&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Default: 0.0.0.0 </td>
 </tr>
 <tr>
-<td></td><td style="line-height: 130%; word-wrap: break-word;">The IP address for the Kubelet to serve on (set to <code>0.0.0.0</code> or <code>::</code> for listening in gll interfaces and IP families)  (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's <code>--config</code> flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
+<td></td><td style="line-height: 130%; word-wrap: break-word;">The IP address for the Kubelet to serve on (set to <code>0.0.0.0</code> or <code>::</code> for listening in all interfaces and IP families)  (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's <code>--config</code> flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)</td>
 </tr>
    
 <tr>
@@ -10,9 +10,9 @@ aka:
 tags:
 - fundamental
 ---
- An abstraction used by Kubernetes to support multiple virtual clusters on the same physical {{< glossary_tooltip text="cluster" term_id="cluster" >}}.
+ An abstraction used by Kubernetes to support isolation of groups of resources within a single {{< glossary_tooltip text="cluster" term_id="cluster" >}}.

 <!--more--> 

-Namespaces are used to organize objects in a cluster and provide a way to divide cluster resources. Names of resources need to be unique within a namespace, but not across namespaces.
+Namespaces are used to organize objects in a cluster and provide a way to divide cluster resources. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects _(e.g. Deployments, Services, etc)_ and not for cluster-wide objects _(e.g. StorageClass, Nodes, PersistentVolumes, etc)_.

@@ -142,6 +142,22 @@ For information about enabling IPVS mode with kubeadm see:
 For information about passing flags to control plane components see:
 - [control-plane-flags](/docs/setup/production-environment/tools/kubeadm/control-plane-flags/)

+### Running kubeadm without an Internet connection {#without-internet-connection}
+
+For running kubeadm without an Internet connection you have to pre-pull the required control-plane images.
+
+You can list and pull the images using the `kubeadm config images` sub-command:
+
+```shell
+kubeadm config images list
+kubeadm config images pull
+```
+
+You can pass `--config` to the above commands with a [kubeadm configuration file](#config-file)
+to control the `kubernetesVersion` and `imageRepository` fields.
+
+All default `k8s.gcr.io` images that kubeadm requires support multiple architectures.
+
 ### Using custom images {#custom-images}

 By default, kubeadm pulls images from `k8s.gcr.io`. If the
@@ -151,13 +167,24 @@ requested Kubernetes version is a CI label (such as `ci/latest`)
 You can override this behavior by using [kubeadm with a configuration file](#config-file).
 Allowed customization are:

+* To provide `kubernetesVersion` which affects the version of the images.
 * To provide an alternative `imageRepository` to be used instead of
  `k8s.gcr.io`.
-* To set `useHyperKubeImage` to `true` to use the HyperKube image.
-* To provide a specific `imageRepository` and `imageTag` for etcd or DNS add-on.
+* To provide a specific `imageRepository` and `imageTag` for etcd or CoreDNS.

-Please note that the configuration field `kubernetesVersion` or the command line flag
-`--kubernetes-version` affect the version of the images.
+Image paths between the default `k8s.gcr.io` and a custom repository specified using
+`imageRepository` may differ for backwards compatibility reasons. For example,
+one image might have a subpath at `k8s.gcr.io/subpath/image`, but be defaulted
+to `my.customrepository.io/image` when using a custom repository.
+
+To ensure you push the images to your custom repository in paths that kubeadm
+can consume, you must:
+
+* Pull images from the defaults paths at `k8s.gcr.io` using `kubeadm config images {list|pull}`.
+* Push images to the paths from `kubeadm config images list --config=config.yaml`,
+where `config.yaml` contains the custom `imageRepository`, and/or `imageTag`
+for etcd and CoreDNS.
+* Pass the same `config.yaml` to `kubeadm init`.

 ### Uploading control-plane certificates to the cluster

@@ -209,19 +236,6 @@ value to the kubelet.

 Be aware that overriding the hostname can [interfere with cloud providers](https://github.com/kubernetes/website/pull/8873).

-### Running kubeadm without an internet connection
-
-For running kubeadm without an internet connection you have to pre-pull the required control-plane images.
-
-You can list and pull the images using the `kubeadm config images` sub-command:
-
-```shell
-kubeadm config images list
-kubeadm config images pull
-```
-
-All images that kubeadm requires such as `k8s.gcr.io/kube-*`, `k8s.gcr.io/etcd` and `k8s.gcr.io/pause` support multiple architectures.
-
 ### Automating kubeadm

 Rather than copying the token you obtained from `kubeadm init` to each node, as
@@ -287,7 +287,7 @@ If your configuration is not using the latest version it is **recommended** that
 the [kubeadm config migrate](/docs/reference/setup-tools/kubeadm/kubeadm-config/) command.

 For more information on the fields and usage of the configuration you can navigate to our
-[API reference](/docs/reference/config-api/kubeadm-config.v1beta2/).
+[API reference](/docs/reference/config-api/kubeadm-config.v1beta3/).

 ## {{% heading "whatsnext" %}}

@@ -15,7 +15,7 @@ behind one command, with support for both planning an upgrade and actually perfo

 ## kubeadm upgrade guidance

-The steps for performing a upgrade using kubeadm are outlined in [this document](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).
+The steps for performing an upgrade using kubeadm are outlined in [this document](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).
 For older versions of kubeadm, please refer to older documentation sets of the Kubernetes website.

 You can use `kubeadm upgrade diff` to see the changes that would be applied to static pod manifests.
@@ -8,8 +8,7 @@ no_list: true
 ---

 <!-- overview -->
-Kubernetes contains several built-in tools and external tools that are commonly used or relevant that may as well be seen as required for Kubernetes to function.
-
+Kubernetes contains several tools to help you work with the Kubernetes system.

 <!-- body -->

@@ -25,10 +24,14 @@ development and testing purposes.
 to a Kubernetes cluster, troubleshoot them, and manage the cluster and its resources itself.

 ## Helm
+{{% thirdparty-content single="true" %}}

 [Helm](https://helm.sh/) is a tool for managing packages of pre-configured
 Kubernetes resources. These packages are known as _Helm charts_.

+Helm is a third party managed tool for managing packages of pre-configured
+Kubernetes resources, aka Kubernetes charts.
+
 Use Helm to:

 * Find and use popular software packaged as Kubernetes charts
@@ -109,7 +109,7 @@ objects.

 All of this is best illustrated by examples.  Imagine a Kubernetes release,
 version X, which introduces a new API group.  A new Kubernetes release is made
-every approximately 3 months (4 per year).  The following table describes which
+every approximately 4 months (3 per year).  The following table describes which
 API versions are supported in a series of subsequent releases.

 <table>
@@ -38,11 +38,13 @@ etcd also implements mutual TLS to authenticate clients and peers.

 ## Where certificates are stored

-If you install Kubernetes with kubeadm, certificates are stored in `/etc/kubernetes/pki`. All paths in this documentation are relative to that directory.
+If you install Kubernetes with kubeadm, most certificates are stored in `/etc/kubernetes/pki`. All paths in this documentation are relative to that directory, with the exception of user account certificates which kubeadm places in `/etc/kubernetes`.

 ## Configure certificates manually

-If you don't want kubeadm to generate the required certificates, you can create them in either of the following ways.
+If you don't want kubeadm to generate the required certificates, you can create them using a single root CA or by providing all certificates. See [Certificates](/docs/tasks/administer-cluster/certificates/) for details on creating your own certificate authority.
+See [Certificate Management with kubeadm](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/) for more on managing certificates.
+

 ### Single root CA

@@ -57,7 +59,16 @@ Required CAs:
 | front-proxy-ca.crt,key | kubernetes-front-proxy-ca | For the [front-end proxy](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) |

 On top of the above CAs, it is also necessary to get a public/private key pair for service account management, `sa.key` and `sa.pub`.
+The following example illustrates the CA key and certificate files shown in the previous table:

+```
+/etc/kubernetes/pki/ca.crt
+/etc/kubernetes/pki/ca.key
+/etc/kubernetes/pki/etcd/ca.crt
+/etc/kubernetes/pki/etcd/ca.key
+/etc/kubernetes/pki/front-proxy-ca.crt
+/etc/kubernetes/pki/front-proxy-ca.key
+```
 ### All certificates

 If you don't wish to copy the CA private keys to your cluster, you can generate all certificates yourself.
@@ -127,6 +138,32 @@ Same considerations apply for the service account key pair:
 |  sa.key                      |                             | kube-controller-manager | --service-account-private-key-file   |
 |                              | sa.pub                      | kube-apiserver          | --service-account-key-file           |

+The following example illustrates the file paths [from the previous tables](/docs/setup/best-practices/certificates/#certificate-paths) you need to provide if you are generating all of your own keys and certificates:
+
+```
+/etc/kubernetes/pki/etcd/ca.key
+/etc/kubernetes/pki/etcd/ca.crt
+/etc/kubernetes/pki/apiserver-etcd-client.key
+/etc/kubernetes/pki/apiserver-etcd-client.crt
+/etc/kubernetes/pki/ca.key
+/etc/kubernetes/pki/ca.crt
+/etc/kubernetes/pki/apiserver.key
+/etc/kubernetes/pki/apiserver.crt
+/etc/kubernetes/pki/apiserver-kubelet-client.key
+/etc/kubernetes/pki/apiserver-kubelet-client.crt
+/etc/kubernetes/pki/front-proxy-ca.key
+/etc/kubernetes/pki/front-proxy-ca.crt
+/etc/kubernetes/pki/front-proxy-client.key
+/etc/kubernetes/pki/front-proxy-client.crt
+/etc/kubernetes/pki/etcd/server.key
+/etc/kubernetes/pki/etcd/server.crt
+/etc/kubernetes/pki/etcd/peer.key
+/etc/kubernetes/pki/etcd/peer.crt
+/etc/kubernetes/pki/etcd/healthcheck-client.key
+/etc/kubernetes/pki/etcd/healthcheck-client.crt
+/etc/kubernetes/pki/sa.key
+/etc/kubernetes/pki/sa.pub
+```
 ## Configure certificates for user accounts

 You must manually configure these administrator account and service accounts:
@@ -146,7 +183,7 @@ The value of `<nodeName>` for `kubelet.conf` **must** match precisely the value

 1. Run `kubectl` as follows for each config:

-```shell
+```
 KUBECONFIG=<filename> kubectl config set-cluster default-cluster --server=https://<host ip>:6443 --certificate-authority <path-to-kubernetes-ca> --embed-certs
 KUBECONFIG=<filename> kubectl config set-credentials <credential-name> --client-key <path-to-key>.pem --client-certificate <path-to-cert>.pem --embed-certs
 KUBECONFIG=<filename> kubectl config set-context default-system --cluster default-cluster --user <credential-name>
@@ -162,4 +199,11 @@ These files are used as follows:
 | controller-manager.conf | kube-controller-manager | Must be added to manifest in `manifests/kube-controller-manager.yaml` |
 | scheduler.conf          | kube-scheduler          | Must be added to manifest in `manifests/kube-scheduler.yaml`          |

+The following files illustrate full paths to the files listed in the previous table:

+```
+/etc/kubernetes/admin.conf
+/etc/kubernetes/kubelet.conf
+/etc/kubernetes/controller-manager.conf
+/etc/kubernetes/scheduler.conf
+```
@@ -9,7 +9,7 @@ weight: 40
 <!-- overview -->

 This page covers how to customize the components that kubeadm deploys. For control plane components
-you can use flags in the `ClusteConfiguration` structure or patches per-node. For the kubelet
+you can use flags in the `ClusterConfiguration` structure or patches per-node. For the kubelet
 and kube-proxy you can use `KubeletConfiguration` and `KubeProxyConfiguration`, accordingly.

 All of these options are possible via the kubeadm configuration API.
@@ -83,6 +83,19 @@ kubeadm to tell it what to do. This crashloop is expected and normal.
 After you initialize your control-plane, the kubelet runs normally.
 {{< /note >}}

+### Preparing the required container images
+
+This step is optional and only applies in case you wish `kubeadm init` and `kubeadm join`
+to not download the default container images which are hosted at `k8s.gcr.io`.
+
+Kubeadm has commands that can help you pre-pull the required images
+when creating a cluster without an internet connection on its nodes.
+See [Running kubeadm without an internet connection](/docs/reference/setup-tools/kubeadm/kubeadm-init#without-internet-connection) for more details.
+
+Kubeadm allows you to use a custom image repository for the required images.
+See [Using custom images](docs/reference/setup-tools/kubeadm/kubeadm-init#custom-images)
+for more details.
+
 ### Initializing your control-plane node

 The control-plane node is the machine where the control plane components run, including
@@ -108,8 +121,6 @@ with the default gateway to set the advertise address for this particular contro
 To use a different network interface, specify the `--apiserver-advertise-address=<ip-address>` argument
 to `kubeadm init`. To deploy an IPv6 Kubernetes cluster using IPv6 addressing, you
 must specify an IPv6 address, for example `--apiserver-advertise-address=fd00::101`
-1. (Optional) Run `kubeadm config images pull` prior to `kubeadm init` to verify
-connectivity to the gcr.io container image registry.

 To initialize the control-plane node run:

@@ -93,6 +93,10 @@ kind: JoinConfiguration
 discovery:
  bootstrapToken:
    apiServerEndpoint: 10.100.0.1:6443
+    token: "clvldh.vjjwg16ucnhp94qr"
+    caCertHashes:
+    - "sha256:a4863cde706cfc580a439f842cc65d5ef112b7b2be31628513a9881cf0d9fe0e"
+    # change auth info above to match the actual token and CA certificate hash for your cluster
 nodeRegistration:
  kubeletExtraArgs:
    node-ip: 10.100.0.3,fd00:1:2:3::3
@@ -109,6 +113,10 @@ controlPlane:
 discovery:
  bootstrapToken:
    apiServerEndpoint: 10.100.0.1:6443
+    token: "clvldh.vjjwg16ucnhp94qr"
+    caCertHashes:
+    - "sha256:a4863cde706cfc580a439f842cc65d5ef112b7b2be31628513a9881cf0d9fe0e"
+    # change auth info above to match the actual token and CA certificate hash for your cluster
 nodeRegistration:
  kubeletExtraArgs:
    node-ip: 10.100.0.4,fd00:1:2:3::4
@@ -118,7 +126,7 @@ nodeRegistration:
 `advertiseAddress` in JoinConfiguration.controlPlane specifies the IP address that the API Server will advertise it is listening on. The value of `advertiseAddress` equals the `--apiserver-advertise-address` flag of `kubeadm join`.

 ```shell
-kubeadm join --config=kubeadm-config.yaml ...
+kubeadm join --config=kubeadm-config.yaml
 ```

 ### Create a single-stack cluster
@@ -150,4 +158,4 @@ networking:

 * [Validate IPv4/IPv6 dual-stack](/docs/tasks/network/validate-dual-stack) networking
 * Read about [Dual-stack](/docs/concepts/services-networking/dual-stack/) cluster networking
-* Learn more about the kubeadm [configuration format](/docs/reference/config-api/kubeadm-config.v1beta2/)
+* Learn more about the kubeadm [configuration format](/docs/reference/config-api/kubeadm-config.v1beta3/)
@@ -77,7 +77,7 @@ option. Your cluster requirements may need a different configuration.
    - Make sure the address of the load balancer always matches
      the address of kubeadm's `ControlPlaneEndpoint`.

-    - Read the [Options for Software Load Balancing](https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing)
+    - Read the [Options for Software Load Balancing](https://git.k8s.io/kubeadm/docs/ha-considerations.md#options-for-software-load-balancing)
      guide for more details.

 1.  Add the first control plane nodes to the load balancer and test the
@@ -262,7 +262,7 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6
  When using DigitalOcean, it can be the public one (assigned to `eth0`) or
  the private one (assigned to `eth1`) should you want to use the optional
  private network. The `kubeletExtraArgs` section of the kubeadm
-  [`NodeRegistrationOptions` structure](/docs/reference/config-api/kubeadm-config.v1beta2/#kubeadm-k8s-io-v1beta2-NodeRegistrationOptions)
+  [`NodeRegistrationOptions` structure](/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-NodeRegistrationOptions)
  can be used for this.

  Then restart `kubelet`:
@@ -336,7 +336,7 @@ Alternatively, you can try separating the `key=value` pairs like so:
 `--apiserver-extra-args "enable-admission-plugins=LimitRanger,enable-admission-plugins=NamespaceExists"`
 but this will result in the key `enable-admission-plugins` only having the value of `NamespaceExists`.

-A known workaround is to use the kubeadm [configuration file](/docs/reference/config-api/kubeadm-config.v1beta2/).
+A known workaround is to use the kubeadm [configuration file](/docs/reference/config-api/kubeadm-config.v1beta3/).

 ## kube-proxy scheduled before node is initialized by cloud-controller-manager

@@ -2,6 +2,7 @@
 title: Set up Ingress on Minikube with the NGINX Ingress Controller
 content_type: task
 weight: 100
+min-kubernetes-server-version: 1.19
 ---

 <!-- overview -->
@@ -17,23 +18,21 @@ This page shows you how to set up a simple Ingress which routes requests to Serv


 {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
+If you are using an older Kubernetes version, switch to the documentation
+for that version.


+### Create a Minikube cluster
+
+Using Katacoda
+: {{< kat-button >}}
+
+Locally
+: If you already [installed Minikube](/docs/tasks/tools/#minikube)
+  locally, run `minikube start` to create a cluster.

 <!-- steps -->

-## Create a Minikube cluster
-
-1. Click **Launch Terminal**
-
-    {{< kat-button >}}
-
-1. (Optional) If you installed Minikube locally, run the following command:
-
-    ```shell
-    minikube start
-    ```
-
 ## Enable the Ingress controller

 1. To enable the NGINX Ingress controller, run the following command:
@@ -45,14 +44,14 @@ This page shows you how to set up a simple Ingress which routes requests to Serv
 1. Verify that the NGINX Ingress controller is running


-  {{< tabs name="tab_with_md" >}}
-  {{% tab name="minikube v1.19 or later" %}}
+   {{< tabs name="tab_with_md" >}}
+   {{% tab name="minikube v1.19 or later" %}}
 ```shell
 kubectl get pods -n ingress-nginx
 ```
-  {{< note >}}This can take up to a minute.{{< /note >}}
+   {{< note >}}It can take up to a minute before you see these pods running OK.{{< /note >}}

-Output:
+   The output is similar to:

 ```
 NAME                                        READY   STATUS      RESTARTS    AGE
@@ -60,15 +59,14 @@ ingress-nginx-admission-create-g9g49        0/1     Completed   0          11m
 ingress-nginx-admission-patch-rqp78         0/1     Completed   1          11m
 ingress-nginx-controller-59b45fb494-26npt   1/1     Running     0          11m
 ```
-    {{% /tab %}}
-
-  {{% tab name="minikube v1.18.1 or earlier" %}}
+   {{% /tab %}}
+   {{% tab name="minikube v1.18.1 or earlier" %}}
 ```shell
 kubectl get pods -n kube-system
 ```
-{{< note >}}This can take up to a minute.{{< /note >}}
+   {{< note >}}It can take up to a minute before you see these pods running OK.{{< /note >}}

-Output:
+   The output is similar to:

 ```
 NAME                                        READY     STATUS    RESTARTS   AGE
@@ -79,133 +77,121 @@ kubernetes-dashboard-5498ccf677-b8p5h       1/1       Running   0          2m
 nginx-ingress-controller-5984b97644-rnkrg   1/1       Running   0          1m
 storage-provisioner                         1/1       Running   0          2m
 ```
-    {{% /tab %}}
-      {{< /tabs >}}
-  
-
-
-    ```shell
-    kubectl get pods -n ingress-nginx
-    ```
-
-    {{< note >}}This can take up to a minute.{{< /note >}}
-
-    Output:
-
-    ```shell
-    NAME                                        READY   STATUS      RESTARTS   AGE
-    ingress-nginx-admission-create-2tgrf        0/1     Completed   0          3m28s
-    ingress-nginx-admission-patch-68b98         0/1     Completed   0          3m28s
-    ingress-nginx-controller-59b45fb494-lzmw2   1/1     Running     0          3m28s
-    ```

+  Make sure that you see a Pod with a name that starts with `nginx-ingress-controller-`.
+   {{% /tab %}}
+   {{< /tabs >}}

 ## Deploy a hello, world app

 1. Create a Deployment using the following command:

-    ```shell
-    kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
-    ```
+   ```shell
+   kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
+   ```

-    Output:
+   The output should be:

-    ```shell
-    deployment.apps/web created
-    ```
+   ```
+   deployment.apps/web created
+   ```

 1. Expose the Deployment:

-    ```shell
-    kubectl expose deployment web --type=NodePort --port=8080
-    ```
+   ```shell
+   kubectl expose deployment web --type=NodePort --port=8080
+   ```

-    Output:
+   The output should be:

-    ```shell
-    service/web exposed
-    ```
+   ```
+   service/web exposed
+   ```

 1. Verify the Service is created and is available on a node port:

    ```shell
-    kubectl get service web
-    ```
+   kubectl get service web
+   ```

-    Output:
+   The output is similar to:

-    ```shell
-    NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
-    web       NodePort   10.104.133.249   <none>        8080:31637/TCP   12m
-    ```
+   ```
+   NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
+   web       NodePort   10.104.133.249   <none>        8080:31637/TCP   12m
+   ```

-1. Visit the service via NodePort:
+1. Visit the Service via NodePort:

-    ```shell
-    minikube service web --url
-    ```
+   ```shell
+   minikube service web --url
+   ```

-    Output:
+   The output is similar to:

-    ```shell
-    http://172.17.0.15:31637
-    ```
+   ```
+   http://172.17.0.15:31637
+   ```

-    {{< note >}}Katacoda environment only: at the top of the terminal panel, click the plus sign, and then click **Select port to view on Host 1**. Enter the NodePort, in this case `31637`, and then click **Display Port**.{{< /note >}}
+   {{< note >}}Katacoda environment only: at the top of the terminal panel, click the plus sign, and then click **Select port to view on Host 1**. Enter the NodePort, in this case `31637`, and then click **Display Port**.{{< /note >}}

-    Output:
+   The output is similar to:

-    ```shell
-    Hello, world!
-    Version: 1.0.0
-    Hostname: web-55b8c6998d-8k564
-    ```
+   ```
+   Hello, world!
+   Version: 1.0.0
+   Hostname: web-55b8c6998d-8k564
+   ```

-    You can now access the sample app via the Minikube IP address and NodePort. The next step lets you access
-    the app using the Ingress resource.
+   You can now access the sample app via the Minikube IP address and NodePort. The next step lets you access
+   the app using the Ingress resource.

-## Create an Ingress resource
+## Create an Ingress

-The following file is an Ingress resource that sends traffic to your Service via hello-world.info.
+The following manifest defines an Ingress that sends traffic to your Service via hello-world.info.

 1. Create `example-ingress.yaml` from the following file:

-  {{< codenew file="service/networking/example-ingress.yaml" >}}
+   {{< codenew file="service/networking/example-ingress.yaml" >}}

-1. Create the Ingress resource by running the following command:
+1. Create the Ingress object by running the following command:

-    ```shell
-    kubectl apply -f https://k8s.io/examples/service/networking/example-ingress.yaml
-    ```
+   ```shell
+   kubectl apply -f https://k8s.io/examples/service/networking/example-ingress.yaml
+   ```

-    Output:
+   The output should be:

-    ```shell
-    ingress.networking.k8s.io/example-ingress created
-    ```
+   ```
+   ingress.networking.k8s.io/example-ingress created
+   ```

 1. Verify the IP address is set:

-    ```shell
-    kubectl get ingress
-    ```
+   ```shell
+   kubectl get ingress
+   ```

-    {{< note >}}This can take a couple of minutes.{{< /note >}}
+   {{< note >}}This can take a couple of minutes.{{< /note >}}

-    ```shell
-    NAME              CLASS    HOSTS              ADDRESS        PORTS   AGE
-    example-ingress   <none>   hello-world.info   172.17.0.15    80      38s
-    ```
+   You should see an IPv4 address in the ADDRESS column; for example:

-1. Add the following line to the bottom of the `/etc/hosts` file.
+   ```
+   NAME              CLASS    HOSTS              ADDRESS        PORTS   AGE
+   example-ingress   <none>   hello-world.info   172.17.0.15    80      38s
+   ```

-    {{< note >}}If you are running Minikube locally, use `minikube ip` to get the external IP. The IP address displayed within the ingress list will be the internal IP.{{< /note >}}
+1. Add the following line to the bottom of the `/etc/hosts` file on
+   your computer (you will need adminstrator access):

    ```
    172.17.0.15 hello-world.info
    ```

-    This sends requests from hello-world.info to Minikube.
+   {{< note >}}If you are running Minikube locally, use `minikube ip` to get the external IP. The IP address displayed within the ingress list will be the internal IP.{{< /note >}}
+
+    After you make this change, your web browser sends requests for
+    hello-world.info URLs to Minikube.

 1. Verify that the Ingress controller is directing traffic:

@@ -213,9 +199,9 @@ The following file is an Ingress resource that sends traffic to your Service via
    curl hello-world.info
    ```

-    Output:
+    You should see:

-    ```shell
+    ```
    Hello, world!
    Version: 1.0.0
    Hostname: web-55b8c6998d-8k564
@@ -223,34 +209,35 @@ The following file is an Ingress resource that sends traffic to your Service via

    {{< note >}}If you are running Minikube locally, you can visit hello-world.info from your browser.{{< /note >}}

-## Create Second Deployment
+## Create a second Deployment

-1. Create a v2 Deployment using the following command:
+1. Create another Deployment using the following command:

-    ```shell
-    kubectl create deployment web2 --image=gcr.io/google-samples/hello-app:2.0
-    ```
-    Output:
+   ```shell
+   kubectl create deployment web2 --image=gcr.io/google-samples/hello-app:2.0
+   ```
+   The output should be:

-    ```shell
-    deployment.apps/web2 created
-    ```
+   ```
+   deployment.apps/web2 created
+   ```

-1. Expose the Deployment:
+1. Expose the second Deployment:

-    ```shell
-    kubectl expose deployment web2 --port=8080 --type=NodePort
-    ```
+   ```shell
+   kubectl expose deployment web2 --port=8080 --type=NodePort
+   ```

-    Output:
+   The output should be:

-    ```shell
-    service/web2 exposed
-    ```
+   ```
+   service/web2 exposed
+   ```

-## Edit Ingress
+## Edit the existing Ingress {#edit-ingress}

-1. Edit the existing `example-ingress.yaml` and add the following lines:
+1. Edit the existing `example-ingress.yaml` manifest, and add the
+   following lines at the end:

    ```yaml
          - path: /v2
@@ -264,47 +251,47 @@ The following file is an Ingress resource that sends traffic to your Service via

 1. Apply the changes:

-    ```shell
-    kubectl apply -f example-ingress.yaml
-    ```
+   ```shell
+   kubectl apply -f example-ingress.yaml
+   ```

-    Output:
+   You should see:

-    ```shell
-    ingress.networking/example-ingress configured
-    ```
+   ```
+   ingress.networking/example-ingress configured
+   ```

-## Test Your Ingress
+## Test your Ingress

 1. Access the 1st version of the Hello World app.

-    ```shell
-    curl hello-world.info
-    ```
+   ```shell
+   curl hello-world.info
+   ```

-    Output:
+   The output is similar to:

-    ```shell
-    Hello, world!
-    Version: 1.0.0
-    Hostname: web-55b8c6998d-8k564
-    ```
+   ```
+   Hello, world!
+   Version: 1.0.0
+   Hostname: web-55b8c6998d-8k564
+   ```

 1. Access the 2nd version of the Hello World app.

-    ```shell
-    curl hello-world.info/v2
-    ```
+   ```shell
+   curl hello-world.info/v2
+   ```

-    Output:
+   The output is similar to:

-    ```shell
-    Hello, world!
-    Version: 2.0.0
-    Hostname: web2-75cd47646f-t8cjk
-    ```
+   ```
+   Hello, world!
+   Version: 2.0.0
+   Hostname: web2-75cd47646f-t8cjk
+   ```

-    {{< note >}}If you are running Minikube locally, you can visit hello-world.info and hello-world.info/v2 from your browser.{{< /note >}}
+   {{< note >}}If you are running Minikube locally, you can visit hello-world.info and hello-world.info/v2 from your browser.{{< /note >}}



@@ -315,5 +302,3 @@ The following file is an Ingress resource that sends traffic to your Service via
 * Read more about [Ingress Controllers](/docs/concepts/services-networking/ingress-controllers/)
 * Read more about [Services](/docs/concepts/services-networking/service/)

-
-
@@ -88,11 +88,11 @@ kubectl patch pv <your-pv-name> -p "{\"spec\":{\"persistentVolumeReclaimPolicy\"
 * Learn more about [PersistentVolumes](/docs/concepts/storage/persistent-volumes/).
 * Learn more about [PersistentVolumeClaims](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims).

-### Reference
+### References {#reference}

-* [PersistentVolume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolume-v1-core)
-* [PersistentVolumeClaim](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolumeclaim-v1-core)
-* See the `persistentVolumeReclaimPolicy` field of [PersistentVolumeSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#persistentvolumeclaim-v1-core).
+* {{< api-reference page="config-and-storage-resources/persistent-volume-v1" >}}
+  * Pay attention to the `.spec.persistentVolumeReclaimPolicy` [field](https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/persistent-volume-v1/#PersistentVolumeSpec) of PersistentVolume.
+* {{< api-reference page="config-and-storage-resources/persistent-volume-claim-v1" >}}



@@ -325,6 +325,10 @@ Here is an example:
 ```shell
 ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379 snapshot restore snapshotdb
 ```
+Another example for restoring using etcdutl options:
+```shell
+ETCDCTL_API=3 etcdctl --data-dir <data-dir-location> snapshot restore snapshotdb
+```

 For more information and examples on restoring a cluster from a snapshot file, see
 [etcd disaster recovery documentation](https://etcd.io/docs/current/op-guide/recovery/#restoring-a-cluster).
@@ -67,7 +67,7 @@ If the `nslookup` command fails, check the following:
 ### Check the local DNS configuration first

 Take a look inside the resolv.conf file.
-(See [Inheriting DNS from the node](/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node) and
+(See [Customizing DNS Service](/docs/tasks/administer-cluster/dns-custom-nameservers) and
 [Known issues](#known-issues) below for more information)

 ```shell
@@ -58,8 +58,7 @@ resources:
 Each `resources` array item is a separate config and contains a complete configuration. The
 `resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`)
 that should be encrypted. The `providers` array is an ordered list of the possible encryption
-providers. Only one provider type may be specified per entry (`identity` or `aescbc` may be provided,
-but not both in the same item).
+providers. Only one provider type may be specified per entry (`identity` or `aescbc` may be provided, but not both in the same item).

 The first provider in the list is used to encrypt resources going into storage. When reading
 resources from storage each provider that matches the stored data attempts to decrypt the data in
@@ -78,9 +77,9 @@ read that resource will fail until it is deleted or a valid decryption key is pr
 Name | Encryption | Strength | Speed | Key Length | Other Considerations
 -----|------------|----------|-------|------------|---------------------
 `identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written.
-`aescbc` | AES-CBC with PKCS#7 padding | Strongest | Fast | 32-byte | The recommended choice for encryption at rest but may be slightly slower than `secretbox`.
 `secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review.
 `aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented.
+`aescbc` | AES-CBC with PKCS#7 padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks.
 `kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes |  The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/)

 Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
@@ -215,5 +214,3 @@ and restart all `kube-apiserver` processes. Then run:
 kubectl get secrets --all-namespaces -o json | kubectl replace -f -
 ```
 to force all secrets to be decrypted.
-
-
@@ -88,6 +88,7 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc
    yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
 {{% /tab %}}
 {{< /tabs >}}
+<br />

 -  Verify that the download works and has the expected version:

@@ -166,11 +167,10 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no

 ### Upgrade kubelet and kubectl

-  Upgrade the kubelet and kubectl
+-  Upgrade the kubelet and kubectl:

 {{< tabs name="k8s_install_kubelet" >}}
-{{< tab name="Ubuntu, Debian or HypriotOS" >}}
-    <pre>
+{{% tab name="Ubuntu, Debian or HypriotOS" %}}
    # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version
    apt-mark unhold kubelet kubectl && \
    apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \
@@ -179,15 +179,13 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no
    # since apt-get version 1.1 you can also use the following method
    apt-get update && \
    apt-get install -y --allow-change-held-packages kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00
-    </pre>
-{{< /tab >}}
-{{< tab name="CentOS, RHEL or Fedora" >}}
-    <pre>
+{{% /tab %}}
+{{% tab name="CentOS, RHEL or Fedora" %}}
    # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version
    yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
-    </pre>
-{{< /tab >}}
+{{% /tab %}}
 {{< /tabs >}}
+<br />

 -  Restart the kubelet:

@@ -268,6 +266,7 @@ without compromising the minimum required capacity for running your workloads.
    yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes
 {{% /tab %}}
 {{< /tabs >}}
+<br />

 -  Restart the kubelet:

@@ -27,8 +27,7 @@ If you are just looking for how to run a pod as a non-root user, see [SecurityCo
 * [Enable systemd with user session](https://rootlesscontaine.rs/getting-started/common/login/)
 * [Configure several sysctl values, depending on host Linux distribution](https://rootlesscontaine.rs/getting-started/common/sysctl/)
 * [Ensure that your unprivileged user is listed in `/etc/subuid` and `/etc/subgid`](https://rootlesscontaine.rs/getting-started/common/subuid/)
-
-* `KubeletInUserNamespace` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+* Enable the `KubeletInUserNamespace` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)

 <!-- steps -->

@@ -86,9 +85,10 @@ Rootless Docker/Podman or LXC/LXD, you are all set, and you can go to the next s
 Otherwise you have to create a user namespace by yourself, by calling `unshare(2)` with `CLONE_NEWUSER`.

 A user namespace can be also unshared by using command line tools such as:
+
+- [`unshare(1)`](https://man7.org/linux/man-pages/man1/unshare.1.html)
 - [RootlessKit](https://github.com/rootless-containers/rootlesskit)
 - [become-root](https://github.com/giuseppe/become-root)
- [`unshare(1)`](https://man7.org/linux/man-pages/man1/unshare.1.html)

 After unsharing the user namespace, you will also have to unshare other namespaces such as mount namespace.

@@ -123,29 +123,37 @@ On your node, systemd must already be configured to allow delegation; for more d
 Containers documentation.

 ### Configuring network
+
 {{% thirdparty-content %}}

 The network namespace of the Node components has to have a non-loopback interface, which can be for example configured with
-slirp4netns, VPNKit, or lxc-user-nic.
+[slirp4netns](https://github.com/rootless-containers/slirp4netns),
+[VPNKit](https://github.com/moby/vpnkit), or
+[lxc-user-nic(1)](https://www.man7.org/linux/man-pages/man1/lxc-user-nic.1.html).

 The network namespaces of the Pods can be configured with regular CNI plugins.
 For multi-node networking, Flannel (VXLAN, 8472/UDP) is known to work.

 Ports such as the kubelet port (10250/TCP) and `NodePort` service ports have to be exposed from the Node network namespace to
-the host with an external port forwarder, such as RootlessKit, slirp4netns, or socat.
+the host with an external port forwarder, such as RootlessKit, slirp4netns, or
+[socat(1)](https://linux.die.net/man/1/socat).

-You can use the port forwarder from K3s; see https://github.com/k3s-io/k3s/blob/v1.21.2+k3s1/pkg/rootlessports/controller.go
+You can use the port forwarder from K3s.
+See [Running K3s in Rootless Mode](https://rancher.com/docs/k3s/latest/en/advanced/#known-issues-with-rootless-mode)
+for more details.

 ### Configuring CRI

-The kubelet relies on a container runtime. You should deploy a container runtime such as containerd or CRI-O and ensure that it is running within the user namespace before the kubelet starts.
+The kubelet relies on a container runtime. You should deploy a container runtime such as
+containerd or CRI-O and ensure that it is running within the user namespace before the kubelet starts.

 {{< tabs name="cri" >}}
 {{% tab name="containerd" %}}

 Running CRI plugin of containerd in a user namespace is supported since containerd 1.4.

-Running containerd within a user namespace requires the following configuration:
+Running containerd within a user namespace requires the following configurations
+in `/etc/containerd/containerd-config.toml`.

 ```toml
 version = 2
@@ -175,7 +183,7 @@ Running CRI-O in a user namespace is supported since CRI-O 1.22.

 CRI-O requires an environment variable `_CRIO_ROOTLESS=1` to be set.

-The following configuration is also recommended:
+The following configurations (in `/etc/crio/crio.conf`) are also recommended:

 ```toml
 [crio]
@@ -197,8 +205,8 @@ The following configuration is also recommended:
 Running kubelet in a user namespace requires the following configuration:

 ```yaml
-kind: KubeletConfiguration
 apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
 featureGates:
  KubeletInUserNamespace: true
 # We use cgroupfs that is delegated by systemd, so we do not use "systemd" driver
@@ -206,22 +214,23 @@ featureGates:
 cgroupDriver: "cgroupfs"
 ```

-When the `KubeletInUserNamespace` feature gate is enabled, kubelet ignores errors that may happen during setting the following sysctl values:
+When the `KubeletInUserNamespace` feature gate is enabled, the kubelet ignores errors
+that may happen during setting the following sysctl values on the node.
+
 - `vm.overcommit_memory`
 - `vm.panic_on_oom`
 - `kernel.panic`
 - `kernel.panic_on_oops`
 - `kernel.keys.root_maxkeys`
 - `kernel.keys.root_maxbytes`.
-   (these are sysctl values for the host, not for the containers).

 Within a user namespace, the kubelet also ignores any error raised from trying to open `/dev/kmsg`.
 This feature gate also allows kube-proxy to ignore an error during setting `RLIMIT_NOFILE`.

 The `KubeletInUserNamespace` feature gate was introduced in Kubernetes v1.22 with "alpha" status.

-Running kubelet in a user namespace without using this feature gate is also possible by mounting a specially crafted proc filesystem,
-but not officially supported.
+Running kubelet in a user namespace without using this feature gate is also possible
+by mounting a specially crafted proc filesystem, but not officially supported.

 ### Configuring kube-proxy

@@ -251,9 +260,11 @@ For more on this, see the [Caveats and Future work](https://rootlesscontaine.rs/
 on the rootlesscontaine.rs website.

 ## {{% heading "seealso" %}}
+
 - [rootlesscontaine.rs](https://rootlesscontaine.rs/)
 - [Rootless Containers 2020 (KubeCon NA 2020)](https://www.slideshare.net/AkihiroSuda/kubecon-na-2020-containerd-rootless-containers-2020)
 - [Running kind with Rootless Docker](https://kind.sigs.k8s.io/docs/user/rootless/)
 - [Usernetes](https://github.com/rootless-containers/usernetes)
 - [Running K3s with rootless mode](https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental)
 - [KEP-2033: Kubelet-in-UserNS (aka Rootless mode)](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless)
+
--- a/Show More
+++ b/Show More