Merge pull request #30795 from jlbutler/dev-1.23-ref-doc

Generate single-page API reference for 1.23
2021-12-07 18:10:49 -05:00 · 2021-12-07 16:26:34 -05:00 · 2021-12-07 13:21:01 -05:00 · 2021-12-07 09:28:42 -05:00 · 2021-12-07 09:23:02 -05:00 · 2021-12-06 18:15:42 -05:00
754 changed files with 69004 additions and 12007 deletions
@@ -3,6 +3,9 @@
 # When modifying this file, consider the security implications of
 # allowing listed reviewers / approvals to modify or remove any
 # configured GitHub Actions.
+#
+options:
+  no_parent_owners: true

 reviewers:
 - sig-docs-leads
@@ -33,7 +33,7 @@ exhaustive, and do not form part of our licenses.
     material not subject to the license. This includes other CC-
     licensed material, or material used under an exception or
     limitation to copyright. More considerations for licensors:
-	wiki.creativecommons.org/Considerations_for_licensors
+    wiki.creativecommons.org/Considerations_for_licensors

     Considerations for the public: By using one of our public
     licenses, a licensor grants the public permission to use the
@@ -48,9 +48,9 @@ exhaustive, and do not form part of our licenses.
     rights in the material. A licensor may make special requests,
     such as asking that all changes be marked or described.
     Although not required by our licenses, you are encouraged to
-     respect those requests where reasonable. More_considerations
-     for the public: 
-	wiki.creativecommons.org/Considerations_for_licensees
+     respect those requests where reasonable. More considerations
+     for the public:
+    wiki.creativecommons.org/Considerations_for_licensees

 =======================================================================

@@ -378,7 +378,7 @@ Section 8 -- Interpretation.
 Creative Commons is not a party to its public
 licenses. Notwithstanding, Creative Commons may elect to apply one of
 its public licenses to material it publishes and in those instances
-will be considered the "Licensor." The text of the Creative Commons
+will be considered the “Licensor.” The text of the Creative Commons
 public licenses is dedicated to the public domain under the CC0 Public
 Domain Dedication. Except for the limited purpose of indicating that
 material is shared under a Creative Commons public license or as
@@ -393,3 +393,4 @@ the avoidance of doubt, this paragraph does not form part of the
 public licenses.

 Creative Commons may be contacted at creativecommons.org.
+
@@ -18,6 +18,7 @@ aliases:
    - annajung
    - bradtopol
    - celestehorgan
+    - divya-mohan0209
    - jimangel
    - jlbutler
    - kbhawkey
@@ -31,11 +32,14 @@ aliases:
    - bradtopol
    - celestehorgan
    - daminisatya
+    - divya-mohan0209
    - jimangel
    - kbhawkey
+    - mehabhalodiya
    - onlydole
    - rajeshdeshpande02
    - sftim
+    - shannonxtreme
    - tengqm
  sig-docs-es-owners: # Admins for Spanish content
    - raelga
@@ -79,6 +83,7 @@ aliases:
    - anubha-v-ardhan
    - divya-mohan0209
    - mittalyashu
+    - verma-kunal
  sig-docs-id-owners: # Admins for Indonesian content
    - ariscahyadi
    - danninov
@@ -133,6 +138,7 @@ aliases:
    - yoonian
    - ysyukr
  sig-docs-leads: # Website chairs and tech leads
+    - divya-mohan0209
    - jimangel
    - kbhawkey
    - onlydole
@@ -162,6 +168,7 @@ aliases:
    - xichengliudui
    # zhangxiaoyu-zidif
  sig-docs-pt-owners: # Admins for Portuguese content
+    - edsoncelio
    - femrtnz
    - jailton
    - jcjesus
@@ -170,6 +177,7 @@ aliases:
    - rikatz
    - yagonobre
  sig-docs-pt-reviews: # PR reviews for Portugese content
+    - edsoncelio
    - femrtnz
    - jailton
    - jcjesus
@@ -217,12 +225,12 @@ aliases:
  # authoritative source: git.k8s.io/community/OWNERS_ALIASES
  committee-steering: # provide PR approvals for announcements
    - cblecker
-    - derekwaynecarr
    - dims
+    - justaugustus
    - liggitt
    - mrbobbytables
-    - nikhita
    - parispittman
+    - tpepper
  # authoritative source: https://git.k8s.io/sig-release/OWNERS_ALIASES
  sig-release-leads:
    - cpanato # SIG Technical Lead
@@ -7,7 +7,7 @@

 ## डॉक्स में योगदान देना

-आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में कोई भी परिवर्तन करना चाहते हैं, और जब आप उन परिवर्तनों को हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं।
+आप अपने GitHub खाते में इस रिपॉजिटरी की एक copy बनाने के लिए स्क्रीन के ऊपरी-दाएँ क्षेत्र में **Fork** बटन पर क्लिक करें। इस copy को *Fork* कहा जाता है। अपने fork में परिवर्तन करने के बाद जब आप उनको हमारे पास भेजने के लिए तैयार हों, तो अपने fork पर जाएं और हमें इसके बारे में बताने के लिए एक नया pull request बनाएं।

 एक बार जब आपका pull request बन जाता है, तो एक कुबरनेट्स समीक्षक स्पष्ट, कार्रवाई योग्य प्रतिक्रिया प्रदान करने की जिम्मेदारी लेगा। pull request के मालिक के रूप में, **यह आपकी जिम्मेदारी है कि आप कुबरनेट्स समीक्षक द्वारा प्रदान की गई प्रतिक्रिया को संबोधित करने के लिए अपने pull request को संशोधित करें।**

@@ -23,9 +23,12 @@

 ## `README.md`'s स्थानीयकरण कुबरनेट्स प्रलेखन

-आप पर हिंदी स्थानीयकरण के maintainers तक पहुँच सकते हैं:
+आप हिंदी स्थानीयकरण के मैन्टेनरों तक पहुँच सकते हैं:

+* Anubhav Vardhan ([Slack](https://kubernetes.slack.com/archives/D0261C0A3R8), [Twitter](https://twitter.com/anubha_v_ardhan), [GitHub](https://github.com/anubha-v-ardhan))
+* Divya Mohan ([Slack](https://kubernetes.slack.com/archives/D027R7BE804), [Twitter](https://twitter.com/Divya_Mohan02), [GitHub](https://github.com/divya-mohan0209))
 * Yashu Mittal ([Twitter](https://twitter.com/mittalyashu77), [GitHub](https://github.com/mittalyashu))
+
 * [Slack channel](https://kubernetes.slack.com/messages/kubernetes-docs-hi)

 ## स्थानीय रूप से डॉकर का उपयोग करके साइट चलाना
@@ -146,7 +146,8 @@ Learn more about SIG Docs Kubernetes community and meetings on the [community pa

 You can also reach the maintainers of this project at:

- [Slack](https://kubernetes.slack.com/messages/sig-docs) [Get an invite for this Slack](https://slack.k8s.io/)
+- [Slack](https://kubernetes.slack.com/messages/sig-docs)
+  - [Get an invite for this Slack](https://slack.k8s.io/)
 - [Mailing List](https://groups.google.com/forum/#!forum/kubernetes-sig-docs)

 ## Contributing to the docs
@@ -10,5 +10,6 @@
 # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE
 # INSTRUCTIONS AT https://kubernetes.io/security/

+divya-mohan0209
 jimangel
-sftim
+sftim
@@ -810,11 +810,10 @@ section#cncf {
  }
 }

-.td-search {
-  header > .header-filler {
-    height: $hero-padding-top;
-    background-color: black;
-  }
+// Header filler size adjustment
+
+.header-hero.filler {
+  height: $hero-padding-top;
 }

 // Docs specific
@@ -859,17 +858,6 @@ section#cncf {

 /* DOCUMENTATION */

-body.td-documentation {
-    header > .header-filler {
-      height: $hero-padding-top;
-      background-color: black;
-    }
-    /* Special case for if an announcement is active */
-    header section#announcement ~ .header-filler {
-      display: none;
-    }
-}
-
 // nav-tabs and tab-content
 .nav-tabs {
  border-bottom: none !important;
@@ -26,6 +26,10 @@ $announcement-size-adjustment: 8px;
  }
 }

+.header-hero #quickstartButton.button {
+  margin-top: 1em;
+}
+
 section {
  .main-section {
    @media only screen and (min-width: 1024px) {
@@ -34,8 +38,11 @@ section {
  }
 }

-.td-outer {
-  padding: 0 !important;
+body {
+  header + .td-outer {
+    min-height: 50vh;
+    height: auto;
+  }
 }


@@ -313,37 +320,68 @@ main {

 // blockquotes and callouts

-.td-content, body {
-  blockquote.callout {
+body {
+  .alert {
+    // Override Docsy styles
    padding: 0.4rem 0.4rem 0.4rem 1rem;
-    border: 1px solid #eee;
-    border-left-width: 0.5em;
+    border-top: 1px solid #eee;
+    border-bottom: 1px solid #eee;
+    border-right: 1px solid #eee;
+    border-radius: 0.25em;
+    border-left-width: 0.5em; // fallback in case calc() is missing
    background: #fff;
    color: #000;
    margin-top: 0.5em;
    margin-bottom: 0.5em;
  }
-  blockquote.callout {
-    border-radius: calc(1em/3);
+  // Set minimum width and radius for alert color
+  .alert {
+      border-left-width: calc(max(0.5em, 4px));
+      border-top-left-radius: calc(max(0.5em, 4px));
+      border-bottom-left-radius: calc(max(0.5em, 4px));
  }
-  .callout.caution {
+  .alert.callout.caution {
    border-left-color: #f0ad4e;
  }
-
-  .callout.note {
+  .alert.callout.note {
    border-left-color: #428bca;
  }
-
-  .callout.warning {
+  .alert.callout.warning {
    border-left-color: #d9534f;
  }
+  .alert.third-party-content {
+    border-left-color: #444;
+  }

-  h1:first-of-type + blockquote.callout {
+  h1:first-of-type + .alert.callout {
    margin-top: 1.5em;
  }
 }

-.deprecation-warning {
+// Special color for third party content disclaimers
+.alert.third-party-content { border-left-color: #222 };
+
+// Highlight disclaimer when targeted as a fragment
+
+#third-party-content-disclaimer {
+  color: #000;
+  background: #f8f9fa;
+  transition: all 0.5s ease;
+}
+
+@keyframes disclaimer-highlight {
+  from { background: #f8f922; color: #000; }
+  50% { background: #f8f944; color: #000; }
+  to { background: #f8f9cb; color: #000; }
+}
+
+#third-party-content-disclaimer:target {
+  color: #000;
+  animation: disclaimer-highlight 1.25s ease;
+  background: #f8f9cb;
+}
+
+.deprecation-warning, .pageinfo.deprecation-warning {
    padding: 20px;
    margin: 20px 0;
    background-color: #faf5b6;
@@ -354,6 +392,12 @@ body.td-home .deprecation-warning, body.td-blog .deprecation-warning, body.td-do
    border-radius: 3px;
 }

+
+.td-documentation .td-content > .highlight {
+  max-width: initial;
+  width: 100%;
+}
+
 body.td-home #deprecation-warning {
    max-width: 1000px;
    margin-top: 2.5rem;
@@ -524,34 +568,6 @@ main.content {
  }
 }

-/* ANNOUNCEMENTS */
-section#fp-announcement ~ .header-hero {
-    padding: $announcement-size-adjustment  0;
-
-    > div {
-       margin-top: $announcement-size-adjustment;
-       margin-bottom: $announcement-size-adjustment;
-    }
-
-    h1, h2, h3, h4, h5 {
-      margin: $announcement-size-adjustment 0;
-    }
-}
-
-section#announcement ~ .header-hero {
-    padding: #{$announcement-size-adjustment / 2} 0;
-
-    > div {
-       margin-top: #{$announcement-size-adjustment / 2};
-       margin-bottom: #{$announcement-size-adjustment / 2};
-       padding-bottom: #{$announcement-size-adjustment / 2};
-    }
-
-    h1, h2, h3, h4, h5 {
-      margin: #{$announcement-size-adjustment / 2} 0;
-    }
-}
-
 /* DOCUMENTATION */

 /* Don't show lead text */
@@ -577,15 +593,15 @@ body.td-documentation {

@media print {
  /* Do not print announcements */
-  #announcement, section#announcement, #fp-announcement, section#fp-announcement {
+  #announcement {
    display: none;
  }
 }

-#announcement, #fp-announcement {
+#announcement {
  > * {
    color: inherit;
-    background: inherit;
+    background: transparent;
  }

  a {
@@ -599,42 +615,97 @@ body.td-documentation {
  }
 }

-#announcement {
-  padding-top: 105px;
-  padding-bottom: 25px;
-}
-
 .header-hero {
  padding-top: 40px;
 }

-/* Extra announcement height only for landscape viewports */
-@media (min-aspect-ratio: 8/9) {
-  #fp-announcement {
-    min-height: 25vh;
-  }
-}
-
-#fp-announcement aside {
-  padding-top: 115px;
-  padding-bottom: 25px;
-}
-
-.announcement {
-  .content {
+#announcement {
+  .announcement-main {
+    margin-left: auto;
+    margin-right: auto;
    margin-bottom: 0px;
+
+    // for padding-top see _size.scss
+    padding-bottom: calc(max(2em, 2rem));
+
+    max-width: calc(min(1200px - 8em, 80vw));
  }


-  > p {
-    .gridPage #announcement .content p,
-    .announcement > h4,
-    .announcement > h3 {
-      color: #ffffff;
+  /* always white */
+  h1, h2, h3, h4, h5, h6, p * {
+    color: #ffffff;
+    background: transparent;
+
+    img.event-logo {
+      display: inline-block;
+      max-height: calc(min(80px, 8em));
+      max-width: calc(min(240px, 33vw));
+      float: right;
    }
  }
 }

+#announcement + .header-hero {
+  padding-top: 2em;
+}
+
+// Extra padding for anything except wide viewports
+@media (min-width: 992px) {
+  #announcement aside { // more specific
+    .announcement-main {
+      padding-top: calc(max(8em, 8rem));
+    }
+  }
+}
+
+@media (max-width: 768px) {
+  #announcement {
+    padding-top: 4rem;
+    padding-bottom: 4rem;
+    .announcement-main, aside .announcement-main {
+      padding-top: calc(min(2rem,2em));
+    }
+  }
+}
+
+@media (max-width: 480px) {
+  #announcement {
+    padding-bottom: 0.5em;
+  }
+  #announcement aside {
+    h1, h2, h3, h4, h5, h6 {
+      img.event-logo {
+        margin-left: auto;
+        margin-right: auto;
+        margin-bottom: 0.75em;
+        display: block;
+        max-height: initial;
+        max-width: calc(min(calc(100vw - 2em), 240px));
+        float: initial;
+      }
+    }
+  }
+}
+
+#announcement + .header-hero.filler {
+  display: none;
+}
+
+@media (min-width: 768px) {
+  #announcement + .header-hero {
+    display: none;
+  }
+}
+
+
+// Match Docsy-imposed max width on text body
+@media (min-width: 1200px) {
+  body.td-blog main .td-content > figure {
+    max-width: 80%;
+  }
+}
+
 .td-content {
  table code {
    background-color: inherit !important;
@@ -642,3 +713,76 @@ body.td-documentation {
    font-size: inherit !important;
  }
 }
+
+/* Force size constraints on figures */
+figure {
+  &.diagram-small img {
+    max-height: clamp(20mm,12em,80vh);
+    margin-left: auto;
+    margin-right: auto;
+    display: block;
+  }
+  &.diagram-medium img {
+    max-height: clamp(25mm,20em,80vh);
+    margin-left: auto;
+    margin-right: auto;
+    display: block;
+  }
+  &.diagram-large img {
+    max-width: clamp(0vw, 95vw, 100%);
+    max-height: calc(80vh - 8rem);
+  }
+}
+
+@media only screen and (min-width: 768px) {
+  figure {
+    &.diagram-small, &.diagram-medium {
+      max-width: 80%;
+    }
+    &.diagram-large {
+      max-width: 100%;
+      width: 100%;
+    }
+    &.diagram-small img {
+      max-width: clamp(30rem, 45ch, 100mm);
+    }
+    &.diagram-medium img {
+      max-width: clamp(50rem, 20ch, 160mm);
+    }
+    &.diagram-large img {
+      max-width: clamp(25vw, 95vw, 100%);
+      max-height: calc(100vh - 10rem);
+    }
+  }
+}
+
+// Indent definition lists
+dl {
+  padding-left: 1.5em;
+
+  // Add vertical space before definitions
+  > *:not(dt) + dt, dt:first-child {
+    margin-top: 1.5em;
+  }
+}
+
+.release-details {
+  padding-left: 2em;
+
+  > :not(p) {
+    font-size: 1.125em;
+  }
+
+  .release-inline-heading, .release-inline-value {
+    display: inline-block
+  }
+
+  .release-inline-value {
+    padding-left: 0.25em;
+  }
+
+  p {
+     margin-top: 1em;
+     margin-bottom: 1em;
+  }
+}
@@ -18,3 +18,11 @@ section,
  line-height: $vendor-strip-height;
  font-size: $vendor-strip-font-size;
 }
+
+#announcement {
+  min-height: $hero-padding-top;
+
+  .announcement-main {
+    padding-top: calc(max(8em, 8rem, #{$hero-padding-top} / 3));
+  }
+}
@@ -8,7 +8,7 @@ options:
  substitution_option: ALLOW_LOOSE
 steps:
    # It's fine to bump the tag to a recent version, as needed
-  - name: "gcr.io/k8s-testimages/gcb-docker-gcloud:v20190906-745fed4"
+  - name: "gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20210917-12df099d55"
    entrypoint: make
    env:
      - DOCKER_CLI_EXPERIMENTAL=enabled
@@ -123,6 +123,7 @@ id = "UA-00000000-0"
 [params]
 copyright_k8s = "The Kubernetes Authors"
 copyright_linux =  "Copyright © 2020 The Linux Foundation ®."
+
 # privacy_policy = "https://policies.google.com/privacy"

 # First one is picked as the Twitter card image if not set on page.
@@ -138,10 +139,10 @@ time_format_default = "January 02, 2006 at 3:04 PM PST"
 description = "Production-Grade Container Orchestration"
 showedit = true

-latest = "v1.22"
+latest = "v1.23"

-fullversion = "v1.22.0"
-version = "v1.22"
+fullversion = "v1.23.0"
+version = "v1.23"
 githubbranch = "main"
 docsbranch = "main"
 deprecated = false
@@ -178,40 +179,40 @@ js = [
 ]

 [[params.versions]]
-fullversion = "v1.22.0"
-version = "v1.22"
-githubbranch = "v1.22.0"
+fullversion = "v1.23.0"
+version = "v1.23"
+githubbranch = "v1.23.0"
 docsbranch = "main"
 url = "https://kubernetes.io"

 [[params.versions]]
-fullversion = "v1.21.4"
+fullversion = "v1.22.4"
+version = "v1.22"
+githubbranch = "v1.22.4"
+docsbranch = "release-1.22"
+url = "https://v1-22.docs.kubernetes.io"
+
+[[params.versions]]
+fullversion = "v1.21.7"
 version = "v1.21"
-githubbranch = "v1.21.4"
+githubbranch = "v1.21.7"
 docsbranch = "release-1.21"
 url = "https://v1-21.docs.kubernetes.io"

 [[params.versions]]
-fullversion = "v1.20.10"
+fullversion = "v1.20.13"
 version = "v1.20"
-githubbranch = "v1.20.10"
+githubbranch = "v1.20.13"
 docsbranch = "release-1.20"
 url = "https://v1-20.docs.kubernetes.io"

 [[params.versions]]
-fullversion = "v1.19.14"
+fullversion = "v1.19.16"
 version = "v1.19"
-githubbranch = "v1.19.14"
+githubbranch = "v1.19.16"
 docsbranch = "release-1.19"
 url = "https://v1-19.docs.kubernetes.io"

-[[params.versions]]
-fullversion = "v1.18.20"
-version = "v1.18"
-githubbranch = "v1.18.20"
-docsbranch = "release-1.18"
-url = "https://v1-18.docs.kubernetes.io"
-
 # User interface configuration
 [params.ui]
 # Enable to show the side bar menu in its compact state.
@@ -42,12 +42,12 @@ Kubernetes ist Open Source und bietet Dir die Freiheit, die Infrastruktur vor Or
        <button id="desktopShowVideoButton" onclick="kub.showVideo()">Video ansehen</button>
        <br>
        <br>
-        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu20" button id="desktopKCButton">Besuche die KubeCon - 13-16 August 2020 in Amsterdam</a>
+        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21" button id="desktopKCButton">Besuche die KubeCon North America vom 11. bis 15. Oktober 2021</a>
        <br>
        <br>
        <br>
        <br>
-        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna20" button id="desktopKCButton">Besuche die KubeCon - 17-20 November 2020 in Boston</a>
+        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22" button id="desktopKCButton">Besuche die KubeCon Europe vom 17. bis 20. Mai 2022</a>
 </div>
 <div id="videoPlayer">
    <iframe data-url="https://www.youtube.com/embed/H06qrNmGqyE?autoplay=1" frameborder="0" allowfullscreen></iframe>
@@ -4,58 +4,253 @@ layout: basic
 cid: community
 ---

-<section id="mainContent">
-    <main>
-        <div class="content">
-            <h3>Die Gewissheit, dass Kubernetes überall und für alle gut funktioniert.</h3>
-            <p>Verbinden Sie sich mit der Kubernetes-Community in unserem <a href="http://slack.k8s.io/">Slack Kanal</a>, <a href="https://discuss.kubernetes.io/">Diskussionsforum</a>, oder beteiligen Sie sich an der <a href="https://groups.google.com/g/kubernetes-dev"> Kubernetes-dev-Google-Gruppe</a>. Eine wöchentliches Community-Meeting findet per Videokonferenz statt, um den Stand der Dinge zu diskutieren, folgen Sie
-		<a href="https://github.com/kubernetes/community/blob/master/events/community-meeting.md">diesen Anweisungen</a> für Informationen wie Sie teilnehmen können.</p>
-            <p>Sie können Kubernetes auch auf der ganzen Welt über unsere
-                <a href="https://www.meetup.com/topics/kubernetes/">Kubernetes Meetup Community</a> und der
-                <a href="https://www.meetup.com/Kubernetes-Cloud-Native-Online-Meetup/">Kubernetes Cloud Native Meetup Community</a> beitreten.</p>
-          </div>
-        <div class="content">
-            <h3>Special Interest Groups (SIGs)</h3>
-            <p>Haben Sie ein besonderes Interesse daran, wie Kubernetes mit einer anderen Technologie arbeitet?  Werfen Sie einen Blick auf unsere kontinuierlich wachsende
-                <a href="https://git.k8s.io/community/sig-list.md">Listen von SIGs</a>, von AWS und Openstack bis hin zu Big Data und Skalierbarkeit, es gibt einen Platz für Sie, an dem Sie mitwirken können, und Anweisungen zur Gründung einer neuen SIG finden, wenn Ihr besonderes Interesse (noch) nicht abgedeckt ist.
-            </p>
+<div class="newcommunitywrapper">
+<div class="banner1">
+  <img src="/images/community/kubernetes-community-final-02.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;padding-left:0px" class="desktop">
+  <img src="/images/community/kubernetes-community-02-mobile.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;padding-left:0px" class="mobile">
+</div>

-            <p>Als Mitglied der Kubernetes-Community sind Sie herzlich eingeladen, an allen SIG-Treffen teilzunehmen, die Sie interessieren. Eine Registrierung ist nicht erforderlich.</p>
+<div class="intro">
+<br class="mobile">
+<p>Die Kubernetes-Community - Nutzer, Mitwirkende und die Kultur, die wir gemeinsam aufgebaut haben - ist einer der Hauptgründe für den kometenhaften Aufstieg dieses Open-Source-Projekts. Unsere Kultur und unsere Werte wachsen und entwickeln sich mit dem Wachstum und der Veränderung des Projekts selbst. Wir alle arbeiten gemeinsam an der ständigen Verbesserung des Projekts und der Art und Weise, wie wir daran arbeiten.
+<br><br>Wir sind die Leute, die Probleme und Pull-Requests einreichen, an SIG-Treffen (Special Interest Groups), Kubernetes-Treffen und der KubeCon teilnehmen, sich für die Einführung und Innovation von Kubernetes einsetzen, <code>kubectl get pods</code> ausführen und auf tausend andere wichtige Arten beitragen. Lies weiter, um zu erfahren, wie Du dich engagieren und Teil dieser faszinierenden Gemeinschaft werden kannst.</p>
+<br class="mobile">
+</div>

-        </div>
+<div class="community__navbar">

-        <div class="content">
-          <h3>Verhaltensregeln</h3>
-          <p>Die Kubernetes-Community schätzt Respekt und Inklusivität und setzt einen <a href="code-of-conduct/">Verhaltenskodex</a>
-            in allen Interaktionen durch. Wenn Sie einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder Sitzung,
-            in Slack oder in einem anderen Kommunikationsmechanismus feststellen, wenden Sie sich
-            bitte an das <a href="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct">Kubernetes Code of Conduct Committee</a> <a href="mailto:conduct@kubernetes.io">conduct@kubernetes.io</a>. Ihre Anonymität wird geschützt.
-          </p>
-         </div>
-    </main>
-</section>
+<a href="#values">Gemeinschaftswerte</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#conduct">Verhaltenskodex </a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#videos">Videos</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#discuss">Diskussionen</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#events">Veranstaltungen und meetups</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="#news">Neuigkeiten</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
+<a href="/releases">Releases</a>

-<section id="talkToUs">
-    <main>
-        <h3>Talk to Us!</h3>
-        <h4>Wir würden uns freuen, von Ihnen zu hören, wie Sie Kubernetes verwenden<br>und was wir tun können, um es besser zu machen.</h4>
-        <div id="bigSocial">
-            <div>
-                <a href="https://twitter.com/kubernetesio">@kubernetesio</a>
-                <p>Erhalten Sie die neuesten Nachrichten und Updates.</p>
-            </div>
-            <div>
-                <a href="https://github.com/kubernetes/kubernetes">Github Project</a>
-                <p>Informieren Sie sich über das Projekt und erwägen Sie, einen Beitrag zu leisten.</p>
-            </div>
-            <div>
-                <a href="http://slack.k8s.io/">#kubernetes-users</a>
-                <p>Unser Slack-Kanal ist der beste Weg, um unsere Ingenieure zu kontaktieren und Ihre Ideen mit ihnen zu teilen.</p>
-            </div>
-            <div>
-                <a href="http://stackoverflow.com/questions/tagged/kubernetes">Stack Overflow</a>
-                <p>Unser Benutzerforum ist ein großartiger Ort, um Community-Support zu erhalten.</p>
-            </div>
-        </div>
-    </main>
-</section>
+</div>
+<br class="mobile"><br class="mobile">
+<div class="imagecols">
+<br class="mobile">
+  <div class="imagecol">
+    <img src="/images/community/kubernetes-community-final-03.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%" class="desktop">
+  </div>
+
+  <div class="imagecol">
+    <img src="/images/community/kubernetes-community-final-04.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%" class="desktop">
+  </div>
+
+  <div class="imagecol" style="margin-right:0% important">
+    <img src="/images/community/kubernetes-community-final-05.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;margin-right:0% important" class="desktop">
+  </div>
+  <img src="/images/community/kubernetes-community-04-mobile.jpg" alt="Kubernetes-Konferenz Galerie" style="width:100%;margin-bottom:3%" class="mobile">
+<a name="values"></a>
+</div>
+
+<div><a name="values"></a></div>
+<div class="conduct">
+<div class="conducttext">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<div class="conducttextnobutton" style="margin-bottom:2%"><h1>Gemeinschaftswerte</h1>
+Die Werte der Kubernetes-Community sind der Grundstein für den anhaltenden Erfolg des Projekts.<br>
+Diese Prinzipien leiten jeden Aspekt des Kubernetes-Projekts.
+<br>
+<a href="/community/values/">
+<br class="mobile"><br class="mobile">
+<span class="fullbutton">
+  MEHR ERFAHREN
+</span>
+</a>
+</div><a name="conduct"></a>
+</div>
+</div>
+
+
+
+<div class="conduct">
+<div class="conducttext">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<div class="conducttextnobutton" style="margin-bottom:2%"><h1>Verhaltenskodex</h1>
+Die Kubernetes-Gemeinschaft legt Wert auf Respekt und Inklusivität und setzt bei allen Interaktionen einen Verhaltenskodex durch. Wenn Du einen Verstoß gegen den Verhaltenskodex bei einer Veranstaltung oder einem Treffen, in Slack oder in einem anderen Kommunikationsmechanismus bemerkst, wende dich an das Kubernetes Code of Conduct Committee unter <a href="mailto:conduct@kubernetes.io" style="color:#0662EE;font-weight:300">conduct@kubernetes.io</a>. Alle Berichte werden vertraulich behandelt. Du kannst&nbsp;<a href="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct" style="color:#0662EE;font-weight:300">hier</a> mehr über den Ausschuss erfahren.
+<br>
+<a href="https://kubernetes.io/de/community/code-of-conduct/">
+<br class="mobile"><br class="mobile">
+
+<span class="fullbutton">
+MEHR ERFAHREN
+</span>
+</a>
+</div><a name="videos"></a>
+</div>
+</div>
+
+
+
+<div class="videos">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<h1 style="margin-top:0px">Videos</h1>
+
+<div style="margin-bottom:4%;font-weight:300;text-align:center;padding-left:10%;padding-right:10%">Wir sind auf YouTube, und zwar oft. Abonniere uns für eine Vielzahl von&nbsp;Themen.</div>
+
+<div class="videocontainer">
+
+<div class="video">
+
+  <iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg" title="Monatliche Bürozeiten" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
+
+<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3azFUvYJjGn45YbF6C-uIg">
+<div class="videocta">
+Monatliche Bürozeiten ansehen&nbsp;&#9654;</div>
+</a>
+</div>
+
+<div class="video">
+  <iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ" title="Wöchentliche Treffen der Gemeinschaft" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
+<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ">
+<div class="videocta">
+Wöchentliche Treffen der Gemeinschaft ansehen&nbsp;&#9654;
+</div>
+</a>
+</div>
+
+<div class="video">
+
+<iframe width="100%" height="250" src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F" title="Vortrag eines Mitglieds der Gemeinschaft" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
+
+<a href="https://www.youtube.com/playlist?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F">
+<div class="videocta">
+Vortrag eines Mitglieds der Gemeinschaft ansehen&nbsp;&#9654;
+</div>
+
+</a>
+<a name="discuss"></a>
+</div>
+</div>
+</div>
+
+
+<div class="resources">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<h1 style="padding-top:1%">Diskussionen</h1>
+
+<div style="font-weight:300;text-align:center">Wir reden gerne und viel. Triff uns auf einer dieser Plattformen und beteilige dich an den Diskussionen.</div>
+
+<div class="resourcecontainer">
+
+<div class="resourcebox">
+<img src="/images/community/discuss.png" alt=Forum" style="width:80%;padding-bottom:2%">
+<a href="https://discuss.kubernetes.io/" style="color:#0662EE;display:block;margin-top:1%">
+forum&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">
+Themenbezogene technische Diskussionen, die eine Brücke zu Docs, StackOverflow und vielem mehr schlagen.
+</div>
+</div>
+
+<div class="resourcebox">
+<img src="/images/community/twitter.png" alt="Twitter" style="width:80%;padding-bottom:2%">
+<a href="https://twitter.com/kubernetesio" style="color:#0662EE;display:block;margin-top:1%">
+twitter&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">Echtzeit-Ankündigungen von Blogeinträgen, Veranstaltungen, Neuigkeiten und Ideen
+</div>
+</div>
+
+<div class="resourcebox">
+<img src="/images/community/github.png" alt="GitHub" style="width:80%;padding-bottom:2%">
+<a href="https://github.com/kubernetes/kubernetes" style="color:#0662EE;display:block;margin-top:1%">
+github&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">
+Die gesamte Projekt- und Problemverfolgung und natürlich der Code
+</div>
+</div>
+
+<div class="resourcebox">
+<img src="/images/community/stack.png" alt="Stack Overflow" style="width:80%;padding-bottom:2%">
+<a href="https://stackoverflow.com/search?q=kubernetes" style="color:#0662EE;display:block;margin-top:1%">
+stack overflow&nbsp;&#9654;
+</a>
+<div class="resourceboxtext" style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">
+  Technische Problemlösung für jeden Anwendungsfall
+  <a name="events"></a>
+</div>
+</div>
+
+<!--
+<div class="resourcebox">
+
+<img src="/images/community/slack.png" style="width:80%">
+
+slack&nbsp;&#9654;
+
+<div class="resourceboxtext" style="font-size:11px;text-transform:none !important;font-weight:200;line-height:1.4em;color:#333333;margin-top:4%">
+With 170+ channels, you'll find one that fits your needs.
+</div>
+
+</div>-->
+
+</div>
+</div>
+<div class="events">
+  <br class="mobile"><br class="mobile">
+  <br class="tablet"><br class="tablet">
+<div class="eventcontainer">
+ <h1 style="color:white !important">Bevorstehende Veranstaltungen</h1>
+    {{< upcoming-events >}}
+</div>
+</div>
+
+<div class="meetups">
+<div class="meetupcol">
+<div class="meetuptext">
+<h1 style="text-align:left">Globale Gemeinschaft</h1>
+Mit mehr als 150 Treffen auf der ganzen Welt, Tendenz steigend, solltest du deine lokalen Kube-Leute finden. Wenn keins in der Nähe ist, nimm die Sache in die Hand und gründe dein eigenes.
+</div>
+<a href="https://www.meetup.com/topics/kubernetes/">
+<div class="button">
+EIN MEETUP FINDEN
+</div>
+</a>
+<a name="news"></a>
+</div>
+</div>
+
+
+<!--
+<div class="contributor">
+<div class="contributortext">
+<br>
+<h1 style="text-align:left">
+New Contributors Site
+ </h1>
+Text about new contributors site.
+
+<br><br>
+
+<div class="button">
+VISIT SITE
+</div>
+</div>
+</div>
+
+-->
+
+<div class="news">
+<br class="mobile"><br class="mobile">
+<br class="tablet"><br class="tablet">
+<h1 style="margin-bottom:2%">Aktuelle Neuigkeiten</h1>
+
+<br>
+<div class="twittercol1">
+<a class="twitter-timeline" data-tweet-limit="1" href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">Tweets von kubernetesio</a> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
+</div>
+
+<br>
+<br><br><br><br>
+</div>
+
+</div>
@@ -50,6 +50,8 @@ Bevor Sie die einzelnen Lernprogramme durchgehen, möchten Sie möglicherweise e

 * [AppArmor](/docs/tutorials/clusters/apparmor/)

+* [seccomp](/docs/tutorials/clusters/seccomp/)
+
 ## Services

 * [Source IP verwenden](/docs/tutorials/services/source-ip/)
@@ -0,0 +1,201 @@
+/*  SECTIONS  */
+.section {
+	clear: both;
+	padding: 0px;
+	margin-bottom: 2em;
+}
+
+.kcsp_section {
+	clear: both;
+	padding: 0px;
+	margin-bottom: 2em;
+}
+
+/*  COLUMN SETUP  */
+.col {
+	display: block;
+	float:left;
+	margin: 1% 0 1% 1.6%;
+	background-color: #f9f9f9;
+}
+.col:first-child { margin-left: 0; }
+
+
+/*  GROUPING  */
+.group:before,
+.group:after {
+	content:"";
+	display:table;
+}
+.group:after {
+	clear:both;
+}
+.group {
+    zoom:1; /* For IE 6/7 */
+}
+
+/*  GRID OF THREE  */
+.span_3_of_3 {
+	width: 35%;
+	background-color: #f9f9f9;
+	padding: 20px;
+}
+.span_2_of_3 {
+	width: 35%;
+	background-color: #f9f9f9;
+	padding: 20px;
+}
+.span_1_of_3 {
+	width: 35%;
+	background-color: #f9f9f9;
+	padding: 20px;
+}
+
+.col-container {
+    display: table; /* Make the container element behave like a table */
+    width: 100%; /* Set full-width to expand the whole page */
+		padding-bottom: 30px;
+}
+
+.col-nav {
+    display: table-cell; /* Make elements inside the container behave like table cells */
+		width: 18%;
+		background-color: #f9f9f9;
+		padding: 20px;
+		border: 5px solid white;
+}
+
+/*  GO FULL WIDTH AT LESS THAN 480 PIXELS */
+
+@media only screen and (max-width: 480px) {
+	.col { margin: 1% 0 1% 0%;}
+	.span_3_of_3, .span_2_of_3, .span_1_of_3 { width: 100%; }
+}
+
+@media only screen and (max-width: 650px) {
+    .col-nav {
+        display: block;
+        width: 100%;
+    }
+}
+
+.button{
+  max-width: 100%;
+	box-sizing: border-box;
+	margin: 0;
+	display: inline-block;
+	border-radius: 6px;
+	padding: 0 20px;
+	line-height: 40px;
+	color: #ffffff;
+	font-size: 16px;
+	background-color: #3371e3;
+	text-decoration: none;
+ }
+
+h5 {
+	font-size: 16px;
+	line-height: 1.5em;
+	margin-bottom: 2em;
+}
+
+#usersGrid a {
+	display: inline-block;
+	background-color: #f9f9f9;
+}
+
+#ktpContainer, #distContainer, #kcspContainer, #isvContainer, #servContainer {
+	position: relative;
+	width: 100%;
+	display: flex;
+	justify-content: space-between;
+	flex-wrap: wrap;
+}
+
+#isvContainer {
+	margin-bottom: 80px;
+}
+
+#kcspContainer {
+	margin-bottom: 80px;
+}
+
+#distContainer {
+	margin-bottom: 80px;
+}
+
+#ktpContainer {
+	margin-bottom: 80px;
+}
+
+.partner-box {
+	position: relative;
+	width: 47%;
+	max-width: 48%;
+	min-width: 48%;
+	margin-bottom: 20px;
+	padding: 20px;
+	flex: 1;
+	display: flex;
+	justify-content: left;
+	align-items: flex-start;
+}
+
+.partner-box img {
+	background-color: #f9f9f9;
+}
+
+.partner-box > div {
+	margin-left: 30px;
+}
+
+.partner-box a {
+	color: #3576E3;
+}
+
+@media screen and (max-width: 1024px) {
+	.partner-box {
+		flex-direction: column;
+		justify-content: flex-start;
+	}
+
+	.partner-box > div {
+		margin: 20px 0 0;
+	}
+}
+
+@media screen and (max-width: 568px) {
+	#ktpContainer, #distContainter, #kcspContainer, #isvContainer, #servContainer {
+		justify-content: center;
+	}
+
+	.partner-box {
+		flex-direction: column;
+		justify-content: flex-start;
+		width: 100%;
+		max-width: 100%;
+		min-width: 100%;
+	}
+
+	.partner-box > div {
+		margin: 20px 0 0;
+	}
+}
+
+@media screen and (max-width: 568px) {
+	#ktpContainer, #distContainer, #kcspContainer, #isvContainer, #servContainer {
+		justify-content: center;
+	}
+
+	.partner-box {
+		flex-direction: column;
+		justify-content: flex-start;
+		width: 100%;
+		max-width: 100%;
+		min-width: 100%;
+	}
+
+	.partner-box > div {
+		margin: 20px 0 0;
+	}
+}
@@ -1,91 +1,53 @@
 ---
 title: Partner
 bigheader: Kubernetes Partner
-abstract: Entwicklung des Kubernetes-Ökosystems.
+abstract: Erweiterung des Kubernetes-Ökosystems.
 class: gridPage
 cid: partners
 ---

 <section id="users">
-    <main>
-        <h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, dynamische Codebasis zu schaffen, die ein Spektrum von aufeinander abgestimmten Plattformen unterstützt.</h5>
-        <div class="col-container">
-          <div class="col-nav">
-            <center>
-              <h5>
-                <b>Kubernetes zertifizierte Service Provider</b>
-              </h5>
-              <br>Geprüfte Service Provider mit großer Erfahrung, die Unternehmen bei der erfolgreichen Einführung von Kubernetes unterstützen.
-              <br><br><br>
-              <button id="kcsp" class="button" onClick="updateSrc(this.id)">KCSP-Partner anzeigen</button>
-              <br><br>Interessiert daran, ein <a href="https://www.cncf.io/certification/kcsp/">KCSP</a> zu werden?
-            </center>
-          </div>
-          <div class="col-nav">
-            <center>
-              <h5>
-                <b>Kubernetes-Distributionen, gehostete Plattformen und zertifizierte Installateure</b>
-              </h5>Software-Konformität stellt sicher, dass die Kubernetes-Versionen aller Hersteller die erforderlichen APIs unterstützen.
-              <br><br><br>
-              <button id="conformance" class="button" onClick="updateSrc(this.id)">Zertifizierte Partner anzeigen</button>
-              <br><br>Interessiert daran, <a href="https://www.cncf.io/certification/software-conformance/">Kubernetes zertifiziert</a> zu werden?
-            </center>
-          </div>
-          <div class="col-nav">
-            <center>
-              <h5><b>Kubernetes Training Partner</b></h5>
-              <br>Geprüfte Schulungsanbieter, die über umfassende Erfahrung in Cloud Native Technologietrainings verfügen.
-              <br><br><br><br>
-              <button id="ktp" class="button" onClick="updateSrc(this.id)">KTP Partner anzeigen</button>
-              <br><br>Interessiert daran, ein <a href="https://www.cncf.io/certification/training/">KTP</a> zu werden?
-            </center>
-          </div>
-        </div>
-<script src="https://code.jquery.com/jquery-3.3.1.min.js" integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
-<script type="text/javascript">
-
-			var defaultLink = "https://landscape.cncf.io/category=kubernetes-certified-service-provider&format=card-mode&grouping=category&embed=yes";
-			var firstLink = "https://landscape.cncf.io/category=certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer&format=card-mode&grouping=category&embed=yes";
-      var secondLink = "https://landscape.cncf.io/category=kubernetes-training-partner&format=card-mode&grouping=category&embed=yes";
-
-      function updateSrc(buttonId) {
-      	if (buttonId == "kcsp") {
-        		$("#landscape").attr("src",defaultLink);
-            window.location.hash = "#kcsp";
-        }
-        if (buttonId == "conformance") {
-         		$("#landscape").attr("src",firstLink);
-            window.location.hash = "#conformance";
-        }
-        if (buttonId == "ktp") {
-        		$("#landscape").attr("src",secondLink);
-            window.location.hash = "#ktp";
-        }
-      }
-
-      // Automatically load the correct iframe based on the URL fragment
-      document.addEventListener('DOMContentLoaded', function() {
-        var showContent = "kcsp";
-        if (window.location.hash) {
-          console.log('hash is:', window.location.hash.substring(1));
-          showContent = window.location.hash.substring(1);
-        }
-        updateSrc(showContent);
-      });
-</script>
-<body>
-    <div id="frameHolder">
-      <iframe id="landscape" frameBorder="0" scrolling="no" style="width: 1px; min-width: 100%" src=""></iframe>
-      <script src="https://landscape.cncf.io/iframeResizer.js"></script>
+    <h5>Kubernetes arbeitet mit Partnern zusammen, um eine starke, lebendige Codebasis zu schaffen, die ein Spektrum von ergänzenden Plattformen unterstützt.</h5>
+    <div class="col-container">
+      <div class="col-nav">
+        <center>
+          <h5>
+            <b>Kubernetes-zertifizierte Service-Anbieter</b>
+          </h5>
+          <br>Geprüfte Dienstleister mit umfassender Erfahrung bei der erfolgreichen Einführung von Kubernetes in Unternehmen.
+          <br><br><br>
+          <button class="button landscape-trigger landscape-default" data-landscape-types="kubernetes-certified-service-provider" id="kcsp">KCSP Partner anzeigen</button>
+          <br><br>Interessiert daran, ein
+          <a href="https://www.cncf.io/certification/kcsp/">KCSP</a> zu werden?
+        </center>
+      </div>
+      <div class="col-nav">
+        <center>
+          <h5>
+            <b>Zertifizierte Kubernetes-Distributionen, gehostete Plattformen und Installationssysteme</b>
+          </h5>Die Softwarekonformität stellt sicher, dass die Kubernetes-Version eines jeden Anbieters die erforderlichen APIs unterstützt.
+          <br><br><br>
+          <button class="button landscape-trigger" data-landscape-types="certified-kubernetes-distribution,certified-kubernetes-hosted,certified-kubernetes-installer" id="conformance">Konforme Partner anzeigen</button>
+          <br><br>Interessiert daran,
+          <a href="https://www.cncf.io/certification/software-conformance/">Kubernetes Zertifiziert</a> zu werden?
+        </center>
+      </div>
+      <div class="col-nav">
+        <center>
+          <h5>
+            <b>Kubernetes Schulungspartner</b>
+          </h5>
+          <br>Geprüfte Schulungsanbieter mit umfassender Erfahrung in der Weiterbildung im Bereich Cloud Native Technology.
+          <br><br><br>
+          <button class="button landscape-trigger" data-landscape-types="kubernetes-training-partner" id="ktp">KTP Partner anzeigen</button>
+          <br><br>Interessiert daran, ein
+          <a href="https://www.cncf.io/certification/training/">KTP</a> zu werden?
+        </center>
+      </div>
    </div>
-</body>
-    </main>
+    {{< cncf-landscape helpers=true >}}
 </section>

 <style>
    {{< include "partner-style.css" >}}
-</style>
-
-<script>
-    {{< include "partner-script.js" >}}
-</script>
+</style>
@@ -0,0 +1,137 @@
+---
+title: Schulungen
+bigheader: Kubernetes Schulungen und Zertifizierungen
+abstract: Schulungsprogramme, Zertifizierungen und Partner.
+layout: basic
+cid: training
+class: training
+---
+
+<section class="call-to-action">
+  <div class="main-section">
+    <div class="call-to-action" id="cta-certification">
+      <div class="cta-text">
+        <h2>Gestalte deine Cloud Native Karriere</h2>
+        <p>Kubernetes ist das Herzstück der Cloud Native-Bewegung. Mit den Schulungen und Zertifizierungen der Linux Foundation und unserer Schulungspartner kannst Du in deine Karriere investieren, Kubernetes lernen und deine Cloud Native-Projekte zum Erfolg führen.</p>
+      </div>
+      <div class="logo-certification cta-image" id="logo-kcnf">
+        <img src="/images/training/kubernetes-kcnf-white.svg" />
+      </div>
+      <div class="logo-certification cta-image" id="logo-cka">
+        <img src="/images/training/kubernetes-cka-white.svg"/>
+      </div>
+      <div class="logo-certification cta-image" id="logo-ckad">
+        <img src="/images/training/kubernetes-ckad-white.svg"/>
+      </div>
+      <div class="logo-certification cta-image" id="logo-cks">
+        <img src="/images/training/kubernetes-cks-white.svg"/>
+      </div>
+    </div>
+  </div>
+</section>
+
+<section>
+  <div class="main-section padded">
+    <center>
+      <h2>Nimm an einen kostenlosen Kurs bei edX teil</h2>
+    </center>
+  <div class="col-container">
+    <div class="col-nav">
+      <center>
+        <h5>
+          <b>Einf&uuml;hrung in Kubernetes <br> &nbsp;</b>
+        </h5>
+        <p>M&ouml;chtest Du Kubernetes lernen? Erfahre alles über dieses leistungsstarke System zur Verwaltung von Containeranwendungen.</p>
+        <br>
+        <a href="https://www.edx.org/course/introduction-to-kubernetes" target="_blank" class="button">Zum Kurs</a>
+      </center>
+    </div>
+    <div class="col-nav">
+      <center>
+        <h5>
+          <b>Einführung in Cloud-Infrastruktur Technologien</b>
+        </h5>
+        <p>Lerne die Grundlagen für den Aufbau und die Verwaltung von Cloud-Technologien direkt von der Linux Foundation, dem Marktführer im Bereich Open Source.</p>
+        <br>
+        <a href="https://www.edx.org/course/introduction-to-cloud-infrastructure-technologies" target="_blank" class="button">Zum Kurs</a>
+      </center>
+    </div>
+    <div class="col-nav">
+      <center>
+        <h5>
+          <b>Einf&uuml;hrung in Linux</b>
+        </h5>
+        <p>Du hast nie Linux gelernt? Willst du eine Auffrischung? Erarbeite dir gute Linux-Kenntnisse über die grafische Oberfläche und die Kommandozeile der wichtigsten Linux-Distributionen.</p>
+        <br>
+        <a href="https://www.edx.org/course/introduction-to-linux" target="_blank" class="button">Zum Kurs</a>
+      </center>
+    </div>
+  </div>
+</section>
+
+<div class="padded lighter-gray-bg">
+  <div class="main-section two-thirds-centered">
+    <center>
+      <h2>Mit der Linux Foundation lernen</h2>
+      <p>Die Linux Foundation bietet Kurse für alle Aspekte der Entwicklung und des Betriebs von Kubernetes-Anwendungen an, die entweder von Lehrkräften geleitet werden oder zum Selbststudium geeignet sind.</p>
+      <br/><br/>
+      <a href="https://training.linuxfoundation.org/training/course-catalog/?_sft_technology=kubernetes" target="_blank" class="button">Kurse anzeigen</a>
+    </center>
+  </div>
+</div>
+
+<section id="get-certified">
+  <div class="main-section padded">
+    <h2>Werde Kubernetes zertifiziert</h2>
+    <div class="col-container">
+      <div class="col-nav">
+        <h5>
+          <b>Kubernetes and Cloud Native Associate (KCNA)</b>
+        </h5>
+        <p>Die Prüfung zum Kubernetes and Cloud Native Associate (KCNA) weist die grundlegenden Kenntnisse und Fähigkeiten eines Benutzers in Kubernetes und dem breiteren Cloud Native-Ökosystem nach.</p>
+        <p>Ein zertifizierter KCNA bestätigt konzeptionelles Wissen über das gesamte Cloud Native Ecosystem, mit besonderem Fokus auf Kubernetes.</p>
+        <br>
+        <a href="https://training.linuxfoundation.org/certification/kubernetes-cloud-native-associate/" target="_blank" class="button">Zur Zertifizierung</a>
+      </div>
+      <div class="col-nav">
+          <h5>
+            <b>Certified Kubernetes Application Developer (CKAD)</b>
+          </h5>
+          <p>Die Prüfung zum Certified Kubernetes Application Developer (Zertifizierter Kubernetes-Anwendungsentwickler) bescheinigt, dass Teilnehmer Cloud Native-Anwendungen für Kubernetes entwerfen, erstellen, konfigurieren und bereitstellen können.</p>
+          <p>Ein CKAD kann Anwendungsressourcen definieren und zentrale Elemente verwenden, um skalierbare Anwendungen und Tools in Kubernetes zu erstellen, zu überwachen und Fehler zu beheben.</p>
+          <br>
+          <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-application-developer-ckad/" target="_blank" class="button">Zur Zertifizierung</a>
+      </div>
+      <div class="col-nav">
+          <h5>
+            <b>Certified Kubernetes Administrator (CKA)</b>
+          </h5>
+          <p>Das Certified Kubernetes Administrator (CKA)-Programm garantiert, dass CKAs die Fähigkeiten, das Wissen und die Kompetenz besitzen, um die Aufgaben eines Kubernetes-Administrators zu erfüllen.</p>
+          <p>Ein zertifizierter Kubernetes-Administrator hat nachgewiesen, dass er in der Lage ist, grundlegende Installationen durchzuführen sowie Kubernetes-Cluster in einer Produktionsumgebung zu konfigurieren und zu verwalten.</p>
+          <br>
+          <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-administrator-cka/" target="_blank" class="button">Zur Zertifizierung</a>
+      </div>
+      <div class="col-nav">
+          <h5>
+            <b>Certified Kubernetes Security Specialist (CKS)</b>
+          </h5>
+          <p>Das Programm Certified Kubernetes Security Specialist (CKS) bietet die Gewissheit, dass der Zertifikatsinhaber mit einem breiten Spektrum an Best Practices vertraut ist und diese beherrscht. Die CKS-Zertifizierung umfasst Fähigkeiten zur Sicherung von Container-basierten Anwendungen und Kubernetes-Plattformen während der Erstellung, Bereitstellung und Laufzeit.</p>
+          <p><em>Kandidaten für den CKS müssen über eine aktuelle Zertifizierung als Certified Kubernetes Administrator (CKA) verfügen, um nachzuweisen, dass sie über ausreichende Kubernetes-Kenntnisse verfügen, bevor sie sich für den CKS anmelden.</em></p>
+          <br>
+          <a href="https://training.linuxfoundation.org/certification/certified-kubernetes-security-specialist/" target="_blank" class="button">Zur Zertifizierung</a>
+      </div>
+    </div>
+  </div>
+</section>
+
+<div class="padded lighter-gray-bg">
+  <div class="main-section two-thirds-centered">
+    <center>
+      <h2>Kubernetes Schulungspartner</h2>
+      <p>Unser Netzwerk von Kubernetes-Schulungspartnern bietet Schulungsangebote für Kubernetes- und Cloud Native-Projekte.</p>
+    </center>
+  </div>
+  <div class="main-section landscape-section">
+    {{< cncf-landscape helpers=false category="kubernetes-training-partner" >}}
+  </div>
+</div>
@@ -43,12 +43,12 @@ Kubernetes is open source giving you the freedom to take advantage of on-premise
        <button id="desktopShowVideoButton" onclick="kub.showVideo()">Watch Video</button>
        <br>
        <br>
-        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21" button id="desktopKCButton">Attend KubeCon North America on October 11-15, 2021</a>
-        <br>
-        <br>
-        <br>
-        <br>
        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22" button id="desktopKCButton">Attend KubeCon Europe on May 17-20, 2022</a>
+        <br>
+        <br>
+        <br>
+        <br>
+        <a href="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21" button id="desktopKCButton">Attend KubeCon North America on October 24-28, 2022</a>
 </div>
 <div id="videoPlayer">
    <iframe data-url="https://www.youtube.com/embed/H06qrNmGqyE?autoplay=1" frameborder="0" allowfullscreen></iframe>
@@ -4,6 +4,8 @@ title:  'Health checking gRPC servers on Kubernetes'
 date: 2018-10-01
 ---

+_Built-in gRPC probes were introduced in Kubernetes 1.23. To learn more, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)._
+
 **Author**: [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google)

 [gRPC](https://grpc.io) is on its way to becoming the lingua franca for
@@ -144,10 +144,9 @@ ways to address it.
 - Specifically add an iptables rule to drop the packets that are marked as
  *INVALID*, so it won’t reach to client pod and cause harm.

-The fix is drafted (https://github.com/kubernetes/kubernetes/pull/74840), but
-unfortunately it didn’t catch the v1.14 release window. However, for the users
-that are affected by this bug, there is a way to mitigate the problem by applying
-the following rule in your cluster.
+The [fix](https://github.com/kubernetes/kubernetes/pull/74840) is available in v1.15+.
+However, for the users that are affected by this bug, there is a way to mitigate the
+problem by applying the following rule in your cluster.

 ```yaml
 apiVersion: extensions/v1beta1
@@ -29,7 +29,7 @@ They join continuing members Christoph Blecker ([@cblecker](https://github.com/c
  * Josh Berkus ([@jberkus](https://github.com/jberkus)), Red Hat
 * Thanks to the Emeritus Steering Committee Members. Your prior service is appreciated by the community:
  * Aaron Crickenberger ([@spiffxp](https://github.com/spiffxp)), Google
-  * and Lachlan Evenson([@lachie8e)](https://github.com/lachie8e)), Microsoft
+  * and Lachlan Evenson([@lachie83)](https://github.com/lachie83)), Microsoft
 * And thank you to all the candidates who came forward to run for election. As [Jorge Castro put it](https://twitter.com/castrojo/status/1315718627639820288?s=20): we are spoiled with capable, kind, and selfless volunteers who put the needs of the project first.

 ## Get Involved with the Steering Committee
@@ -28,7 +28,7 @@ as cgroups v2 and user namespaces are being implemented in these newer CRI
 runtimes. Removing support for the dockershim will allow further development in
 those areas.

-[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1985-remove-dockershim
+[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim

 ### Can I still use Docker in Kubernetes 1.20?

@@ -42,9 +42,11 @@ startup if using Docker as the runtime.

 Given the impact of this change, we are using an extended deprecation timeline.
 It will not be removed before Kubernetes 1.22, meaning the earliest release without
-dockershim would be 1.23 in late 2021. We will be working closely with vendors
-and other ecosystem groups to ensure a smooth transition and will evaluate things
-as the situation evolves.
+dockershim would be 1.23 in late 2021. 
+_Update_: removal of dockershim is scheduled for Kubernetes v1.24, see 
+[Dockershim Removal Kubernetes Enhancement Proposal][drkep].
+We will be working closely with vendors and other ecosystem groups to ensure a smooth transition and will evaluate 
+things as the situation evolves.


 ### Can I still use dockershim after it is removed from Kubernetes?
@@ -190,9 +190,9 @@ kubectl get configmap
 No resources found in default namespace.
 ```

-To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=false option for `kubectl delete` to delete an object and orphan its children. 
+To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=orphan option for `kubectl delete` to delete an object and orphan its children. 

-In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=false, the parent is deleted but the child still exists:
+In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=orphan, the parent is deleted but the child still exists:

 ```
 kubectl get configmap
@@ -200,7 +200,7 @@ NAME           DATA   AGE
 mymap-child    0      13m8s
 mymap-parent   0      13m8s

-kubectl delete --cascade=false configmap/mymap-parent
+kubectl delete --cascade=orphan configmap/mymap-parent
 configmap "mymap-parent" deleted

 kubectl get configmap
@@ -121,7 +121,7 @@ deploymentApplyConfig.Spec.Template.Spec.WithContainers(corev1ac.Container().
 )

 // apply
-applied, err := deploymentClient.Apply(ctx, extractedDeployment, metav1.ApplyOptions{FieldManager: fieldMgr})
+applied, err := deploymentClient.Apply(ctx, deploymentApplyConfig, metav1.ApplyOptions{FieldManager: fieldMgr})
 ```

 For developers using Custom Resource Definitions (CRDs), the Kubebuilder apply support will provide the same capabilities. Documentation will be included in the Kubebuilder book when available.
@@ -28,7 +28,7 @@ metadata:
  name: shared-cache
 spec:
  accessModes:
-  - ReadWriteMany # Allow many pods to access shared-cache simultaneously.
+  - ReadWriteMany # Allow many nodes to access shared-cache simultaneously.
  resources:
    requests:
      storage: 1Gi
@@ -0,0 +1,72 @@
+---
+layout: blog
+title: "Spotlight on SIG Node"
+date: 2021-09-27
+slug: sig-node-spotlight-2021
+---
+
+**Author:** Dewan Ahmed, Red Hat
+
+## Introduction
+
+In Kubernetes, a _Node_ is a representation of a single machine in your cluster. [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) owns that very important Node component and supports various subprojects such as Kubelet, Container Runtime Interface (CRI) and more to support how the pods and host resources interact. In this blog, we have summarized our conversation with [Elana Hashman (EH)](https://twitter.com/ehashdn) & [Sergey Kanzhelev (SK)](https://twitter.com/SergeyKanzhelev), who walk us through the various aspects of being a part of the SIG and share some insights about how others can get involved.
+
+## A summary of our conversation
+
+### Could you tell us a little about what SIG Node does?
+
+SK: SIG Node is a vertical SIG responsible for the components that support the controlled interactions between the pods and host resources. We manage the lifecycle of pods that are scheduled to a node. This SIG's focus is to enable a broad set of workload types, including workloads with hardware specific or performance sensitive requirements. All while maintaining isolation boundaries between pods on a node, as well as the pod and the host. This SIG maintains quite a few components and has many external dependencies (like container runtimes or operating system features), which makes the complexity we deal with huge. We tame the complexity and aim to continuously improve node reliability.
+
+### "SIG Node is a vertical SIG" could you explain a bit more?
+
+EH: There are two kinds of SIGs: horizontal and vertical. Horizontal SIGs are concerned with a particular function of every component in Kubernetes: for example, SIG Security considers security aspects of every component in Kubernetes, or SIG Instrumentation looks at the logs, metrics, traces and events of every component in Kubernetes. Such SIGs don't tend to own a lot of code.
+
+Vertical SIGs, on the other hand, own a single component, and are responsible for approving and merging patches to that code base. SIG Node owns the "Node" vertical, pertaining to the kubelet and its lifecycle. This includes the code for the kubelet itself, as well as the node controller, the container runtime interface, and related subprojects like the node problem detector. 
+
+### How did the CI subproject start? Is this specific to SIG Node and how does it help the SIG?
+
+SK: The subproject started as a follow up after one of the releases was blocked by numerous test failures of critical tests. These tests haven’t started falling all at once, rather continuous lack of attention led to slow degradation of tests quality. SIG Node was always prioritizing quality and reliability, and forming of the subproject was a way to highlight this priority.
+
+### As the 3rd largest SIG in terms of number of issues and PRs, how does your SIG juggle so much work?
+
+EH: It helps to be organized. When I increased my contributions to the SIG in January of 2021, I found myself overwhelmed by the volume of pull requests and issues and wasn't sure where to start. We were already tracking test-related issues and pull requests on the CI subproject board, but that was missing a lot of our bugfixes and feature work. So I began putting together a triage board for the rest of our pull requests, which allowed me to sort each one by status and what actions to take, and documented its use for other contributors. We closed or merged over 500 issues and pull requests tracked by our two boards in each of the past two releases. The Kubernetes devstats showed that we have significantly increased our velocity as a result.
+
+In June, we ran our first bug scrub event to work through the backlog of issues filed against SIG Node, ensuring they were properly categorized. We closed over 130 issues over the course of this 48 hour global event, but as of writing we still have 333 open issues. 
+
+### Why should new and existing contributors consider joining SIG Node?
+
+SK: Being a SIG Node contributor gives you skills and recognition that are rewarding and useful. Understanding under the hood of a kubelet helps architecting better apps, tune and optimize those apps, and gives leg up in issues troubleshooting. If you are a new contributor, SIG Node gives you the foundational knowledge that is key to understanding why other Kubernetes components are designed the way they are. Existing contributors may benefit as many features will require SIG Node changes one way or another. So being a SIG Node contributor helps building features in other SIGs faster.
+
+SIG Node maintains numerous components, many of which have dependency on external projects or OS features. This makes the onboarding process quite lengthy and demanding. But if you are up for a challenge, there is always a place for you, and a group of people to support. 
+
+### What do you do to help new contributors get started?
+
+EH: Getting started in SIG Node can be intimidating, since there is so much work to be done, our SIG meetings are very large, and it can be hard to find a place to start.
+
+I always encourage new contributors to work on things that they have some investment in already. In SIG Node, that might mean volunteering to help fix a bug that you have personally been affected by, or helping to triage bugs you care about by priority.
+
+To come up to speed on any open source code base, there are two strategies you can take: start by exploring a particular issue deeply, and follow that to expand the edges of your knowledge as needed, or briefly review as many issues and change requests as you possibly can to get a higher level picture of how the component works. Ultimately, you will need to do both if you want to become a Node reviewer or approver.
+
+[Davanum Srinivas](https://twitter.com/dims) and I each ran a cohort of group mentoring to help teach new contributors the skills to become Node reviewers, and if there's interest we can work to find a mentor to run another session. I also encourage new contributors to attend our Node CI Subproject meeting: it's a smaller audience and we don't record the triage sessions, so it can be a less intimidating way to get started with the SIG. 
+
+### Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn?
+
+SK: SIG Node works on many workstreams in very different areas. All of these areas are on system level. For the typical code contributions you need to have a passion for building and utilizing low level APIs and writing performant and reliable components. Being a contributor you will learn how to debug and troubleshoot, profile, and monitor these components, as well as user workload that is run by these components. Often, with the limited to no access to Nodes, as they are running production workloads.
+
+The other way of contribution is to help document SIG node features. This type of contribution requires a deep understanding of features, and ability to explain them in simple terms.
+
+Finally, we are always looking for feedback on how best to run your workload. Come and  explain specifics of it, and what features in SIG Node components may help to run it better. 
+
+### What are you getting positive feedback on, and what’s coming up next for SIG Node?
+
+EH: Over the past year SIG Node has adopted some new processes to help manage our feature development and Kubernetes enhancement proposals, and other SIGs have looked to us for inspiration in managing large workloads. I hope that this is an area we can continue to provide leadership in and further iterate on.
+
+We have a great balance of new features and deprecations in flight right now. Deprecations of unused or difficult to maintain features help us keep technical debt and maintenance load under control, and examples include the dockershim and DynamicKubeletConfiguration deprecations. New features will unlock additional functionality in end users' clusters, and include exciting features like support for cgroups v2, swap memory, graceful node shutdowns, and device management policies.
+
+### Any closing thoughts/resources you’d like to share?
+
+SK/EH: It takes time and effort to get to any open source community. SIG Node may overwhelm you at first with the number of participants, volume of work, and project scope. But it is totally worth it. Join our welcoming community! [SIG Node GitHub Repo](https://github.com/kubernetes/community/tree/master/sig-node) contains many useful resources including Slack, mailing list and other contact info. 
+
+## Wrap Up
+
+SIG Node hosted a [KubeCon + CloudNativeCon Europe 2021 talk](https://www.youtube.com/watch?v=z5aY4e2RENA) with an intro and deep dive to their awesome SIG. Join the SIG's meetings to find out about the most recent research results, what the plans are for the forthcoming year, and how to get involved in the upstream Node team as a contributor!
@@ -0,0 +1,243 @@
+---
+layout: blog
+title: "How to Handle Data Duplication in Data-Heavy Kubernetes Environments"
+date: 2021-09-29
+slug: how-to-handle-data-duplication-in-data-heavy-kubernetes-environments 
+---
+
+**Authors:**
+Augustinas Stirbis (CAST AI)
+
+## Why Duplicate Data?
+
+It’s convenient to create a copy of your application with a copy of its state for each team. 
+For example, you might want a separate database copy to test some significant schema changes 
+or develop other disruptive operations like bulk insert/delete/update...
+
+**Duplicating data takes a lot of time.** That’s because you need first to download 
+all the data from a source block storage provider to compute and then send 
+it back to a storage provider again. There’s a lot of network traffic and CPU/RAM used in this process.
+Hardware acceleration by offloading certain expensive operations to dedicated hardware is 
+**always a huge performance boost**. It reduces the time required to complete an operation by orders 
+of magnitude.
+
+## Volume Snapshots to the rescue
+
+Kubernetes introduced [VolumeSnapshots](/docs/concepts/storage/volume-snapshots/) as alpha in 1.12,
+beta in 1.17, and the Generally Available version in 1.20. 
+VolumeSnapshots use specialized APIs from storage providers to duplicate volume of data.
+
+Since data is already in the same storage device (array of devices), duplicating data is usually 
+a metadata operation for storage providers with local snapshots (majority of on-premise storage providers).
+All you need to do is point a new disk to an immutable snapshot and only 
+save deltas (or let it do a full-disk copy). As an operation that is inside the storage back-end,
+it’s much quicker and usually doesn’t involve sending traffic over the network.
+Public Clouds storage providers under the hood work a bit differently. They save snapshots
+to Object Storage and then copy back from Object storage to Block storage when "duplicating" disk.
+Technically there is a lot of Compute and network resources spent on Cloud providers side,
+but from Kubernetes user perspective VolumeSnapshots work the same way whether is it local or
+remote snapshot storage provider and no Compute and Network resources are involved in this operation.
+
+## Sounds like we have our solution, right?
+
+Actually, VolumeSnapshots are namespaced, and Kubernetes protects namespaced data from 
+being shared between tenants (Namespaces). This Kubernetes limitation is a conscious design 
+decision so that a Pod running in a different namespace can’t mount another application’s
+[PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) (PVC).
+
+One way around it would be to create multiple volumes with duplicate data in one namespace.
+However, you could easily reference the wrong copy.
+
+So the idea is to separate teams/initiatives by namespaces to avoid that and generally 
+limit access to the production namespace.
+
+## Solution? Creating a Golden Snapshot externally
+
+Another way around this design limitation is to create Snapshot externally (not through Kubernetes).
+This is also called pre-provisioning a snapshot manually. Next, I will import it 
+as a multi-tenant golden snapshot that can be used for many namespaces. Below illustration will be 
+for AWS EBS (Elastic Block Storage) and GCE PD (Persistent Disk) services.
+
+### High-level plan for preparing the Golden Snapshot
+
+1. Identify Disk (EBS/Persistent Disk) that you want to clone with data in the cloud provider
+2. Make a Disk Snapshot (in cloud provider console)
+3. Get Disk Snapshot ID
+
+### High-level plan for cloning data for each team
+
+1. Create Namespace “sandbox01”
+2. Import Disk Snapshot (ID) as VolumeSnapshotContent to Kubernetes
+3. Create VolumeSnapshot in the Namespace "sandbox01" mapped to VolumeSnapshotContent
+4. Create the PersistentVolumeClaim from VolumeSnapshot
+5. Install Deployment or StatefulSet with PVC
+
+## Step 1: Identify Disk
+
+First, you need to identify your golden source. In my case, it’s a PostgreSQL database
+on PersistentVolumeClaim “postgres-pv-claim” in the “production” namespace.
+
+```terminal
+kubectl -n <namespace> get pvc <pvc-name> -o jsonpath='{.spec.volumeName}'
+```
+
+The output will look similar to:
+```
+pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d90
+```
+
+## Step 2: Prepare your golden source
+
+You need to do this once or every time you want to refresh your golden data.
+
+### Make a Disk Snapshot
+
+Go to AWS EC2 or GCP Compute Engine console and search for an EBS volume
+(on AWS) or Persistent Disk (on GCP), that has a label matching the last output.
+In this case I saw: `pvc-3096b3ba-38b6-4fd1-a42f-ec99176ed0d9`.
+
+Click on Create snapshot and give it a name. You can do it in Console manually,
+in AWS CloudShell / Google Cloud Shell, or in the terminal. To create a snapshot in the
+terminal you must have the AWS CLI tool (`aws`) or Google's CLI (`gcloud`)
+installed and configured.
+
+Here’s the command to create snapshot on GCP:
+
+```terminal
+gcloud compute disks snapshot <cloud-disk-id> --project=<gcp-project-id> --snapshot-names=<set-new-snapshot-name> --zone=<availability-zone> --storage-location=<region>
+```
+{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/create-volume-snapshot-gcp.png" alt="Screenshot of a terminal showing volume snapshot creation on GCP" title="GCP snapshot creation" >}}
+
+
+GCP identifies the disk by its PVC name, so it’s direct mapping. In AWS, you need to 
+find volume by the CSIVolumeName AWS tag with PVC name value first that will be used for snapshot creation.
+
+{{< figure src="/images/blog/2021-09-07-data-duplication-in-data-heavy-k8s-env/identify-volume-aws.png" alt="Screenshot of AWS web console, showing EBS volume identification" title="Identify disk ID on AWS" >}}
+
+Mark done Volume (volume-id) ```vol-00c7ecd873c6fb3ec``` and ether create EBS snapshot in AWS Console, or use ```aws cli```.
+
+```terminal
+aws ec2 create-snapshot --volume-id '<volume-id>' --description '<set-new-snapshot-name>' --tag-specifications 'ResourceType=snapshot'
+```
+
+## Step 3: Get your Disk Snapshot ID
+
+In AWS, the command above will output something similar to:
+```terminal
+"SnapshotId": "snap-09ed24a70bc19bbe4"
+```
+
+If you’re using the GCP cloud, you can get the snapshot ID from the gcloud command by querying for the snapshot’s given name:
+
+```terminal
+gcloud compute snapshots --project=<gcp-project-id> describe <new-snapshot-name> | grep id:
+```
+You should get similar output to:
+```
+id: 6645363163809389170
+```
+
+## Step 4: Create a development environment for each team
+
+Now I have my Golden Snapshot, which is immutable data. Each team will get a copy 
+of this data, and team members can modify it as they see fit, given that a new EBS/persistent 
+disk will be created for each team.
+
+Below I will define a manifest for each namespace. To save time, you can replace
+the namespace name (such as changing “sandbox01” → “sandbox42”) using tools
+such as `sed` or `yq`, with Kubernetes-aware templating tools like
+[Kustomize](/docs/tasks/manage-kubernetes-objects/kustomization/),
+or using variable substitution in a CI/CD pipeline.
+
+Here's an example manifest:
+
+```yaml
+---
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshotContent
+metadata:
+ name: postgresql-orders-db-sandbox01
+ namespace: sandbox01
+spec:
+ deletionPolicy: Retain
+ driver: pd.csi.storage.gke.io
+ source:
+   snapshotHandle: 'gcp/projects/staging-eu-castai-vt5hy2/global/snapshots/6645363163809389170'
+ volumeSnapshotRef:
+   kind: VolumeSnapshot
+   name: postgresql-orders-db-snap
+   namespace: sandbox01
+---
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshot
+metadata:
+ name: postgresql-orders-db-snap
+ namespace: sandbox01
+spec:
+ source:
+   volumeSnapshotContentName: postgresql-orders-db-sandbox01
+```
+
+In Kubernetes, VolumeSnapshotContent (VSC) objects are not namespaced.
+However, I need a separate VSC for each different namespace to use, so the
+`metadata.name` of each VSC must also be different. To make that straightfoward,
+I used the target namespace as part of the name.
+
+Now it’s time to replace the driver field with the CSI (Container Storage Interface) driver
+installed in your K8s cluster. Major cloud providers have CSI driver for block storage that
+support VolumeSnapshots but quite often CSI drivers are not installed by default, consult
+with your Kubernetes provider. 
+
+That manifest above defines a VSC that works on GCP.
+On AWS, driver and SnashotHandle values might look like:
+
+```YAML
+  driver: ebs.csi.aws.com
+  source:
+    snapshotHandle: "snap-07ff83d328c981c98"
+```
+
+At this point, I need to use the *Retain* policy, so that the CSI driver doesn’t try to
+delete my manually created EBS disk snapshot.
+
+For GCP, you will have to build this string by hand - add a full project ID and snapshot ID.
+For AWS, it’s just a plain snapshot ID.
+
+VSC also requires specifying which VolumeSnapshot (VS) will use it, so VSC and VS are
+referencing each other.
+
+Now I can create PersistentVolumeClaim from VS above. It’s important to set this first:
+
+
+```yaml
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: postgres-pv-claim
+ namespace: sandbox01
+spec:
+ dataSource:
+   kind: VolumeSnapshot
+   name: postgresql-orders-db-snap
+   apiGroup: snapshot.storage.k8s.io
+ accessModes:
+   - ReadWriteOnce
+ resources:
+   requests:
+     storage: 21Gi
+```
+
+If default StorageClass has [WaitForFirstConsumer](https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode) policy,
+then the actual Cloud Disk will be created from the Golden Snapshot only when some Pod bounds that PVC.
+
+Now I assign that PVC to my Pod (in my case, it’s Postgresql) as I would with any other PVC.
+
+```terminal
+kubectl -n <namespace> get volumesnapshotContent,volumesnapshot,pvc,pod
+```
+
+Both VS and VSC should be *READYTOUSE* true, PVC bound, and the Pod (from Deployment or StatefulSet) running.
+
+**To keep on using data from my Golden Snapshot, I just need to repeat this for the
+next namespace and voilà! No need to waste time and compute resources on the duplication process.**
@@ -0,0 +1,417 @@
+---
+layout: blog
+title: A Closer Look at NSA/CISA Kubernetes Hardening Guidance
+date: 2021-10-05
+slug: nsa-cisa-kubernetes-hardening-guidance
+---
+
+**Authors:** Jim Angel (Google), Pushkar Joglekar (VMware), and Savitha
+Raghunathan (Red Hat)
+
+{{% alert title="Disclaimer" %}}
+The open source tools listed in this article are to serve as examples only 
+and are in no way a direct recommendation from the Kubernetes community or authors.
+{{% /alert %}}
+
+## Background
+
+USA's National Security Agency (NSA) and the Cybersecurity and Infrastructure
+Security Agency (CISA)
+released, "[Kubernetes Hardening Guidance](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF)"
+on August 3rd, 2021. The guidance details threats to Kubernetes environments
+and provides secure configuration guidance to minimize risk.
+
+The following sections of this blog correlate to the sections in the NSA/CISA guidance. 
+Any missing sections are skipped because of limited opportunities to add 
+anything new to the existing content. 
+
+_Note_: This blog post is not a substitute for reading the guide. Reading the published 
+guidance is recommended before proceeding as the following content is 
+complementary.
+
+## Introduction and Threat Model
+
+Note that the threats identified as important by the NSA/CISA, or the intended audience of this guidance, may be different from the threats that other enterprise users of Kubernetes consider important. This section 
+is still useful for organizations that care about data, resource theft and 
+service unavailability.
+
+The guidance highlights the following three sources of compromises:
+
+- Supply chain risks
+- Malicious threat actors
+- Insider threats (administrators, users, or cloud service providers)
+
+The [threat model](https://en.wikipedia.org/wiki/Threat_model) tries to take a step back and review threats that not only
+exist within the boundary of a Kubernetes cluster but also include the underlying
+infrastructure and surrounding workloads that Kubernetes does not manage.
+
+For example, when a workload outside the cluster shares the same physical
+network, it has access to the kubelet and to control plane components: etcd, controller manager, scheduler and API
+server. Therefore, the guidance recommends having network level isolation
+separating Kubernetes clusters from other workloads that do not need connectivity
+to Kubernetes control plane nodes. Specifically, scheduler, controller-manager,
+etcd only need to be accessible to the API server. Any interactions with Kubernetes
+from outside the cluster can happen by providing access to API server port.
+
+List of ports and protocols for each of these components are
+defined in [Ports and Protocols](/docs/reference/ports-and-protocols/)
+within the Kubernetes documentation.
+
+> Special note: kube-scheduler and kube-controller-manager uses different ports than the ones mentioned in the guidance
+
+The [Threat modelling](https://cnsmap.netlify.app/threat-modelling) section
+from the CNCF [Cloud Native Security Whitepaper + Map](https://github.com/cncf/tag-security/tree/main/security-whitepaper)
+provides another perspective on approaching threat modelling Kubernetes, from a
+cloud native lens.
+
+## Kubernetes Pod security
+
+Kubernetes by default does not guarantee strict workload isolation between pods
+running in the same node in a cluster. However, the guidance provides several 
+techniques to enhance existing isolation and reduce the attack surface in case of a 
+compromise.
+
+### "Non-root" containers and "rootless" container engines
+
+Several best practices related to basic security principle of least privilege
+i.e. provide only the permissions are needed; no more, no less, are worth a
+second look.
+
+The guide recommends setting non-root user at build time instead of relying on
+setting `runAsUser` at runtime in your Pod spec. This is a good practice and provides
+some level of defense in depth. For example, if the container image is built with user `10001`
+and the Pod spec misses adding the `runAsuser` field in its `Deployment` object. In this
+case there are certain edge cases that are worth exploring for awareness:
+
+1. Pods can fail to start, if the user defined at build time is different from
+   the one defined in pod spec and some files are as a result inaccessible.
+2. Pods can end up sharing User IDs unintentionally. This can be problematic
+   even if the User IDs are non-zero in a situation where a container escape to
+   host file system is possible. Once the attacker has access to the host file
+   system, they get access to all the file resources that are owned by other
+   unrelated pods that share the same UID.
+3. Pods can end up sharing User IDs, with other node level processes not managed
+   by Kubernetes e.g. node level daemons for auditing, vulnerability scanning,
+   telemetry. The threat is similar to the one above where host file system
+   access can give attacker full access to these node level daemons without
+   needing to be root on the node.
+
+However, none of these cases will have as severe an impact as a container
+running as root being able to escape as a root user on the host, which can provide 
+an attacker with complete control of the worker node, further allowing lateral
+movement to other worker or control plane nodes.
+
+Kubernetes 1.22 introduced
+an [alpha feature](/docs/tasks/administer-cluster/kubelet-in-userns/)
+that specifically reduces the impact of such a control plane component running
+as root user to a non-root user through user namespaces.
+
+That ([alpha stage](/docs/reference/command-line-tools-reference/feature-gates/#feature-stages)) support for user namespaces / rootless mode is available with
+the following container runtimes:
+
+- [Docker Engine](https://docs.docker.com/engine/security/rootless/)
+- [Podman](https://developers.redhat.com/blog/2020/09/25/rootless-containers-with-podman-the-basics)
+
+Some distributions support running in rootless mode, like the following:
+
+- [kind](https://kind.sigs.k8s.io/docs/user/rootless/)
+- [k3s](https://rancher.com/docs/k3s/latest/en/advanced/#running-k3s-with-rootless-mode-experimental)
+- [Usernetes](https://github.com/rootless-containers/usernetes)
+
+### Immutable container filesystems
+
+The NSA/CISA Kubernetes Hardening Guidance highlights an often overlooked feature `readOnlyRootFileSystem`, with a
+working example in [Appendix B](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=42). This example limits execution and tampering of
+containers at runtime. Any read/write activity can then be limited to few
+directories by using `tmpfs` volume mounts.
+
+However, some applications that modify the container filesystem at runtime, like exploding a WAR or JAR file at container startup, 
+could face issues when enabling this feature. To avoid this issue, consider making minimal changes to the filesystem at runtime 
+when possible.
+
+### Building secure container images
+
+Kubernetes Hardening Guidance also recommends running a scanner at deploy time as an admission controller,
+to prevent vulnerable or misconfigured pods from running in the cluster.
+Theoretically, this sounds like a good approach but there are several caveats to
+consider before this can be implemented in practice:
+
+- Depending on network bandwidth, available resources and scanner of choice,
+  scanning for vulnerabilities for an image can take an indeterminate amount of
+  time. This could lead to slower or unpredictable pod start up times, which
+  could result in spikes of unavailability when apps are serving peak load.
+- If the policy that allows or denies pod startup is made using incorrect or
+  incomplete data it could result in several false positive or false negative
+  outcomes like the following:
+  - inside a container image, the `openssl` package is detected as vulnerable. However,
+    the application is written in Golang and uses the Go `crypto` package for TLS. Therefore, this vulnerability
+    is not in the code execution path and as such has minimal impact if it
+    remains unfixed.
+  - A vulnerability is detected in the `openssl` package for a Debian base image.
+    However, the upstream Debian community considers this as a Minor impact
+    vulnerability and as a result does not release a patch fix for this
+    vulnerability. The owner of this image is now stuck with a vulnerability that
+    cannot be fixed and a cluster that does not allow the image to run because
+    of predefined policy that does not take into account whether the fix for a
+    vulnerability is available or not
+  - A Golang app is built on top of a [distroless](https://github.com/GoogleContainerTools/distroless)
+    image, but it is compiled with a Golang version that uses a vulnerable [standard library](https://pkg.go.dev/std).
+    The scanner has
+    no visibility into golang version but only on OS level packages. So it 
+    allows the pod to run in the cluster in spite of the image containing an 
+    app binary built on vulnerable golang.
+
+To be clear, relying on vulnerability scanners is absolutely a good idea but
+policy definitions should be flexible enough to allow:
+
+- Creation of exception lists for images or vulnerabilities through labelling
+- Overriding the severity with a risk score based on impact of a vulnerability
+- Applying the same policies at build time to catch vulnerable images with
+  fixable vulnerabilities before they can be deployed into Kubernetes clusters
+
+Special considerations like offline vulnerability database fetch, may also be
+needed, if the clusters run in an air-gapped environment and the scanners
+require internet access to update the vulnerability database.
+
+### Pod Security Policies
+
+Since Kubernetes v1.21, the [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) 
+API and related features are [deprecated](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/),
+but some of the guidance in this section will still apply for the next few years, until cluster operators
+upgrade their clusters to newer Kubernetes versions.
+
+The Kubernetes project is working on a replacement for PodSecurityPolicy.
+Kubernetes v1.22 includes an alpha feature called [Pod Security Admission](/docs/concepts/security/pod-security-admission/)
+that is intended to allow enforcing a minimum level of isolation between pods.
+
+The built-in isolation levels for Pod Security Admission are derived
+from [Pod Security Standards](/docs/concepts/security/pod-security-standards/), which is a superset of all the components mentioned in Table I [page 10](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=17) of
+the guidance.
+
+Information about migrating from PodSecurityPolicy to the Pod Security
+Admission feature is available
+in
+[Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/).
+
+One important behavior mentioned in the guidance that remains the same between
+Pod Security Policy and its replacement is that enforcing either of them does
+not affect pods that are already running. With both PodSecurityPolicy and Pod Security Admission,
+the enforcement happens during the pod creation
+stage.
+
+### Hardening container engines
+
+Some container workloads are less trusted than others but may need to run in the
+same cluster. In those cases, running them on dedicated nodes that include
+hardened container runtimes that provide stricter pod isolation boundaries can
+act as a useful security control.
+
+Kubernetes supports
+an API called [RuntimeClass](/docs/concepts/containers/runtime-class/) that is
+stable / GA (and, therefore, enabled by default) stage as of Kubernetes v1.20.
+RuntimeClass allows you to ensure that Pods requiring strong isolation are scheduled onto
+nodes that can offer it.
+
+Some third-party projects that you can use in conjunction with RuntimeClass are:
+
+- [kata containers](https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-runtime-class-for-kata-containers)
+- [gvisor](https://gvisor.dev/docs/user_guide/containerd/quick_start/)
+
+As discussed here and in the guidance, many features and tooling exist in and around
+Kubernetes that can enhance the isolation boundaries between
+pods. Based on relevant threats and risk posture, you should pick and choose
+between them, instead of trying to apply all the recommendations. Having said that, cluster
+level isolation i.e. running workloads in dedicated clusters, remains the strictest workload 
+isolation mechanism, in spite of improvements mentioned earlier here and in the guide.
+
+## Network Separation and Hardening
+
+Kubernetes Networking can be tricky and this section focuses on how to secure
+and harden the relevant configurations. The guide identifies the following as key
+takeaways: 
+- Using NetworkPolicies to create isolation between resources,
+- Securing the control plane
+- Encrypting traffic and sensitive data
+
+### Network Policies
+
+Network policies can be created with the help of network plugins. In order to
+make the creation and visualization easier for users, Cilium supports
+a [web GUI tool](https://editor.cilium.io). That web GUI lets you create Kubernetes
+NetworkPolicies (a generic API that nevertheless requires a compatible CNI plugin),
+and / or Cilium network policies (CiliumClusterwideNetworkPolicy and CiliumNetworkPolicy,
+which only work in clusters that use the Cilium CNI plugin).
+You can use these APIs to restrict network traffic between pods, and therefore minimize the
+attack vector.
+
+Another scenario that is worth exploring is the usage of external IPs. Some
+services, when misconfigured, can create random external IPs. An attacker can take
+advantage of this misconfiguration and easily intercept traffic. This vulnerability
+has been reported
+in [CVE-2020-8554](https://www.cvedetails.com/cve/CVE-2020-8554/).
+Using [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook)
+can mitigate this vulnerability by preventing the services from using random
+external IPs. [externalip-webhook](https://github.com/kubernetes-sigs/externalip-webhook) 
+only allows creation of services that don't require external IPs or whose 
+external IPs are within the range specified by the administrator.
+
+> CVE-2020-8554 - Kubernetes API server in all versions allow an attacker
+> who is able to create a ClusterIP service and set the `spec.externalIPs` field,
+> to intercept traffic to that IP address. Additionally, an attacker who is able to
+> patch the `status` (which is considered a privileged operation and should not
+> typically be granted to users) of a LoadBalancer service can set the
+> `status.loadBalancer.ingress.ip` to similar effect.
+
+### Resource Policies
+
+In addition to configuring ResourceQuotas and limits, consider restricting how many process
+IDs (PIDs) a given Pod can use, and also to reserve some PIDs for node-level use to avoid
+resource exhaustion. More details to apply these limits can be
+found in [Process ID Limits And Reservations](/docs/concepts/policy/pid-limiting/).
+
+### Control Plane Hardening
+
+In the next section, the guide covers control plane hardening. It is worth
+noting that
+from [Kubernetes 1.20](https://github.com/kubernetes/kubernetes/issues/91506),
+insecure port from API server, has been removed.
+
+### Etcd
+
+As a general rule, the etcd server should be configured to only trust
+certificates assigned to the API server. It limits the attack surface and prevents a
+malicious attacker from gaining access to the cluster. It might be beneficial to
+use a separate CA for etcd, as it by default trusts all the certificates issued
+by the root CA.
+
+### Kubeconfig Files
+
+In addition to specifying the token and certificates directly, `.kubeconfig`
+supports dynamic retrieval of temporary tokens using auth provider plugins.
+Beware of the possibility of malicious
+shell [code execution](https://banzaicloud.com/blog/kubeconfig-security/) in a
+`kubeconfig` file. Once attackers gain access to the cluster, they can steal ssh
+keys/secrets or more.
+
+### Secrets
+Kubernetes [Secrets](/docs/concepts/configuration/secret/) is the native way of managing secrets as a Kubernetes
+API object. However, in some scenarios such as a desire to have a single source of truth for all app secrets, irrespective of whether they run on Kubernetes or not, secrets can be managed loosely coupled with 
+Kubernetes and consumed by pods through side-cars or init-containers with minimal usage of Kubernetes Secrets API. 
+
+[External secrets providers](https://github.com/external-secrets/kubernetes-external-secrets)
+and [csi-secrets-store](https://github.com/kubernetes-sigs/secrets-store-csi-driver)
+are some of these alternatives to Kubernetes Secrets
+
+## Log Auditing
+
+The NSA/CISA guidance stresses monitoring and alerting based on logs. The key points
+include logging at the host level, application level, and on the cloud. When
+running Kubernetes in production, it's important to understand who's
+responsible, and who's accountable, for each layer of logging.
+
+### Kubernetes API auditing
+
+One area that deserves more focus is what exactly should alert or be logged. The
+document outlines a sample policy in [Appendix L: Audit Policy](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=55) that logs all
+RequestResponse's including metadata and request / response bodies. While helpful for a demo, it may not be practical for production.
+
+Each organization needs to evaluate their
+own threat model and build an audit policy that complements or helps troubleshooting incident response. Think
+about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug-application-cluster/audit/#audit-policy).
+It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point. 
+
+Audit logging configurations can also be tested with
+kind following these [instructions](https://kind.sigs.k8s.io/docs/user/auditing).
+
+### Streaming logs and auditing
+
+Logging is important for threat and anomaly detection. As the document outlines,
+it's a best practice to scan and alert on logs as close to real time as possible
+and to protect logs from tampering if a compromise occurs. It's important to
+reflect on the various levels of logging and identify the critical areas such as
+API endpoints.
+
+Kubernetes API audit logging can stream to a webhook and there's an example in [Appendix N: Webhook configuration](https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF#page=58). Using a webhook could be a method that
+stores logs off cluster and/or centralizes all audit logs. Once logs are
+centrally managed, look to enable alerting based on critical events. Also ensure
+you understand what the baseline is for normal activities.
+
+### Alert identification
+
+While the guide stressed the importance of notifications, there is not a blanket
+event list to alert from. The alerting requirements vary based on your own
+requirements and threat model. Examples include the following events:
+
+- Changes to the `securityContext` of a Pod
+- Updates to admission controller configs
+- Accessing certain files / URLs
+
+### Additional logging resources
+
+- [Seccomp Security Profiles and You: A Practical Guide - Duffie Cooley](https://www.youtube.com/watch?v=OPuu8wsu2Zc)
+- [TGI Kubernetes 119: Gatekeeper and OPA](https://www.youtube.com/watch?v=ZJgaGJm9NJE)
+- [Abusing The Lack of Kubernetes Auditing Policies](https://www.lacework.com/blog/hiding-in-plaintext-sight-abusing-the-lack-of-kubernetes-auditing-policies/)
+- [Enable seccomp for all workloads with a new v1.22 alpha feature](https://kubernetes.io/blog/2021/08/25/seccomp-default/)
+- [This Week in Cloud Native: Auditing / Pod Security](https://www.twitch.tv/videos/1147889860)
+
+## Upgrading and Application Security practices
+
+Kubernetes releases three times per year, so upgrade-related toil is a common problem for
+people running production clusters. In addition to this, operators must
+regularly upgrade the underlying node's operating system and running
+applications. This is a best practice to ensure continued support and to reduce
+the likelihood of bugs or vulnerabilities.
+
+Kubernetes supports the three most recent stable releases. While each Kubernetes
+release goes through a large number of tests before being published, some
+teams aren't comfortable running the latest stable release until some time has
+passed. No matter what version you're running, ensure that patch upgrades
+happen frequently or automatically. More information can be found in
+the [version skew](/releases/version-skew-policy/) policy
+pages.
+
+When thinking about how you'll manage node OS upgrades, consider ephemeral
+nodes. Having the ability to destroy and add nodes allows your team to respond
+quicker to node issues. In addition, having deployments that tolerate node
+instability (and a culture that encourages frequent deployments) allows for
+easier cluster upgrades.
+
+Additionally, it's worth reiterating from the guidance that periodic 
+vulnerability scans and penetration tests can be performed on the various system
+components to proactively look for insecure configurations and vulnerabilities.
+
+### Finding release & security information
+
+To find the most recent Kubernetes supported versions, refer to
+[https://k8s.io/releases](https://k8s.io/releases), which includes minor versions. It's good to stay up to date with
+your minor version patches.
+
+If you're running a managed Kubernetes offering, look for their release
+documentation and find their various security channels.
+
+Subscribe to
+the [Kubernetes Announce mailing list](https://groups.google.com/g/kubernetes-announce).
+The Kubernetes Announce mailing list is searchable for terms such
+as "[Security Advisories](https://groups.google.com/g/kubernetes-announce/search?q=%5BSecurity%20Advisory%5D)".
+You can set up alerts and email notifications as long as you know what key
+words to alert on.
+
+## Conclusion
+
+In summary, it is fantastic to see security practitioners sharing this
+level of detailed guidance in public. This guidance further highlights
+Kubernetes going mainstream and how securing Kubernetes clusters and the
+application containers running on Kubernetes continues to need attention and focus of
+practitioners. Only a few weeks after the guidance was published, an open source
+tool [kubescape](https://github.com/armosec/kubescape) to validate cluster
+against this guidance became available.
+
+This tool can be a great starting point to check the current state of your
+clusters, after which you can use the information in this blog post and in the guidance to assess 
+where improvements can be made.
+
+Finally, it is worth reiterating that not all controls in this guidance will
+make sense for all practitioners. The best way to know which controls matter is
+to rely on the threat model of your own Kubernetes environment.
+
+_A special shout out and thanks to Rory McCune (@raesene) for his inputs to this blog post_
@@ -0,0 +1,141 @@
+---
+layout: blog
+title: "Introducing ClusterClass and Managed Topologies in Cluster API"
+date: 2021-10-08
+slug: capi-clusterclass-and-managed-topologies
+---
+
+**Author:** Fabrizio Pandini (VMware)
+
+The [Cluster API community](https://cluster-api.sigs.k8s.io/) is happy to announce the implementation of *ClusterClass and Managed Topologies*, a new feature that will greatly simplify how you can provision, upgrade, and operate multiple Kubernetes clusters in a declarative way.
+
+## A little bit of context…
+
+Before getting into the details, let's take a step back and look at the history of Cluster API.
+
+The [Cluster API project](https://github.com/kubernetes-sigs/cluster-api/) started three years ago, and the first releases focused on extensibility and implementing a declarative API that allows a seamless experience across infrastructure providers. This was a success with many cloud providers: AWS, Azure, Digital Ocean, GCP, Metal3, vSphere and still counting.
+
+With extensibility addressed, the focus shifted to features, like automatic control plane and etcd management, health-based machine remediation, machine rollout strategies and more.
+
+Fast forwarding to 2021, with lots of companies using Cluster API to manage fleets of Kubernetes clusters running workloads in production, the community focused its effort on stabilization of both code, APIs, documentation, and on extensive test signals which inform Kubernetes releases.
+
+With solid foundations in place, and a vibrant and welcoming community that still continues to grow, it was time to plan another iteration on our UX for both new and advanced users.
+
+Enter ClusterClass and Managed Topologies, tada!
+
+## ClusterClass
+
+As the name suggests, ClusterClass and managed topologies are built in two parts.
+
+The idea behind ClusterClass is simple: define the shape of your cluster once, and reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away.
+
+![Defining a ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/clusterclass.svg)
+
+ClusterClass, at its heart, is a collection of Cluster and Machine templates. You can use it as a “stamp” that can be leveraged to create many clusters of a similar shape.
+
+```yaml
+---
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: ClusterClass
+metadata:
+  name: my-amazing-cluster-class
+spec:
+  controlPlane:
+    ref:
+      apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+      kind: KubeadmControlPlaneTemplate
+      name: high-availability-control-plane
+    machineInfrastructure:
+      ref:
+        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+        kind: DockerMachineTemplate
+        name: control-plane-machine
+  workers:
+    machineDeployments:
+      - class: type1-workers
+        template:
+          bootstrap:
+            ref:
+              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+              kind: KubeadmConfigTemplate
+              name: type1-bootstrap
+          infrastructure:
+            ref:
+              apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+              kind: DockerMachineTemplate
+              name: type1-machine
+      - class: type2-workers
+        template:
+          bootstrap:
+            ref:
+              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
+              kind: KubeadmConfigTemplate
+              name: type2-bootstrap
+          infrastructure:
+            ref:
+              kind: DockerMachineTemplate
+              apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+              name: type2-machine
+  infrastructure:
+    ref:
+      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
+      kind: DockerClusterTemplate
+      name: cluster-infrastructure
+
+```
+
+The possibilities are endless; you can get a default ClusterClass from the community, “off-the-shelf” classes from your vendor of choice, “certified” classes from the platform admin in your company, or even create custom ones for advanced scenarios.
+
+## Managed Topologies
+
+Managed Topologies let you put the power of ClusterClass into action.
+
+Given a ClusterClass, you can create many Clusters of a similar shape by providing a single resource, the Cluster.
+
+![Create a Cluster with ClusterClass](/images/blog/2021-10-08-clusterclass-and-managed-topologies/create-cluster.svg)
+
+Here is an example:
+
+```yaml
+---
+apiVersion: cluster.x-k8s.io/v1beta1
+ kind: Cluster
+ metadata:
+   name: my-amazing-cluster
+   namespace: bar
+ spec:
+   topology: # define a managed topology
+     class: my-amazing-cluster-class # use the ClusterClass mentioned earlier
+     version: v1.21.2
+     controlPlane:
+       replicas: 3
+     workers:
+       machineDeployments:
+       - class: type1-workers
+         name: big-pool-of-machines
+         replicas: 5
+       - class: type2-workers
+         name: small-pool-of-machines
+         replicas: 1
+```
+
+But there is more than simplified cluster creation. Now the Cluster acts as a single control point for your entire topology.
+
+All the power of Cluster API, extensibility, lifecycle automation, stability, all the features required for managing an enterprise grade Kubernetes cluster on the infrastructure provider of your choice are now at your fingertips: you can create your Cluster, add new machines, upgrade to the next Kubernetes version, and all from a single place.
+
+It is just as simple as it looks!
+
+## What’s next
+
+While the amazing Cluster API community is working hard to deliver the first version of ClusterClass and managed topologies later this year, we are already looking forward to what comes next for the project and its ecosystem.
+
+There are a lot of great ideas and opportunities ahead!
+
+We want to make managed topologies even more powerful and flexible, allowing users to dynamically change bits of a ClusterClass according to the specific needs of a Cluster; this will ensure the same simple and intuitive UX for solving complex problems like e.g. selecting machine image for a specific Kubernetes version and for a specific region of your infrastructure provider, or injecting proxy configurations in the entire Cluster, and so on.
+
+Stay tuned for what comes next, and if you have any questions, comments or suggestions:
+
+* Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T)
+* Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents
+* Join our [Zoom meeting](https://zoom.us/j/861487554), every Wednesday at 10:00 Pacific Time
+* Check out the [ClusterClass tutorial](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-classes.html) in the Cluster API book.
@@ -0,0 +1,241 @@
+---
+layout: blog
+title: "Use KPNG to Write Specialized kube-proxiers"
+date: 2021-10-18
+slug: use-kpng-to-write-specialized-kube-proxiers
+---
+
+**Author**: Lars Ekman (Ericsson)
+
+The post will show you how to create a specialized service kube-proxy
+style network proxier using Kubernetes Proxy NG
+[kpng](https://github.com/kubernetes-sigs/kpng) without interfering
+with the existing kube-proxy. The kpng project aims at renewing the
+the default Kubernetes Service implementation, the "kube-proxy". An
+important feature of kpng is that it can be used as a library to
+create proxiers outside K8s. While this is useful for CNI-plugins that
+replaces the kube-proxy it also opens the possibility for anyone to
+create a proxier for a special purpose.
+
+
+## Define a service that uses a specialized proxier
+
+```
+apiVersion: v1
+kind: Service
+metadata:
+  name: kpng-example
+  labels:
+    service.kubernetes.io/service-proxy-name: kpng-example
+spec:
+  clusterIP: None
+  ipFamilyPolicy: RequireDualStack
+  externalIPs:
+  - 10.0.0.55
+  - 1000::55
+  selector:
+    app: kpng-alpine
+  ports:
+  - port: 6000
+```
+
+If the `service.kubernetes.io/service-proxy-name` label is defined the
+`kube-proxy` will ignore the service. A custom controller can watch
+services with the label set to it's own name, "kpng-example" in
+this example, and setup specialized load-balancing.
+
+The `service.kubernetes.io/service-proxy-name` label is [not
+new](https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioservice-proxy-name),
+but so far is has been quite hard to write a specialized proxier.
+
+The common use for a specialized proxier is assumed to be handling
+external traffic for some use-case not supported by K8s. In that
+case `ClusterIP` is not needed, so we use a "headless" service in this
+example.
+
+
+## Specialized proxier using kpng
+
+A [kpng](https://github.com/kubernetes-sigs/kpng) based proxier
+consists of the `kpng` controller handling all the K8s api related
+functions, and a "backend" implementing the load-balancing. The
+backend can be linked with the `kpng` controller binary or be a
+separate program communicating with the controller using gRPC.
+
+```
+kpng kube --service-proxy-name=kpng-example to-api
+```
+
+This starts the `kpng` controller and tell it to watch only services
+with the "kpng-example" service proxy name. The "to-api" parameter
+will open a gRPC server for backends.
+
+You can test this yourself outside your cluster. Please see the example
+below.
+
+Now we start a backend that simply prints the updates from the
+controller.
+
+```
+$ kubectl apply -f kpng-example.yaml
+$ kpng-json | jq     # (this is the backend)
+{
+  "Service": {
+    "Namespace": "default",
+    "Name": "kpng-example",
+    "Type": "ClusterIP",
+    "IPs": {
+      "ClusterIPs": {},
+      "ExternalIPs": {
+        "V4": [
+          "10.0.0.55"
+        ],
+        "V6": [
+          "1000::55"
+        ]
+      },
+      "Headless": true
+    },
+    "Ports": [
+      {
+        "Protocol": 1,
+        "Port": 6000,
+        "TargetPort": 6000
+      }
+    ]
+  },
+  "Endpoints": [
+    {
+      "IPs": {
+        "V6": [
+          "1100::202"
+        ]
+      },
+      "Local": true
+    },
+    {
+      "IPs": {
+        "V4": [
+          "11.0.2.2"
+        ]
+      },
+      "Local": true
+    },
+    {
+      "IPs": {
+        "V4": [
+          "11.0.1.2"
+        ]
+      }
+    },
+    {
+      "IPs": {
+        "V6": [
+          "1100::102"
+        ]
+      }
+    }
+  ]
+}
+```
+
+A real backend would use some mechanism to load-balance traffic from
+the external IPs to the endpoints.
+
+
+
+## Writing a backend
+
+The `kpng-json` backend looks like this:
+
+```go
+package main
+import (
+        "os"
+        "encoding/json"
+        "sigs.k8s.io/kpng/client"
+)
+func main() {
+        client.Run(jsonPrint)
+}
+func jsonPrint(items []*client.ServiceEndpoints) {
+        enc := json.NewEncoder(os.Stdout)
+        for _, item := range items {
+                _ = enc.Encode(item)
+        }
+}
+```
+
+(yes, that is the entire program)
+
+A real backend would of course be much more complex, but this
+illustrates how `kpng` let you focus on load-balancing.
+
+You can have several backends connected to a `kpng` controller, so
+during development or debug it can be useful to let something like the
+`kpng-json` backend run in parallel with your real backend.
+
+
+## Example
+
+
+The complete example can be found [here](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
+
+As an example we implement an "all-ip" backend. It direct all traffic
+for the externalIPs to a local endpoint, regardless of ports and upper
+layer protocols. There is a
+[KEP](https://github.com/kubernetes/enhancements/pull/2611) for this
+function and this example is a much simplified version.
+
+To direct all traffic from an external address to a local POD [only
+one iptables rule is
+needed](https://github.com/kubernetes/enhancements/pull/2611#issuecomment-895061013),
+for instance;
+
+```
+ip6tables -t nat -A PREROUTING -d 1000::55/128 -j DNAT --to-destination 1100::202
+```
+
+As you can see the addresses are in the call to the backend and all it
+have to do is:
+
+* Extract the addresses with `Local: true`
+* Setup iptables rules for the `ExternalIPs`
+
+A script doing that may look like:
+
+```
+xip=$(cat /tmp/out | jq -r .Service.IPs.ExternalIPs.V6[0])
+podip=$(cat /tmp/out | jq -r '.Endpoints[]|select(.Local == true)|select(.IPs.V6 != null)|.IPs.V6[0]')
+ip6tables -t nat -A PREROUTING -d $xip/128 -j DNAT --to-destination $podip
+```
+
+Assuming the JSON output above is stored in `/tmp/out` ([jq](https://stedolan.github.io/jq/) is an *awesome* program!).
+
+
+As this is an example we make it really simple for ourselves by using
+a minor variation of the `kpng-json` backend above. Instead of just
+printing, a program is called and the JSON output is passed as `stdin`
+to that program. The backend can be tested stand-alone:
+
+```
+CALLOUT=jq kpng-callout
+```
+
+Where `jq` can be replaced with your own program or script. A script
+may look like the example above.  For more info and the complete
+example please see [https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec](https://github.com/kubernetes-sigs/kpng/tree/master/examples/pipe-exec).
+
+
+## Summary
+
+While [kpng](https://github.com/kubernetes-sigs/kpng) is in early
+stage of development this post wants to show how you may build your
+own specialized K8s proxiers in the future. The only thing your
+applications need to do is to add the
+`service.kubernetes.io/service-proxy-name` label in the Service
+manifest.
+
+It is a tedious process to get new features into the `kube-proxy` and
+it is not unlikely that they will be rejected, so to write a
+specialized proxier may be the only option.
@@ -0,0 +1,56 @@
+---
+layout: blog
+title: "Announcing the 2021 Steering Committee Election Results"
+date: 2021-11-08
+slug: steering-committee-results-2021
+---
+
+**Author**: Kaslin Fields
+
+The [2021 Steering Committee Election](https://github.com/kubernetes/community/tree/master/events/elections/2021) is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2021. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.
+
+This community body is significant since it oversees the governance of the entire Kubernetes project. With that great power comes great responsibility. You can learn more about the steering committee’s role in their [charter](https://github.com/kubernetes/steering/blob/master/charter.md).
+
+## Results
+
+Congratulations to the elected committee members whose two year terms begin immediately (listed in alphabetical order by GitHub handle):
+
+* **Christoph Blecker ([@cblecker](https://github.com/cblecker)), Red Hat**
+* **Stephen Augustus ([@justaugustus](https://github.com/justaugustus)), Cisco**
+* **Paris Pittman ([@parispittman](https://github.com/parispittman)), Apple**
+* **Tim Pepper ([@tpepper](https://github.com/tpepper)), VMware**
+
+They join continuing members:
+
+* **Davanum Srinivas ([@dims](https://github.com/dims)), VMware**
+* **Jordan Liggitt ([@liggitt](https://github.com/liggitt)), Google**
+* **Bob Killen ([@mrbobbytables](https://github.com/mrbobbytables)), Google**
+
+Paris Pittman and Christoph Blecker are returning Steering Committee Members.
+
+## Big Thanks
+
+Thank you and congratulations on a successful election to this round’s election officers:
+
+* Alison Dowdney, ([@alisondy](https://github.com/alisondy))
+* Noah Kantrowitz ([@coderanger](https://github.com/coderanger))
+* Josh Berkus ([@jberkus](https://github.com/jberkus))
+
+Special thanks to Arnaud Meukam ([@ameukam](https://github.com/ameukam)), k8s-infra liaison, who enabled our voting software on community-owned infrastructure.
+
+Thanks to the Emeritus Steering Committee Members. Your prior service is appreciated by the community:
+
+* Derek Carr ([@derekwaynecarr](https://github.com/derekwaynecarr))
+* Nikhita Raghunath ([@nikhita](https://github.com/nikhita))
+
+And thank you to all the candidates who came forward to run for election.
+
+## Get Involved with the Steering Committee
+
+This governing body, like all of Kubernetes, is open to all. You can follow along with Steering Committee [backlog items](https://github.com/kubernetes/steering/projects/1) and weigh in by filing an issue or creating a PR against their [repo](https://github.com/kubernetes/steering). They have an open meeting on [the first Monday at 9:30am PT of every month](https://github.com/kubernetes/steering) and regularly attend Meet Our Contributors. They can also be contacted at their public mailing list steering@kubernetes.io.
+
+You can see what the Steering Committee meetings are all about by watching past meetings on the [YouTube Playlist](https://www.youtube.com/playlist?list=PL69nYSiGNLP1yP1B_nd9-drjoxp0Q14qM).
+
+---
+
+_This post was written by the [Upstream Marketing Working Group](https://github.com/kubernetes/community/tree/master/communication/marketing-team#contributor-marketing). If you want to write stories about the Kubernetes community, learn more about us._
@@ -0,0 +1,238 @@
+---
+layout: blog
+title: 'Non-root Containers And Devices'
+date: 2021-11-09
+slug: non-root-containers-and-devices
+---
+
+**Author:** Mikko Ylinen (Intel)
+
+The user/group ID related security settings in Pod's `securityContext` trigger a problem when users want to
+deploy containers that use accelerator devices (via [Kubernetes Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)) on Linux. In this blog
+post I talk about the problem and describe the work done so far to address it. It's not meant to be a long story about getting the [k/k issue](https://github.com/kubernetes/kubernetes/issues/92211) fixed.
+
+Instead, this post aims to raise awareness of the issue and to highlight important device use-cases too. This is needed as Kubernetes works on new related features such as support for user namespaces.
+
+## Why non-root containers can't use devices and why it matters
+One of the key security principles for running containers in Kubernetes is the
+principle of least privilege. The Pod/container `securityContext` specifies the config
+options to set, e.g., Linux capabilities, MAC policies, and user/group ID values to achieve this.
+
+Furthermore, the cluster admins are supported with tools like [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) (deprecated) or
+[Pod Security Admission](/docs/concepts/security/pod-security-admission/) (alpha) to enforce the desired security settings for pods that are being deployed in
+the cluster. These settings could, for instance, require that containers must be `runAsNonRoot` or
+that they are forbidden from running with root's group ID in `runAsGroup` or `supplementalGroups`.
+
+In Kubernetes, the kubelet builds the list of [`Device`](https://pkg.go.dev/k8s.io/cri-api@v0.22.1/pkg/apis/runtime/v1#Device) resources to be made available to a container
+(based on inputs from the Device Plugins) and the list is included in the CreateContainer CRI message
+sent to the CRI container runtime. Each `Device` contains little information: host/container device
+paths and the desired devices cgroups permissions.
+
+The [OCI Runtime Spec for Linux Container Configuration](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md)
+expects that in addition to the devices cgroup fields, more detailed information about the devices
+must be provided:
+
+```yaml
+{
+        "type": "<string>",
+        "path": "<string>",
+        "major": <int64>,
+        "minor": <int64>,
+        "fileMode": <uint32>,
+        "uid": <uint32>,
+        "gid": <uint32>
+},
+```
+
+The CRI container runtimes (containerd, CRI-O) are responsible for obtaining this information
+from the host for each `Device`. By default, the runtimes copy the host device's user and group IDs:
+
+- `uid` (uint32, OPTIONAL) - id of device owner in the container namespace.  
+- `gid` (uint32, OPTIONAL) - id of device group in the container namespace.
+
+Similarly, the runtimes prepare other mandatory `config.json` sections based on the CRI fields,
+including the ones defined in `securityContext`: `runAsUser`/`runAsGroup`, which become part of the POSIX
+platforms user structure via:
+
+- `uid` (int, REQUIRED) specifies the user ID in the container namespace.  
+- `gid` (int, REQUIRED) specifies the group ID in the container namespace.  
+- `additionalGids` (array of ints, OPTIONAL) specifies additional group IDs in the container namespace to be added to the process.
+
+However, the resulting `config.json` triggers a problem when trying to run containers with
+both devices added and with non-root uid/gid set via `runAsUser`/`runAsGroup`: the container user process
+has no permission to use the device even when its group id (gid, copied from host) was permissive to
+non-root groups. This is because the container user does not belong to that host group (e.g., via `additionalGids`).
+
+Being able to run applications that use devices as non-root user is normal and expected to work so that
+the security principles can be met. Therefore, several alternatives were considered to get the gap filled with what the PodSec/CRI/OCI supports today.
+
+## What was done to solve the issue?
+You might have noticed from the problem definition that it would at least be possible to workaround
+the problem by manually adding the device gid(s) to `supplementalGroups`, or in
+the case of just one device, set `runAsGroup` to the device's group id. However, this is problematic because the device gid(s) may have
+different values depending on the nodes' distro/version in the cluster. For example, with GPUs the following commands for different distros and versions return different gids:
+
+Fedora 33:
+```
+$ ls -l /dev/dri/
+total 0
+drwxr-xr-x. 2 root root         80 19.10. 10:21 by-path
+crw-rw----+ 1 root video  226,   0 19.10. 10:42 card0
+crw-rw-rw-. 1 root render 226, 128 19.10. 10:21 renderD128
+$ grep -e video -e render /etc/group
+video:x:39:
+render:x:997:
+```
+
+Ubuntu 20.04:
+```
+$ ls -l /dev/dri/
+total 0
+drwxr-xr-x 2 root root         80 19.10. 17:36 by-path
+crw-rw---- 1 root video  226,   0 19.10. 17:36 card0
+crw-rw---- 1 root render 226, 128 19.10. 17:36 renderD128
+$ grep -e video -e render /etc/group
+video:x:44:
+render:x:133:
+```
+
+Which number to choose in your `securityContext`? Also, what if the `runAsGroup`/`runAsUser` values cannot be hard-coded because
+they are automatically assigned during pod admission time via external security policies?
+
+Unlike volumes with `fsGroup`, the devices have no official notion of `deviceGroup`/`deviceUser` that the CRI runtimes (or kubelet)
+would be able to use. We considered using container annotations set by the device plugins (e.g., `io.kubernetes.cri.hostDeviceSupplementalGroup/`) to get custom OCI `config.json` uid/gid values.
+This would have required changes to all existing device plugins which was not ideal.
+
+Instead, a solution that is *seamless* to end-users without getting the device plugin vendors involved was preferred. The selected approach was
+to re-use `runAsUser` and `runAsGroup` values in `config.json` for devices:
+
+```yaml
+{
+        "type": "c",
+        "path": "/dev/foo",
+        "major": 123,
+        "minor": 4,
+        "fileMode": 438,
+        "uid": <runAsUser>,
+        "gid": <runAsGroup>
+},
+```
+
+With `runc` OCI runtime (in non-rootless mode), the device is created (`mknod(2)`) in
+the container namespace and the ownership is changed to `runAsUser`/`runAsGroup` using `chmod(2)`.
+
+{{< note >}}
+[Rootless mode](/docs/tasks/administer-cluster/kubelet-in-userns/) and devices is not supported.
+{{</note>}}
+Having the ownership updated in the container namespace is justified as the user process is the only one accessing the device. Only `runAsUser`/`runAsGroup`
+are taken into account, and, e.g., the `USER` setting in the container is currently ignored.
+
+While it is likely that the "faulty" deployments (i.e., non-root `securityContext` + devices) do not exist, to be absolutely sure no
+deployments break, an opt-in config entry in both containerd and CRI-O to enable the new behavior was added. The following:
+
+`device_ownership_from_security_context (bool)`
+
+defaults to `false` and must be enabled to use the feature.
+
+## See non-root containers using devices after the fix
+To demonstrate the new behavior, let's use a Data Plane Development Kit (DPDK) application using hardware accelerators, Kubernetes CPU manager, and HugePages as an example. The cluster runs containerd with:
+
+```toml
+[plugins]
+  [plugins."io.containerd.grpc.v1.cri"]
+    device_ownership_from_security_context = true
+```
+
+or CRI-O with:
+
+```toml
+[crio.runtime]
+device_ownership_from_security_context = true
+```
+
+and the `Guaranteed` QoS Class Pod that runs DPDK's crypto-perf test utility with this YAML:
+
+```yaml
+...
+metadata:
+  name: qat-dpdk
+spec:
+  securityContext:
+    runAsUser: 1000
+    runAsGroup: 2000
+    fsGroup: 3000
+  containers:
+  - name: crypto-perf
+    image: intel/crypto-perf:devel
+    ...
+    resources:
+      requests:
+        cpu: "3"
+        memory: "128Mi"
+        qat.intel.com/generic: '4'
+        hugepages-2Mi: "128Mi"
+      limits:
+        cpu: "3"
+        memory: "128Mi"
+        qat.intel.com/generic: '4'
+        hugepages-2Mi: "128Mi"
+  ...
+```
+
+To verify the results, check the user and group ID that the container runs as:
+
+```
+$ kubectl exec -it qat-dpdk -c crypto-perf -- id
+```
+
+They are set to non-zero values as expected:
+
+```
+uid=1000 gid=2000 groups=2000,3000
+```
+
+Next, check the device node permissions (`qat.intel.com/generic` exposes `/dev/vfio/` devices) are accessible to `runAsUser`/`runAsGroup`:
+
+```
+$ kubectl exec -it qat-dpdk -c crypto-perf -- ls -la /dev/vfio
+total 0
+drwxr-xr-x 2 root root      140 Sep  7 10:55 .
+drwxr-xr-x 7 root root      380 Sep  7 10:55 ..
+crw------- 1 1000 2000 241,   0 Sep  7 10:55 58
+crw------- 1 1000 2000 241,   2 Sep  7 10:55 60
+crw------- 1 1000 2000 241,  10 Sep  7 10:55 68
+crw------- 1 1000 2000 241,  11 Sep  7 10:55 69
+crw-rw-rw- 1 1000 2000  10, 196 Sep  7 10:55 vfio
+```
+
+Finally, check the non-root container is also allowed to create HugePages:
+
+```
+$ kubectl exec -it qat-dpdk -c crypto-perf -- ls -la /dev/hugepages/
+```
+
+`fsGroup` gives a `runAsUser` writable HugePages emptyDir mountpoint:
+
+```
+total 0
+drwxrwsr-x 2 root 3000   0 Sep  7 10:55 .
+drwxr-xr-x 7 root root 380 Sep  7 10:55 ..
+```
+
+## Help us test it and provide feedback!
+The functionality described here is expected to help with cluster security and the configurability of device permissions. To allow
+non-root containers to use devices requires cluster admins to opt-in to the functionality by setting
+`device_ownership_from_security_context = true`. To make it a default setting, please test it and provide your feedback (via SIG-Node meetings or issues)!
+The flag is available in CRI-O v1.22 release and queued for containerd v1.6.
+
+More work is needed to get it *properly* supported. It is known to work with `runc` but it also needs to be made to function
+with other OCI runtimes too, where applicable. For instance, Kata Containers supports device passthrough and allows it to make devices
+available to containers in VM sandboxes too.
+
+Moreover, the additional challenge comes with support of user names and devices. This problem is still [open](https://github.com/kubernetes/enhancements/pull/2101)
+and requires more brainstorming.
+
+Finally, it needs to be understood whether `runAsUser`/`runAsGroup` are enough or if device specific settings similar to `fsGroups` are needed in PodSpec/CRI v2.
+
+## Thanks
+My thanks goes to Mike Brown (IBM, containerd), Peter Hunt (Redhat, CRI-O), and Alexander Kanevskiy (Intel) for providing all the feedback and good conversations.
@@ -0,0 +1,59 @@
+---
+layout: blog
+title: "Dockershim removal is coming. Are you ready?"
+date: 2021-11-12
+slug: are-you-ready-for-dockershim-removal
+---
+
+**Author:** Sergey Kanzhelev, Google. With reviews from Davanum Srinivas, Elana Hashman, Noah Kantrowitz, Rey Lejano.
+
+Last year we announced that Dockershim is being deprecated: [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/).
+Our current plan is to remove dockershim from the Kubernetes codebase soon.
+We are looking for feedback from you whether you are ready for dockershim
+removal and to ensure that you are ready when the time comes.
+**Please fill out this survey: https://forms.gle/svCJmhvTv78jGdSx8**.
+
+The dockershim component that enables Docker as a Kubernetes container runtime is
+being deprecated in favor of runtimes that directly use the [Container Runtime Interface](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)
+created for Kubernetes. Many Kubernetes users have migrated to
+other container runtimes without problems. However we see that dockershim is
+still very popular. You may see some public numbers in recent [Container Report](https://www.datadoghq.com/container-report/#8) from DataDog.
+Some Kubernetes hosting vendors just recently enabled other runtimes support
+(especially for Windows nodes). And we know that many third party tools vendors
+are still not ready: [migrating telemetry and security agents](/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents/#telemetry-and-security-agent-vendors).
+
+At this point, we believe that there is feature parity between Docker and the
+other runtimes. Many end-users have used our [migration guide](/docs/tasks/administer-cluster/migrating-from-dockershim/)
+and are running production workload using these different runtimes. The plan of
+record today is that dockershim will be removed in version 1.24, slated for 
+release around April of next year. For those developing or running alpha and
+beta versions, dockershim will be removed in December at the beginning of the
+1.24 release development cycle.
+
+There is only one month left to give us feedback. We want you to tell us how
+ready you are.
+
+**We are collecting opinions through this survey: [https://forms.gle/svCJmhvTv78jGdSx8](https://forms.gle/svCJmhvTv78jGdSx8)**
+To better understand preparedness for the dockershim removal, our survey is
+asking the version of Kubernetes you are currently using, and an estimate of
+when you think you will adopt Kubernetes 1.24. All the aggregated information
+on dockershim removal readiness will be published.
+Free form comments will be reviewed by SIG Node leadership. If you want to
+discuss any details of migrating from dockershim, report bugs or adoption
+blockers, you can use one of the SIG Node contact options any time:
+https://github.com/kubernetes/community/tree/master/sig-node#contact
+
+Kubernetes is a mature project. This deprecation is another
+step in the effort to get away from permanent beta features and providing more
+stability and compatibility guarantees. With the migration from dockershim you
+will get more flexibility and choice of container runtime features as well as
+less dependencies of your apps on specific underlying technology. Please take
+time to review the [dockershim migration documentation](/docs/tasks/administer-cluster/migrating-from-dockershim/)
+and consult your Kubernetes hosting vendor (if you have one) what container runtime options are available for you.
+Read up [container runtime documentation with instructions on how to use containerd and CRI-O](/docs/setup/production-environment/container-runtimes/#container-runtimes)
+to help prepare you when you're ready to upgrade to 1.24. CRI-O, containerd, and
+Docker with [Mirantis cri-dockerd](https://github.com/Mirantis/cri-dockerd) are
+not the only container runtime options, we encourage you to explore the [CNCF landscape on container runtimes](https://landscape.cncf.io/card-mode?category=container-runtime&grouping=category)
+in case another suits you better.
+
+Thank you!
@@ -0,0 +1,87 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- Generated by CodeCogs with dvisvgm 2.9.1 -->
+<svg version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='389.559851pt' height='13.50934pt' viewBox='-.239051 -.240635 389.559851 13.50934'>
+<defs>
+<path id='g1-61' d='M8.069738-3.873474C8.237111-3.873474 8.452304-3.873474 8.452304-4.088667C8.452304-4.315816 8.249066-4.315816 8.069738-4.315816H1.028144C.860772-4.315816 .645579-4.315816 .645579-4.100623C.645579-3.873474 .848817-3.873474 1.028144-3.873474H8.069738ZM8.069738-1.649813C8.237111-1.649813 8.452304-1.649813 8.452304-1.865006C8.452304-2.092154 8.249066-2.092154 8.069738-2.092154H1.028144C.860772-2.092154 .645579-2.092154 .645579-1.876961C.645579-1.649813 .848817-1.649813 1.028144-1.649813H8.069738Z'/>
+<path id='g1-91' d='M2.988792 2.988792V2.546451H1.829141V-8.524035H2.988792V-8.966376H1.3868V2.988792H2.988792Z'/>
+<path id='g1-93' d='M1.853051-8.966376H.251059V-8.524035H1.41071V2.546451H.251059V2.988792H1.853051V-8.966376Z'/>
+<path id='g0-58' d='M2.199751-.573848C2.199751-.920548 1.912827-1.159651 1.625903-1.159651C1.279203-1.159651 1.0401-.872727 1.0401-.585803C1.0401-.239103 1.327024 0 1.613948 0C1.960648 0 2.199751-.286924 2.199751-.573848Z'/>
+<path id='g0-97' d='M3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665ZM4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.944458 4.27995-1.075965 4.387547-1.542217C4.471233-1.841096 4.531009-2.10411 4.62665-2.450809C5.068991-4.244085 5.176588-4.674471 5.176588-4.746202C5.176588-4.913574 5.045081-5.045081 4.865753-5.045081C4.483188-5.045081 4.387547-4.62665 4.363636-4.483188Z'/>
+<path id='g0-99' d='M4.674471-4.495143C4.447323-4.495143 4.339726-4.495143 4.172354-4.351681C4.100623-4.291905 3.969116-4.112578 3.969116-3.921295C3.969116-3.682192 4.148443-3.53873 4.375592-3.53873C4.662516-3.53873 4.985305-3.777833 4.985305-4.25604C4.985305-4.829888 4.435367-5.272229 3.610461-5.272229C2.044334-5.272229 .478207-3.56264 .478207-1.865006C.478207-.824907 1.123786 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.542217-.119552 1.279203-.836862 1.279203-1.43462C1.279203-1.853051 1.482441-3.012702 1.912827-3.825654C2.223661-4.387547 2.86924-5.033126 3.622416-5.033126C3.777833-5.033126 4.435367-5.009215 4.674471-4.495143Z'/>
+<path id='g0-100' d='M6.01345-7.998007C6.025405-8.045828 6.049315-8.117559 6.049315-8.177335C6.049315-8.296887 5.929763-8.296887 5.905853-8.296887C5.893898-8.296887 5.308095-8.249066 5.248319-8.237111C5.045081-8.225156 4.865753-8.201245 4.65056-8.18929C4.351681-8.16538 4.267995-8.153425 4.267995-7.938232C4.267995-7.81868 4.363636-7.81868 4.531009-7.81868C5.116812-7.81868 5.128767-7.711083 5.128767-7.591532C5.128767-7.519801 5.104857-7.424159 5.092902-7.388294L4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.71731 4.244085-.968369 4.327771-1.303113L6.01345-7.998007ZM3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665Z'/>
+<path id='g0-101' d='M2.139975-2.773599C2.462765-2.773599 3.275716-2.797509 3.849564-3.012702C4.758157-3.359402 4.841843-4.052802 4.841843-4.267995C4.841843-4.794022 4.387547-5.272229 3.598506-5.272229C2.343213-5.272229 .537983-4.136488 .537983-2.008468C.537983-.753176 1.255293 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.685679-.119552 1.327024-.657534 1.327024-1.542217C1.327024-1.709589 1.327024-2.008468 1.506351-2.773599H2.139975ZM1.566127-3.012702C2.080199-4.853798 3.21594-5.033126 3.598506-5.033126C4.124533-5.033126 4.483188-4.722291 4.483188-4.267995C4.483188-3.012702 2.570361-3.012702 2.068244-3.012702H1.566127Z'/>
+<path id='g0-105' d='M3.383313-1.709589C3.383313-1.769365 3.335492-1.817186 3.263761-1.817186C3.156164-1.817186 3.144209-1.78132 3.084433-1.578082C2.773599-.490162 2.283437-.119552 1.888917-.119552C1.745455-.119552 1.578082-.155417 1.578082-.514072C1.578082-.836862 1.721544-1.195517 1.853051-1.554172L2.689913-3.777833C2.725778-3.873474 2.809465-4.088667 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 1.984558-3.93325 1.888917-3.670237L1.052055-1.446575C.980324-1.255293 .908593-1.06401 .908593-.848817C.908593-.310834 1.279203 .119552 1.853051 .119552C2.952927 .119552 3.383313-1.625903 3.383313-1.709589ZM3.287671-7.460025C3.287671-7.639352 3.144209-7.854545 2.881196-7.854545C2.606227-7.854545 2.295392-7.591532 2.295392-7.280697C2.295392-6.981818 2.546451-6.886177 2.689913-6.886177C3.012702-6.886177 3.287671-7.197011 3.287671-7.460025Z'/>
+<path id='g0-109' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-4.124533 5.057036-3.88543 4.913574-3.323537L4.614695-2.092154C4.519054-1.733499 4.291905-.848817 4.267995-.71731C4.220174-.537983 4.148443-.227148 4.148443-.179328C4.148443-.011955 4.27995 .119552 4.459278 .119552C4.817933 .119552 4.877709-.155417 4.985305-.585803L5.702615-3.443088C5.726526-3.53873 6.348194-5.033126 7.663263-5.033126C8.141469-5.033126 8.308842-4.674471 8.308842-4.196264C8.308842-3.526775 7.84259-2.223661 7.579577-1.506351C7.47198-1.219427 7.412204-1.06401 7.412204-.848817C7.412204-.310834 7.782814 .119552 8.356663 .119552C9.468493 .119552 9.886924-1.637858 9.886924-1.709589C9.886924-1.769365 9.839103-1.817186 9.767372-1.817186C9.659776-1.817186 9.647821-1.78132 9.588045-1.578082C9.313076-.621669 8.870735-.119552 8.392528-.119552C8.272976-.119552 8.081694-.131507 8.081694-.514072C8.081694-.824907 8.225156-1.207472 8.272976-1.338979C8.488169-1.912827 9.026152-3.323537 9.026152-4.016936C9.026152-4.734247 8.607721-5.272229 7.699128-5.272229C6.898132-5.272229 6.252553-4.817933 5.774346-4.112578C5.738481-4.758157 5.34396-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.207472-5.272229 .932503-5.104857 .705355-4.65056C.490162-4.220174 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.327771 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/>
+<path id='g0-110' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-3.514819 4.566874-2.15193 4.327771-1.506351C4.220174-1.219427 4.160399-1.06401 4.160399-.848817C4.160399-.310834 4.531009 .119552 5.104857 .119552C6.216687 .119552 6.635118-1.637858 6.635118-1.709589C6.635118-1.769365 6.587298-1.817186 6.515567-1.817186C6.40797-1.817186 6.396015-1.78132 6.336239-1.578082C6.06127-.597758 5.606974-.119552 5.140722-.119552C5.021171-.119552 4.829888-.131507 4.829888-.514072C4.829888-.812951 4.961395-1.171606 5.033126-1.338979C5.272229-1.996513 5.774346-3.335492 5.774346-4.016936C5.774346-4.734247 5.355915-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.171606-5.272229 .908593-5.045081 .705355-4.638605C.490162-4.208219 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.824907-4.351681 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/>
+<path id='g0-111' d='M5.451557-3.287671C5.451557-4.423412 4.710336-5.272229 3.622416-5.272229C2.044334-5.272229 .490162-3.550685 .490162-1.865006C.490162-.729265 1.231382 .119552 2.319303 .119552C3.90934 .119552 5.451557-1.601993 5.451557-3.287671ZM2.331258-.119552C1.733499-.119552 1.291158-.597758 1.291158-1.43462C1.291158-1.984558 1.578082-3.203985 1.912827-3.801743C2.450809-4.722291 3.120299-5.033126 3.610461-5.033126C4.196264-5.033126 4.65056-4.554919 4.65056-3.718057C4.65056-3.239851 4.399502-1.960648 3.945205-1.231382C3.455044-.430386 2.797509-.119552 2.331258-.119552Z'/>
+<path id='g0-112' d='M.514072 1.518306C.430386 1.876961 .382565 1.972603-.107597 1.972603C-.251059 1.972603-.37061 1.972603-.37061 2.199751C-.37061 2.223661-.358655 2.319303-.227148 2.319303C-.071731 2.319303 .095641 2.295392 .251059 2.295392H.765131C1.016189 2.295392 1.625903 2.319303 1.876961 2.319303C1.948692 2.319303 2.092154 2.319303 2.092154 2.10411C2.092154 1.972603 2.008468 1.972603 1.80523 1.972603C1.255293 1.972603 1.219427 1.888917 1.219427 1.793275C1.219427 1.649813 1.75741-.406476 1.829141-.681445C1.960648-.3467 2.283437 .119552 2.905106 .119552C4.25604 .119552 5.71457-1.637858 5.71457-3.395268C5.71457-4.495143 5.092902-5.272229 4.196264-5.272229C3.431133-5.272229 2.785554-4.531009 2.654047-4.363636C2.558406-4.961395 2.092154-5.272229 1.613948-5.272229C1.267248-5.272229 .992279-5.104857 .765131-4.65056C.549938-4.220174 .382565-3.490909 .382565-3.443088S.430386-3.335492 .514072-3.335492C.609714-3.335492 .621669-3.347447 .6934-3.622416C.872727-4.327771 1.099875-5.033126 1.578082-5.033126C1.853051-5.033126 1.948692-4.841843 1.948692-4.483188C1.948692-4.196264 1.912827-4.076712 1.865006-3.861519L.514072 1.518306ZM2.582316-3.730012C2.666002-4.064757 3.000747-4.411457 3.19203-4.578829C3.323537-4.698381 3.718057-5.033126 4.172354-5.033126C4.698381-5.033126 4.937484-4.507098 4.937484-3.88543C4.937484-3.311582 4.60274-1.960648 4.303861-1.338979C4.004981-.6934 3.455044-.119552 2.905106-.119552C2.092154-.119552 1.960648-1.147696 1.960648-1.195517C1.960648-1.231382 1.984558-1.327024 1.996513-1.3868L2.582316-3.730012Z'/>
+<path id='g0-113' d='M5.272229-5.152677C5.272229-5.212453 5.224408-5.260274 5.164633-5.260274C5.068991-5.260274 4.60274-4.829888 4.375592-4.411457C4.160399-4.94944 3.789788-5.272229 3.275716-5.272229C1.924782-5.272229 .466252-3.526775 .466252-1.75741C.466252-.573848 1.159651 .119552 1.972603 .119552C2.606227 .119552 3.132254-.358655 3.383313-.633624L3.395268-.621669L2.940971 1.171606L2.833375 1.601993C2.725778 1.960648 2.546451 1.960648 1.984558 1.972603C1.853051 1.972603 1.733499 1.972603 1.733499 2.199751C1.733499 2.283437 1.80523 2.319303 1.888917 2.319303C2.056289 2.319303 2.271482 2.295392 2.438854 2.295392H3.658281C3.837609 2.295392 4.040847 2.319303 4.220174 2.319303C4.291905 2.319303 4.435367 2.319303 4.435367 2.092154C4.435367 1.972603 4.339726 1.972603 4.160399 1.972603C3.598506 1.972603 3.56264 1.888917 3.56264 1.793275C3.56264 1.733499 3.574595 1.721544 3.610461 1.566127L5.272229-5.152677ZM3.58655-1.422665C3.526775-1.219427 3.526775-1.195517 3.359402-.968369C3.096389-.633624 2.570361-.119552 2.008468-.119552C1.518306-.119552 1.243337-.561893 1.243337-1.267248C1.243337-1.924782 1.613948-3.263761 1.841096-3.765878C2.247572-4.60274 2.809465-5.033126 3.275716-5.033126C4.064757-5.033126 4.220174-4.052802 4.220174-3.957161C4.220174-3.945205 4.184309-3.789788 4.172354-3.765878L3.58655-1.422665Z'/>
+<path id='g0-114' d='M4.65056-4.889664C4.27995-4.817933 4.088667-4.554919 4.088667-4.291905C4.088667-4.004981 4.315816-3.90934 4.483188-3.90934C4.817933-3.90934 5.092902-4.196264 5.092902-4.554919C5.092902-4.937484 4.722291-5.272229 4.124533-5.272229C3.646326-5.272229 3.096389-5.057036 2.594271-4.327771C2.510585-4.961395 2.032379-5.272229 1.554172-5.272229C1.08792-5.272229 .848817-4.913574 .705355-4.65056C.502117-4.220174 .32279-3.502864 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.339726 1.0401-5.033126 1.518306-5.033126C1.80523-5.033126 1.888917-4.829888 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.338979 .119552 1.566127 .035866 1.637858-.203238C1.673724-.298879 2.116065-2.10411 2.187796-2.379078C2.247572-2.642092 2.319303-2.893151 2.379078-3.156164C2.426899-3.323537 2.47472-3.514819 2.510585-3.670237C2.546451-3.777833 2.86924-4.363636 3.16812-4.62665C3.311582-4.758157 3.622416-5.033126 4.112578-5.033126C4.303861-5.033126 4.495143-4.99726 4.65056-4.889664Z'/>
+<path id='g0-115' d='M2.725778-2.391034C2.929016-2.355168 3.251806-2.283437 3.323537-2.271482C3.478954-2.223661 4.016936-2.032379 4.016936-1.458531C4.016936-1.08792 3.682192-.119552 2.295392-.119552C2.044334-.119552 1.147696-.155417 .908593-.812951C1.3868-.753176 1.625903-1.123786 1.625903-1.3868C1.625903-1.637858 1.458531-1.769365 1.219427-1.769365C.956413-1.769365 .609714-1.566127 .609714-1.028144C.609714-.32279 1.327024 .119552 2.283437 .119552C4.100623 .119552 4.638605-1.219427 4.638605-1.841096C4.638605-2.020423 4.638605-2.355168 4.25604-2.737733C3.957161-3.024658 3.670237-3.084433 3.024658-3.21594C2.701868-3.287671 2.187796-3.395268 2.187796-3.93325C2.187796-4.172354 2.402989-5.033126 3.53873-5.033126C4.040847-5.033126 4.531009-4.841843 4.65056-4.411457C4.124533-4.411457 4.100623-3.957161 4.100623-3.945205C4.100623-3.694147 4.327771-3.622416 4.435367-3.622416C4.60274-3.622416 4.937484-3.753923 4.937484-4.25604S4.483188-5.272229 3.550685-5.272229C1.984558-5.272229 1.566127-4.040847 1.566127-3.550685C1.566127-2.642092 2.450809-2.450809 2.725778-2.391034Z'/>
+<path id='g0-116' d='M2.402989-4.805978H3.502864C3.730012-4.805978 3.849564-4.805978 3.849564-5.021171C3.849564-5.152677 3.777833-5.152677 3.53873-5.152677H2.486675L2.929016-6.898132C2.976837-7.065504 2.976837-7.089415 2.976837-7.173101C2.976837-7.364384 2.82142-7.47198 2.666002-7.47198C2.570361-7.47198 2.295392-7.436115 2.199751-7.053549L1.733499-5.152677H.609714C.37061-5.152677 .263014-5.152677 .263014-4.925529C.263014-4.805978 .3467-4.805978 .573848-4.805978H1.637858L.848817-1.649813C.753176-1.231382 .71731-1.111831 .71731-.956413C.71731-.394521 1.111831 .119552 1.78132 .119552C2.988792 .119552 3.634371-1.625903 3.634371-1.709589C3.634371-1.78132 3.58655-1.817186 3.514819-1.817186C3.490909-1.817186 3.443088-1.817186 3.419178-1.769365C3.407223-1.75741 3.395268-1.745455 3.311582-1.554172C3.060523-.956413 2.510585-.119552 1.817186-.119552C1.458531-.119552 1.43462-.418431 1.43462-.681445C1.43462-.6934 1.43462-.920548 1.470486-1.06401L2.402989-4.805978Z'/>
+<path id='g0-117' d='M4.076712-.6934C4.23213-.02391 4.805978 .119552 5.092902 .119552C5.475467 .119552 5.762391-.131507 5.953674-.537983C6.156912-.968369 6.312329-1.673724 6.312329-1.709589C6.312329-1.769365 6.264508-1.817186 6.192777-1.817186C6.085181-1.817186 6.073225-1.75741 6.025405-1.578082C5.810212-.753176 5.595019-.119552 5.116812-.119552C4.758157-.119552 4.758157-.514072 4.758157-.669489C4.758157-.944458 4.794022-1.06401 4.913574-1.566127C4.99726-1.888917 5.080946-2.211706 5.152677-2.546451L5.642839-4.495143C5.726526-4.794022 5.726526-4.817933 5.726526-4.853798C5.726526-5.033126 5.583064-5.152677 5.403736-5.152677C5.057036-5.152677 4.97335-4.853798 4.901619-4.554919C4.782067-4.088667 4.136488-1.518306 4.052802-1.099875C4.040847-1.099875 3.574595-.119552 2.701868-.119552C2.080199-.119552 1.960648-.657534 1.960648-1.099875C1.960648-1.78132 2.295392-2.737733 2.606227-3.53873C2.749689-3.921295 2.809465-4.076712 2.809465-4.315816C2.809465-4.829888 2.438854-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.578829 1.374844-5.033126 1.829141-5.033126C1.948692-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.303113-2.10411 1.243337-1.649813 1.243337-1.291158C1.243337-.071731 2.163885 .119552 2.654047 .119552C3.419178 .119552 3.837609-.406476 4.076712-.6934Z'/>
+<path id='g0-121' d='M3.144209 1.338979C2.82142 1.793275 2.355168 2.199751 1.769365 2.199751C1.625903 2.199751 1.052055 2.175841 .872727 1.625903C.908593 1.637858 .968369 1.637858 .992279 1.637858C1.350934 1.637858 1.590037 1.327024 1.590037 1.052055S1.362889 .681445 1.183562 .681445C.992279 .681445 .573848 .824907 .573848 1.41071C.573848 2.020423 1.08792 2.438854 1.769365 2.438854C2.964882 2.438854 4.172354 1.338979 4.507098 .011955L5.678705-4.65056C5.69066-4.710336 5.71457-4.782067 5.71457-4.853798C5.71457-5.033126 5.571108-5.152677 5.391781-5.152677C5.284184-5.152677 5.033126-5.104857 4.937484-4.746202L4.052802-1.231382C3.993026-1.016189 3.993026-.992279 3.897385-.860772C3.658281-.526027 3.263761-.119552 2.689913-.119552C2.020423-.119552 1.960648-.777086 1.960648-1.099875C1.960648-1.78132 2.283437-2.701868 2.606227-3.56264C2.737733-3.90934 2.809465-4.076712 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.033126 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.243337-1.960648 1.243337-1.566127 1.243337-1.279203C1.243337-.143462 2.056289 .119552 2.654047 .119552C3.000747 .119552 3.431133 .011955 3.849564-.430386L3.861519-.418431C3.682192 .286924 3.56264 .753176 3.144209 1.338979Z'/>
+</defs>
+<g id='page1' transform='matrix(1.13 0 0 1.13 -63.986043 -64.41)'>
+<use x='56.413267' y='65.753425' xlink:href='#g0-109'/>
+<use x='66.652534' y='65.753425' xlink:href='#g0-101'/>
+<use x='72.077974' y='65.753425' xlink:href='#g0-109'/>
+<use x='82.317241' y='65.753425' xlink:href='#g0-111'/>
+<use x='87.944679' y='65.753425' xlink:href='#g0-114'/>
+<use x='93.545152' y='65.753425' xlink:href='#g0-121'/>
+<use x='99.681804' y='65.753425' xlink:href='#g0-58'/>
+<use x='102.933465' y='65.753425' xlink:href='#g0-109'/>
+<use x='113.172732' y='65.753425' xlink:href='#g0-105'/>
+<use x='117.166164' y='65.753425' xlink:href='#g0-110'/>
+<use x='127.474599' y='65.753425' xlink:href='#g1-61'/>
+<use x='139.90008' y='65.753425' xlink:href='#g0-112'/>
+<use x='145.775223' y='65.753425' xlink:href='#g0-111'/>
+<use x='151.402661' y='65.753425' xlink:href='#g0-100'/>
+<use x='157.485354' y='65.753425' xlink:href='#g0-58'/>
+<use x='160.737015' y='65.753425' xlink:href='#g0-115'/>
+<use x='166.251021' y='65.753425' xlink:href='#g0-112'/>
+<use x='172.126164' y='65.753425' xlink:href='#g0-101'/>
+<use x='177.551604' y='65.753425' xlink:href='#g0-99'/>
+<use x='182.589592' y='65.753425' xlink:href='#g0-58'/>
+<use x='185.841254' y='65.753425' xlink:href='#g0-99'/>
+<use x='190.879242' y='65.753425' xlink:href='#g0-111'/>
+<use x='196.50668' y='65.753425' xlink:href='#g0-110'/>
+<use x='203.494285' y='65.753425' xlink:href='#g0-116'/>
+<use x='207.721445' y='65.753425' xlink:href='#g0-97'/>
+<use x='213.866389' y='65.753425' xlink:href='#g0-105'/>
+<use x='217.859822' y='65.753425' xlink:href='#g0-110'/>
+<use x='224.847427' y='65.753425' xlink:href='#g0-101'/>
+<use x='230.272867' y='65.753425' xlink:href='#g0-114'/>
+<use x='235.873341' y='65.753425' xlink:href='#g0-115'/>
+<use x='241.387347' y='65.753425' xlink:href='#g1-91'/>
+<use x='244.639008' y='65.753425' xlink:href='#g0-105'/>
+<use x='248.63244' y='65.753425' xlink:href='#g1-93'/>
+<use x='251.884101' y='65.753425' xlink:href='#g0-58'/>
+<use x='255.135763' y='65.753425' xlink:href='#g0-114'/>
+<use x='260.736236' y='65.753425' xlink:href='#g0-101'/>
+<use x='266.161676' y='65.753425' xlink:href='#g0-115'/>
+<use x='271.675682' y='65.753425' xlink:href='#g0-111'/>
+<use x='277.303119' y='65.753425' xlink:href='#g0-117'/>
+<use x='283.965559' y='65.753425' xlink:href='#g0-114'/>
+<use x='289.566032' y='65.753425' xlink:href='#g0-99'/>
+<use x='294.604021' y='65.753425' xlink:href='#g0-101'/>
+<use x='300.029461' y='65.753425' xlink:href='#g0-115'/>
+<use x='305.543467' y='65.753425' xlink:href='#g0-58'/>
+<use x='308.795128' y='65.753425' xlink:href='#g0-114'/>
+<use x='314.395601' y='65.753425' xlink:href='#g0-101'/>
+<use x='319.821042' y='65.753425' xlink:href='#g0-113'/>
+<use x='325.440198' y='65.753425' xlink:href='#g0-117'/>
+<use x='332.102638' y='65.753425' xlink:href='#g0-101'/>
+<use x='337.528078' y='65.753425' xlink:href='#g0-115'/>
+<use x='343.042083' y='65.753425' xlink:href='#g0-116'/>
+<use x='347.269243' y='65.753425' xlink:href='#g0-115'/>
+<use x='352.783249' y='65.753425' xlink:href='#g1-91'/>
+<use x='356.03491' y='65.753425' xlink:href='#g0-109'/>
+<use x='366.274177' y='65.753425' xlink:href='#g0-101'/>
+<use x='371.699617' y='65.753425' xlink:href='#g0-109'/>
+<use x='381.938884' y='65.753425' xlink:href='#g0-111'/>
+<use x='387.566322' y='65.753425' xlink:href='#g0-114'/>
+<use x='393.166795' y='65.753425' xlink:href='#g0-121'/>
+<use x='399.303447' y='65.753425' xlink:href='#g1-93'/>
+</g>
+</svg>
@@ -0,0 +1,118 @@
+---
+layout: blog
+title: 'Quality-of-Service for Memory Resources'
+date: 2021-11-26
+slug: qos-memory-resources
+---
+
+**Authors:** Tim Xu (Tencent Cloud)
+
+Kubernetes v1.22, released in August 2021, introduced a new alpha feature that improves how Linux nodes implement memory resource requests and limits.
+
+In prior releases, Kubernetes did not support memory quality guarantees. 
+For example, if you set container resources as follows:
+```
+apiVersion: v1
+kind: Pod
+metadata:
+  name: example
+spec:
+  containers:
+  - name: nginx
+    resources:
+      requests:
+        memory: "64Mi"
+        cpu: "250m"
+      limits:
+        memory: "64Mi"
+        cpu: "500m"
+```
+`spec.containers[].resources.requests`(e.g. cpu, memory) is designed for scheduling. When you create a Pod, the Kubernetes scheduler selects a node for the Pod to run on. Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for Pods. The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled Containers is less than the capacity of the node.
+
+`spec.containers[].resources.limits` is passed to the container runtime when the kubelet starts a container. CPU is considered a "compressible" resource. If your app starts hitting your CPU limits, Kubernetes starts throttling your container, giving your app potentially worse performance. However, it won’t be terminated. That is what "compressible" means.
+
+In cgroup v1, and prior to this feature, the container runtime never took into account and effectively ignored spec.containers[].resources.requests["memory"]. This is unlike CPU, in which the container runtime consider both requests and limits. Furthermore, memory actually can't be compressed in cgroup v1. Because there is no way to throttle memory usage, if a container goes past its memory limit it will be terminated by the kernel with an OOM (Out of Memory) kill.
+
+Fortunately, cgroup v2 brings a new design and implementation to achieve full protection on memory. The new feature relies on cgroups v2 which most current operating system releases for Linux already provide. With this experimental feature, [quality-of-service for pods and containers](/docs/tasks/configure-pod-container/quality-service-pod/) extends to cover not just CPU time but memory as well.
+
+## How does it work?
+Memory QoS uses the memory controller of cgroup v2 to guarantee memory resources in Kubernetes. Memory requests and limits of containers in pod are used to set specific interfaces `memory.min` and `memory.high` provided by the memory controller. When `memory.min` is set to memory requests, memory resources are reserved and never reclaimed by the kernel; this is how Memory QoS ensures the availability of memory for Kubernetes pods. And if memory limits are set in the container, this means that the system needs to limit container memory usage, Memory QoS uses `memory.high` to throttle workload approaching it's memory limit, ensuring that the system is not overwhelmed by instantaneous memory allocation.
+
+![](./memory-qos-cal.svg)
+
+The following table details the specific functions of these two parameters and how they correspond to Kubernetes container resources.
+
+<table>
+    <tr>
+        <th style="text-align:center">File</th>
+        <th style="text-align:center">Description</th>
+   </tr>
+   <tr>
+        <td>memory.min</td>
+        <td><code>memory.min</code> specifies a minimum amount of memory the cgroup must always retain, i.e., memory that can never be reclaimed by the system. If the cgroup's memory usage reaches this low limit and can’t be increased, the system OOM killer will be invoked.
+        <br>
+        <br>
+        <i>We map it to the container's memory request</i>
+        </td>
+   </tr>
+   <tr>
+       <td>memory.high</td>
+       <td><code>memory.high</code> is the memory usage throttle limit. This is the main mechanism to control a cgroup's memory use. If a cgroup's memory use goes over the high boundary specified here, the cgroup’s processes are throttled and put under heavy reclaim pressure. The default is max, meaning there is no limit. 
+       <br>
+       <br>
+       <i>We use a formula to calculate <code>memory.high</code>, depending on container's memory limit or node allocatable memory (if container's memory limit is empty) and a throttling factor. Please refer to the KEP for more details on the formula.</i>
+       </td>
+   </tr>
+</table>
+
+When container memory requests are made, kubelet passes `memory.min` to the back-end CRI runtime (possibly containerd, cri-o) via the `Unified` field in CRI during container creation. The `memory.min` in container level cgroup will be set to:  
+
+![](./container-memory-min.svg)  
+<sub>i: the i<sup>th</sup> container in one pod</sub>
+
+Since the `memory.min` interface requires that the ancestor cgroup directories are all set, the pod and node cgroup directories need to be set correctly. 
+
+`memory.min` in pod level cgroup:  
+![](./pod-memory-min.svg)  
+<sub>i: the i<sup>th</sup> container in one pod</sub>
+
+`memory.min` in node level cgroup:  
+![](./node-memory-min.svg)  
+<sub>i: the i<sup>th</sup> pod in one node, j: the j<sup>th</sup> container in one pod</sub>
+
+Kubelet will manage the cgroup hierarchy of the pod level and node level cgroups directly using runc libcontainer library, while container cgroup limits are managed by the container runtime.
+
+For memory limits, in addition to the original way of limiting memory usage, Memory QoS adds an additional feature of throttling memory allocation. A throttling factor is introduced as a multiplier (default is 0.8). If the result of multiplying memory limits by the factor is greater than memory requests, kubelet will set `memory.high` to the value and use `Unified` via CRI. And if the container does not specify memory limits, kubelet will use node allocatable memory instead. The `memory.high` in container level cgroup is set to:
+
+![](./container-memory-high.svg)  
+<sub>i: the i<sup>th</sup> container in one pod</sub>
+
+This can can help improve stability when pod memory usage increases, ensuring that memory is throttled as it approaches the memory limit.
+
+## How do I use it?
+Here are the prerequisites for enabling Memory QoS on your Linux node, some of these are related to [Kubernetes support for cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2254-cgroup-v2).
+
+1. Kubernetes since v1.22
+2. [runc](https://github.com/opencontainers/runc) since v1.0.0-rc93; [containerd](https://containerd.io/) since 1.4; [cri-o](https://cri-o.io/) since 1.20
+3. Linux kernel minimum version: 4.15, recommended version: 5.2+
+4. Linux image with cgroupv2 enabled or enabling cgroupv2 unified_cgroup_hierarchy manually
+
+OCI runtimes such as runc and crun already support cgroups v2 [`Unified`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#unified), and Kubernetes CRI has also made the desired changes to support passing [`Unified`](https://github.com/kubernetes/kubernetes/pull/102578). However, CRI Runtime support is required as well. Memory QoS in Alpha phase is designed to support containerd and cri-o. Related PR [Feature: containerd-cri support LinuxContainerResources.Unified #5627](https://github.com/containerd/containerd/pull/5627) has been merged and will be released in containerd 1.6. CRI-O [implement kube alpha features for 1.22 #5207](https://github.com/cri-o/cri-o/pull/5207) is still in WIP.
+
+With those prerequisites met, you can enable the memory QoS feature gate (see [Set kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/)).
+    
+## How can I learn more?
+
+You can find more details as follows:
+- [Support Memory QoS with cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos/#readme)
+- [cgroup v2](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2254-cgroup-v2/#readme)
+
+## How do I get involved?
+You can reach SIG Node by several means:
+- Slack: [#sig-node](https://kubernetes.slack.com/messages/sig-node)
+- [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-node)
+- [Open Community Issues/PRs](https://github.com/kubernetes/community/labels/sig%2Fnode)
+
+You can also contact me directly:
+- GitHub / Slack: @xiaoxubeii
+- Email: xiaoxubeii@gmail.com
@@ -0,0 +1,98 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- Generated by CodeCogs with dvisvgm 2.9.1 -->
+<svg version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='446.671233pt' height='30.306891pt' viewBox='-.239051 -.248234 446.671233 30.306891'>
+<defs>
+<path id='g0-88' d='M15.135243 16.737235L16.581818 12.911582H16.282939C15.816687 14.154919 14.54944 14.96787 13.174595 15.326526C12.923537 15.386301 11.75193 15.697136 9.456538 15.697136H2.247572L8.332752 8.5599C8.416438 8.464259 8.440349 8.428394 8.440349 8.368618C8.440349 8.344707 8.440349 8.308842 8.356663 8.18929L2.785554 .573848H9.336986C10.938979 .573848 12.026899 .74122 12.134496 .765131C12.780075 .860772 13.820174 1.06401 14.764633 1.661768C15.063512 1.853051 15.876463 2.391034 16.282939 3.359402H16.581818L15.135243 0H1.004234C.729265 0 .71731 .011955 .681445 .083686C.669489 .119552 .669489 .3467 .669489 .478207L6.993773 9.133748L.800996 16.390535C.681445 16.533998 .681445 16.593773 .681445 16.605729C.681445 16.737235 .789041 16.737235 1.004234 16.737235H15.135243Z'/>
+<path id='g1-105' d='M2.375093-4.97335C2.375093-5.148692 2.247572-5.276214 2.064259-5.276214C1.857036-5.276214 1.625903-5.084932 1.625903-4.845828C1.625903-4.670486 1.753425-4.542964 1.936737-4.542964C2.14396-4.542964 2.375093-4.734247 2.375093-4.97335ZM1.211457-2.048319L.781071-.948443C.74122-.828892 .70137-.73325 .70137-.597758C.70137-.207223 1.004234 .079701 1.42665 .079701C2.199751 .079701 2.526526-1.036115 2.526526-1.139726C2.526526-1.219427 2.462765-1.243337 2.406974-1.243337C2.311333-1.243337 2.295392-1.187547 2.271482-1.107846C2.088169-.470237 1.761395-.143462 1.44259-.143462C1.346949-.143462 1.251308-.183313 1.251308-.398506C1.251308-.589788 1.307098-.73325 1.41071-.980324C1.490411-1.195517 1.570112-1.41071 1.657783-1.625903L1.904857-2.271482C1.976588-2.454795 2.072229-2.701868 2.072229-2.83736C2.072229-3.235866 1.753425-3.514819 1.346949-3.514819C.573848-3.514819 .239103-2.399004 .239103-2.295392C.239103-2.223661 .294894-2.191781 .358655-2.191781C.462267-2.191781 .470237-2.239601 .494147-2.319303C.71731-3.076463 1.083935-3.291656 1.323039-3.291656C1.43462-3.291656 1.514321-3.251806 1.514321-3.028643C1.514321-2.948941 1.506351-2.83736 1.42665-2.598257L1.211457-2.048319Z'/>
+<path id='g1-106' d='M3.291656-4.97335C3.291656-5.124782 3.172105-5.276214 2.980822-5.276214C2.741719-5.276214 2.534496-5.053051 2.534496-4.845828C2.534496-4.694396 2.654047-4.542964 2.84533-4.542964C3.084433-4.542964 3.291656-4.766127 3.291656-4.97335ZM1.625903 .398506C1.506351 .884682 1.115816 1.40274 .629639 1.40274C.502117 1.40274 .382565 1.370859 .366625 1.362889C.613699 1.243337 .645579 1.028144 .645579 .956413C.645579 .765131 .502117 .661519 .334745 .661519C.103611 .661519-.111582 .860772-.111582 1.123786C-.111582 1.42665 .183313 1.625903 .637609 1.625903C1.123786 1.625903 2.000498 1.323039 2.239601 .366625L2.956912-2.486675C2.980822-2.582316 2.996762-2.646077 2.996762-2.765629C2.996762-3.203985 2.646077-3.514819 2.183811-3.514819C1.338979-3.514819 .844832-2.399004 .844832-2.295392C.844832-2.223661 .900623-2.191781 .964384-2.191781C1.052055-2.191781 1.060025-2.215691 1.115816-2.335243C1.354919-2.885181 1.761395-3.291656 2.1599-3.291656C2.327273-3.291656 2.422914-3.180075 2.422914-2.917061C2.422914-2.805479 2.399004-2.693898 2.375093-2.582316L1.625903 .398506Z'/>
+<path id='g3-61' d='M8.069738-3.873474C8.237111-3.873474 8.452304-3.873474 8.452304-4.088667C8.452304-4.315816 8.249066-4.315816 8.069738-4.315816H1.028144C.860772-4.315816 .645579-4.315816 .645579-4.100623C.645579-3.873474 .848817-3.873474 1.028144-3.873474H8.069738ZM8.069738-1.649813C8.237111-1.649813 8.452304-1.649813 8.452304-1.865006C8.452304-2.092154 8.249066-2.092154 8.069738-2.092154H1.028144C.860772-2.092154 .645579-2.092154 .645579-1.876961C.645579-1.649813 .848817-1.649813 1.028144-1.649813H8.069738Z'/>
+<path id='g3-91' d='M2.988792 2.988792V2.546451H1.829141V-8.524035H2.988792V-8.966376H1.3868V2.988792H2.988792Z'/>
+<path id='g3-93' d='M1.853051-8.966376H.251059V-8.524035H1.41071V2.546451H.251059V2.988792H1.853051V-8.966376Z'/>
+<path id='g2-58' d='M2.199751-.573848C2.199751-.920548 1.912827-1.159651 1.625903-1.159651C1.279203-1.159651 1.0401-.872727 1.0401-.585803C1.0401-.239103 1.327024 0 1.613948 0C1.960648 0 2.199751-.286924 2.199751-.573848Z'/>
+<path id='g2-97' d='M3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665ZM4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.944458 4.27995-1.075965 4.387547-1.542217C4.471233-1.841096 4.531009-2.10411 4.62665-2.450809C5.068991-4.244085 5.176588-4.674471 5.176588-4.746202C5.176588-4.913574 5.045081-5.045081 4.865753-5.045081C4.483188-5.045081 4.387547-4.62665 4.363636-4.483188Z'/>
+<path id='g2-99' d='M4.674471-4.495143C4.447323-4.495143 4.339726-4.495143 4.172354-4.351681C4.100623-4.291905 3.969116-4.112578 3.969116-3.921295C3.969116-3.682192 4.148443-3.53873 4.375592-3.53873C4.662516-3.53873 4.985305-3.777833 4.985305-4.25604C4.985305-4.829888 4.435367-5.272229 3.610461-5.272229C2.044334-5.272229 .478207-3.56264 .478207-1.865006C.478207-.824907 1.123786 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.542217-.119552 1.279203-.836862 1.279203-1.43462C1.279203-1.853051 1.482441-3.012702 1.912827-3.825654C2.223661-4.387547 2.86924-5.033126 3.622416-5.033126C3.777833-5.033126 4.435367-5.009215 4.674471-4.495143Z'/>
+<path id='g2-100' d='M6.01345-7.998007C6.025405-8.045828 6.049315-8.117559 6.049315-8.177335C6.049315-8.296887 5.929763-8.296887 5.905853-8.296887C5.893898-8.296887 5.308095-8.249066 5.248319-8.237111C5.045081-8.225156 4.865753-8.201245 4.65056-8.18929C4.351681-8.16538 4.267995-8.153425 4.267995-7.938232C4.267995-7.81868 4.363636-7.81868 4.531009-7.81868C5.116812-7.81868 5.128767-7.711083 5.128767-7.591532C5.128767-7.519801 5.104857-7.424159 5.092902-7.388294L4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.71731 4.244085-.968369 4.327771-1.303113L6.01345-7.998007ZM3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665Z'/>
+<path id='g2-101' d='M2.139975-2.773599C2.462765-2.773599 3.275716-2.797509 3.849564-3.012702C4.758157-3.359402 4.841843-4.052802 4.841843-4.267995C4.841843-4.794022 4.387547-5.272229 3.598506-5.272229C2.343213-5.272229 .537983-4.136488 .537983-2.008468C.537983-.753176 1.255293 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.685679-.119552 1.327024-.657534 1.327024-1.542217C1.327024-1.709589 1.327024-2.008468 1.506351-2.773599H2.139975ZM1.566127-3.012702C2.080199-4.853798 3.21594-5.033126 3.598506-5.033126C4.124533-5.033126 4.483188-4.722291 4.483188-4.267995C4.483188-3.012702 2.570361-3.012702 2.068244-3.012702H1.566127Z'/>
+<path id='g2-105' d='M3.383313-1.709589C3.383313-1.769365 3.335492-1.817186 3.263761-1.817186C3.156164-1.817186 3.144209-1.78132 3.084433-1.578082C2.773599-.490162 2.283437-.119552 1.888917-.119552C1.745455-.119552 1.578082-.155417 1.578082-.514072C1.578082-.836862 1.721544-1.195517 1.853051-1.554172L2.689913-3.777833C2.725778-3.873474 2.809465-4.088667 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 1.984558-3.93325 1.888917-3.670237L1.052055-1.446575C.980324-1.255293 .908593-1.06401 .908593-.848817C.908593-.310834 1.279203 .119552 1.853051 .119552C2.952927 .119552 3.383313-1.625903 3.383313-1.709589ZM3.287671-7.460025C3.287671-7.639352 3.144209-7.854545 2.881196-7.854545C2.606227-7.854545 2.295392-7.591532 2.295392-7.280697C2.295392-6.981818 2.546451-6.886177 2.689913-6.886177C3.012702-6.886177 3.287671-7.197011 3.287671-7.460025Z'/>
+<path id='g2-106' d='M4.184309-3.789788C4.23213-3.981071 4.23213-4.148443 4.23213-4.196264C4.23213-4.889664 3.718057-5.272229 3.180075-5.272229C1.972603-5.272229 1.327024-3.526775 1.327024-3.443088C1.327024-3.383313 1.374844-3.335492 1.446575-3.335492C1.542217-3.335492 1.554172-3.383313 1.613948-3.502864C2.092154-4.662516 2.689913-5.033126 3.144209-5.033126C3.395268-5.033126 3.526775-4.901619 3.526775-4.483188C3.526775-4.196264 3.490909-4.076712 3.443088-3.861519L2.307347 .645579C2.080199 1.530262 1.518306 2.199751 .860772 2.199751C.812951 2.199751 .561893 2.199751 .334745 2.080199C.621669 2.020423 .848817 1.793275 .848817 1.506351C.848817 1.315068 .705355 1.123786 .442341 1.123786C.131507 1.123786-.155417 1.3868-.155417 1.745455C-.155417 2.235616 .37061 2.438854 .860772 2.438854C1.685679 2.438854 2.773599 1.829141 3.072478 .633624L4.184309-3.789788ZM4.674471-7.460025C4.674471-7.758904 4.423412-7.854545 4.27995-7.854545C3.957161-7.854545 3.682192-7.543711 3.682192-7.280697C3.682192-7.10137 3.825654-6.886177 4.088667-6.886177C4.363636-6.886177 4.674471-7.149191 4.674471-7.460025Z'/>
+<path id='g2-109' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-4.124533 5.057036-3.88543 4.913574-3.323537L4.614695-2.092154C4.519054-1.733499 4.291905-.848817 4.267995-.71731C4.220174-.537983 4.148443-.227148 4.148443-.179328C4.148443-.011955 4.27995 .119552 4.459278 .119552C4.817933 .119552 4.877709-.155417 4.985305-.585803L5.702615-3.443088C5.726526-3.53873 6.348194-5.033126 7.663263-5.033126C8.141469-5.033126 8.308842-4.674471 8.308842-4.196264C8.308842-3.526775 7.84259-2.223661 7.579577-1.506351C7.47198-1.219427 7.412204-1.06401 7.412204-.848817C7.412204-.310834 7.782814 .119552 8.356663 .119552C9.468493 .119552 9.886924-1.637858 9.886924-1.709589C9.886924-1.769365 9.839103-1.817186 9.767372-1.817186C9.659776-1.817186 9.647821-1.78132 9.588045-1.578082C9.313076-.621669 8.870735-.119552 8.392528-.119552C8.272976-.119552 8.081694-.131507 8.081694-.514072C8.081694-.824907 8.225156-1.207472 8.272976-1.338979C8.488169-1.912827 9.026152-3.323537 9.026152-4.016936C9.026152-4.734247 8.607721-5.272229 7.699128-5.272229C6.898132-5.272229 6.252553-4.817933 5.774346-4.112578C5.738481-4.758157 5.34396-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.207472-5.272229 .932503-5.104857 .705355-4.65056C.490162-4.220174 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.327771 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/>
+<path id='g2-110' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-3.514819 4.566874-2.15193 4.327771-1.506351C4.220174-1.219427 4.160399-1.06401 4.160399-.848817C4.160399-.310834 4.531009 .119552 5.104857 .119552C6.216687 .119552 6.635118-1.637858 6.635118-1.709589C6.635118-1.769365 6.587298-1.817186 6.515567-1.817186C6.40797-1.817186 6.396015-1.78132 6.336239-1.578082C6.06127-.597758 5.606974-.119552 5.140722-.119552C5.021171-.119552 4.829888-.131507 4.829888-.514072C4.829888-.812951 4.961395-1.171606 5.033126-1.338979C5.272229-1.996513 5.774346-3.335492 5.774346-4.016936C5.774346-4.734247 5.355915-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.171606-5.272229 .908593-5.045081 .705355-4.638605C.490162-4.208219 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.824907-4.351681 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/>
+<path id='g2-111' d='M5.451557-3.287671C5.451557-4.423412 4.710336-5.272229 3.622416-5.272229C2.044334-5.272229 .490162-3.550685 .490162-1.865006C.490162-.729265 1.231382 .119552 2.319303 .119552C3.90934 .119552 5.451557-1.601993 5.451557-3.287671ZM2.331258-.119552C1.733499-.119552 1.291158-.597758 1.291158-1.43462C1.291158-1.984558 1.578082-3.203985 1.912827-3.801743C2.450809-4.722291 3.120299-5.033126 3.610461-5.033126C4.196264-5.033126 4.65056-4.554919 4.65056-3.718057C4.65056-3.239851 4.399502-1.960648 3.945205-1.231382C3.455044-.430386 2.797509-.119552 2.331258-.119552Z'/>
+<path id='g2-112' d='M.514072 1.518306C.430386 1.876961 .382565 1.972603-.107597 1.972603C-.251059 1.972603-.37061 1.972603-.37061 2.199751C-.37061 2.223661-.358655 2.319303-.227148 2.319303C-.071731 2.319303 .095641 2.295392 .251059 2.295392H.765131C1.016189 2.295392 1.625903 2.319303 1.876961 2.319303C1.948692 2.319303 2.092154 2.319303 2.092154 2.10411C2.092154 1.972603 2.008468 1.972603 1.80523 1.972603C1.255293 1.972603 1.219427 1.888917 1.219427 1.793275C1.219427 1.649813 1.75741-.406476 1.829141-.681445C1.960648-.3467 2.283437 .119552 2.905106 .119552C4.25604 .119552 5.71457-1.637858 5.71457-3.395268C5.71457-4.495143 5.092902-5.272229 4.196264-5.272229C3.431133-5.272229 2.785554-4.531009 2.654047-4.363636C2.558406-4.961395 2.092154-5.272229 1.613948-5.272229C1.267248-5.272229 .992279-5.104857 .765131-4.65056C.549938-4.220174 .382565-3.490909 .382565-3.443088S.430386-3.335492 .514072-3.335492C.609714-3.335492 .621669-3.347447 .6934-3.622416C.872727-4.327771 1.099875-5.033126 1.578082-5.033126C1.853051-5.033126 1.948692-4.841843 1.948692-4.483188C1.948692-4.196264 1.912827-4.076712 1.865006-3.861519L.514072 1.518306ZM2.582316-3.730012C2.666002-4.064757 3.000747-4.411457 3.19203-4.578829C3.323537-4.698381 3.718057-5.033126 4.172354-5.033126C4.698381-5.033126 4.937484-4.507098 4.937484-3.88543C4.937484-3.311582 4.60274-1.960648 4.303861-1.338979C4.004981-.6934 3.455044-.119552 2.905106-.119552C2.092154-.119552 1.960648-1.147696 1.960648-1.195517C1.960648-1.231382 1.984558-1.327024 1.996513-1.3868L2.582316-3.730012Z'/>
+<path id='g2-113' d='M5.272229-5.152677C5.272229-5.212453 5.224408-5.260274 5.164633-5.260274C5.068991-5.260274 4.60274-4.829888 4.375592-4.411457C4.160399-4.94944 3.789788-5.272229 3.275716-5.272229C1.924782-5.272229 .466252-3.526775 .466252-1.75741C.466252-.573848 1.159651 .119552 1.972603 .119552C2.606227 .119552 3.132254-.358655 3.383313-.633624L3.395268-.621669L2.940971 1.171606L2.833375 1.601993C2.725778 1.960648 2.546451 1.960648 1.984558 1.972603C1.853051 1.972603 1.733499 1.972603 1.733499 2.199751C1.733499 2.283437 1.80523 2.319303 1.888917 2.319303C2.056289 2.319303 2.271482 2.295392 2.438854 2.295392H3.658281C3.837609 2.295392 4.040847 2.319303 4.220174 2.319303C4.291905 2.319303 4.435367 2.319303 4.435367 2.092154C4.435367 1.972603 4.339726 1.972603 4.160399 1.972603C3.598506 1.972603 3.56264 1.888917 3.56264 1.793275C3.56264 1.733499 3.574595 1.721544 3.610461 1.566127L5.272229-5.152677ZM3.58655-1.422665C3.526775-1.219427 3.526775-1.195517 3.359402-.968369C3.096389-.633624 2.570361-.119552 2.008468-.119552C1.518306-.119552 1.243337-.561893 1.243337-1.267248C1.243337-1.924782 1.613948-3.263761 1.841096-3.765878C2.247572-4.60274 2.809465-5.033126 3.275716-5.033126C4.064757-5.033126 4.220174-4.052802 4.220174-3.957161C4.220174-3.945205 4.184309-3.789788 4.172354-3.765878L3.58655-1.422665Z'/>
+<path id='g2-114' d='M4.65056-4.889664C4.27995-4.817933 4.088667-4.554919 4.088667-4.291905C4.088667-4.004981 4.315816-3.90934 4.483188-3.90934C4.817933-3.90934 5.092902-4.196264 5.092902-4.554919C5.092902-4.937484 4.722291-5.272229 4.124533-5.272229C3.646326-5.272229 3.096389-5.057036 2.594271-4.327771C2.510585-4.961395 2.032379-5.272229 1.554172-5.272229C1.08792-5.272229 .848817-4.913574 .705355-4.65056C.502117-4.220174 .32279-3.502864 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.339726 1.0401-5.033126 1.518306-5.033126C1.80523-5.033126 1.888917-4.829888 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.338979 .119552 1.566127 .035866 1.637858-.203238C1.673724-.298879 2.116065-2.10411 2.187796-2.379078C2.247572-2.642092 2.319303-2.893151 2.379078-3.156164C2.426899-3.323537 2.47472-3.514819 2.510585-3.670237C2.546451-3.777833 2.86924-4.363636 3.16812-4.62665C3.311582-4.758157 3.622416-5.033126 4.112578-5.033126C4.303861-5.033126 4.495143-4.99726 4.65056-4.889664Z'/>
+<path id='g2-115' d='M2.725778-2.391034C2.929016-2.355168 3.251806-2.283437 3.323537-2.271482C3.478954-2.223661 4.016936-2.032379 4.016936-1.458531C4.016936-1.08792 3.682192-.119552 2.295392-.119552C2.044334-.119552 1.147696-.155417 .908593-.812951C1.3868-.753176 1.625903-1.123786 1.625903-1.3868C1.625903-1.637858 1.458531-1.769365 1.219427-1.769365C.956413-1.769365 .609714-1.566127 .609714-1.028144C.609714-.32279 1.327024 .119552 2.283437 .119552C4.100623 .119552 4.638605-1.219427 4.638605-1.841096C4.638605-2.020423 4.638605-2.355168 4.25604-2.737733C3.957161-3.024658 3.670237-3.084433 3.024658-3.21594C2.701868-3.287671 2.187796-3.395268 2.187796-3.93325C2.187796-4.172354 2.402989-5.033126 3.53873-5.033126C4.040847-5.033126 4.531009-4.841843 4.65056-4.411457C4.124533-4.411457 4.100623-3.957161 4.100623-3.945205C4.100623-3.694147 4.327771-3.622416 4.435367-3.622416C4.60274-3.622416 4.937484-3.753923 4.937484-4.25604S4.483188-5.272229 3.550685-5.272229C1.984558-5.272229 1.566127-4.040847 1.566127-3.550685C1.566127-2.642092 2.450809-2.450809 2.725778-2.391034Z'/>
+<path id='g2-116' d='M2.402989-4.805978H3.502864C3.730012-4.805978 3.849564-4.805978 3.849564-5.021171C3.849564-5.152677 3.777833-5.152677 3.53873-5.152677H2.486675L2.929016-6.898132C2.976837-7.065504 2.976837-7.089415 2.976837-7.173101C2.976837-7.364384 2.82142-7.47198 2.666002-7.47198C2.570361-7.47198 2.295392-7.436115 2.199751-7.053549L1.733499-5.152677H.609714C.37061-5.152677 .263014-5.152677 .263014-4.925529C.263014-4.805978 .3467-4.805978 .573848-4.805978H1.637858L.848817-1.649813C.753176-1.231382 .71731-1.111831 .71731-.956413C.71731-.394521 1.111831 .119552 1.78132 .119552C2.988792 .119552 3.634371-1.625903 3.634371-1.709589C3.634371-1.78132 3.58655-1.817186 3.514819-1.817186C3.490909-1.817186 3.443088-1.817186 3.419178-1.769365C3.407223-1.75741 3.395268-1.745455 3.311582-1.554172C3.060523-.956413 2.510585-.119552 1.817186-.119552C1.458531-.119552 1.43462-.418431 1.43462-.681445C1.43462-.6934 1.43462-.920548 1.470486-1.06401L2.402989-4.805978Z'/>
+<path id='g2-117' d='M4.076712-.6934C4.23213-.02391 4.805978 .119552 5.092902 .119552C5.475467 .119552 5.762391-.131507 5.953674-.537983C6.156912-.968369 6.312329-1.673724 6.312329-1.709589C6.312329-1.769365 6.264508-1.817186 6.192777-1.817186C6.085181-1.817186 6.073225-1.75741 6.025405-1.578082C5.810212-.753176 5.595019-.119552 5.116812-.119552C4.758157-.119552 4.758157-.514072 4.758157-.669489C4.758157-.944458 4.794022-1.06401 4.913574-1.566127C4.99726-1.888917 5.080946-2.211706 5.152677-2.546451L5.642839-4.495143C5.726526-4.794022 5.726526-4.817933 5.726526-4.853798C5.726526-5.033126 5.583064-5.152677 5.403736-5.152677C5.057036-5.152677 4.97335-4.853798 4.901619-4.554919C4.782067-4.088667 4.136488-1.518306 4.052802-1.099875C4.040847-1.099875 3.574595-.119552 2.701868-.119552C2.080199-.119552 1.960648-.657534 1.960648-1.099875C1.960648-1.78132 2.295392-2.737733 2.606227-3.53873C2.749689-3.921295 2.809465-4.076712 2.809465-4.315816C2.809465-4.829888 2.438854-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.578829 1.374844-5.033126 1.829141-5.033126C1.948692-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.303113-2.10411 1.243337-1.649813 1.243337-1.291158C1.243337-.071731 2.163885 .119552 2.654047 .119552C3.419178 .119552 3.837609-.406476 4.076712-.6934Z'/>
+<path id='g2-121' d='M3.144209 1.338979C2.82142 1.793275 2.355168 2.199751 1.769365 2.199751C1.625903 2.199751 1.052055 2.175841 .872727 1.625903C.908593 1.637858 .968369 1.637858 .992279 1.637858C1.350934 1.637858 1.590037 1.327024 1.590037 1.052055S1.362889 .681445 1.183562 .681445C.992279 .681445 .573848 .824907 .573848 1.41071C.573848 2.020423 1.08792 2.438854 1.769365 2.438854C2.964882 2.438854 4.172354 1.338979 4.507098 .011955L5.678705-4.65056C5.69066-4.710336 5.71457-4.782067 5.71457-4.853798C5.71457-5.033126 5.571108-5.152677 5.391781-5.152677C5.284184-5.152677 5.033126-5.104857 4.937484-4.746202L4.052802-1.231382C3.993026-1.016189 3.993026-.992279 3.897385-.860772C3.658281-.526027 3.263761-.119552 2.689913-.119552C2.020423-.119552 1.960648-.777086 1.960648-1.099875C1.960648-1.78132 2.283437-2.701868 2.606227-3.56264C2.737733-3.90934 2.809465-4.076712 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.033126 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.243337-1.960648 1.243337-1.566127 1.243337-1.279203C1.243337-.143462 2.056289 .119552 2.654047 .119552C3.000747 .119552 3.431133 .011955 3.849564-.430386L3.861519-.418431C3.682192 .286924 3.56264 .753176 3.144209 1.338979Z'/>
+</defs>
+<g id='page1' transform='matrix(1.13 0 0 1.13 -63.986043 -66.444003)'>
+<use x='56.413267' y='69.937791' xlink:href='#g2-109'/>
+<use x='66.652534' y='69.937791' xlink:href='#g2-101'/>
+<use x='72.077974' y='69.937791' xlink:href='#g2-109'/>
+<use x='82.317241' y='69.937791' xlink:href='#g2-111'/>
+<use x='87.944679' y='69.937791' xlink:href='#g2-114'/>
+<use x='93.545152' y='69.937791' xlink:href='#g2-121'/>
+<use x='99.681804' y='69.937791' xlink:href='#g2-58'/>
+<use x='102.933465' y='69.937791' xlink:href='#g2-109'/>
+<use x='113.172732' y='69.937791' xlink:href='#g2-105'/>
+<use x='117.166164' y='69.937791' xlink:href='#g2-110'/>
+<use x='127.474599' y='69.937791' xlink:href='#g3-61'/>
+<use x='139.90008' y='58.580327' xlink:href='#g0-88'/>
+<use x='147.092819' y='83.774682' xlink:href='#g1-105'/>
+<use x='159.161195' y='58.580327' xlink:href='#g0-88'/>
+<use x='165.85349' y='83.774682' xlink:href='#g1-106'/>
+<use x='178.422309' y='69.937791' xlink:href='#g2-112'/>
+<use x='184.297452' y='69.937791' xlink:href='#g2-111'/>
+<use x='189.92489' y='69.937791' xlink:href='#g2-100'/>
+<use x='196.007582' y='69.937791' xlink:href='#g3-91'/>
+<use x='199.259244' y='69.937791' xlink:href='#g2-105'/>
+<use x='203.252676' y='69.937791' xlink:href='#g3-93'/>
+<use x='206.504337' y='69.937791' xlink:href='#g2-58'/>
+<use x='209.755998' y='69.937791' xlink:href='#g2-115'/>
+<use x='215.270004' y='69.937791' xlink:href='#g2-112'/>
+<use x='221.145147' y='69.937791' xlink:href='#g2-101'/>
+<use x='226.570587' y='69.937791' xlink:href='#g2-99'/>
+<use x='231.608576' y='69.937791' xlink:href='#g2-58'/>
+<use x='234.860237' y='69.937791' xlink:href='#g2-99'/>
+<use x='239.898226' y='69.937791' xlink:href='#g2-111'/>
+<use x='245.525663' y='69.937791' xlink:href='#g2-110'/>
+<use x='252.513269' y='69.937791' xlink:href='#g2-116'/>
+<use x='256.740429' y='69.937791' xlink:href='#g2-97'/>
+<use x='262.885373' y='69.937791' xlink:href='#g2-105'/>
+<use x='266.878805' y='69.937791' xlink:href='#g2-110'/>
+<use x='273.866411' y='69.937791' xlink:href='#g2-101'/>
+<use x='279.291851' y='69.937791' xlink:href='#g2-114'/>
+<use x='284.892324' y='69.937791' xlink:href='#g2-115'/>
+<use x='290.40633' y='69.937791' xlink:href='#g3-91'/>
+<use x='293.657991' y='69.937791' xlink:href='#g2-106'/>
+<use x='299.173487' y='69.937791' xlink:href='#g3-93'/>
+<use x='302.425148' y='69.937791' xlink:href='#g2-58'/>
+<use x='305.676809' y='69.937791' xlink:href='#g2-114'/>
+<use x='311.277283' y='69.937791' xlink:href='#g2-101'/>
+<use x='316.702723' y='69.937791' xlink:href='#g2-115'/>
+<use x='322.216728' y='69.937791' xlink:href='#g2-111'/>
+<use x='327.844166' y='69.937791' xlink:href='#g2-117'/>
+<use x='334.506606' y='69.937791' xlink:href='#g2-114'/>
+<use x='340.107079' y='69.937791' xlink:href='#g2-99'/>
+<use x='345.145068' y='69.937791' xlink:href='#g2-101'/>
+<use x='350.570508' y='69.937791' xlink:href='#g2-115'/>
+<use x='356.084513' y='69.937791' xlink:href='#g2-58'/>
+<use x='359.336175' y='69.937791' xlink:href='#g2-114'/>
+<use x='364.936648' y='69.937791' xlink:href='#g2-101'/>
+<use x='370.362088' y='69.937791' xlink:href='#g2-113'/>
+<use x='375.981245' y='69.937791' xlink:href='#g2-117'/>
+<use x='382.643684' y='69.937791' xlink:href='#g2-101'/>
+<use x='388.069124' y='69.937791' xlink:href='#g2-115'/>
+<use x='393.58313' y='69.937791' xlink:href='#g2-116'/>
+<use x='397.81029' y='69.937791' xlink:href='#g2-115'/>
+<use x='403.324295' y='69.937791' xlink:href='#g3-91'/>
+<use x='406.575957' y='69.937791' xlink:href='#g2-109'/>
+<use x='416.815224' y='69.937791' xlink:href='#g2-101'/>
+<use x='422.240664' y='69.937791' xlink:href='#g2-109'/>
+<use x='432.479931' y='69.937791' xlink:href='#g2-111'/>
+<use x='438.107368' y='69.937791' xlink:href='#g2-114'/>
+<use x='443.707842' y='69.937791' xlink:href='#g2-121'/>
+<use x='449.844493' y='69.937791' xlink:href='#g3-93'/>
+</g>
+</svg>
@@ -0,0 +1,97 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<!-- Generated by CodeCogs with dvisvgm 2.9.1 -->
+<svg version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink' width='411.32491pt' height='36.683306pt' viewBox='-.239051 -.232683 411.32491 36.683306'>
+<defs>
+<path id='g0-88' d='M15.135243 16.737235L16.581818 12.911582H16.282939C15.816687 14.154919 14.54944 14.96787 13.174595 15.326526C12.923537 15.386301 11.75193 15.697136 9.456538 15.697136H2.247572L8.332752 8.5599C8.416438 8.464259 8.440349 8.428394 8.440349 8.368618C8.440349 8.344707 8.440349 8.308842 8.356663 8.18929L2.785554 .573848H9.336986C10.938979 .573848 12.026899 .74122 12.134496 .765131C12.780075 .860772 13.820174 1.06401 14.764633 1.661768C15.063512 1.853051 15.876463 2.391034 16.282939 3.359402H16.581818L15.135243 0H1.004234C.729265 0 .71731 .011955 .681445 .083686C.669489 .119552 .669489 .3467 .669489 .478207L6.993773 9.133748L.800996 16.390535C.681445 16.533998 .681445 16.593773 .681445 16.605729C.681445 16.737235 .789041 16.737235 1.004234 16.737235H15.135243Z'/>
+<path id='g3-48' d='M3.897385-2.542466C3.897385-3.395268 3.809714-3.913325 3.5467-4.423412C3.196015-5.124782 2.550436-5.300125 2.11208-5.300125C1.107846-5.300125 .74122-4.550934 .629639-4.327771C.342715-3.745953 .326775-2.956912 .326775-2.542466C.326775-2.016438 .350685-1.211457 .73325-.573848C1.099875 .01594 1.689664 .167372 2.11208 .167372C2.494645 .167372 3.180075 .047821 3.57858-.74122C3.873474-1.315068 3.897385-2.024408 3.897385-2.542466ZM2.11208-.055791C1.841096-.055791 1.291158-.183313 1.123786-1.020174C1.036115-1.474471 1.036115-2.223661 1.036115-2.638107C1.036115-3.188045 1.036115-3.745953 1.123786-4.184309C1.291158-4.99726 1.912827-5.076961 2.11208-5.076961C2.383064-5.076961 2.933001-4.941469 3.092403-4.216189C3.188045-3.777833 3.188045-3.180075 3.188045-2.638107C3.188045-2.16787 3.188045-1.45056 3.092403-1.004234C2.925031-.167372 2.375093-.055791 2.11208-.055791Z'/>
+<path id='g3-61' d='M5.826152-2.654047C5.945704-2.654047 6.105106-2.654047 6.105106-2.83736S5.913823-3.020672 5.794271-3.020672H.781071C.661519-3.020672 .470237-3.020672 .470237-2.83736S.629639-2.654047 .749191-2.654047H5.826152ZM5.794271-.964384C5.913823-.964384 6.105106-.964384 6.105106-1.147696S5.945704-1.331009 5.826152-1.331009H.749191C.629639-1.331009 .470237-1.331009 .470237-1.147696S.661519-.964384 .781071-.964384H5.794271Z'/>
+<path id='g1-105' d='M2.375093-4.97335C2.375093-5.148692 2.247572-5.276214 2.064259-5.276214C1.857036-5.276214 1.625903-5.084932 1.625903-4.845828C1.625903-4.670486 1.753425-4.542964 1.936737-4.542964C2.14396-4.542964 2.375093-4.734247 2.375093-4.97335ZM1.211457-2.048319L.781071-.948443C.74122-.828892 .70137-.73325 .70137-.597758C.70137-.207223 1.004234 .079701 1.42665 .079701C2.199751 .079701 2.526526-1.036115 2.526526-1.139726C2.526526-1.219427 2.462765-1.243337 2.406974-1.243337C2.311333-1.243337 2.295392-1.187547 2.271482-1.107846C2.088169-.470237 1.761395-.143462 1.44259-.143462C1.346949-.143462 1.251308-.183313 1.251308-.398506C1.251308-.589788 1.307098-.73325 1.41071-.980324C1.490411-1.195517 1.570112-1.41071 1.657783-1.625903L1.904857-2.271482C1.976588-2.454795 2.072229-2.701868 2.072229-2.83736C2.072229-3.235866 1.753425-3.514819 1.346949-3.514819C.573848-3.514819 .239103-2.399004 .239103-2.295392C.239103-2.223661 .294894-2.191781 .358655-2.191781C.462267-2.191781 .470237-2.239601 .494147-2.319303C.71731-3.076463 1.083935-3.291656 1.323039-3.291656C1.43462-3.291656 1.514321-3.251806 1.514321-3.028643C1.514321-2.948941 1.506351-2.83736 1.42665-2.598257L1.211457-2.048319Z'/>
+<path id='g1-110' d='M1.594022-1.307098C1.617933-1.42665 1.697634-1.729514 1.721544-1.849066C1.833126-2.279452 1.833126-2.287422 2.016438-2.550436C2.279452-2.940971 2.654047-3.291656 3.188045-3.291656C3.474969-3.291656 3.642341-3.124284 3.642341-2.749689C3.642341-2.311333 3.307597-1.40274 3.156164-1.012204C3.052553-.749191 3.052553-.70137 3.052553-.597758C3.052553-.143462 3.427148 .079701 3.769863 .079701C4.550934 .079701 4.877709-1.036115 4.877709-1.139726C4.877709-1.219427 4.813948-1.243337 4.758157-1.243337C4.662516-1.243337 4.646575-1.187547 4.622665-1.107846C4.431382-.454296 4.096638-.143462 3.793773-.143462C3.666252-.143462 3.602491-.223163 3.602491-.406476S3.666252-.765131 3.745953-.964384C3.865504-1.267248 4.216189-2.183811 4.216189-2.630137C4.216189-3.227895 3.801743-3.514819 3.227895-3.514819C2.582316-3.514819 2.16787-3.124284 1.936737-2.82142C1.880946-3.259776 1.530262-3.514819 1.123786-3.514819C.836862-3.514819 .637609-3.331507 .510087-3.084433C.318804-2.709838 .239103-2.311333 .239103-2.295392C.239103-2.223661 .294894-2.191781 .358655-2.191781C.462267-2.191781 .470237-2.223661 .526027-2.430884C.621669-2.82142 .765131-3.291656 1.099875-3.291656C1.307098-3.291656 1.354919-3.092403 1.354919-2.917061C1.354919-2.773599 1.315068-2.622167 1.251308-2.359153C1.235367-2.295392 1.115816-1.825156 1.083935-1.713574L.789041-.518057C.757161-.398506 .70934-.199253 .70934-.167372C.70934 .01594 .860772 .079701 .964384 .079701C1.107846 .079701 1.227397-.01594 1.283188-.111582C1.307098-.159402 1.370859-.430386 1.41071-.597758L1.594022-1.307098Z'/>
+<path id='g4-61' d='M8.069738-3.873474C8.237111-3.873474 8.452304-3.873474 8.452304-4.088667C8.452304-4.315816 8.249066-4.315816 8.069738-4.315816H1.028144C.860772-4.315816 .645579-4.315816 .645579-4.100623C.645579-3.873474 .848817-3.873474 1.028144-3.873474H8.069738ZM8.069738-1.649813C8.237111-1.649813 8.452304-1.649813 8.452304-1.865006C8.452304-2.092154 8.249066-2.092154 8.069738-2.092154H1.028144C.860772-2.092154 .645579-2.092154 .645579-1.876961C.645579-1.649813 .848817-1.649813 1.028144-1.649813H8.069738Z'/>
+<path id='g4-91' d='M2.988792 2.988792V2.546451H1.829141V-8.524035H2.988792V-8.966376H1.3868V2.988792H2.988792Z'/>
+<path id='g4-93' d='M1.853051-8.966376H.251059V-8.524035H1.41071V2.546451H.251059V2.988792H1.853051V-8.966376Z'/>
+<path id='g2-58' d='M2.199751-.573848C2.199751-.920548 1.912827-1.159651 1.625903-1.159651C1.279203-1.159651 1.0401-.872727 1.0401-.585803C1.0401-.239103 1.327024 0 1.613948 0C1.960648 0 2.199751-.286924 2.199751-.573848Z'/>
+<path id='g2-97' d='M3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665ZM4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.944458 4.27995-1.075965 4.387547-1.542217C4.471233-1.841096 4.531009-2.10411 4.62665-2.450809C5.068991-4.244085 5.176588-4.674471 5.176588-4.746202C5.176588-4.913574 5.045081-5.045081 4.865753-5.045081C4.483188-5.045081 4.387547-4.62665 4.363636-4.483188Z'/>
+<path id='g2-99' d='M4.674471-4.495143C4.447323-4.495143 4.339726-4.495143 4.172354-4.351681C4.100623-4.291905 3.969116-4.112578 3.969116-3.921295C3.969116-3.682192 4.148443-3.53873 4.375592-3.53873C4.662516-3.53873 4.985305-3.777833 4.985305-4.25604C4.985305-4.829888 4.435367-5.272229 3.610461-5.272229C2.044334-5.272229 .478207-3.56264 .478207-1.865006C.478207-.824907 1.123786 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.542217-.119552 1.279203-.836862 1.279203-1.43462C1.279203-1.853051 1.482441-3.012702 1.912827-3.825654C2.223661-4.387547 2.86924-5.033126 3.622416-5.033126C3.777833-5.033126 4.435367-5.009215 4.674471-4.495143Z'/>
+<path id='g2-100' d='M6.01345-7.998007C6.025405-8.045828 6.049315-8.117559 6.049315-8.177335C6.049315-8.296887 5.929763-8.296887 5.905853-8.296887C5.893898-8.296887 5.308095-8.249066 5.248319-8.237111C5.045081-8.225156 4.865753-8.201245 4.65056-8.18929C4.351681-8.16538 4.267995-8.153425 4.267995-7.938232C4.267995-7.81868 4.363636-7.81868 4.531009-7.81868C5.116812-7.81868 5.128767-7.711083 5.128767-7.591532C5.128767-7.519801 5.104857-7.424159 5.092902-7.388294L4.363636-4.483188C4.23213-4.794022 3.90934-5.272229 3.287671-5.272229C1.936737-5.272229 .478207-3.526775 .478207-1.75741C.478207-.573848 1.171606 .119552 1.984558 .119552C2.642092 .119552 3.203985-.394521 3.53873-.789041C3.658281-.083686 4.220174 .119552 4.578829 .119552S5.224408-.095641 5.439601-.526027C5.630884-.932503 5.798257-1.661768 5.798257-1.709589C5.798257-1.769365 5.750436-1.817186 5.678705-1.817186C5.571108-1.817186 5.559153-1.75741 5.511333-1.578082C5.332005-.872727 5.104857-.119552 4.614695-.119552C4.267995-.119552 4.244085-.430386 4.244085-.669489C4.244085-.71731 4.244085-.968369 4.327771-1.303113L6.01345-7.998007ZM3.598506-1.422665C3.53873-1.219427 3.53873-1.195517 3.371357-.968369C3.108344-.633624 2.582316-.119552 2.020423-.119552C1.530262-.119552 1.255293-.561893 1.255293-1.267248C1.255293-1.924782 1.625903-3.263761 1.853051-3.765878C2.259527-4.60274 2.82142-5.033126 3.287671-5.033126C4.076712-5.033126 4.23213-4.052802 4.23213-3.957161C4.23213-3.945205 4.196264-3.789788 4.184309-3.765878L3.598506-1.422665Z'/>
+<path id='g2-101' d='M2.139975-2.773599C2.462765-2.773599 3.275716-2.797509 3.849564-3.012702C4.758157-3.359402 4.841843-4.052802 4.841843-4.267995C4.841843-4.794022 4.387547-5.272229 3.598506-5.272229C2.343213-5.272229 .537983-4.136488 .537983-2.008468C.537983-.753176 1.255293 .119552 2.343213 .119552C3.969116 .119552 4.99726-1.147696 4.99726-1.303113C4.99726-1.374844 4.925529-1.43462 4.877709-1.43462C4.841843-1.43462 4.829888-1.422665 4.722291-1.315068C3.957161-.298879 2.82142-.119552 2.367123-.119552C1.685679-.119552 1.327024-.657534 1.327024-1.542217C1.327024-1.709589 1.327024-2.008468 1.506351-2.773599H2.139975ZM1.566127-3.012702C2.080199-4.853798 3.21594-5.033126 3.598506-5.033126C4.124533-5.033126 4.483188-4.722291 4.483188-4.267995C4.483188-3.012702 2.570361-3.012702 2.068244-3.012702H1.566127Z'/>
+<path id='g2-105' d='M3.383313-1.709589C3.383313-1.769365 3.335492-1.817186 3.263761-1.817186C3.156164-1.817186 3.144209-1.78132 3.084433-1.578082C2.773599-.490162 2.283437-.119552 1.888917-.119552C1.745455-.119552 1.578082-.155417 1.578082-.514072C1.578082-.836862 1.721544-1.195517 1.853051-1.554172L2.689913-3.777833C2.725778-3.873474 2.809465-4.088667 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 1.984558-3.93325 1.888917-3.670237L1.052055-1.446575C.980324-1.255293 .908593-1.06401 .908593-.848817C.908593-.310834 1.279203 .119552 1.853051 .119552C2.952927 .119552 3.383313-1.625903 3.383313-1.709589ZM3.287671-7.460025C3.287671-7.639352 3.144209-7.854545 2.881196-7.854545C2.606227-7.854545 2.295392-7.591532 2.295392-7.280697C2.295392-6.981818 2.546451-6.886177 2.689913-6.886177C3.012702-6.886177 3.287671-7.197011 3.287671-7.460025Z'/>
+<path id='g2-109' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-4.124533 5.057036-3.88543 4.913574-3.323537L4.614695-2.092154C4.519054-1.733499 4.291905-.848817 4.267995-.71731C4.220174-.537983 4.148443-.227148 4.148443-.179328C4.148443-.011955 4.27995 .119552 4.459278 .119552C4.817933 .119552 4.877709-.155417 4.985305-.585803L5.702615-3.443088C5.726526-3.53873 6.348194-5.033126 7.663263-5.033126C8.141469-5.033126 8.308842-4.674471 8.308842-4.196264C8.308842-3.526775 7.84259-2.223661 7.579577-1.506351C7.47198-1.219427 7.412204-1.06401 7.412204-.848817C7.412204-.310834 7.782814 .119552 8.356663 .119552C9.468493 .119552 9.886924-1.637858 9.886924-1.709589C9.886924-1.769365 9.839103-1.817186 9.767372-1.817186C9.659776-1.817186 9.647821-1.78132 9.588045-1.578082C9.313076-.621669 8.870735-.119552 8.392528-.119552C8.272976-.119552 8.081694-.131507 8.081694-.514072C8.081694-.824907 8.225156-1.207472 8.272976-1.338979C8.488169-1.912827 9.026152-3.323537 9.026152-4.016936C9.026152-4.734247 8.607721-5.272229 7.699128-5.272229C6.898132-5.272229 6.252553-4.817933 5.774346-4.112578C5.738481-4.758157 5.34396-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.207472-5.272229 .932503-5.104857 .705355-4.65056C.490162-4.220174 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.327771 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/>
+<path id='g2-110' d='M2.462765-3.502864C2.486675-3.574595 2.785554-4.172354 3.227895-4.554919C3.53873-4.841843 3.945205-5.033126 4.411457-5.033126C4.889664-5.033126 5.057036-4.674471 5.057036-4.196264C5.057036-3.514819 4.566874-2.15193 4.327771-1.506351C4.220174-1.219427 4.160399-1.06401 4.160399-.848817C4.160399-.310834 4.531009 .119552 5.104857 .119552C6.216687 .119552 6.635118-1.637858 6.635118-1.709589C6.635118-1.769365 6.587298-1.817186 6.515567-1.817186C6.40797-1.817186 6.396015-1.78132 6.336239-1.578082C6.06127-.597758 5.606974-.119552 5.140722-.119552C5.021171-.119552 4.829888-.131507 4.829888-.514072C4.829888-.812951 4.961395-1.171606 5.033126-1.338979C5.272229-1.996513 5.774346-3.335492 5.774346-4.016936C5.774346-4.734247 5.355915-5.272229 4.447323-5.272229C3.383313-5.272229 2.82142-4.519054 2.606227-4.220174C2.570361-4.901619 2.080199-5.272229 1.554172-5.272229C1.171606-5.272229 .908593-5.045081 .705355-4.638605C.490162-4.208219 .32279-3.490909 .32279-3.443088S.37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.824907-4.351681 1.0401-5.033126 1.518306-5.033126C1.793275-5.033126 1.888917-4.841843 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.350934 .119552 1.518306 .047821 1.613948-.131507C1.637858-.191283 1.745455-.609714 1.80523-.848817L2.068244-1.924782L2.462765-3.502864Z'/>
+<path id='g2-111' d='M5.451557-3.287671C5.451557-4.423412 4.710336-5.272229 3.622416-5.272229C2.044334-5.272229 .490162-3.550685 .490162-1.865006C.490162-.729265 1.231382 .119552 2.319303 .119552C3.90934 .119552 5.451557-1.601993 5.451557-3.287671ZM2.331258-.119552C1.733499-.119552 1.291158-.597758 1.291158-1.43462C1.291158-1.984558 1.578082-3.203985 1.912827-3.801743C2.450809-4.722291 3.120299-5.033126 3.610461-5.033126C4.196264-5.033126 4.65056-4.554919 4.65056-3.718057C4.65056-3.239851 4.399502-1.960648 3.945205-1.231382C3.455044-.430386 2.797509-.119552 2.331258-.119552Z'/>
+<path id='g2-112' d='M.514072 1.518306C.430386 1.876961 .382565 1.972603-.107597 1.972603C-.251059 1.972603-.37061 1.972603-.37061 2.199751C-.37061 2.223661-.358655 2.319303-.227148 2.319303C-.071731 2.319303 .095641 2.295392 .251059 2.295392H.765131C1.016189 2.295392 1.625903 2.319303 1.876961 2.319303C1.948692 2.319303 2.092154 2.319303 2.092154 2.10411C2.092154 1.972603 2.008468 1.972603 1.80523 1.972603C1.255293 1.972603 1.219427 1.888917 1.219427 1.793275C1.219427 1.649813 1.75741-.406476 1.829141-.681445C1.960648-.3467 2.283437 .119552 2.905106 .119552C4.25604 .119552 5.71457-1.637858 5.71457-3.395268C5.71457-4.495143 5.092902-5.272229 4.196264-5.272229C3.431133-5.272229 2.785554-4.531009 2.654047-4.363636C2.558406-4.961395 2.092154-5.272229 1.613948-5.272229C1.267248-5.272229 .992279-5.104857 .765131-4.65056C.549938-4.220174 .382565-3.490909 .382565-3.443088S.430386-3.335492 .514072-3.335492C.609714-3.335492 .621669-3.347447 .6934-3.622416C.872727-4.327771 1.099875-5.033126 1.578082-5.033126C1.853051-5.033126 1.948692-4.841843 1.948692-4.483188C1.948692-4.196264 1.912827-4.076712 1.865006-3.861519L.514072 1.518306ZM2.582316-3.730012C2.666002-4.064757 3.000747-4.411457 3.19203-4.578829C3.323537-4.698381 3.718057-5.033126 4.172354-5.033126C4.698381-5.033126 4.937484-4.507098 4.937484-3.88543C4.937484-3.311582 4.60274-1.960648 4.303861-1.338979C4.004981-.6934 3.455044-.119552 2.905106-.119552C2.092154-.119552 1.960648-1.147696 1.960648-1.195517C1.960648-1.231382 1.984558-1.327024 1.996513-1.3868L2.582316-3.730012Z'/>
+<path id='g2-113' d='M5.272229-5.152677C5.272229-5.212453 5.224408-5.260274 5.164633-5.260274C5.068991-5.260274 4.60274-4.829888 4.375592-4.411457C4.160399-4.94944 3.789788-5.272229 3.275716-5.272229C1.924782-5.272229 .466252-3.526775 .466252-1.75741C.466252-.573848 1.159651 .119552 1.972603 .119552C2.606227 .119552 3.132254-.358655 3.383313-.633624L3.395268-.621669L2.940971 1.171606L2.833375 1.601993C2.725778 1.960648 2.546451 1.960648 1.984558 1.972603C1.853051 1.972603 1.733499 1.972603 1.733499 2.199751C1.733499 2.283437 1.80523 2.319303 1.888917 2.319303C2.056289 2.319303 2.271482 2.295392 2.438854 2.295392H3.658281C3.837609 2.295392 4.040847 2.319303 4.220174 2.319303C4.291905 2.319303 4.435367 2.319303 4.435367 2.092154C4.435367 1.972603 4.339726 1.972603 4.160399 1.972603C3.598506 1.972603 3.56264 1.888917 3.56264 1.793275C3.56264 1.733499 3.574595 1.721544 3.610461 1.566127L5.272229-5.152677ZM3.58655-1.422665C3.526775-1.219427 3.526775-1.195517 3.359402-.968369C3.096389-.633624 2.570361-.119552 2.008468-.119552C1.518306-.119552 1.243337-.561893 1.243337-1.267248C1.243337-1.924782 1.613948-3.263761 1.841096-3.765878C2.247572-4.60274 2.809465-5.033126 3.275716-5.033126C4.064757-5.033126 4.220174-4.052802 4.220174-3.957161C4.220174-3.945205 4.184309-3.789788 4.172354-3.765878L3.58655-1.422665Z'/>
+<path id='g2-114' d='M4.65056-4.889664C4.27995-4.817933 4.088667-4.554919 4.088667-4.291905C4.088667-4.004981 4.315816-3.90934 4.483188-3.90934C4.817933-3.90934 5.092902-4.196264 5.092902-4.554919C5.092902-4.937484 4.722291-5.272229 4.124533-5.272229C3.646326-5.272229 3.096389-5.057036 2.594271-4.327771C2.510585-4.961395 2.032379-5.272229 1.554172-5.272229C1.08792-5.272229 .848817-4.913574 .705355-4.65056C.502117-4.220174 .32279-3.502864 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.549938-3.335492 .561893-3.347447 .633624-3.622416C.812951-4.339726 1.0401-5.033126 1.518306-5.033126C1.80523-5.033126 1.888917-4.829888 1.888917-4.483188C1.888917-4.220174 1.769365-3.753923 1.685679-3.383313L1.350934-2.092154C1.303113-1.865006 1.171606-1.327024 1.111831-1.111831C1.028144-.800996 .896638-.239103 .896638-.179328C.896638-.011955 1.028144 .119552 1.207472 .119552C1.338979 .119552 1.566127 .035866 1.637858-.203238C1.673724-.298879 2.116065-2.10411 2.187796-2.379078C2.247572-2.642092 2.319303-2.893151 2.379078-3.156164C2.426899-3.323537 2.47472-3.514819 2.510585-3.670237C2.546451-3.777833 2.86924-4.363636 3.16812-4.62665C3.311582-4.758157 3.622416-5.033126 4.112578-5.033126C4.303861-5.033126 4.495143-4.99726 4.65056-4.889664Z'/>
+<path id='g2-115' d='M2.725778-2.391034C2.929016-2.355168 3.251806-2.283437 3.323537-2.271482C3.478954-2.223661 4.016936-2.032379 4.016936-1.458531C4.016936-1.08792 3.682192-.119552 2.295392-.119552C2.044334-.119552 1.147696-.155417 .908593-.812951C1.3868-.753176 1.625903-1.123786 1.625903-1.3868C1.625903-1.637858 1.458531-1.769365 1.219427-1.769365C.956413-1.769365 .609714-1.566127 .609714-1.028144C.609714-.32279 1.327024 .119552 2.283437 .119552C4.100623 .119552 4.638605-1.219427 4.638605-1.841096C4.638605-2.020423 4.638605-2.355168 4.25604-2.737733C3.957161-3.024658 3.670237-3.084433 3.024658-3.21594C2.701868-3.287671 2.187796-3.395268 2.187796-3.93325C2.187796-4.172354 2.402989-5.033126 3.53873-5.033126C4.040847-5.033126 4.531009-4.841843 4.65056-4.411457C4.124533-4.411457 4.100623-3.957161 4.100623-3.945205C4.100623-3.694147 4.327771-3.622416 4.435367-3.622416C4.60274-3.622416 4.937484-3.753923 4.937484-4.25604S4.483188-5.272229 3.550685-5.272229C1.984558-5.272229 1.566127-4.040847 1.566127-3.550685C1.566127-2.642092 2.450809-2.450809 2.725778-2.391034Z'/>
+<path id='g2-116' d='M2.402989-4.805978H3.502864C3.730012-4.805978 3.849564-4.805978 3.849564-5.021171C3.849564-5.152677 3.777833-5.152677 3.53873-5.152677H2.486675L2.929016-6.898132C2.976837-7.065504 2.976837-7.089415 2.976837-7.173101C2.976837-7.364384 2.82142-7.47198 2.666002-7.47198C2.570361-7.47198 2.295392-7.436115 2.199751-7.053549L1.733499-5.152677H.609714C.37061-5.152677 .263014-5.152677 .263014-4.925529C.263014-4.805978 .3467-4.805978 .573848-4.805978H1.637858L.848817-1.649813C.753176-1.231382 .71731-1.111831 .71731-.956413C.71731-.394521 1.111831 .119552 1.78132 .119552C2.988792 .119552 3.634371-1.625903 3.634371-1.709589C3.634371-1.78132 3.58655-1.817186 3.514819-1.817186C3.490909-1.817186 3.443088-1.817186 3.419178-1.769365C3.407223-1.75741 3.395268-1.745455 3.311582-1.554172C3.060523-.956413 2.510585-.119552 1.817186-.119552C1.458531-.119552 1.43462-.418431 1.43462-.681445C1.43462-.6934 1.43462-.920548 1.470486-1.06401L2.402989-4.805978Z'/>
+<path id='g2-117' d='M4.076712-.6934C4.23213-.02391 4.805978 .119552 5.092902 .119552C5.475467 .119552 5.762391-.131507 5.953674-.537983C6.156912-.968369 6.312329-1.673724 6.312329-1.709589C6.312329-1.769365 6.264508-1.817186 6.192777-1.817186C6.085181-1.817186 6.073225-1.75741 6.025405-1.578082C5.810212-.753176 5.595019-.119552 5.116812-.119552C4.758157-.119552 4.758157-.514072 4.758157-.669489C4.758157-.944458 4.794022-1.06401 4.913574-1.566127C4.99726-1.888917 5.080946-2.211706 5.152677-2.546451L5.642839-4.495143C5.726526-4.794022 5.726526-4.817933 5.726526-4.853798C5.726526-5.033126 5.583064-5.152677 5.403736-5.152677C5.057036-5.152677 4.97335-4.853798 4.901619-4.554919C4.782067-4.088667 4.136488-1.518306 4.052802-1.099875C4.040847-1.099875 3.574595-.119552 2.701868-.119552C2.080199-.119552 1.960648-.657534 1.960648-1.099875C1.960648-1.78132 2.295392-2.737733 2.606227-3.53873C2.749689-3.921295 2.809465-4.076712 2.809465-4.315816C2.809465-4.829888 2.438854-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.578829 1.374844-5.033126 1.829141-5.033126C1.948692-5.033126 2.139975-5.021171 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.303113-2.10411 1.243337-1.649813 1.243337-1.291158C1.243337-.071731 2.163885 .119552 2.654047 .119552C3.419178 .119552 3.837609-.406476 4.076712-.6934Z'/>
+<path id='g2-121' d='M3.144209 1.338979C2.82142 1.793275 2.355168 2.199751 1.769365 2.199751C1.625903 2.199751 1.052055 2.175841 .872727 1.625903C.908593 1.637858 .968369 1.637858 .992279 1.637858C1.350934 1.637858 1.590037 1.327024 1.590037 1.052055S1.362889 .681445 1.183562 .681445C.992279 .681445 .573848 .824907 .573848 1.41071C.573848 2.020423 1.08792 2.438854 1.769365 2.438854C2.964882 2.438854 4.172354 1.338979 4.507098 .011955L5.678705-4.65056C5.69066-4.710336 5.71457-4.782067 5.71457-4.853798C5.71457-5.033126 5.571108-5.152677 5.391781-5.152677C5.284184-5.152677 5.033126-5.104857 4.937484-4.746202L4.052802-1.231382C3.993026-1.016189 3.993026-.992279 3.897385-.860772C3.658281-.526027 3.263761-.119552 2.689913-.119552C2.020423-.119552 1.960648-.777086 1.960648-1.099875C1.960648-1.78132 2.283437-2.701868 2.606227-3.56264C2.737733-3.90934 2.809465-4.076712 2.809465-4.315816C2.809465-4.817933 2.450809-5.272229 1.865006-5.272229C.765131-5.272229 .32279-3.53873 .32279-3.443088C.32279-3.395268 .37061-3.335492 .454296-3.335492C.561893-3.335492 .573848-3.383313 .621669-3.550685C.908593-4.554919 1.362889-5.033126 1.829141-5.033126C1.936737-5.033126 2.139975-5.033126 2.139975-4.638605C2.139975-4.327771 2.008468-3.981071 1.829141-3.526775C1.243337-1.960648 1.243337-1.566127 1.243337-1.279203C1.243337-.143462 2.056289 .119552 2.654047 .119552C3.000747 .119552 3.431133 .011955 3.849564-.430386L3.861519-.418431C3.682192 .286924 3.56264 .753176 3.144209 1.338979Z'/>
+</defs>
+<g id='page1' transform='matrix(1.13 0 0 1.13 -63.986043 -62.281577)'>
+<use x='56.413267' y='73.369366' xlink:href='#g2-109'/>
+<use x='66.652534' y='73.369366' xlink:href='#g2-101'/>
+<use x='72.077974' y='73.369366' xlink:href='#g2-109'/>
+<use x='82.317241' y='73.369366' xlink:href='#g2-111'/>
+<use x='87.944679' y='73.369366' xlink:href='#g2-114'/>
+<use x='93.545152' y='73.369366' xlink:href='#g2-121'/>
+<use x='99.681804' y='73.369366' xlink:href='#g2-58'/>
+<use x='102.933465' y='73.369366' xlink:href='#g2-109'/>
+<use x='113.172732' y='73.369366' xlink:href='#g2-105'/>
+<use x='117.166164' y='73.369366' xlink:href='#g2-110'/>
+<use x='127.474599' y='73.369366' xlink:href='#g4-61'/>
+<use x='145.965287' y='58.425345' xlink:href='#g1-110'/>
+<use x='139.90008' y='62.011901' xlink:href='#g0-88'/>
+<use x='141.682474' y='87.206256' xlink:href='#g1-105'/>
+<use x='144.565614' y='87.206256' xlink:href='#g3-61'/>
+<use x='151.15212' y='87.206256' xlink:href='#g3-48'/>
+<use x='159.161195' y='73.369366' xlink:href='#g2-112'/>
+<use x='165.036338' y='73.369366' xlink:href='#g2-111'/>
+<use x='170.663775' y='73.369366' xlink:href='#g2-100'/>
+<use x='176.746468' y='73.369366' xlink:href='#g2-58'/>
+<use x='179.998129' y='73.369366' xlink:href='#g2-115'/>
+<use x='185.512135' y='73.369366' xlink:href='#g2-112'/>
+<use x='191.387278' y='73.369366' xlink:href='#g2-101'/>
+<use x='196.812718' y='73.369366' xlink:href='#g2-99'/>
+<use x='201.850707' y='73.369366' xlink:href='#g2-58'/>
+<use x='205.102368' y='73.369366' xlink:href='#g2-99'/>
+<use x='210.140357' y='73.369366' xlink:href='#g2-111'/>
+<use x='215.767794' y='73.369366' xlink:href='#g2-110'/>
+<use x='222.7554' y='73.369366' xlink:href='#g2-116'/>
+<use x='226.982559' y='73.369366' xlink:href='#g2-97'/>
+<use x='233.127504' y='73.369366' xlink:href='#g2-105'/>
+<use x='237.120936' y='73.369366' xlink:href='#g2-110'/>
+<use x='244.108542' y='73.369366' xlink:href='#g2-101'/>
+<use x='249.533982' y='73.369366' xlink:href='#g2-114'/>
+<use x='255.134455' y='73.369366' xlink:href='#g2-115'/>
+<use x='260.648461' y='73.369366' xlink:href='#g4-91'/>
+<use x='263.900122' y='73.369366' xlink:href='#g2-105'/>
+<use x='267.893554' y='73.369366' xlink:href='#g4-93'/>
+<use x='271.145216' y='73.369366' xlink:href='#g2-58'/>
+<use x='274.396877' y='73.369366' xlink:href='#g2-114'/>
+<use x='279.99735' y='73.369366' xlink:href='#g2-101'/>
+<use x='285.42279' y='73.369366' xlink:href='#g2-115'/>
+<use x='290.936796' y='73.369366' xlink:href='#g2-111'/>
+<use x='296.564234' y='73.369366' xlink:href='#g2-117'/>
+<use x='303.226673' y='73.369366' xlink:href='#g2-114'/>
+<use x='308.827147' y='73.369366' xlink:href='#g2-99'/>
+<use x='313.865135' y='73.369366' xlink:href='#g2-101'/>
+<use x='319.290575' y='73.369366' xlink:href='#g2-115'/>
+<use x='324.804581' y='73.369366' xlink:href='#g2-58'/>
+<use x='328.056242' y='73.369366' xlink:href='#g2-114'/>
+<use x='333.656716' y='73.369366' xlink:href='#g2-101'/>
+<use x='339.082156' y='73.369366' xlink:href='#g2-113'/>
+<use x='344.701312' y='73.369366' xlink:href='#g2-117'/>
+<use x='351.363752' y='73.369366' xlink:href='#g2-101'/>
+<use x='356.789192' y='73.369366' xlink:href='#g2-115'/>
+<use x='362.303198' y='73.369366' xlink:href='#g2-116'/>
+<use x='366.530357' y='73.369366' xlink:href='#g2-115'/>
+<use x='372.044363' y='73.369366' xlink:href='#g4-91'/>
+<use x='375.296024' y='73.369366' xlink:href='#g2-109'/>
+<use x='385.535291' y='73.369366' xlink:href='#g2-101'/>
+<use x='390.960731' y='73.369366' xlink:href='#g2-109'/>
+<use x='401.199998' y='73.369366' xlink:href='#g2-111'/>
+<use x='406.827436' y='73.369366' xlink:href='#g2-114'/>
+<use x='412.427909' y='73.369366' xlink:href='#g2-121'/>
+<use x='418.564561' y='73.369366' xlink:href='#g4-93'/>
+</g>
+</svg>
@@ -0,0 +1,343 @@
+---
+layout: blog
+title: "Contribution, containers and cricket: the Kubernetes 1.22 release interview"
+date: 2021-12-01
+---
+
+**Author**: Craig Box (Google)
+
+The Kubernetes release train rolls on, and we look ahead to the release of 1.23 next week. [As is our tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the process that brought us the previous version.
+
+The release team for 1.22 was led by [Savitha Raghunathan](https://twitter.com/coffeeartgirl), who was, at the time, a Senior Platform Engineer at MathWorks. [I spoke to Savitha](https://kubernetespodcast.com/episode/157-kubernetes-1.22/) on the [Kubernetes Podcast from Google](https://kubernetespodcast.com/), the weekly<super>*</super> show covering the Kubernetes and Cloud Native ecosystem.
+
+Our release conversations shine a light on the team that puts together each Kubernetes release. Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/) so you catch the story of 1.23.
+
+And in case you're interested in why the show has been on a hiatus the last few weeks, all will be revealed in the next episode!
+
+*This transcript has been lightly edited and condensed for clarity.*
+
+---
+
+**CRAIG BOX: Welcome to the show, Savitha.**
+
+SAVITHA RAGHUNATHAN: Hey, Craig. Thanks for having me on the show. How are you today?
+
+**CRAIG BOX: I'm very well, thank you. I've interviewed a lot of people on the show, and you're actually the first person who's asked that of me.**
+
+SAVITHA RAGHUNATHAN: I'm glad. It's something that I always do. I just want to make sure the other person is good and happy.
+
+**CRAIG BOX: That's very kind of you. Thank you for kicking off on a wonderful foot there. I want to ask first of all — you grew up in Chennai. My association with Chennai is the [Super Kings cricket team](https://en.wikipedia.org/wiki/Chennai_Super_Kings). Was cricket part of your upbringing?**
+
+SAVITHA RAGHUNATHAN: Yeah. Actually, a lot. My mom loves watching cricket. I have a younger brother, and when we were growing up, we used to play cricket on the terrace. Everyone surrounding me, my best friends — and even now, my partner — loves watching cricket, too. Cricket is a part of my life.
+
+I stopped watching it a while ago, but I still enjoy a good game.
+
+**CRAIG BOX: It's probably a bit harder in the US. Everything's in a different time zone. I find, with my cricket team being on the other side of the world, that it's a lot easier when they're playing near me, as opposed to trying to keep up with what they're doing when they're playing at 3:00 in the morning.**
+
+SAVITHA RAGHUNATHAN: That is actually one of the things that made me lose touch with cricket. I'm going to give you a piece of interesting information. I never supported Chennai Super Kings. I always supported [Royal Challengers of Bangalore](https://en.wikipedia.org/wiki/Royal_Challengers_Bangalore).
+
+I once went to the stadium, and it was a match between the Chennai Super Kings and the RCB. I was the only one who was cheering whenever the RCB hit a 6, or when they were scoring. I got the stares of thousands of people looking at me. I'm like, "what are you doing?" My friends are like, "you're going to get us killed! Just stop screaming!"
+
+**CRAIG BOX: I hear you. As a New Zealander in the UK, there are a lot of international cricket matches I've been to where I am one of the few people dressed in the full beige kit. But I have to ask, why an affiliation with a different team?**
+
+SAVITHA RAGHUNATHAN: I'm not sure. When the IPL came out, I really liked Virat Kohli. He was playing for RCB at that time, and I think pretty much that's it.
+
+**CRAIG BOX: Well, what I know about the Chennai Super Kings is that their coach is New Zealand's finest batsmen and [air conditioning salesman](https://www.youtube.com/watch?v=vSZAaUCAclw), [Stephen Fleming](https://en.wikipedia.org/wiki/Stephen_Fleming).**
+
+SAVITHA RAGHUNATHAN: Oh, really?
+
+**CRAIG BOX: Yeah, he's a dead ringer for the guy who played the [yellow Wiggle](https://s1.reutersmedia.net/resources/r/?m=02&d=20061130&t=2&i=153531&w=&fh=545px&fw=&ll=&pl=&sq=&r=153531) back in the day.**
+
+SAVITHA RAGHUNATHAN: Oh, interesting. I remember the name, but I cannot put the picture and the name together. I stopped watching cricket once I moved to the States. Then, all my focus was on studies and extracurriculars. I have always been an introvert. The campus — it was a new thing for me — they had international festivals.
+
+And every week, they'd have some kind of new thing going on, so I'd go check them out. I wouldn't participate, but I did go out and check them out. That was a big feat for me around that time because a lot of people — and still, even now, a lot of people — they kind of scare me. I don't know how to make a conversation with everyone.
+
+I'll just go and say, "hi, how are you? OK, I'm good. I'm just going to move on". And I'll just go to the next person. And after two hours, I'm out of that place.
+
+**CRAIG BOX: Perhaps a pleasant side effect of the last 12 months — a lot fewer gatherings of people.**
+
+SAVITHA RAGHUNATHAN: Could be that, but I'm so excited about KubeCon. But when I think about it, I'm like "oh my God. There's going to be a lot of people. What am I going to do? I'm going to meet all my friends over there".
+
+Sometimes I have social anxiety like, what's going to happen?
+
+**CRAIG BOX: What's going to happen is you're going to ask them how they are at the beginning, and they're immediately going to be set at ease.**
+
+SAVITHA RAGHUNATHAN: *laughs* I hope so.
+
+**CRAIG BOX: Let's talk a little bit, then, about your transition from India to the US. You did your undergraduate degree in computer science at the SSN College of Engineering. How did you end up at Arizona State?**
+
+SAVITHA RAGHUNATHAN: I always wanted to pursue higher studies when I was in India, and I didn't have the opportunity immediately. Once I graduated from my school there, I went and I worked for a couple of years. My aim was always to get out of there and come here, do my graduate studies.
+
+Eventually, I want to do a PhD. I have an idea of what I want to do. I always wanted to keep studying. If there's an option that I could just keep studying and not do work or anything of that sort, I'd just pick that other one — I'll just keep studying.
+
+But unfortunately, you need money and other things to live and sustain in this world. So I'm like, OK, I'll take a break from studies, and I will work for a while.
+
+**CRAIG BOX: The road to success is littered with dreams of PhDs. I have a lot of friends who thought that that was the path they were going to take, and they've had a beautiful career and probably aren't going to go back to study. Did you use the [Matlab](https://en.wikipedia.org/wiki/MATLAB) software at all while you were going through your schooling?**
+
+SAVITHA RAGHUNATHAN: No, unfortunately. That is a question that everyone asks. I have not used Matlab. I haven't used it even now. I don't use it for work. I didn't have any necessity for my school work. I didn't have anything to do with Matlab. I never analysed, or did data processing, or anything, with Matlab. So unfortunately, no.
+
+Everyone asks me like, you're working at [MathWorks](https://en.wikipedia.org/wiki/MathWorks). Have you used Matlab? I'm like, no.
+
+**CRAIG BOX: Fair enough. Nor have I. But it's been around since the late 1970s, so I imagine there are a lot of people who will have come across it at some point. Do you work with a lot of people who have been working on it that whole time?**
+
+SAVITHA RAGHUNATHAN: Kind of. Not all the time, but I get to meet some folks who work on the product itself. Most of my interactions are with the infrastructure team and platform engineering teams at MathWorks. One other interesting fact is that when I joined the company — MathWorks has an extensive internal curriculum for training and learning, which I really love. They have an "Intro to Matlab" course, and that's on my bucket of things to do.
+
+It was like 500 years ago. I added it, and I never got to it. I'm like, OK, maybe this year at least I want to get to it and I want to learn something new. My partner used Matlab extensively. He misses it right now at his current employer. And he's like, "you have the entire licence! You have access to the entire suite and you haven't used it?" I'm like, "no!"
+
+**CRAIG BOX: Well, I have bad news for the idea of you doing a PhD, I'm sorry.**
+
+SAVITHA RAGHUNATHAN: Another thing is that none of my family knew about the company MathWorks and Matlab. The only person who knew was my younger brother. He was so proud. He was like, "oh my God".
+
+When he was 12 years old, he started getting involved in robotics and all that stuff. That's how he got introduced to Matlab. He goes absolutely bananas for the swag. So all the t-shirts, all the hoodies — any swag that I get from MathWorks goes to him, without saying.
+
+Over the five, six years, the things that I've got — there was only one sweatshirt that I kept for myself. Everything else I've just given to him. And he cherishes it. He's the only one in my family who knew about Matlab and MathWorks.
+
+Now, everyone knows, because I'm working there. They were initially like, I don't even know that company name. Is it like Amazon? I'm like, no, we make software that can send people to the moon. And we also make software that can do amazing robotic surgeries and even make a car drive on its own. That's something that I take immense pride in.
+
+I know I don't directly work on the product, but I'm enabling the people who are creating the product. I'm really, really proud of that.
+
+**CRAIG BOX: I think Jeff Bezos is working on at least two out of three of those disciplines that you mentioned before, so it's maybe a little bit like Amazon. One thing I've always thought about Matlab is that, because it's called Matlab, it solves that whole problem where [Americans call it math, and the rest of the world call it maths](https://www.grammar.com/math_vs._maths). Why do Americans think there's only one math?**
+
+SAVITHA RAGHUNATHAN: Definitely. I had trouble — growing up in India, it's always British English. And I had so much trouble when I moved here. So many things changed.
+
+One of the things is maths. I always got used to writing maths, physics, and everything.
+
+**CRAIG BOX: They don't call it "physic" in the US, do they?**
+
+SAVITHA RAGHUNATHAN: No, no, they don't. Luckily, they don't. That still stays "physics". But math — I had trouble. It's maths. Even when you do the full abbreviations like mathematics and you are still calling it math, I'm like, mm.
+
+**CRAIG BOX: They can do the computer science abbreviation thing and call it math-7-S or whatever the number of letters is.**
+
+SAVITHA RAGHUNATHAN: Just like Kubernetes. K-8-s.
+
+**CRAIG BOX: Your path to Kubernetes is through MathWorks. They started out as a company making software which was distributed in a physical sense — boxed copies, if you will. I understand now there is a cloud version. Can I assume that that is where the two worlds intersect?**
+
+SAVITHA RAGHUNATHAN: Kind of. I have interaction with the team that supports Matlab on the cloud, but I don't get to work with them on a day-to-day basis. They use Docker containers, and they are building the platform using Kubernetes. So yeah, a little bit of that.
+
+**CRAIG BOX: So what exactly is the platform that you are engineering day to day?**
+
+SAVITHA RAGHUNATHAN: Providing Kubernetes as a platform, obviously — that goes without saying — to some of the internal development teams. In the future we might expand it to more teams within the company. That is a focus area right now, so that's what we are doing. In the process, we might even get to work with the people who are deploying Matlab on the cloud, which is exciting.
+
+**CRAIG BOX: Now, your path to contribution to Kubernetes, you've said before, was through [fixing a 404 error on the Kubernetes.io website](https://github.com/kubernetes/website/pull/15588). Do you remember what the page was?**
+
+SAVITHA RAGHUNATHAN: I do. I was going to something for work, and I came across this changelog. In Kubernetes there's a nice page — once you got to the release page, there would be a long list of changelogs.
+
+One of the things that I fixed was, the person who worked on the feature had changed their GitHub handle, and that wasn't reflected on this page. So that was my first. I got curious and clicked on the links. One of the links was the handle, and that went to a 404. And I was like "Yeah, I'll just fix that. They have done all the hard work. They can get the credit that's due".
+
+It was easy. It wasn't overwhelming for me to pick it up as my first issue. Before that I logged on around Kubernetes for about six to eight months without doing anything because it was just a lot.
+
+**CRAIG BOX: One of the other things that you said about your initial contribution is that you had to learn how to use Git. As a very powerful tool, I find Git is a high barrier to entry for even contributing code to a project. When you want to contribute a blog post or documentation or a fix like you did before, I find it almost impossible to think how a new user would come along and do that. What was your process? Do you think that there's anything we can do to make that barrier lower for new contributors?**
+
+SAVITHA RAGHUNATHAN: Of course. There are more and more tutorials available these days. There is a new contributor workshop. They actually have a [GitHub workflow section](https://www.kubernetes.dev/docs/guide/github-workflow/), [how to do a pull request](https://www.kubernetes.dev/docs/guide/pull-requests/) and stuff like that. I know a couple of folks from SIG Docs that are working on which Git commands that you need, or how to get to writing something small and getting it committed. But more tutorials or more links to intro to Git would definitely help.
+
+The thing is also, someone like a documentation writer — they don't actually want to know the entirety of Git. Honestly, it's an ocean. I don't know how to do it. Most of the time, I still ask for help even though I work with Git on a day to day basis. There are several articles and a lot of help is available already within the community. Maybe we could just add a couple more to [kubernetes.dev](https://kubernetes.dev/). That is an amazing site for all the new contributors and existing contributors who want to build code, who want to write documentation.
+
+We could just add a tutorial there like, "hey, don't know Git, you are new to Git? You just need to know these main things".
+
+**CRAIG BOX: I find it a shame, to be honest, that people need to use Git for that, by comparison to Wikipedia where you can come along, and even though it might be written in Markdown or something like it, it seems like the barrier is a lot lower. Similar to you, I always have to look up anything more complicated than the five or six Git commands that I use on a day to day basis. Even to do simple things, I basically just go and follow a recipe which I find on the internet.**
+
+SAVITHA RAGHUNATHAN: This is how I got introduced to one of the amazing mentors in Kubernetes. Everyone knows him by his handle, Dims. It was my second PR to the Kubernetes website, and I made a mistake. I destroyed the Git history. I could not push my reviews and comments — I addressed them. I couldn't push them back.
+
+My immediate thought was to delete it and recreate, do another pull request. But then I was like, "what happens to others who have already put effort into reviewing them?" I asked for help, and Dims was there.
+
+I would say I just got lucky he was there. And he was like, "OK, let me walk you through". We did troubleshooting through Slack messages. I copied and pasted all the errors. Every single command that he said, I copied and pasted. And then he was like, "OK, run this one. Try this one. And do this one".
+
+Finally, I got it fixed. So you know what I did? I went and I stored the command history somewhere local for the next time when I run into this problem. Luckily, I haven't. But I find the contributors so helpful. They are busy. They have a lot of things to do, but they take moments to stop and help someone who's new.
+
+That is also another part of the reason why I stay — I want to contribute more. It's mainly the community. It's the Kubernetes community. I know you asked me about Git, and I just took the conversation to the Kubernetes community. That's how my brain works.
+
+**CRAIG BOX: A lot of people in the community do that and think that's fantastic, obviously, people like Dims who are just floating around on Slack and seem to have endless time. I don't know how they do it.**
+
+SAVITHA RAGHUNATHAN: I really want to know the secret for endless time. If I only had 48 hours in a day. I would sleep for 16 hours, and I would use the rest of the time for doing the things that I want.
+
+**CRAIG BOX: If I had a chance to sleep up to 48 hours a day, I think it'd be a lot more than 16.**
+
+**Now, one of the areas that you've been contributing to Kubernetes is in the release team. In 1.18, you were a shadow for the docs role. You led that role in 1.19. And you were a release lead shadow for versions 1,20 and 1.21 before finally leading this release, 1.22, which we will talk about soon.**
+
+**How did you get involved? And how did you decide which roles to take as you went through that process?**
+
+SAVITHA RAGHUNATHAN: That is a topic I love to talk about. This was fresh when I started learning about Kubernetes and using Kubernetes at work. And I got so much help from the community, I got interested in contributing back.
+
+At the first KubeCon that I attended in 2018, in Seattle, they had a speed mentoring session. Now they call it "pod mentoring". I went to the session, and said, "hey, I want to contribute. I don't know where to start". And I got a lot of information on how to get started.
+
+One of the places was SIG Release and the release team. I came back and diligently attended all the SIG Release meetings for four to six months. And in between, I applied to the Kubernetes release team — 1.14 and 1.15. I didn't get through. So I took a little bit of a break, and I focused on doing some documentation work. Then I applied for 1.18.
+
+Since I was already working on some kinds of — not like full fledged "documentation" documentation, I still don't write. I eventually want to write something really nice and full fledged documentation like other awesome folks.
+
+**CRAIG BOX: You'll need a lot more than 48 hours in your day to do that.**
+
+SAVITHA RAGHUNATHAN: *laughing* That's how I applied for the docs role, because I know a little bit about the website. I've done a few pull requests and commits. That's how I got started. I applied for that one role, and I got selected for the 1.18 team. That's how my journey just took off.
+
+And the next release, I was leading the documentation team. And as everyone knows, the pandemic hit. It was one of the longest releases. I could lean back on the community. I would just wait for the release team meetings.
+
+It was my way of coping with the pandemic. It took my mind off. It was actually more than a release team, they were people. They were all people first, and we took care of each other. So it felt good.
+
+And then, I became a release lead shadow for 1.20 and 1.21 because I wanted to know more. I wanted to learn more. I wasn't ready. I still don't feel ready, but I have led 1.22. So if I could do it, anyone could do it.
+
+**CRAIG BOX: How much of this work is day job?**
+
+SAVITHA RAGHUNATHAN: I am lucky to be blessed with an awesome team. I do most of my work after work, but there have been times where I have to take meetings and attend to immediate urgent stuff. During the time of exception requests and stuff like that, I take a little bit of time from my work.
+
+My team has been wonderful: they support me in all possible ways, and the management as well. Other than the meetings, I don't do much of the work during the day job. It just takes my focus and attention away too much, and I end up having to spend a lot of time sitting in front of the computer, which I don't like.
+
+Before the pandemic I had a good work life balance. I'd just go to work at 7:00, 7:30, and I'd be back by 4 o'clock. I never touched my laptop ever again. I left all work behind when I came home. So right now, I'm still learning how to get through.
+
+I try to limit the amount of open source work that I do during work time. The release lead shadow and the release lead job — they require a lot of time, effort. So on average, I'd be spending two to three hours post work time on the release activities.
+
+**CRAIG BOX: Before the pandemic, everyone was worried that if we let people work from home, they wouldn't work enough. I think the opposite has actually happened, is that now we're worried that if we let people work from home, they will just get on the computer in the morning and you'll have to pry it out of their hands at midnight.**
+
+SAVITHA RAGHUNATHAN: Yeah, I think the productivity has increased at least twofold, I would say, for everyone, once they started working from home.
+
+**CRAIG BOX: But at the expense of work-life balance, though, because as you say, when you're sitting in the same chair in front of, perhaps, the same computer doing your MathWorks work and then your open source work, they kind of can blur into one perhaps?**
+
+SAVITHA RAGHUNATHAN: That is a challenge. I face it every day. But so many others are also facing it. I implemented a few little tricks to help me. When I used to come back home from work, the first thing I would do is remove my watch. That was an indication that OK, I'm done.
+
+That's the thing that I still do. I just remove my watch, and I just keep it right where my workstation is. And I just close the door so that I never look back. Even going past the room, I don't get a glimpse of my work office. I start implementing tiny little things like that to avoid burnout.
+
+I think I'm still facing a little bit of burnout. I don't know if I have fully recovered from it. I constantly feel like I need a vacation. And I could just take a vacation for like a month or two. If it's possible, I will just do it.
+
+**CRAIG BOX: I do hope that travel opens up for everyone as an opportunity because I know that, for a lot of people, it's not so much they've been working from home but they've been living at work. The idea of taking vacation effectively means, well, I've been stuck in the same place, if I've been under a lockdown. It's hard to justify that. It will be good as things improve worldwide for us to be able to start focusing more on mental health and perhaps getting away from the "everything room," as I sometimes call it.**
+
+SAVITHA RAGHUNATHAN: I'm totally looking forward to it. I hope that travel opens up and I could go home and I could meet my siblings and my aunt and my parents.
+
+**CRAIG BOX: Catch a cricket match?**
+
+SAVITHA RAGHUNATHAN: Yeah. Probably yes, if I have company and if there is anything interesting happening around the time. I don't mind going back to the Chepauk Stadium and catching a match or two.
+
+**CRAIG BOX: Let's turn now to the recently released [Kubernetes 1.22](https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/). Congratulations on the launch.**
+
+SAVITHA RAGHUNATHAN: Thank you.
+
+**CRAIG BOX: Each launch comes with a theme and a mascot or a logo. What is the theme for this release?**
+
+SAVITHA RAGHUNATHAN: The theme for the release is reaching new peaks. I am fascinated with a lot of space travel and chasing stars, the Milky Way. The best place to do that is over the top of a mountain. So that is the release logo, basically. It's a mountain — Mount Rainier. On top of that, there is a Kubernetes flag, and it's overlooking the Milky Way.
+
+It's also symbolic that with every release, that we are achieving something new, bigger, and better, and we are making the release awesome. So I just wanted to incorporate that into the team as to say, we are achieving new things with every release. That's the "reaching new peaks" theme.
+
+**CRAIG BOX: The last couple of releases have both been incrementally larger — as a result, perhaps, of the fact there are now only three releases per year rather than four. There were also changes to the process, where the work has been driven a lot more by the SIGs than by the release team having to go and ask the SIGs what was going on. What can you say about the size and scope of the 1.22 release?**
+
+SAVITHA RAGHUNATHAN: The 1.22 release is the largest release to date. We have 56 enhancements if I'm not wrong, and we have a good amount of features that's graduated as stable. You can now say that Kubernetes as a project has become more mature because you see new features coming in. At the same time, you see the features that weren't used getting deprecated — we have like three deprecations in this release.
+
+Aside from that fact, we also have a big team that's supporting one of the longest releases. This is the first official release cycle after the cadence KEP got approved. Officially, we are at four months, even though 1.19 was six months, and 1.21 was like 3 and 1/2 months, I think, this is the first one after the official KEP approval.
+
+**CRAIG BOX: What changes did you make to the process knowing that you had that extra month?**
+
+SAVITHA RAGHUNATHAN: One of the things the community had asked for is more time for development. We tried to incorporate that in the release schedule. We had about six weeks between the enhancements freeze and the code freeze. That's one.
+
+It might not be visible to everyone, but one of the things that I wanted to make sure of was the health of the team — since it was a long, long release, we had time to plan out, and not have everyone work during the weekends or during their evenings or time off. That actually helped everyone keep their sanity, and also in making good progress and delivering good results at the end of the release. That's one of the process improvements that I'd call out.
+
+We got better by making a post during the exception request process. Everyone works around the world. People from the UK start a little earlier than the people in the US East Coast. The West Coast starts three hours later than the East Coast. We used to make a post every Friday evening saying "hey, we actually received this many requests. We have addressed a number of them. We are waiting on a couple, or whatever. All the release team members are done for the day. We will see you around on Monday. Have a good weekend." Something like that.
+
+We set the expectations from the community as well. We understand things are really important and urgent, but we are done. This gave everyone their time back. They don't have to worry over the weekend thinking like, hey, what's happening? What's happening in the release? They could spend time with their family, or they could do whatever they want to do, like go on a hike, or just sit and watch TV.
+
+There have been weekends that I just did that. I just binge-watched a series. That's what I did.
+
+**CRAIG BOX: Any recommendations?**
+
+SAVITHA RAGHUNATHAN: I'm a big fan of Marvel, so I have watched the new [Loki](https://en.wikipedia.org/wiki/Loki_(TV_series)), which I really love. Loki is one of my favourite characters in Marvel. And I also liked [WandaVision](https://en.wikipedia.org/wiki/WandaVision). That was good, too.
+
+**CRAIG BOX: I've not seen Loki yet, but I've heard it described as the best series of Doctor Who in the last few years.**
+
+SAVITHA RAGHUNATHAN: Really?
+
+**CRAIG BOX: There must be an element of time-travelling in there if that's how people are describing it.**
+
+SAVITHA RAGHUNATHAN: You should really go and watch it whenever you have time. It's really amazing. I might go back and watch it again because I might have missed bits and pieces. That always happens in Marvel movies and the episodes; you need to watch them a couple of times to catch, "oh, this is how they relate".
+
+**CRAIG BOX: Yes, the mark of good media that you want to immediately go back and watch it again once you've seen it.**
+
+**Let's look now at some of the new features in Kubernetes 1.22. A couple of things that have graduated to general availability — server-side apply, external credential providers, a couple of new security features — the replacement for pod security policy has been announced, and seccomp is now available by default.**
+
+**Do you have any favourite features in 1.22 that you'd like to discuss?**
+
+SAVITHA RAGHUNATHAN: I have a lot of them. All my favourite features are related to security. OK, one of them is not security, but a major theme of my favourite KEPs is security. I'll start with the [default seccomp](https://github.com/kubernetes/enhancements/issues/2413). I think it will help make clusters secure by default, and may assist in preventing more vulnerabilities, which means less headaches for the cluster administrators.
+
+This is close to my heart because the base of the MathWorks platform is provisioning Kubernetes clusters. Knowing that they are secure by default will definitely provide me with some good sleep. And also, I'm paranoid about security most of the time. I'm super interested in making everything secure. It might get in the way of making the users of the platform angry because it's not usable in any way.
+
+My next one is [rootless Kubelet](https://github.com/kubernetes/enhancements/issues/2033). That feature's going to enable the cluster admin, the platform developers to deploy Kubernetes components to run in a user namespace. And I think that is also a great addition.
+
+Like you mention, the most awaited drop in for the PSP replacement is here. It's [pod admission control](https://github.com/kubernetes/enhancements/issues/2579). It lets cluster admins apply the pod security standards. And I think it's just not related to the cluster admins. I might have to go back and check on that. Anyone can probably use it — the developers and the admins alike.
+
+It also supports various modes, which is most welcome. There are times where you don't want to just cut the users off because they are trying to do something which is not securely correct. You just want to warn them, hey, this is what you are doing. This might just cause a security issue later, so you might want to correct it. But you just don't want to cut them off from using the platform, or them trying to attempt to do something — deploy their workload and get their day-to-day job done. That is something that I really like, that it also supports a warning mechanism.
+
+Another one which is not security is [node swap support](https://github.com/kubernetes/enhancements/issues/2400). Kubernetes didn't have support for swap before, but it is taken into consideration now. This is an alpha feature. With this, you can take advantage of the swap, which is provisioned on the Linux VMs.
+
+Some of the workloads — when they are deployed, they might need a lot of swap for the start-up — example, like Node and Java applications, which I just took out of their KEP user stories. So if anyone's interested, they can go and look in the KEP. That's useful. And it also increases the node stability and whatnot. So I think it's going to be beneficial for a lot of folks.
+
+We know how Java and containers work. I think it has gotten better, but five years ago, it was so hard to get a Java application to fit in a small container. It always needed a lot of memory, swap, and everything to start up and run. I think this will help the users and help the admins and keep the cost low, and it will tie into so many other things as well. I'm excited about that feature.
+
+Another feature that I want to just call out — I don't use Windows that much, but I just want to give a shout out to the folks who are doing an amazing job bringing all the Kubernetes features to Windows as well, to give a seamless experience.
+
+One of the things is [Windows privileged containers](https://github.com/kubernetes/enhancements/issues/1981). I think it went alpha this release. And that is a wonderful addition, if you ask me. It can take advantage of whatever that's happening on the Linux side. And they can also port it over and see, OK, I can now run Windows containers in a privileged mode.
+
+So whatever they are trying to achieve, they can do it. So that's a noteworthy mention. I need to give a shout out for the folks who work and make things happen in the Windows ecosystem as well.
+
+**CRAIG BOX: One of the things that's great about the release process is the continuity between groups and teams. There's always an emeritus advisor who was a lead from a previous release. One thing that I always ask when I do these interviews is, what is the advice that you give to the next person? When [we talked to Nabarun for the 1.21 interview](https://kubernetespodcast.com/episode/146-kubernetes-1.21/), he said that his advice to you would be "do, delegate, and defer". Figure out what you can do, figure out what you can ask other people to do, and figure out what doesn't need to be done. Were you able to take that advice on board?**
+
+SAVITHA RAGHUNATHAN: Yeah, you won't believe it. [I have it right here stuck to my monitor.](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/3)
+
+**CRAIG BOX: Next to your Git cheat sheet?**
+
+SAVITHA RAGHUNATHAN: *laughs* Absolutely. I just have it stuck there. I just took a look at it.
+
+**CRAIG BOX: Someone that you will have been able to delegate and defer to is Rey Lejano from Rancher Labs and SUSE, who is the release lead to be for 1.23.**
+
+SAVITHA RAGHUNATHAN: I want to tell Rey to beware of the team's mental health. Schedule in such a way that it avoids burnout. Check in, and make sure that everyone is doing good. If they need some kind of help, create a safe space where they can actually ask for help, if they want to step back, if they need someone to cover.
+
+I think that is most important. The releases are successful based on the thousands and thousands of contributors. But when it comes to a release team, you need to have a healthy team where people feel they are in a good place and they just want to make good contributions, which means they want to be heard. That's one thing that I want to tell Rey.
+
+Also collaborate and learn from each other. I constantly learn. I think the team was 39 folks, including me. Every day I learned something or the other, even starting from how to interact.
+
+Sometimes I have learned more leadership skills from my release lead shadows. They are awesome, and they are mature. I constantly learn from them, and I admire them a lot.
+
+It also helps to have good, strong individuals in the team who can step up and help when needed. For example, unfortunately, we lost one of our teammates after the start of the release cycle. That was tragic. His name was [Peeyush Gupta](https://github.com/cncf/memorials/blob/main/peeyush-gupta.md). He was an awesome and wonderful human — very warm.
+
+I didn't get more of a chance to interact with him. I had exchanged a few Slack messages, but I got his warm personality. I just want to take a couple of seconds to remember him. He was awesome.
+
+After we lost him, we had this strong person from the team step up and lead the communications, who had never been a part of the release team before at all. He was a shadow for the first time. His name is Jesse Butler. So he stepped up, and he just took it away. He ran the comms show for 1.22.
+
+That's what the community is about. You take care of team members, and the team will take care of you. So that's one other thing that I want to let Rey know, and maybe whoever — I think it's applicable overall.
+
+**CRAIG BOX: There's a link to a [family education fund for Peeyush Gupta](https://milaap.org/fundraisers/support-peeyush-gupta-family-education), which you can find in the show notes.**
+
+**Five releases in a row now you've been a member of the release team. Will you be putting your feet up now for 1.23?**
+
+SAVITHA RAGHUNATHAN: I am going to take a break for a while. In the future, I want to be contributing, if not the release team, the SIG Release and the release management effort. But right now, I have been there for five releases. And I feel like, OK, I just need a little bit of fresh air.
+
+And also the pandemic and the burnout has caught up, so I'm going to take a break from certain contributions. You will see me in the future. I will be around, but I might not be actively participating in the release team activities. I will be around the community. Anyone can reach out to me. They all know my Slack, so they can just reach out to me via Slack or Twitter.
+
+**CRAIG BOX: Yes, your Twitter handle is CoffeeArtGirl. Does that mean that you'll be spending some time working on your lattes?**
+
+SAVITHA RAGHUNATHAN: I am very bad at making lattes. The coffee art means that I used to [make art with coffee](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/1). You get  instant coffee powder and just mix it with water. You get the colours, very beautiful brown colours. I used to make art using that.
+
+And I love coffee. So I just combined all the words together. And I had to come up with it in a span of one hour or so because I was joining this 'meet our contributors' panel. And Paris asked me, "do you have a Twitter handle?" I was planning to create one, but I didn't have the time.
+
+I'm like, well, let me just think what I could just come up with real quick. So I just came up with that. So that's the story behind my Twitter handle. Everyone's interested in it. You are not the first person you have asked me or mentioned about it. So many others are like, why coffee art?
+
+**CRAIG BOX: And you are also interested in art with perhaps other materials?**
+
+SAVITHA RAGHUNATHAN: Yes. My interests keep changing. I used to do pebble art. It's just collecting pebbles from wherever I go, and I used to paint on them. I used to use watercolour, but I want to come back to watercolour sometime.
+
+My recent interests are coloured pencils, which came back. When I was very young, I used to do a lot of coloured pencils. And then I switched to watercolours and oil painting. So I just go around in circles.
+
+One of the hobbies that I picked up during a pandemic is crochet. I made a scarf for Mother's Day. My mum and my dad were here last year. They got stuck because of the pandemic, and they couldn't go back home. So they stayed with me for 10 months. That is the jackpot that I had, that I got to spend so much time with my parents after I moved to the US.
+
+**CRAIG BOX: And they got rewarded with a scarf.**
+
+SAVITHA RAGHUNATHAN: Yeah.
+
+**CRAIG BOX: One to share between them.**
+
+SAVITHA RAGHUNATHAN: I started making a blanket for my dad. And it became so heavy, I might have to just pick up some lighter yarn. I still don't know the differences between different kinds of yarns, but I'm getting better.
+
+I started out because I wanted to make these little toys. They call them [amigurumi](https://en.wikipedia.org/wiki/Amigurumi) in the crochet world. I wanted to make them. That's why I started out. I'm trying. I made [a little cat](https://twitter.com/KubernetesPod/status/1423188323347177474/photo/2) which doesn't look like a cat, but it is a cat. I have to tell everyone that it's a cat so that they don't mock me later, but.
+
+**CRAIG BOX: It's an artistic interpretation of a cat.**
+
+SAVITHA RAGHUNATHAN: It definitely is!
+
+---
+
+_[Savitha Raghunathan](https://twitter.com/coffeeartgirl), now a Senior Software Engineer at Red Hat, served as the Kubernetes 1.22 release team lead._
+
+_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._
@@ -19,6 +19,7 @@ cid: community

 <div class="community__navbar">

+<a href="https://www.kubernetes.dev/">Contributor Community</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <a href="#values">Community Values</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <a href="#conduct">Code of conduct </a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <a href="#videos">Videos</a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
@@ -43,11 +43,11 @@ The controllers inside the cloud controller manager include:

 ### Node controller

-The node controller is responsible for creating {{< glossary_tooltip text="Node" term_id="node" >}} objects
+The node controller is responsible for updating {{< glossary_tooltip text="Node" term_id="node" >}} objects
 when new servers are created in your cloud infrastructure. The node controller obtains information about the
 hosts running inside your tenancy with the cloud provider. The node controller performs the following functions:

-1. Initialize a Node object for each server that the controller discovers through the cloud provider API.
+1. Update a Node object with the corresponding server's unique identifier obtained from the cloud provider API.
 2. Annotating and labelling the Node object with cloud-specific information, such as the region the node
   is deployed into and the resources (CPU, memory, etc) that it has available.
 3. Obtain the node's hostname and network addresses.
@@ -0,0 +1,51 @@
+---
+title: Container Runtime Interface (CRI)
+content_type: concept
+weight: 50
+---
+
+<!-- overview -->
+
+The CRI is a plugin interface which enables the kubelet to use a wide variety of
+container runtimes, without having a need to recompile the cluster components.
+
+You need a working
+{{<glossary_tooltip text="container runtime" term_id="container-runtime">}} on
+each Node in your cluster, so that the
+{{< glossary_tooltip text="kubelet" term_id="kubelet" >}} can launch
+{{< glossary_tooltip text="Pods" term_id="pod" >}} and their containers.
+
+{{< glossary_definition term_id="container-runtime-interface" length="all" >}}
+
+<!-- body -->
+
+## The API {#api}
+
+{{< feature-state for_k8s_version="v1.23" state="stable" >}}
+
+The kubelet acts as a client when connecting to the container runtime via gRPC.
+The runtime and image service endpoints have to be available in the container
+runtime, which can be configured separately within the kubelet by using the
+`--image-service-endpoint` and `--container-runtime-endpoint` [command line
+flags](/docs/reference/command-line-tools-reference/kubelet)
+
+For Kubernetes v{{< skew currentVersion >}}, the kubelet prefers to use CRI `v1`.
+If a container runtime does not support `v1` of the CRI, then the kubelet tries to
+negotiate any older supported version.
+The v{{< skew currentVersion >}} kubelet can also negotiate CRI `v1alpha2`, but
+this version is considered as deprecated.
+If the kubelet cannot negotiate a supported CRI version, the kubelet gives up
+and doesn't register as a node.
+
+## Upgrading
+
+When upgrading Kubernetes, then the kubelet tries to automatically select the
+latest CRI version on restart of the component. If that fails, then the fallback
+will take place as mentioned above. If a gRPC re-dial was required because the
+container runtime has been upgraded, then the container runtime must also
+support the initially selected version or the redial is expected to fail. This
+requires a restart of the kubelet.
+
+## {{% heading "whatsnext" %}}
+
+- Learn more about the CRI [protocol definition](https://github.com/kubernetes/cri-api/blob/c75ef5b/pkg/apis/runtime/v1/api.proto)
@@ -72,7 +72,8 @@ The name of a Node object must be a valid
 The [name](/docs/concepts/overview/working-with-objects/names#names) identifies a Node. Two Nodes
 cannot have the same name at the same time. Kubernetes also assumes that a resource with the same
 name is the same object. In case of a Node, it is implicitly assumed that an instance using the
-same name will have the same state (e.g. network settings, root disk contents). This may lead to
+same name will have the same state (e.g. network settings, root disk contents)
+and attributes like node labels. This may lead to
 inconsistencies if an instance was modified without changing its name. If the Node needs to be
 replaced or updated significantly, the existing Node object needs to be removed from API server
 first and re-added after the update.
@@ -98,6 +99,21 @@ When the [Node authorization mode](/docs/reference/access-authn-authz/node/) and
 [NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) are enabled,
 kubelets are only authorized to create/modify their own Node resource.

+{{< note >}}
+As mentioned in the [Node name uniqueness](#node-name-uniqueness) section,
+when Node configuration needs to be updated, it is a good practice to re-register
+the node with the API server. For example, if the kubelet being restarted with
+the new set of `--node-labels`, but the same Node name is used, the change will
+not take an effect, as labels are being set on the Node registration.
+
+Pods already scheduled on the Node may misbehave or cause issues if the Node
+configuration will be changed on kubelet restart. For example, already running
+Pod may be tainted against the new labels assigned to the Node, while other
+Pods, that are incompatible with that Pod will be scheduled based on this new
+label.  Node re-registration ensures all Pods will be drained and properly
+re-scheduled.
+{{< /note >}}
+
 ### Manual Node administration

 You can create and modify Node objects using
@@ -386,7 +402,7 @@ Graceful node shutdown is controlled with the `GracefulNodeShutdown`
 enabled by default in 1.21.

 Note that by default, both configuration options described below,
-`ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are set to zero,
+`shutdownGracePeriod` and `shutdownGracePeriodCriticalPods` are set to zero,
 thus not activating Graceful node shutdown functionality.
 To activate the feature, the two kubelet config settings should be configured appropriately and set to non-zero values.

@@ -396,32 +412,116 @@ During a graceful shutdown, kubelet terminates pods in two phases:
 2. Terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) running on the node.

 Graceful node shutdown feature is configured with two [`KubeletConfiguration`](/docs/tasks/administer-cluster/kubelet-config-file/) options:
-* `ShutdownGracePeriod`:
+* `shutdownGracePeriod`:
  * Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
-* `ShutdownGracePeriodCriticalPods`:
-  * Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `ShutdownGracePeriod`.
+* `shutdownGracePeriodCriticalPods`:
+  * Specifies the duration used to terminate [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) during a node shutdown. This value should be less than `shutdownGracePeriod`.

-For example, if `ShutdownGracePeriod=30s`, and
-`ShutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by
+For example, if `shutdownGracePeriod=30s`, and
+`shutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by
 30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved
 for gracefully terminating normal pods, and the last 10 seconds would be
 reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).

 {{< note >}}
-When pods were evicted during the graceful node shutdown, they are marked as failed.
-Running `kubectl get pods` shows the status of the the evicted pods as `Shutdown`.
+When pods were evicted during the graceful node shutdown, they are marked as shutdown.
+Running `kubectl get pods` shows the status of the the evicted pods as `Terminated`.
 And `kubectl describe pod` indicates that the pod was evicted because of node shutdown:

 ```
-Status:         Failed
-Reason:         Shutdown
-Message:        Node is shutting, evicting pods
+Reason:         Terminated
+Message:        Pod was terminated in response to imminent node shutdown.
 ```

-Failed pod objects will be preserved until explicitly deleted or [cleaned up by the GC](/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection).
-This is a change of behavior compared to abrupt node termination.
 {{< /note >}}

+### Pod Priority based graceful node shutdown {#pod-priority-graceful-node-shutdown}
+
+{{< feature-state state="alpha" for_k8s_version="v1.23" >}}
+
+To provide more flexibility during graceful node shutdown around the ordering
+of pods during shutdown, graceful node shutdown honors the PriorityClass for
+Pods, provided that you enabled this feature in your cluster. The feature
+allows allows cluster administers to explicitly define the ordering of pods
+during graceful node shutdown based on [priority
+classes](docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass).
+
+The [Graceful Node Shutdown](#graceful-node-shutdown) feature, as described
+above, shuts down pods in two phases, non-critical pods, followed by critical
+pods. If additional flexibility is needed to explicitly define the ordering of
+pods during shutdown in a more granular way, pod priority based graceful
+shutdown can be used.
+
+When graceful node shutdown honors pod priorities, this makes it possible to do
+graceful node shutdown in multiple phases, each phase shutting down a
+particular priority class of pods. The kubelet can be configured with the exact
+phases and shutdown time per phase.
+
+Assuming the following custom pod [priority
+classes](docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass)
+in a cluster,
+
+|Pod priority class name|Pod priority class value|
+|-------------------------|------------------------|
+|`custom-class-a`         | 100000                 |
+|`custom-class-b`         | 10000                  |
+|`custom-class-c`         | 1000                   |
+|`regular/unset`          | 0                      |
+
+Within the [kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
+the settings for `shutdownGracePeriodByPodPriority` could look like:
+
+|Pod priority class value|Shutdown period|
+|------------------------|---------------|
+| 100000                 |10 seconds     |
+| 10000                  |180 seconds    |
+| 1000                   |120 seconds    |
+| 0                      |60 seconds     |
+
+The corresponding kubelet config YAML configuration would be:
+
+```yaml
+shutdownGracePeriodByPodPriority:
+  - priority: 100000
+    shutdownGracePeriodSeconds: 10
+  - priority: 10000
+    shutdownGracePeriodSeconds: 180
+  - priority: 1000
+    shutdownGracePeriodSeconds: 120
+  - priority: 0
+    shutdownGracePeriodSeconds: 60
+```
+
+The above table implies that any pod with priority value >= 100000 will get
+just 10 seconds to stop, any pod with value >= 10000 and < 100000 will get 180
+seconds to stop, any pod with value >= 1000 and < 10000 will get 120 seconds to stop.
+Finally, all other pods will get 60 seconds to stop.
+
+One doesn't have to specify values corresponding to all of the classes. For
+example, you could instead use these settings:
+
+|Pod priority class value|Shutdown period|
+|------------------------|---------------|
+| 100000                 |300 seconds    |
+| 1000                   |120 seconds    |
+| 0                      |60 seconds     |
+
+
+In the above case, the pods with custom-class-b will go into the same bucket
+as custom-class-c for shutdown.
+
+If there are no pods in a particular range, then the kubelet does not wait
+for pods in that priority range. Instead, the kubelet immediately skips to the
+next priority class value range.
+
+If this feature is enabled and no configuration is provided, then no ordering
+action will be taken.
+
+Using this feature, requires enabling the
+`GracefulNodeShutdownBasedOnPodPriority` feature gate, and setting the kubelet
+config's `ShutdownGracePeriodByPodPriority` to the desired configuration
+containing the pod priority class values and their respective shutdown periods.
+
 ## Swap memory management {#swap-memory}

 {{< feature-state state="alpha" for_k8s_version="v1.22" >}}
@@ -435,6 +535,11 @@ the kubelet, and the `--fail-swap-on` command line flag or `failSwapOn`
 [configuration setting](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
 must be set to false.

+{{< warning >}}
+When the memory swap feature is turned on, Kubernetes data such as the content
+of Secret objects that were written to tmpfs now could be swapped to disk.
+{{< /warning >}}
+
 A user can also optionally configure `memorySwap.swapBehavior` in order to
 specify how a node will use swap memory. For example,

@@ -25,7 +25,7 @@ This page lists some of the available add-ons and links to their respective inst
 * [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is an open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads.
 * [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) is an overlay network provider that can be used with Kubernetes.
 * [Knitter](https://github.com/ZTE/Knitter/) is a plugin to support multiple network interfaces in a Kubernetes pod.
-* [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
+* Multus is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
 * [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) is a networking provider for Kubernetes based on [OVN (Open Virtual Network)](https://github.com/ovn-org/ovn/), a virtual networking implementation that came out of the Open vSwitch (OVS) project. OVN-Kubernetes provides an overlay based networking implementation for Kubernetes, including an OVS based implementation of load balancing and network policy.
 * [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) is OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC), Multiple OVN overlay networking, dynamic subnet creation, dynamic creation of virtual networks, VLAN Provider network, Direct provider network and pluggable with other Multi-network plugins, ideal for edge based cloud native workloads in Multi-cluster networking
 * [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.
@@ -45,6 +45,11 @@ This page lists some of the available add-ons and links to their respective inst
 ## Infrastructure

 * [KubeVirt](https://kubevirt.io/user-guide/#/installation/installation) is an add-on to run virtual machines on Kubernetes. Usually run on bare-metal clusters.
+* The
+  [node problem detector](https://github.com/kubernetes/node-problem-detector)
+  runs on Linux nodes and reports system issues as either
+  [Events](/docs/reference/kubernetes-api/cluster-resources/event-v1/) or
+  [Node conditions](/docs/concepts/architecture/nodes/#condition).

 ## Legacy Add-ons

@@ -26,6 +26,10 @@ fair queuing technique so that, for example, a poorly-behaved
 {{< glossary_tooltip text="controller" term_id="controller" >}} need not
 starve others (even at the same priority level).

+This feature is designed to work well with standard controllers, which
+use informers and react to failures of API requests with exponential
+back-off, and other clients that also work this way.
+
 {{< caution >}}
 Requests classified as "long-running" — primarily watches — are not
 subject to the API Priority and Fairness filter. This is also true for
@@ -102,6 +106,8 @@ name of the matching FlowSchema plus a _flow distinguisher_ — which
 is either the requesting user, the target resource's namespace, or nothing — and the
 system attempts to give approximately equal weight to requests in different
 flows of the same priority level.
+To enable distinct handling of distinct instances, controllers that have
+many instances should authenticate with distinct usernames

 After classifying a request into a flow, the API Priority and Fairness
 feature then may assign the request to a queue.  This assignment uses
@@ -64,7 +64,7 @@ This means that containers within a `Pod` can all reach each other's ports on
 usage, but this is no different from processes in a VM.  This is called the
 "IP-per-pod" model.

-How this is implemented is a detail of the particular container runtime in use.
+How this is implemented is a detail of the particular container runtime in use. Likewise, the networking option you choose may support [dual-stack IPv4/IPv6 networking](/docs/concepts/services-networking/dual-stack/); implementations vary.

 It is possible to request ports on the `Node` itself which forward to your `Pod`
 (called host ports), but this is a very niche operation. How that forwarding is
@@ -91,18 +91,6 @@ imply any preferential status.
 Project [Antrea](https://github.com/vmware-tanzu/antrea) is an opensource Kubernetes networking solution intended to be Kubernetes native. It leverages Open vSwitch as the networking data plane. Open vSwitch is a high-performance programmable virtual switch that supports both Linux and Windows. Open vSwitch enables Antrea to implement Kubernetes Network Policies in a high-performance and efficient manner.
 Thanks to the "programmable" characteristic of Open vSwitch, Antrea is able to implement an extensive set of networking and security features and services on top of Open vSwitch.

-### AOS from Apstra
-
-[AOS](https://www.apstra.com/products/aos/) is an Intent-Based Networking system that creates and manages complex datacenter environments from a simple integrated platform.  AOS leverages a highly scalable distributed design to eliminate network outages while minimizing costs.
-
-The AOS Reference Design currently supports Layer-3 connected hosts that eliminate legacy Layer-2 switching problems.  These Layer-3 hosts can be Linux servers (Debian, Ubuntu, CentOS) that create BGP neighbor relationships directly with the top of rack switches (TORs).  AOS automates the routing adjacencies and then provides fine grained control over the route health injections (RHI) that are common in a Kubernetes deployment.
-
-AOS has a rich set of REST API endpoints that enable Kubernetes to quickly change the network policy based on application requirements.  Further enhancements will integrate the AOS Graph model used for the network design with the workload provisioning, enabling an end to end management system for both private and public clouds.
-
-AOS supports the use of common vendor equipment from manufacturers including Cisco, Arista, Dell, Mellanox, HPE, and a large number of white-box systems and open network operating systems like Microsoft SONiC, Dell OPX, and Cumulus Linux.
-
-Details on how the AOS system works can be accessed here: https://www.apstra.com/products/how-it-works/
-
 ### AWS VPC CNI for Kubernetes

 The [AWS VPC CNI](https://github.com/aws/amazon-vpc-cni-k8s) offers integrated AWS Virtual Private Cloud (VPC) networking for Kubernetes clusters. This CNI plugin offers high throughput and availability, low latency, and minimal network jitter. Additionally, users can apply existing AWS VPC networking and security best practices for building Kubernetes clusters. This includes the ability to use VPC flow logs, VPC routing policies, and security groups for network traffic isolation.
@@ -116,15 +104,6 @@ Additionally, the CNI can be run alongside [Calico for network policy enforcemen

 Azure CNI is available natively in the [Azure Kubernetes Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni).

-
-### Big Cloud Fabric from Big Switch Networks
-
-[Big Cloud Fabric](https://www.bigswitch.com/container-network-automation) is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premises environments. Using unified physical & virtual SDN, Big Cloud Fabric tackles inherent container networking problems such as load balancing, visibility, troubleshooting, security policies & container traffic monitoring.
-
-With the help of the Big Cloud Fabric's virtual pod multi-tenant architecture, container orchestration systems such as Kubernetes, RedHat OpenShift, Mesosphere DC/OS & Docker Swarm will be natively integrated alongside with VM orchestration systems such as VMware, OpenStack & Nutanix. Customers will be able to securely inter-connect any number of these clusters and enable inter-tenant communication between them if needed.
-
-BCF was recognized by Gartner as a visionary in the latest [Magic Quadrant](https://go.bigswitch.com/17GatedDocuments-MagicQuadrantforDataCenterNetworking_Reg.html). One of the BCF Kubernetes on-premises deployments (which includes Kubernetes, DC/OS & VMware running on multiple DCs across different geographic regions) is also referenced [here](https://portworx.com/architects-corner-kubernetes-satya-komala-nio/).
-
 ### Calico

 [Calico](https://docs.projectcalico.org/) is an open source networking and network security solution for containers, virtual machines, and native host-based workloads. Calico supports multiple data planes including: a pure Linux eBPF dataplane, a standard Linux networking dataplane, and a Windows HNS dataplane. Calico provides a full networking stack but can also be used in conjunction with [cloud provider CNIs](https://docs.projectcalico.org/networking/determine-best-networking#calico-compatible-cni-plugins-and-cloud-provider-integrations) to provide network policy enforcement.
@@ -166,7 +145,7 @@ Coil operates with a low overhead compared to bare metal, and allows you to defi

 ### Contiv

-[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan,  classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](https://contiv.io) is all open sourced.
+[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan,  classic l2, or Cisco-SDN/ACI) for various use cases.

 ### Contrail / Tungsten Fabric

@@ -190,49 +169,6 @@ With this toolset DANM is able to provide multiple separated network interfaces,
 network that satisfies the Kubernetes requirements. Many
 people have reported success with Flannel and Kubernetes.

-### Google Compute Engine (GCE)
-
-For the Google Compute Engine cluster configuration scripts, [advanced
-routing](https://cloud.google.com/vpc/docs/routes) is used to
-assign each VM a subnet (default is `/24` - 254 IPs).  Any traffic bound for that
-subnet will be routed directly to the VM by the GCE network fabric.  This is in
-addition to the "main" IP address assigned to the VM, which is NAT'ed for
-outbound internet access.  A linux bridge (called `cbr0`) is configured to exist
-on that subnet, and is passed to docker's `--bridge` flag.
-
-Docker is started with:
-
-```shell
-DOCKER_OPTS="--bridge=cbr0 --iptables=false --ip-masq=false"
-```
-
-This bridge is created by Kubelet (controlled by the `--network-plugin=kubenet`
-flag) according to the `Node`'s `.spec.podCIDR`.
-
-Docker will now allocate IPs from the `cbr-cidr` block.  Containers can reach
-each other and `Nodes` over the `cbr0` bridge.  Those IPs are all routable
-within the GCE project network.
-
-GCE itself does not know anything about these IPs, though, so it will not NAT
-them for outbound internet traffic.  To achieve that an iptables rule is used
-to masquerade (aka SNAT - to make it seem as if packets came from the `Node`
-itself) traffic that is bound for IPs outside the GCE project network
-(10.0.0.0/8).
-
-```shell
-iptables -t nat -A POSTROUTING ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE
-```
-
-Lastly IP forwarding is enabled in the kernel (so the kernel will process
-packets for bridged containers):
-
-```shell
-sysctl net.ipv4.ip_forward=1
-```
-
-The result of all this is that all `Pods` can reach each other and can egress
-traffic to the internet.
-
 ### Jaguar

 [Jaguar](https://gitlab.com/sdnlab/jaguar) is an open source solution for Kubernetes's network based on OpenDaylight. Jaguar provides overlay network using vxlan and Jaguar CNIPlugin provides one IP address per pod.
@@ -267,9 +203,9 @@ Lars Kellogg-Stedman.

 ### Multus (a Multi Network plugin)

-[Multus](https://github.com/Intel-Corp/multus-cni) is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.
+Multus is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.

-Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes.
+Multus supports all [reference plugins](https://github.com/containernetworking/plugins) (eg. [Flannel](https://github.com/containernetworking/cni.dev/blob/main/content/plugins/v0.9/meta/flannel.md), [DHCP](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp), [Macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan)) that implement the CNI specification and 3rd party plugins (eg. [Calico](https://github.com/projectcalico/cni-plugin), [Weave](https://github.com/weaveworks/weave), [Cilium](https://github.com/cilium/cilium), [Contiv](https://github.com/contiv/netplugin)). In addition to it, Multus supports [SRIOV](https://github.com/hustcat/sriov-cni), [DPDK](https://github.com/Intel-Corp/sriov-cni), [OVS-DPDK & VPP](https://github.com/intel/vhost-user-net-plugin) workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes.

 ### OVN4NFV-K8s-Plugin (OVN based CNI controller & plugin)

@@ -281,18 +217,6 @@ Multus supports all [reference plugins](https://github.com/containernetworking/p

 [NSX-T Container Plug-in (NCP)](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) provides integration between NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.

-### Nuage Networks VCS (Virtualized Cloud Services)
-
-[Nuage](https://www.nuagenetworks.net) provides a highly scalable policy-based Software-Defined Networking (SDN) platform. Nuage uses the open source Open vSwitch for the data plane along with a feature rich SDN Controller built on open standards.
-
-The Nuage platform uses overlays to provide seamless policy-based networking between Kubernetes Pods and non-Kubernetes environments (VMs and bare metal servers). Nuage's policy abstraction model is designed with applications in mind and makes it easy to declare fine-grained policies for applications.The platform's real-time analytics engine enables visibility and security monitoring for Kubernetes applications.
-
-### OpenVSwitch
-
-[OpenVSwitch](https://www.openvswitch.org/) is a somewhat more mature but also
-complicated way to build an overlay network.  This is endorsed by several of the
-"Big Shops" for networking.
-
 ### OVN (Open Virtual Networking)

 OVN is an opensource network virtualization solution developed by the
@@ -22,14 +22,62 @@ generates log messages for the Kubernetes system components.

 For more information about klog configuration, see the [Command line tool reference](/docs/reference/command-line-tools-reference/).

-An example of the klog native format:
+Kubernetes is in the process of simplifying logging in its components. The
+following klog command line flags [are
+deprecated](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
+starting with Kubernetes 1.23 and will be removed in a future release:
+
+- `--add-dir-header`
+- `--alsologtostderr`
+- `--log-backtrace-at`
+- `--log-dir`
+- `--log-file`
+- `--log-file-max-size`
+- `--logtostderr`
+- `--one-output`
+- `--skip-headers`
+- `--skip-log-headers`
+- `--stderrthreshold`
+
+Output will always be written to stderr, regardless of the output
+format. Output redirection is expected to be handled by the component which
+invokes a Kubernetes component. This can be a POSIX shell or a tool like
+systemd.
+
+In some cases, for example a distroless container or a Windows system service,
+those options are not available. Then the
+[`kube-log-runner`](https://github.com/kubernetes/kubernetes/blob/d2a8a81639fcff8d1221b900f66d28361a170654/staging/src/k8s.io/component-base/logs/kube-log-runner/README.md)
+binary can be used as wrapper around a Kubernetes component to redirect
+output. A prebuilt binary is included in several Kubernetes base images under
+its traditional name as `/go-runner` and as `kube-log-runner` in server and
+node release archives.
+
+This table shows how `kube-log-runner` invocations correspond to shell redirection:
+
+| Usage                                    | POSIX shell (such as bash) | `kube-log-runner <options> <cmd>`                           |
+| -----------------------------------------|----------------------------|-------------------------------------------------------------|
+| Merge stderr and stdout, write to stdout | `2>&1`                     | `kube-log-runner` (default behavior)                        |
+| Redirect both into log file              | `1>>/tmp/log 2>&1`         | `kube-log-runner -log-file=/tmp/log`                        |
+| Copy into log file and to stdout         | `2>&1 \| tee -a /tmp/log`  | `kube-log-runner -log-file=/tmp/log -also-stdout`           |
+| Redirect only stdout into log file       | `>/tmp/log`                | `kube-log-runner -log-file=/tmp/log -redirect-stderr=false` |
+
+### Klog output
+
+An example of the traditional klog native format:
 ```
 I1025 00:15:15.525108       1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756]
 ```

+The message string may contain line breaks:
+```
+I1025 00:15:15.525108       1 example.go:79] This is a message
+which has a line break.
+```
+
+
 ### Structured Logging

-{{< feature-state for_k8s_version="v1.19" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.23" state="beta" >}}

 {{< warning >}}
 Migration to structured log messages is an ongoing process. Not all log messages are structured in this version. When parsing log files, you must also handle unstructured log messages.
@@ -38,9 +86,11 @@ Log formatting and value serialization are subject to change.
 {{< /warning>}}

 Structured logging introduces a uniform structure in log messages allowing for programmatic extraction of information. You can store and process structured logs with less effort and cost.
-New message format is backward compatible and enabled by default.
+The code which generates a log message determines whether it uses the traditional unstructured klog output
+or structured logging.

-Format of structured logs:
+The default formatting of structured log messages is as text, with a format that
+is backward compatible with traditional klog:

 ```ini
 <klog header> "<message>" <key1>="<value1>" <key2>="<value2>" ...
@@ -52,6 +102,13 @@ Example:
 I1025 00:15:15.525108       1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready"
 ```

+Strings are quoted. Other values are formatted with
+[`%+v`](https://pkg.go.dev/fmt#hdr-Printing), which may cause log messages to
+continue on the next line [depending on the data](https://github.com/kubernetes/kubernetes/issues/106428).
+```
+I1025 00:15:15.525108       1 example.go:116] "Example" data="This is text with a line break\nand \"quotation marks\"." someInt=1 someFloat=0.1 someStruct={StringField: First line,
+second line.}
+```

 ### JSON log format

@@ -82,7 +139,7 @@ Example of JSON log format (pretty printed):

 Keys with special meaning:
 * `ts` - timestamp as Unix time (required, float)
-* `v` - verbosity (required, int, default 0)
+* `v` - verbosity (only for info and not for error messages, int)
 * `err` - error string (optional, string)
 * `msg` - message (required, string)

@@ -139,4 +196,5 @@ The `logrotate` tool rotates logs daily, or once the log size is greater than 10

 * Read about the [Kubernetes Logging Architecture](/docs/concepts/cluster-administration/logging/)
 * Read about [Structured Logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging)
+* Read about [deprecation of klog flags](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
 * Read about the [Conventions for logging severity](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md)
@@ -99,9 +99,10 @@ resource requests/limits of that type for each Container in the Pod.
 Limits and requests for CPU resources are measured in *cpu* units.
 One cpu, in Kubernetes, is equivalent to **1 vCPU/Core** for cloud providers and **1 hyperthread** on bare-metal Intel processors.

-Fractional requests are allowed. A Container with
-`spec.containers[].resources.requests.cpu` of `0.5` is guaranteed half as much
-CPU as one that asks for 1 CPU. The expression `0.1` is equivalent to the
+Fractional requests are allowed. When you define a container with
+`spec.containers[].resources.requests.cpu` set to `0.5`, you are requesting half
+as much CPU time compared to if you asked for `1.0` CPU.
+For CPU resource units, the expression `0.1` is equivalent to the
 expression `100m`, which can be read as "one hundred millicpu". Some people say
 "one hundred millicores", and this is understood to mean the same thing. A
 request with a decimal point, like `0.1`, is converted to `100m` by the API, and
@@ -115,11 +116,11 @@ CPU is always requested as an absolute quantity, never as a relative quantity;

 Limits and requests for `memory` are measured in bytes. You can express memory as
 a plain integer or as a fixed-point number using one of these suffixes:
-E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
+E, P, T, G, M, k, m (millis). You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
 Mi, Ki. For example, the following represent roughly the same value:

 ```shell
-128974848, 129e6, 129M, 123Mi
+128974848, 129e6, 129M,  128974848000m, 123Mi
 ```

 Here's an example.
@@ -236,7 +237,7 @@ The kubelet also uses this kind of storage to hold
 container images, and the writable layers of running containers.

 {{< caution >}}
-If a node fails, the data in its ephemeral storage can be lost.  
+If a node fails, the data in its ephemeral storage can be lost.
 Your applications cannot expect any performance SLAs (disk IOPS for example)
 from local ephemeral storage.
 {{< /caution >}}
@@ -440,7 +441,7 @@ Kubernetes does not use them.
 Quotas are faster and more accurate than directory scanning. When a
 directory is assigned to a project, all files created under a
 directory are created in that project, and the kernel merely has to
-keep track of how many blocks are in use by files in that project.  
+keep track of how many blocks are in use by files in that project.
 If a file is created and deleted, but has an open file descriptor,
 it continues to consume space. Quota tracking records that space accurately
 whereas directory scans overlook the storage used by deleted files.
@@ -55,7 +55,7 @@ DNS server watches the Kubernetes API for new `Services` and creates a set of DN

  If you only need access to the port for debugging purposes, you can use the [apiserver proxy](/docs/tasks/access-application-cluster/access-cluster/#manually-constructing-apiserver-proxy-urls) or [`kubectl port-forward`](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/).

-  If you explicitly need to expose a Pod's port on the node, consider using a [NodePort](/docs/concepts/services-networking/service/#nodeport) Service before resorting to `hostPort`.
+  If you explicitly need to expose a Pod's port on the node, consider using a [NodePort](/docs/concepts/services-networking/service/#type-nodeport) Service before resorting to `hostPort`.

 - Avoid using `hostNetwork`, for the same reasons as `hostPort`.

@@ -212,7 +212,8 @@ to create a Docker registry Secret, you can do:
 kubectl create secret docker-registry secret-tiger-docker \
  --docker-username=tiger \
  --docker-password=pass113 \
-  --docker-email=tiger@acme.com
+  --docker-email=tiger@acme.com \
+  --docker-server=my-registry.example:5000
 ```

 This command creates a Secret of type `kubernetes.io/dockerconfigjson`.
@@ -222,22 +223,28 @@ on the fly:

 ```json
 {
-  "auths": {
-    "https://index.docker.io/v1/": {
-      "username": "tiger",
-      "password": "pass113",
-      "email": "tiger@acme.com",
-      "auth": "dGlnZXI6cGFzczExMw=="
-    }
-  }
+    "apiVersion": "v1",
+    "data": {
+        ".dockerconfigjson": "eyJhdXRocyI6eyJteS1yZWdpc3RyeTo1MDAwIjp7InVzZXJuYW1lIjoidGlnZXIiLCJwYXNzd29yZCI6InBhc3MxMTMiLCJlbWFpbCI6InRpZ2VyQGFjbWUuY29tIiwiYXV0aCI6ImRHbG5aWEk2Y0dGemN6RXhNdz09In19fQ=="
+    },
+    "kind": "Secret",
+    "metadata": {
+        "creationTimestamp": "2021-07-01T07:30:59Z",
+        "name": "secret-tiger-docker",
+        "namespace": "default",
+        "resourceVersion": "566718",
+        "uid": "e15c1d7b-9071-4100-8681-f3a7a2ce89ca"
+    },
+    "type": "kubernetes.io/dockerconfigjson"
 }
+
 ```

 ### Basic authentication Secret

 The `kubernetes.io/basic-auth` type is provided for storing credentials needed
 for basic authentication. When using this Secret type, the `data` field of the
-Secret must contain the following two keys:
+Secret must contain one of the following two keys:

 - `username`: the user name for authentication;
 - `password`: the password or token for authentication.
@@ -59,7 +59,7 @@ Resources consumed by the command are counted against the Container.
 ### Hook handler execution

 When a Container lifecycle management hook is called,
-the Kubernetes management system execute the handler according to the hook action,
+the Kubernetes management system executes the handler according to the hook action,
 `httpGet` and `tcpSocket` are executed by the kubelet process, and `exec` is executed in the container.

 Hook handler calls are synchronous within the context of the Pod containing the Container.
@@ -108,7 +108,7 @@ When you (or a controller) submit a new Pod to the API server, your cluster sets
  `:latest`, `imagePullPolicy` is automatically set to `Always`;
 - if you omit the `imagePullPolicy` field, and you don't specify the tag for the
  container image, `imagePullPolicy` is automatically set to `Always`;
- if you omit the `imagePullPolicy` field, and you don't specify the tag for the
+- if you omit the `imagePullPolicy` field, and you specify the tag for the
  container image that isn't `:latest`, the `imagePullPolicy` is automatically set to
  `IfNotPresent`.

@@ -265,6 +265,73 @@ template needs to include the `.docker/config.json` or mount a drive that contai
 All pods will have read access to images in any private registry once private
 registry keys are added to the `.docker/config.json`.

+### Interpretation of config.json {#config-json}
+
+The interpretation of `config.json` varies between the original Docker
+implementation and the Kubernetes interpretation. In Docker, the `auths` keys
+can only specify root URLs, whereas Kubernetes allows glob URLs as well as
+prefix-matched paths. This means that a `config.json` like this is valid:
+
+```json
+{
+    "auths": {
+        "*my-registry.io/images": {
+            "auth": "…"
+        }
+    }
+}
+```
+
+The root URL (`*my-registry.io`) is matched by using the following syntax:
+
+```
+pattern:
+    { term }
+
+term:
+    '*'         matches any sequence of non-Separator characters
+    '?'         matches any single non-Separator character
+    '[' [ '^' ] { character-range } ']'
+                character class (must be non-empty)
+    c           matches character c (c != '*', '?', '\\', '[')
+    '\\' c      matches character c
+
+character-range:
+    c           matches character c (c != '\\', '-', ']')
+    '\\' c      matches character c
+    lo '-' hi   matches character c for lo <= c <= hi
+```
+
+Image pull operations would now pass the credentials to the CRI container
+runtime for every valid pattern. For example the following container image names
+would match successfully:
+
+- `my-registry.io/images`
+- `my-registry.io/images/my-image`
+- `my-registry.io/images/another-image`
+- `sub.my-registry.io/images/my-image`
+- `a.sub.my-registry.io/images/my-image`
+
+The kubelet performs image pulls sequentially for every found credential. This
+means, that multiple entries in `config.json` are possible, too:
+
+```json
+{
+    "auths": {
+        "my-registry.io/images": {
+            "auth": "…"
+        },
+        "my-registry.io/images/subpath": {
+            "auth": "…"
+        }
+    }
+}
+```
+
+If now a container specifies an image `my-registry.io/images/subpath/my-image`
+to be pulled, then the kubelet will try to download them from both
+authentication sources if one of them fails.
+
 ### Pre-pulled images

 {{< note >}}
@@ -390,3 +457,4 @@ Kubelet will merge any `imagePullSecrets` into a single virtual `.docker/config.

 * Read the [OCI Image Manifest Specification](https://github.com/opencontainers/image-spec/blob/master/manifest.md).
 * Learn about [container image garbage collection](/docs/concepts/architecture/garbage-collection/#container-image-garbage-collection).
+* Learn more about [pulling an Image from a Private Registry](/docs/tasks/configure-pod-container/pull-image-private-registry).
@@ -52,9 +52,9 @@ Flags and configuration files may not always be changeable in a hosted Kubernete
 Extensions are software components that extend and deeply integrate with Kubernetes.
 They adapt it to support new types and new kinds of hardware.

-Most cluster administrators will use a hosted or distribution
-instance of Kubernetes. As a result, most Kubernetes users will not need to
-install extensions and fewer will need to author new ones.
+Many cluster administrators use a hosted or distribution instance of Kubernetes. 
+These clusters come with extensions pre-installed. As a result, most Kubernetes 
+users will not need to install extensions and even fewer users will need to author new ones.

 ## Extension Patterns

@@ -77,7 +77,7 @@ failure.
 In the webhook model, Kubernetes makes a network request to a remote service.
 In the *Binary Plugin* model, Kubernetes executes a binary (program).
 Binary plugins are used by the kubelet (e.g.
-[Flex Volume Plugins](/docs/concepts/storage/volumes/#flexVolume)
+[Flex Volume Plugins](/docs/concepts/storage/volumes/#flexvolume)
 and [Network Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/))
 and by kubectl.

@@ -145,7 +145,7 @@ Kubernetes provides several built-in authentication methods, and an [Authenticat

 ### Authorization

-[Authorization](/docs/reference/access-authn-authz/webhook/) determines whether specific users can read, write, and do other operations on API resources. It works at the level of whole resources -- it doesn't discriminate based on arbitrary object fields. If the built-in authorization options don't meet your needs, and [Authorization webhook](/docs/reference/access-authn-authz/webhook/) allows calling out to user-provided code to make an authorization decision.
+[Authorization](/docs/reference/access-authn-authz/authorization/) determines whether specific users can read, write, and do other operations on API resources. It works at the level of whole resources -- it doesn't discriminate based on arbitrary object fields. If the built-in authorization options don't meet your needs, [Authorization webhook](/docs/reference/access-authn-authz/webhook/) allows calling out to user-provided code to make an authorization decision.


 ### Dynamic Admission Control
@@ -163,6 +163,8 @@ After a request is authorized, if it is a write operation, it also goes through
 ) allow users to mount volume types without built-in support by having the
 Kubelet call a Binary Plugin to mount the volume.

+FlexVolume is deprecated since Kubernetes v1.23. The Out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes. See [Kubernetes Volume Plugin FAQ for Storage Vendors](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors) for more information.
+

 ### Device Plugins

@@ -35,11 +35,11 @@ On their own, custom resources let you store and retrieve structured data.
 When you combine a custom resource with a *custom controller*, custom resources
 provide a true _declarative API_.

-A [declarative API](/docs/concepts/overview/kubernetes-api/)
-allows you to _declare_ or specify the desired state of your resource and tries to
-keep the current state of Kubernetes objects in sync with the desired state.
-The controller interprets the structured data as a record of the user's
-desired state, and continually maintains this state.
+The Kubernetes [declarative API](/docs/concepts/overview/kubernetes-api/)
+enforces a separation of responsibilities. You declare the desired state of
+your resource. The Kubernetes controller keeps the current state of Kubernetes
+objects in sync with your declared desired state. This is in contrast to an
+imperative API, where you *instruct* a server what to do.

 You can deploy and update a custom controller on a running cluster, independently
 of the cluster's lifecycle. Custom controllers can work with any kind of resource,
@@ -148,8 +148,8 @@ and use a controller to handle events.
 Usually, each resource in the Kubernetes API requires code that handles REST requests and manages persistent storage of objects. The main Kubernetes API server handles built-in resources like *pods* and *services*, and can also generically handle custom resources through [CRDs](#customresourcedefinitions).

 The [aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) allows you to provide specialized
-implementations for your custom resources by writing and deploying your own standalone API server.
-The main API server delegates requests to you for the custom resources that you handle,
+implementations for your custom resources by writing and deploying your own API server.
+The main API server delegates requests to your API server for the custom resources that you handle,
 making them available to all of its clients.

 ## Choosing a method for adding custom resources
@@ -197,6 +197,8 @@ service PodResourcesLister {
 }
 ```

+### `List` gRPC endpoint {#grpc-endpoint-list}
+
 The `List` endpoint provides information on resources of running pods, with details such as the
 id of exclusively allocated CPUs, device id as it was reported by device plugins and id of
 the NUMA node where these devices are allocated. Also, for NUMA-based machines, it contains the information about memory and hugepages reserved for a container.
@@ -246,10 +248,35 @@ message ContainerDevices {
    TopologyInfo topology = 3;
 }
 ```
+{{< note >}}
+cpu_ids in the `ContainerResources` in the `List` endpoint correspond to exclusive CPUs allocated
+to a partilar container. If the goal is to evaluate CPUs that belong to the shared pool, the `List`
+endpoint needs to be used in conjunction with the `GetAllocatableResources` endpoint as explained
+below:
+1. Call `GetAllocatableResources` to get a list of all the allocatable CPUs
+2. Call `GetCpuIds` on all `ContainerResources` in the system
+3. Subtract out all of the CPUs from the `GetCpuIds` calls from the `GetAllocatableResources` call
+{{< /note >}}
+
+### `GetAllocatableResources` gRPC endpoint {#grpc-endpoint-getallocatableresources}
+
+{{< feature-state state="beta" for_k8s_version="v1.23" >}}

 GetAllocatableResources provides information on resources initially available on the worker node.
 It provides more information than kubelet exports to APIServer.

+{{< note >}}
+`GetAllocatableResources` should only be used to evaluate [allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)
+resources on a node. If the goal is to evaluate free/unallocated resources it should be used in
+conjunction with the List() endpoint. The result obtained by `GetAllocatableResources` would remain
+the same unless the underlying resources exposed to kubelet change. This happens rarely but when
+it does (for example: hotplug/hotunplug, device health changes), client is expected to call
+`GetAlloctableResources` endpoint.
+However, calling `GetAllocatableResources` endpoint is not sufficient in case of cpu and/or memory
+update and Kubelet needs to be restarted to reflect the correct resource capacity and allocatable.
+{{< /note >}}
+
+
 ```gRPC
 // AllocatableResourcesResponses contains informations about all the devices known by the kubelet
 message AllocatableResourcesResponse {
@@ -259,6 +286,13 @@ message AllocatableResourcesResponse {
 }

 ```
+Starting from Kubernetes v1.23, the `GetAllocatableResources` is enabled by default.
+You can disable it by turning off the
+`KubeletPodResourcesGetAllocatable` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
+
+Preceding Kubernetes v1.23, to enable this feature `kubelet` must be started with the following flag:
+
+`--feature-gates=KubeletPodResourcesGetAllocatable=true`

 `ContainerDevices` do expose the topology information declaring to which NUMA cells the device is affine.
 The NUMA cells are identified using a opaque integer ID, which value is consistent to what device
@@ -31,9 +31,7 @@ built-in automation from the core of Kubernetes. You can use Kubernetes
 to automate deploying and running workloads, *and* you can automate how
 Kubernetes does that.

-Kubernetes' {{< glossary_tooltip text="controllers" term_id="controller" >}}
-concept lets you extend the cluster's behaviour without modifying the code
-of Kubernetes itself.
+Kubernetes' {{< glossary_tooltip text="operator pattern" term_id="operator-pattern" >}} concept lets you extend the cluster's behaviour without modifying the code of Kubernetes itself by linking {{< glossary_tooltip text="controllers" term_id="controller" >}} to one or more custom resources.
 Operators are clients of the Kubernetes API that act as controllers for
 a [Custom Resource](/docs/concepts/extend-kubernetes/api-extension/custom-resources/).

@@ -19,11 +19,7 @@ When you deploy Kubernetes, you get a cluster.
 This document outlines the various components you need to have for
 a complete and working Kubernetes cluster.

-Here's the diagram of a Kubernetes cluster with all the components tied together.
-
-![Components of Kubernetes](/images/docs/components-of-kubernetes.svg)
-
-
+{{< figure src="/images/docs/components-of-kubernetes.svg" alt="Components of Kubernetes" caption="The components of a Kubernetes cluster" class="diagram-large" >}}

 <!-- body -->
 ## Control Plane Components
@@ -37,8 +37,11 @@ if you are writing an application using the Kubernetes API.

 Complete API details are documented using [OpenAPI](https://www.openapis.org/).

-The Kubernetes API server serves an OpenAPI spec via the `/openapi/v2` endpoint.
-You can request the response format using request headers as follows:
+### OpenAPI V2
+
+The Kubernetes API server serves an aggregated OpenAPI v2 spec via the
+`/openapi/v2` endpoint. You can request the response format using
+request headers as follows:

 <table>
  <caption style="display:none">Valid request header values for OpenAPI v2 queries</caption>
@@ -77,6 +80,55 @@ about this format, see the [Kubernetes Protobuf serialization](https://github.co
 Interface Definition Language (IDL) files for each schema located in the Go
 packages that define the API objects.

+### OpenAPI V3
+
+{{< feature-state state="alpha"  for_k8s_version="v1.23" >}}
+
+Kubernetes v1.23 offers initial support for publishing its APIs as OpenAPI v3; this is an
+alpha feature that is disabled by default.
+You can enable the alpha feature by turning on the
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) named `OpenAPIV3`
+for the kube-apiserver component.
+
+With the feature enabled, the Kubernetes API server serves an
+aggregated OpenAPI v3 spec per Kubernetes group version at the
+`/openapi/v3/apis/<group>/<version>` endpoint. Please refer to the
+table below for accepted request headers.
+
+<table>
+  <caption style="display:none">Valid request header values for OpenAPI v3 queries</caption>
+  <thead>
+     <tr>
+        <th>Header</th>
+        <th style="min-width: 50%;">Possible values</th>
+        <th>Notes</th>
+     </tr>
+  </thead>
+  <tbody>
+     <tr>
+        <td><code>Accept-Encoding</code></td>
+        <td><code>gzip</code></td>
+        <td><em>not supplying this header is also acceptable</em></td>
+     </tr>
+     <tr>
+        <td rowspan="3"><code>Accept</code></td>
+        <td><code>application/com.github.proto-openapi.spec.v3@v1.0+protobuf</code></td>
+        <td><em>mainly for intra-cluster use</em></td>
+     </tr>
+     <tr>
+        <td><code>application/json</code></td>
+        <td><em>default</em></td>
+     </tr>
+     <tr>
+        <td><code>*</code></td>
+        <td><em>serves </em><code>application/json</code></td>
+     </tr>
+  </tbody>
+</table>
+
+A discovery endpoint `/openapi/v3` is provided to see a list of all
+group/versions available. This endpoint only returns JSON.
+
 ## Persistence

 Kubernetes stores the serialized state of objects by writing them into
@@ -44,7 +44,8 @@ and the controller deletes the volume.

 ## Owner references, labels, and finalizers {#owners-labels-finalizers}

-Like {{<glossary_tooltip text="labels" term_id="label">}}, [owner references](/concepts/overview/working-with-objects/owners-dependents/)
+Like {{<glossary_tooltip text="labels" term_id="label">}},
+[owner references](/docs/concepts/overview/working-with-objects/owners-dependents/)
 describe the relationships between objects in Kubernetes, but are used for a
 different purpose. When a
 {{<glossary_tooltip text="controller" term_id="controller">}} manages objects
@@ -10,8 +10,7 @@ weight: 30

 <!-- overview -->

-Kubernetes supports multiple virtual clusters backed by the same physical cluster.
-These virtual clusters are called namespaces.
+In Kubernetes, _namespaces_ provides a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects _(e.g. Deployments, Services, etc)_ and not for cluster-wide objects _(e.g. StorageClass, Nodes, PersistentVolumes, etc)_.

 <!-- body -->

@@ -52,7 +52,7 @@ equivalent to "Predicate" and "Scoring" is equivalent to "Priority function".
 One plugin may register at multiple extension points to perform more complex or
 stateful tasks.

-{{< figure src="/images/docs/scheduling-framework-extensions.png" title="scheduling framework extension points" >}}
+{{< figure src="/images/docs/scheduling-framework-extensions.png" title="scheduling framework extension points" class="diagram-large">}}

 ### QueueSort {#queue-sort}

@@ -29,7 +29,7 @@ computing approach to security, which is widely regarded as a best practice for
 software systems.
 {{< /note >}}

-{{< figure src="/images/docs/4c.png" title="The 4C's of Cloud Native Security" >}}
+{{< figure src="/images/docs/4c.png" title="The 4C's of Cloud Native Security" class="diagram-large" >}}

 Each layer of the Cloud Native security model builds upon the next outermost layer.
 The Code layer benefits from strong base (Cloud, Cluster, Container) security layers.
@@ -60,6 +60,7 @@ Amazon Web Services | https://aws.amazon.com/security/ |
 Google Cloud Platform | https://cloud.google.com/security/ |
 IBM Cloud | https://www.ibm.com/cloud/security |
 Microsoft Azure | https://docs.microsoft.com/en-us/azure/security/azure-security |
+Oracle Cloud Infrastructure | https://www.oracle.com/security/ |
 VMWare VSphere | https://www.vmware.com/security/hardening-guides.html |

 {{< /table >}}
@@ -73,10 +74,10 @@ Suggestions for securing your infrastructure in a Kubernetes cluster:
 Area of Concern for Kubernetes Infrastructure | Recommendation |
 --------------------------------------------- | -------------- |
 Network access to API Server (Control plane) | All access to the Kubernetes control plane is not allowed publicly on the internet and is controlled by network access control lists restricted to the set of IP addresses needed to administer the cluster.|
-Network access to Nodes (nodes) | Nodes should be configured to _only_ accept connections (via network access control lists)from the control plane on the specified ports, and accept connections for services in Kubernetes of type NodePort and LoadBalancer. If possible, these nodes should not be exposed on the public internet entirely.
+Network access to Nodes (nodes) | Nodes should be configured to _only_ accept connections (via network access control lists) from the control plane on the specified ports, and accept connections for services in Kubernetes of type NodePort and LoadBalancer. If possible, these nodes should not be exposed on the public internet entirely.
 Kubernetes access to Cloud Provider API | Each cloud provider needs to grant a different set of permissions to the Kubernetes control plane and nodes. It is best to provide the cluster with cloud provider access that follows the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) for the resources it needs to administer. The [Kops documentation](https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md#iam-roles) provides information about IAM policies and roles.
 Access to etcd | Access to etcd (the datastore of Kubernetes) should be limited to the control plane only. Depending on your configuration, you should attempt to use etcd over TLS. More information can be found in the [etcd documentation](https://github.com/etcd-io/etcd/tree/master/Documentation).
-etcd Encryption | Wherever possible it's a good practice to encrypt all drives at rest, but since etcd holds the state of the entire cluster (including Secrets) its disk should especially be encrypted at rest.
+etcd Encryption | Wherever possible it's a good practice to encrypt all storage at rest, and since etcd holds the state of the entire cluster (including Secrets) its disk should especially be encrypted at rest.

 {{< /table >}}

@@ -98,7 +99,7 @@ good information practices, read and follow the advice about
 Depending on the attack surface of your application, you may want to focus on specific
 aspects of security. For example: If you are running a service (Service A) that is critical
 in a chain of other resources and a separate workload (Service B) which is
-vulnerable to a resource exhaustion attack then the risk of compromising Service A
+vulnerable to a resource exhaustion attack, then the risk of compromising Service A
 is high if you do not limit the resources of Service B. The following table lists
 areas of security concerns and recommendations for securing workloads running in Kubernetes:

@@ -107,10 +108,10 @@ Area of Concern for Workload Security | Recommendation |
 RBAC Authorization (Access to the Kubernetes API) | https://kubernetes.io/docs/reference/access-authn-authz/rbac/
 Authentication | https://kubernetes.io/docs/concepts/security/controlling-access/
 Application secrets management (and encrypting them in etcd at rest) | https://kubernetes.io/docs/concepts/configuration/secret/ <br> https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
-Pod Security Policies | https://kubernetes.io/docs/concepts/policy/pod-security-policy/
+Ensuring that pods meet defined Pod Security Standards | https://kubernetes.io/docs/concepts/security/pod-security-standards/#policy-instantiation
 Quality of Service (and Cluster resource management) | https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/
 Network Policies | https://kubernetes.io/docs/concepts/services-networking/network-policies/
-TLS For Kubernetes Ingress | https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
+TLS for Kubernetes Ingress | https://kubernetes.io/docs/concepts/services-networking/ingress/#tls

 ## Container

@@ -136,7 +137,7 @@ are recommendations to protect application code:

 Area of Concern for Code | Recommendation |
 -------------------------| -------------- |
-Access over TLS only | If your code needs to communicate by TCP, perform a TLS handshake with the client ahead of time. With the exception of a few cases, encrypt everything in transit. Going one step further, it's a good idea to encrypt network traffic between services. This can be done through a process known as mutual or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) which performs a two sided verification of communication between two certificate holding services. |
+Access over TLS only | If your code needs to communicate by TCP, perform a TLS handshake with the client ahead of time. With the exception of a few cases, encrypt everything in transit. Going one step further, it's a good idea to encrypt network traffic between services. This can be done through a process known as mutual TLS authentication or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication) which performs a two sided verification of communication between two certificate holding services. |
 Limiting port ranges of communication | This recommendation may be a bit self-explanatory, but wherever possible you should only expose the ports on your service that are absolutely essential for communication or metric gathering. |
 3rd Party Dependency Security | It is a good practice to regularly scan your application's third party libraries for known security vulnerabilities. Each programming language has a tool for performing this check automatically. |
 Static Code Analysis | Most languages provide a way for a snippet of code to be analyzed for any potentially unsafe coding practices. Whenever possible you should perform checks using automated tooling that can scan codebases for common security errors. Some of the tools can be found at: https://owasp.org/www-community/Source_Code_Analysis_Tools |
@@ -13,13 +13,13 @@ min-kubernetes-server-version: v1.22

 <!-- overview -->

-{{< feature-state for_k8s_version="v1.22" state="alpha" >}}
+{{< feature-state for_k8s_version="v1.23" state="beta" >}}

 The Kubernetes [Pod Security Standards](/docs/concepts/security/pod-security-standards/) define
 different isolation levels for Pods. These standards let you define how you want to restrict the
 behavior of pods in a clear, consistent fashion.

-As an Alpha feature, Kubernetes offers a built-in _Pod Security_ {{< glossary_tooltip
+As an Beta feature, Kubernetes offers a built-in _Pod Security_ {{< glossary_tooltip
 text="admission controller" term_id="admission-controller" >}}, the successor
 to [PodSecurityPolicies](/docs/concepts/policy/pod-security-policy/). Pod security restrictions
 are applied at the {{< glossary_tooltip text="namespace" term_id="namespace" >}} level when pods
@@ -32,15 +32,40 @@ The PodSecurityPolicy API is deprecated and will be

 <!-- body -->

-## Enabling the Alpha feature
+## Enabling the `PodSecurity` admission plugin

-Setting pod security controls by namespace is an alpha feature. You must enable the `PodSecurity`
-[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) in order to use it.
+In v1.23, the `PodSecurity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+is a Beta feature and is enabled by default.
+
+In v1.22, the `PodSecurity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+is an Alpha feature and must be enabled in `kube-apiserver` in order to use the built-in admission plugin.

 ```shell
 --feature-gates="...,PodSecurity=true"
 ```

+## Alternative: installing the `PodSecurity` admission webhook {#webhook}
+
+For environments where the built-in `PodSecurity` admission plugin cannot be used,
+either because the cluster is older than v1.22, or the `PodSecurity` feature cannot be enabled,
+the `PodSecurity` admission logic is also available as a Beta [validating admission webhook](https://git.k8s.io/pod-security-admission/webhook).
+
+A pre-built container image, certificate generation scripts, and example manifests
+are available at [https://git.k8s.io/pod-security-admission/webhook](https://git.k8s.io/pod-security-admission/webhook).
+
+To install:
+```shell
+git clone git@github.com:kubernetes/pod-security-admission.git
+cd pod-security-admission/webhook
+make certs
+kubectl apply -k .
+```
+
+{{< note >}}
+The generated certificate is valid for 2 years. Before it expires,
+regenerate the certificate or remove the webhook in favor of the built-in admission plugin.
+{{< /note >}}
+
 ## Pod Security levels

 Pod Security admission places requirements on a Pod's [Security
@@ -52,7 +77,7 @@ page for an in-depth look at those requirements.

 ## Pod Security Admission labels for namespaces

-Provided that you have enabled this feature, you can configure namespaces to define the admission
+Once the feature is enabled or the webhook is installed, you can configure namespaces to define the admission
 control mode you want to use for pod security in each namespace. Kubernetes defines a set of 
 {{< glossary_tooltip term_id="label" text="labels" >}} that you can set to define which of the 
 predefined Pod Security Standard levels you want to use for a namespace. The label you select
@@ -373,6 +373,24 @@ fail validation.
 				</small>
 			</td>
 		</tr>
+		<tr>
+			<td style="white-space: nowrap">Running as Non-root user (v1.23+)</td>
+			<td>
+				<p>Containers must not set <tt>runAsUser</tt> to 0</p>
+				<p><strong>Restricted Fields</strong></p>
+				<ul>
+					<li><code>spec.securityContext.runAsUser</code></li>
+					<li><code>spec.containers[*].securityContext.runAsUser</code></li>
+					<li><code>spec.initContainers[*].securityContext.runAsUser</code></li>
+					<li><code>spec.ephemeralContainers[*].securityContext.runAsUser</code></li>
+				</ul>
+				<p><strong>Allowed Values</strong></p>
+				<ul>
+					<li>any non-zero value</li>
+					<li><code>undefined/null</code></li>
+				</ul>
+			</td>
+		</tr>
 		<tr>
 			<td style="white-space: nowrap">Non-root groups <em>(optional)</em></td>
 			<td>
@@ -466,7 +484,7 @@ of individual policies are not defined here.
 - {{< example file="security/podsecurity-baseline.yaml" >}}Baseline namespace{{< /example >}}
 - {{< example file="security/podsecurity-restricted.yaml" >}}Restricted namespace{{< /example >}}

-[**PodSecurityPolicy**](/docs/concepts/profile/pod-security-profile/) (Deprecated)
+[**PodSecurityPolicy**](/docs/concepts/policy/pod-security-policy/) (Deprecated)

 - {{< example file="policy/privileged-psp.yaml" >}}Privileged{{< /example >}}
 - {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}}
@@ -54,7 +54,7 @@ kubectl get pods -l run=my-nginx -o yaml | grep podIP

 You should be able to ssh into any node in your cluster and curl both IPs. Note that the containers are *not* using port 80 on the node, nor are there any special NAT rules to route traffic to the pod. This means you can run multiple nginx pods on the same node all using the same containerPort and access them from any other pod or node in your cluster using IP. Like Docker, ports can still be published to the host node's interfaces, but the need for this is radically diminished because of the networking model.

-You can read more about [how we achieve this](/docs/concepts/cluster-administration/networking/#how-to-achieve-this) if you're curious.
+You can read more about the [Kubernetes Networking Model](/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model) if you're curious.

 ## Creating a Service

@@ -39,7 +39,7 @@ namespace.

 DNS queries may be expanded using the pod's `/etc/resolv.conf`. Kubelet 
 sets this file for each pod. For example, a query for just `data` may be 
-expanded to `data.test.cluster.local`. The values of the `search` option 
+expanded to `data.test.svc.cluster.local`. The values of the `search` option 
 are used to expand queries. To learn more about DNS queries, see 
 [the `resolv.conf` manual page.](https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html) 

@@ -217,13 +217,13 @@ following pod-specific DNS policies. These policies are specified in the

 - "`Default`": The Pod inherits the name resolution configuration from the node
  that the pods run on.
-  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#inheriting-dns-from-the-node)
+  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers)
  for more details.
 - "`ClusterFirst`": Any DNS query that does not match the configured cluster
  domain suffix, such as "`www.kubernetes.io`", is forwarded to the upstream
  nameserver inherited from the node. Cluster administrators may have extra
  stub-domain and upstream DNS servers configured.
-  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers/#effects-on-pods)
+  See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers)
  for details on how DNS queries are handled in those cases.
 - "`ClusterFirstWithHostNet`": For Pods running with hostNetwork, you should
  explicitly set its DNS policy "`ClusterFirstWithHostNet`".
@@ -16,7 +16,7 @@ weight: 70

 <!-- overview -->

-{{< feature-state for_k8s_version="v1.21" state="beta" >}}
+{{< feature-state for_k8s_version="v1.23" state="stable" >}}

 IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to {{< glossary_tooltip text="Pods" term_id="pod" >}} and {{< glossary_tooltip text="Services" term_id="service" >}}.

@@ -47,8 +47,6 @@ The following prerequisites are needed in order to utilize IPv4/IPv6 dual-stack

 ## Configure IPv4/IPv6 dual-stack

-To use IPv4/IPv6 dual-stack, ensure the `IPv6DualStack` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled for the relevant components of your cluster. (Starting in 1.21, IPv4/IPv6 dual-stack defaults to enabled.)
-
 To configure IPv4/IPv6 dual-stack, set dual-stack cluster network assignments:

   * kube-apiserver:
@@ -65,9 +63,6 @@ An example of an IPv4 CIDR: `10.244.0.0/16` (though you would supply your own ad

 An example of an IPv6 CIDR: `fdXY:IJKL:MNOP:15::/64` (this shows the format but is not a valid address - see [RFC 4193](https://tools.ietf.org/html/rfc4193))

-Starting in 1.21, IPv4/IPv6 dual-stack defaults to enabled.
-You can disable it when necessary by specifying `--feature-gates="IPv6DualStack=false"`
-on the kube-apiserver, kube-controller-manager, kubelet, and kube-proxy command line.
 {{< /note >}}

 ## Services
@@ -81,7 +76,7 @@ set the `.spec.ipFamilyPolicy` field to one of the following values:

 * `SingleStack`: Single-stack service. The control plane allocates a cluster IP for the Service, using the first configured service cluster IP range.
 * `PreferDualStack`:
-  * Allocates IPv4 and IPv6 cluster IPs for the Service. (If the cluster has `--feature-gates="IPv6DualStack=false"`, this setting follows the same behavior as `SingleStack`.)
+  * Allocates IPv4 and IPv6 cluster IPs for the Service.
 * `RequireDualStack`: Allocates Service `.spec.ClusterIPs` from both IPv4 and IPv6 address ranges.
  * Selects the `.spec.ClusterIP` from the list of `.spec.ClusterIPs` based on the address family of the first element in the `.spec.ipFamilies` array.

@@ -124,7 +119,7 @@ These examples demonstrate the behavior of various dual-stack Service configurat

 #### Dual-stack defaults on existing Services

-These examples demonstrate the default behavior when dual-stack is newly enabled on a cluster where Services already exist. (Upgrading an existing cluster to 1.21 will enable dual-stack unless `--feature-gates="IPv6DualStack=false"` is set.)
+These examples demonstrate the default behavior when dual-stack is newly enabled on a cluster where Services already exist. (Upgrading an existing cluster to 1.21 or beyond will enable dual-stack.)

 1. When dual-stack is enabled on a cluster, existing Services (whether `IPv4` or `IPv6`) are configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set `.spec.ipFamilies` to the address family of the existing Service. The existing Service cluster IP will be stored in `.spec.ClusterIPs`.

@@ -28,6 +28,7 @@ Kubernetes as a project supports and maintains [AWS](https://github.com/kubernet
  controller.
 * [Apache APISIX ingress controller](https://github.com/apache/apisix-ingress-controller) is an [Apache APISIX](https://github.com/apache/apisix)-based ingress controller.
 * [Avi Kubernetes Operator](https://github.com/vmware/load-balancer-and-ingress-services-for-kubernetes) provides L4-L7 load-balancing using [VMware NSX Advanced Load Balancer](https://avinetworks.com/).
+* [BFE Ingress Controller](https://github.com/bfenetworks/ingress-bfe) is a [BFE](https://www.bfe-networks.net)-based ingress controller.
 * The [Citrix ingress controller](https://github.com/citrix/citrix-k8s-ingress-controller#readme) works with
  Citrix Application Delivery Controller.
 * [Contour](https://projectcontour.io/) is an [Envoy](https://www.envoyproxy.io/) based ingress controller.
@@ -56,12 +57,11 @@ Kubernetes as a project supports and maintains [AWS](https://github.com/kubernet

 ## Using multiple Ingress controllers

-You may deploy [any number of ingress controllers](https://git.k8s.io/ingress-nginx/docs/user-guide/multiple-ingress.md#multiple-ingress-controllers) 
-within a cluster. When you create an ingress, you should annotate each ingress with the appropriate
-[`ingress.class`](https://git.k8s.io/ingress-gce/docs/faq/README.md#how-do-i-run-multiple-ingress-controllers-in-the-same-cluster) 
-to indicate which ingress controller should be used if more than one exists within your cluster.
+You may deploy any number of ingress controllers using [ingress class](/docs/concepts/services-networking/ingress/#ingress-class)
+within a cluster. Note the `.metadata.name` of your ingress class resource. When you create an ingress you would need that name to specify the `ingressClassName` field on your Ingress object (refer to [IngressSpec v1 reference](/docs/reference/kubernetes-api/service-resources/ingress-v1/#IngressSpec). `ingressClassName` is a replacement of the older [annotation method](/docs/concepts/services-networking/ingress/#deprecated-annotation).

-If you do not define a class, your cloud provider may use a default ingress controller.
+If you do not specify an IngressClass for an Ingress, and your cluster has exactly one IngressClass marked as default, then Kubernetes [applies](/docs/concepts/services-networking/ingress/#default-ingress-class) the cluster's default IngressClass to the Ingress.
+You mark an IngressClass as default by setting the [`ingressclass.kubernetes.io/is-default-class` annotation](/docs/reference/labels-annotations-taints/#ingressclass-kubernetes-io-is-default-class) on that IngressClass, with the string value `"true"`.

 Ideally, all ingress controllers should fulfill this specification, but the various ingress
 controllers operate slightly differently.
@@ -51,7 +51,7 @@ graph LR;
 An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting. An [Ingress controller](/docs/concepts/services-networking/ingress-controllers) is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.

 An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically
-uses a service of type [Service.Type=NodePort](/docs/concepts/services-networking/service/#nodeport) or
+uses a service of type [Service.Type=NodePort](/docs/concepts/services-networking/service/#type-nodeport) or
 [Service.Type=LoadBalancer](/docs/concepts/services-networking/service/#loadbalancer).

 ## Prerequisites
@@ -219,25 +219,98 @@ of the controller that should implement the class.

 {{< codenew file="service/networking/external-lb.yaml" >}}

-IngressClass resources contain an optional parameters field. This can be used to
-reference additional implementation-specific configuration for this class.
+The `.spec.parameters` field of an IngressClass lets you reference another
+resource that provides configuration related to that IngressClass.

-#### Namespace-scoped parameters
+The specific type of parameters to use depends on the ingress controller
+that you specify in the `.spec.controller` field of the IngressClass.

-{{< feature-state for_k8s_version="v1.22" state="beta" >}}
+### IngressClass scope

-`Parameters` field has a `scope` and `namespace` field that can be used to
-reference a namespace-specific resource for configuration of an Ingress class.
-`Scope` field defaults to `Cluster`, meaning, the default is cluster-scoped
-resource. Setting `Scope` to `Namespace` and setting the `Namespace` field
-will reference a parameters resource in a specific namespace:
+Depending on your ingress controller, you may be able to use parameters
+that you set cluster-wide, or just for one namespace.

-Namespace-scoped parameters avoid the need for a cluster-scoped CustomResourceDefinition
-for a parameters resource. This further avoids RBAC-related resources
-that would otherwise be required to grant permissions to cluster-scoped
-resources.
+{{< tabs name="tabs_ingressclass_parameter_scope" >}}
+{{% tab name="Cluster" %}}
+The default scope for IngressClass parameters is cluster-wide.

-{{< codenew file="service/networking/namespaced-params.yaml" >}}
+If you set the `.spec.parameters` field and don't set
+`.spec.parameters.scope`, or if you set `.spec.parameters.scope` to
+`Cluster`, then the IngressClass refers to a cluster-scoped resource.
+The `kind` (in combination the `apiGroup`) of the parameters
+refers to a cluster-scoped API (possibly a custom resource), and
+the `name` of the parameters identifies a specific cluster scoped
+resource for that API.
+
+For example:
+```yaml
+---
+apiVersion: networking.k8s.io/v1
+kind: IngressClass
+metadata:
+  name: external-lb-1
+spec:
+  controller example.com/ingress-controller
+  parameters:
+    # The parameters for this IngressClass are specified in a
+    # ClusterIngressParameter (API group k8s.example.net) named
+    # "external-config-1". This definition tells Kubernetes to
+    # look for a cluster-scoped parameter resource.
+    scope: Cluster
+    apiGroup: k8s.example.net
+    kind: ClusterIngressParameter
+    name: external-config-1
+```
+{{% /tab %}}
+{{% tab name="Namespaced" %}}
+{{< feature-state for_k8s_version="v1.23" state="stable" >}}
+
+If you set the `.spec.parameters` field and set
+`.spec.parameters.scope` to `Namespace`, then the IngressClass refers
+to a namespaced-scoped resource. You must also set the `namespace`
+field within `.spec.parameters` to the namespace that contains
+the parameters you want to use.
+
+The `kind` (in combination the `apiGroup`) of the parameters
+refers to a namespaced API (for example: ConfigMap), and
+the `name` of the parameters identifies a specific resource
+in the namespace you specified in `namespace`.
+
+Namespace-scoped parameters help the cluster operator delegate control over the
+configuration (for example: load balancer settings, API gateway definition)
+that is used for a workload. If you used a cluster-scoped parameter then either:
+
+- the cluster operator team needs to approve a different team's changes every
+  time there's a new configuration change being applied.
+- the cluster operator must define specific access controls, such as
+  [RBAC](/docs/reference/access-authn-authz/rbac/) roles and bindings, that let
+  the application team make changes to the cluster-scoped parameters resource.
+
+The IngressClass API itself is always cluster-scoped.
+
+Here is an example of an IngressClass that refers to parameters that are
+namespaced:
+```yaml
+---
+apiVersion: networking.k8s.io/v1
+kind: IngressClass
+metadata:
+  name: external-lb-2
+spec:
+  controller example.com/ingress-controller
+  parameters:
+    # The parameters for this IngressClass are specified in an
+    # IngressParameter (API group k8s.example.com) named "external-config",
+    # that's in the "external-configuration" configuration namespace.
+    scope: Namespace
+    apiGroup: k8s.example.com
+    kind: IngressParameter
+    namespace: external-configuration
+    name: external-config
+```
+
+{{% /tab %}}
+{{< /tabs >}}

 ### Deprecated annotation

@@ -261,7 +261,7 @@ The following restrictions apply when using this field:
 at a cluster level, you (or your cluster administrator) need to disable the 
 `NetworkPolicyEndPort` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) 
 for the API server with `--feature-gates=NetworkPolicyEndPort=false,…`.
-* The `endPort` field must be equal than or greater to the `port` field.
+* The `endPort` field must be equal to or greater than the `port` field.
 * `endPort` can only be defined if `port` is also defined.
 * Both ports must be numeric.

@@ -68,6 +68,6 @@ When the [feature gate](/docs/reference/command-line-tools-reference/feature-gat

 ## {{% heading "whatsnext" %}}

-* Read about [enabling Topology Aware Hints](/docs/tasks/administer-cluster/enabling-topology-aware-hints)
+* Read about [Topology Aware Hints](/docs/concepts/services-networking/topology-aware-hints)
 * Read about [Service External Traffic Policy](/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip)
 * Read [Connecting Applications with Services](/docs/concepts/services-networking/connect-applications-service/)
@@ -183,6 +183,13 @@ Accessing a Service without a selector works the same as if it had a selector.
 In the example above, traffic is routed to the single endpoint defined in
 the YAML: `192.0.2.42:9376` (TCP).

+{{< note >}}
+The Kubernetes API server does not allow proxying to endpoints that are not mapped to 
+pods. Actions such as `kubectl proxy <service-name>` where the service has no 
+selector will fail due to this constraint. This prevents the Kubernetes API server 
+from being used as a proxy to endpoints the caller may not be authorized to access. 
+{{< /note >}}
+
 An ExternalName Service is a special case of Service that does not have
 selectors and uses DNS names instead. For more information, see the
 [ExternalName](#externalname) section later in this document.
@@ -414,7 +421,7 @@ endpoints, the kube-proxy does not forward any traffic for the relevant Service.
 {{< feature-state for_k8s_version="v1.22" state="alpha" >}}
 If you enable the `ProxyTerminatingEndpoints`
 [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
-`ProxyTerminatingEndpoints` for the kube-proxy, the kube-proxy checks if the node
+for the kube-proxy, the kube-proxy checks if the node
 has local endpoints and whether or not all the local endpoints are marked as terminating.
 If there are local endpoints and **all** of those are terminating, then the kube-proxy ignores
 any external traffic policy of `Local`. Instead, whilst the node-local endpoints remain as all
@@ -444,8 +451,7 @@ variables and DNS.

 When a Pod is run on a Node, the kubelet adds a set of environment variables
 for each active Service.  It supports both [Docker links
-compatible](https://docs.docker.com/userguide/dockerlinks/) variables (see
-[makeLinkVariables](https://releases.k8s.io/{{< param "fullversion" >}}/pkg/kubelet/envvars/envvars.go#L49))
+compatible](https://docs.docker.com/userguide/dockerlinks/) variables (see [makeLinkVariables](https://github.com/kubernetes/kubernetes/blob/dd2d12f6dc0e654c15d5db57a5f9f6ba61192726/pkg/kubelet/envvars/envvars.go#L72))
 and simpler `{SVCNAME}_SERVICE_HOST` and `{SVCNAME}_SERVICE_PORT` variables,
 where the Service name is upper-cased and dashes are converted to underscores.

@@ -544,7 +550,7 @@ The default is `ClusterIP`.
 * `ClusterIP`: Exposes the Service on a cluster-internal IP. Choosing this value
  makes the Service only reachable from within the cluster. This is the
  default `ServiceType`.
-* [`NodePort`](#nodeport): Exposes the Service on each Node's IP at a static port
+* [`NodePort`](#type-nodeport): Exposes the Service on each Node's IP at a static port
  (the `NodePort`). A `ClusterIP` Service, to which the `NodePort` Service
  routes, is automatically created.  You'll be able to contact the `NodePort` Service,
  from outside the cluster,
@@ -562,7 +568,7 @@ The default is `ClusterIP`.
 You can also use [Ingress](/docs/concepts/services-networking/ingress/) to expose your Service. Ingress is not a Service type, but it acts as the entry point for your cluster. It lets you consolidate your routing rules
 into a single resource as it can expose multiple services under the same IP address.

-### Type NodePort {#nodeport}
+### Type NodePort {#type-nodeport}

 If you set the `type` field to `NodePort`, the Kubernetes control plane
 allocates a port from a range specified by `--service-node-port-range` flag (default: 30000-32767).
@@ -1066,6 +1072,9 @@ in those modified security groups.

 {{< /note >}}

+Further documentation on annotations for Elastic IPs and other common use-cases may be found
+in the [AWS Load Balancer Controller documentation](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/).
+
 #### Other CLB annotations on Tencent Kubernetes Engine (TKE)

 There are other annotations for managing Cloud Load Balancers on TKE as shown below.
@@ -1122,7 +1131,7 @@ spec:
 ```

 {{< note >}}
-ExternalName accepts an IPv4 address string, but as a DNS names comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName
+ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName
 is intended to specify a canonical DNS name. To hardcode an IP address, consider using
 [headless Services](#headless-services).
 {{< /note >}}
--- a/Show More
+++ b/Show More