Merge remote-tracking branch 'upstream/main' into dev-1.24

2022-05-02 10:29:49 -07:00
parent 7f13ad092c a8b640bd6f
commit 5ead53b3e8
141 changed files with 5085 additions and 666 deletions
@@ -19,7 +19,7 @@ Security:
 - [Node authorizer](/docs/reference/access-authn-authz/node/) and admission control plugin are new additions that restrict kubelet’s access to secrets, pods and other objects based on its node.
 - [Encryption for Secrets](/docs/tasks/administer-cluster/encrypt-data/), and other resources in etcd, is now available as alpha.&nbsp;
 - [Kubelet TLS bootstrapping](/docs/admin/kubelet-tls-bootstrapping/) now supports client and server certificate rotation.
- [Audit logs](/docs/tasks/debug-application-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit.
+- [Audit logs](/docs/tasks/debug/debug-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit.

 Stateful workloads:  

@@ -117,7 +117,7 @@ To achieve the best possible isolation, each function call would have to happen
 By using Landlock, we could isolate function calls from each other within the same container, making a temporary file created by one function call inaccessible to the next function call, for example. Integration between Landlock and technologies like Kubernetes-based serverless frameworks would be a ripe area for further exploration.  

 ## Auditing kubectl-exec with eBPF
-In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug-application-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster.   
+In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug/debug-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster.   

 While these events log Kubernetes events, they don't currently provide the level of visibility that some may require. For example, while we can see that someone has used `kubectl exec` to enter a container, we are not able to see what commands were executed in that session. With eBPF one can attach a BPF program that would record any commands executed in the `kubectl exec` session and pass those commands to a user-space program that logs those events. We could then play that session back and know the exact sequence of events that took place.
 ## Learn more about eBPF
@@ -66,7 +66,7 @@ There are plenty of [good examples](https://docs.bitnami.com/kubernetes/how-to/c

 Incorrect or excessively permissive RBAC policies are a security threat in case of a compromised pod. Maintaining least privilege, and continuously reviewing and improving RBAC rules, should be considered part of the "technical debt hygiene" that teams build into their development lifecycle.

-[Audit Logging](/docs/tasks/debug-application-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation&#39;s security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started.
+[Audit Logging](/docs/tasks/debug/debug-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation&#39;s security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started.

 For read requests such as get, list, and watch, only the request object is saved in the audit logs; the response object is not. For requests involving sensitive data such as Secret and ConfigMap, only the metadata is exported. For all other requests, both request and response objects are saved in audit logs.

@@ -174,7 +174,7 @@ Cluster-distributed stateful services (e.g., Cassandra) can benefit from splitti

 ## Other considerations

-[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location.
+[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location.

 Your production deployment should utilize an automated installation, configuration and update tool (e.g., [Ansible](https://github.com/kubernetes-incubator/kubespray), [BOSH](https://github.com/cloudfoundry-incubator/kubo-deployment), [Chef](https://github.com/chef-cookbooks/kubernetes), [Juju](/docs/getting-started-guides/ubuntu/installation/), [kubeadm](/docs/reference/setup-tools/kubeadm/), [Puppet](https://forge.puppet.com/puppetlabs/kubernetes), etc.). A manual process will have repeatability issues, be labor intensive, error prone, and difficult to scale. [Certified distributions](https://www.cncf.io/certification/software-conformance/#logos) are likely to include a facility for retaining configuration settings across updates, but if you implement your own install and config toolchain, then retention, backup and recovery of the configuration artifacts is essential. Consider keeping your deployment components and settings under a version control system such as Git.

@@ -177,7 +177,7 @@ group_right() apiserver_request_total

 Metrics are a fast way to check whether deprecated APIs are being used, and at what rate,
 but they don't include enough information to identify particular clients or API objects.
-Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug-application-cluster/audit/)
+Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug/debug-cluster/audit/)
 for requests to deprecated APIs include an audit annotation of `"k8s.io/deprecated":"true"`.
 Administrators can use those audit events to identify specific clients or objects that need to be updated.

@@ -20,7 +20,7 @@ The paper attempts to _not_ focus on any specific [cloud native project](https:/
 When using Kubernetes as a workload orchestrator, some of the security controls this version of the whitepaper recommends are:
 * [Pod Security Policies](/docs/concepts/security/pod-security-policy/): Implement a single source of truth for “least privilege” workloads across the entire cluster
 * [Resource requests and limits](/docs/concepts/configuration/manage-resources-containers/#requests-and-limits): Apply requests (soft constraint) and limits (hard constraint) for shared resources such as memory and CPU
-* [Audit log analysis](/docs/tasks/debug-application-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events
+* [Audit log analysis](/docs/tasks/debug/debug-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events
 * [Control plane authentication and certificate root of trust](/docs/concepts/architecture/control-plane-node-communication/): Enable mutual TLS authentication with a trusted CA for communication within the cluster
 * [Secrets management](/docs/concepts/configuration/secret/): Integrate with a built-in or external secrets store

@@ -155,7 +155,7 @@ runtime where possible.

 Another thing to look out for is anything expecting to run for system maintenance
 or nested inside a container when building images will no longer work. For the
-former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
+former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
 latter you can use newer container build options like [img], [buildah],
 [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker.

@@ -32,7 +32,7 @@ The `kubectl alpha debug` features graduates to beta in 1.20, becoming `kubectl

 Note that as a new built-in command, `kubectl debug` takes priority over any kubectl plugin named “debug”. You must rename the affected plugin.

-Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/).
+Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/).

 ### Beta: API Priority and Fairness

@@ -317,7 +317,7 @@ RequestResponse's including metadata and request / response bodies. While helpfu

 Each organization needs to evaluate their
 own threat model and build an audit policy that complements or helps troubleshooting incident response. Think
-about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug-application-cluster/audit/#audit-policy).
+about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug/debug-cluster/audit/#audit-policy).
 It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point. 

 Audit logging configurations can also be tested with
@@ -0,0 +1,155 @@
+---
+layout: blog
+title: 'Increasing the security bar in Ingress-NGINX v1.2.0'
+date: 2022-04-28
+slug: ingress-nginx-1-2-0
+---
+
+**Authors:** Ricardo Katz (VMware), James Strong (Chainguard)
+
+The [Ingress](/docs/concepts/services-networking/ingress/) may be one of the most targeted components
+of Kubernetes. An Ingress typically defines an HTTP reverse proxy, exposed to the Internet, containing
+multiple websites, and with some privileged access to Kubernetes API (such as to read Secrets relating to
+TLS certificates and their private keys).
+
+While it is a risky component in your architecture, it is still the most popular way to properly expose your services.
+
+Ingress-NGINX has been part of security assessments that figured out we have a big problem: we don't
+do all proper sanitization before turning the configuration into an `nginx.conf` file, which may lead to information
+disclosure risks.
+
+While we understand this risk and the real need to fix this, it's not an easy process to do, so we took another approach to reduce (but not remove!) this risk in the current (v1.2.0) release.
+
+## Meet Ingress NGINX v1.2.0 and the chrooted NGINX process
+
+One of the main challenges is that Ingress-NGINX runs the web proxy server (NGINX) alongside the Ingress
+controller (the component that has access to Kubernetes API that and that creates the `nginx.conf` file).
+
+So, NGINX does have the same access to the filesystem of the controller (and Kubernetes service account token, and other configurations from the container). While splitting those components is our end goal, the project needed a fast response; that lead us to the idea of using `chroot()`.
+
+Let's take a look into what an Ingress-NGINX container looked like before this change:
+
+![Ingress NGINX pre chroot](ingress-pre-chroot.png)
+
+As we can see, the same container (not the Pod, the container!) that provides HTTP Proxy is the one that watches Ingress objects and writes the Container Volume
+
+Now, meet the new architecture:
+
+![Ingress NGINX post chroot](ingress-post-chroot.png)
+
+What does all of this mean? A basic summary is: that we are isolating the NGINX service as a container inside the
+controller container.
+
+While this is not strictly true, to understand what was done here, it's good to understand how
+Linux containers (and underlying mechanisms such as kernel namespaces) work.
+You can read about cgroups in the Kubernetes glossary: [`cgroup`](https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-cgroup) and learn more about cgroups interact with namespaces in the NGINX project article
+[What Are Namespaces and cgroups, and How Do They Work?](https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/).
+(As you read that, bear in mind that Linux kernel namespaces are a different thing from
+[Kubernetes namespaces](/docs/concepts/overview/working-with-objects/namespaces/)).
+
+## Skip the talk, what do I need to use this new approach?
+
+While this increases the security, we made this feature an opt-in in this release so you can have
+time to make the right adjustments in your environment(s). This new feature is only available from
+release v1.2.0 of the Ingress-NGINX controller.
+
+There are two required changes in your deployments to use this feature:
+* Append the suffix "-chroot" to the container image name. For example: `gcr.io/k8s-staging-ingress-nginx/controller-chroot:v1.2.0`
+* In your Pod template for the Ingress controller, find where you add the capability `NET_BIND_SERVICE` and add the capability `SYS_CHROOT`. After you edit the manifest, you'll see a snippet like:
+
+```yaml
+capabilities:
+  drop:
+  - ALL
+  add:
+  - NET_BIND_SERVICE
+  - SYS_CHROOT
+```
+
+If you deploy the controller using the official Helm chart then change the following setting in
+`values.yaml`:
+
+```yaml
+controller:
+  image:
+    chroot: true
+```
+
+Ingress controllers are normally set up cluster-wide (the IngressClass API is cluster scoped). If you manage the
+Ingress-NGINX controller but you're not the overall cluster operator, then check with your cluster admin about
+whether you can use the `SYS_CHROOT` capability, **before** you enable it in your deployment.
+
+## OK, but how does this increase the security of my Ingress controller?
+
+Take the following configuration snippet and imagine, for some reason it was added to your `nginx.conf`:
+```
+location /randomthing/ {
+      alias /;
+      autoindex on;
+}
+```
+
+If you deploy this configuration, someone can call `http://website.example/randomthing` and get some listing (and access) to the whole filesystem of the Ingress controller.
+
+Now, can you spot the difference between chrooted and non chrooted Nginx on the listings below?
+
+| Without extra `chroot()`             | With extra `chroot()` |
+|----------------------------|--------|
+| `bin`                      | `bin`  |
+| `dev`                      | `dev`  |
+| `etc`                      | `etc`  |
+| `home`                     |        |
+| `lib`                      | `lib`  |
+| `media`                    |        |
+| `mnt`                      |        |
+| `opt`                      | `opt`  |
+| `proc`                     | `proc` |
+| `root`                     |        |
+| `run`                      | `run`  |
+| `sbin`                     |        |
+| `srv`                      |        |
+| `sys`                      |        |
+| `tmp`                      | `tmp`  |
+| `usr`                      | `usr`  |
+| `var`                      | `var`  |
+| `dbg`                      |        |
+| `nginx-ingress-controller` |        |
+| `wait-shutdown`            |        |
+
+The one in left side is not chrooted. So NGINX has full access to the filesystem. The one in right side is chrooted, so a new filesystem with only the required files to make NGINX work is created.
+
+## What about other security improvements in this release?
+
+We know that the new `chroot()` mechanism helps address some portion of the risk, but still, someone
+can try to inject commands to read, for example, the `nginx.conf` file and extract sensitive information.
+
+So, another change in this release (this is opt-out!) is the _deep inspector_.
+We know that some directives or regular expressions may be dangerous to NGINX, so the deep inspector
+checks all fields from an Ingress object (during its reconciliation, and also with a
+[validating admission webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook))
+to verify if any fields contains these dangerous directives.
+
+The ingress controller already does this for annotations, and our goal is to move this existing validation to happen inside
+deep inspection as part of a future release.
+
+You can take a look into the existing rules in [https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go](https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go).
+
+Due to the nature of inspecting and matching all strings within relevant Ingress objects, this new feature may consume a bit more CPU. You can disable it by running the ingress controller with the command line argument `--deep-inspect=false`.
+
+## What's next?
+
+This is not our final goal. Our final goal is to split the control plane and the data plane processes.
+In fact, doing so will help us also achieve a [Gateway](https://gateway-api.sigs.k8s.io/) API implementation,
+as we may have a different controller as soon as it "knows" what to provide to the data plane
+(we need some help here!!)
+
+Some other projects in Kubernetes already take this approach
+(like [KPNG](https://github.com/kubernetes-sigs/kpng), the proposed replacement for `kube-proxy`),
+and we plan to align with them and get the same experience for Ingress-NGINX.
+
+## Further reading
+
+If you want to take a look into how chrooting was done in Ingress NGINX, take a look 
+into [https://github.com/kubernetes/ingress-nginx/pull/8337](https://github.com/kubernetes/ingress-nginx/pull/8337)
+The release v1.2.0 containing all the changes can be found at 
+[https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0)
@@ -0,0 +1,319 @@
+---
+layout: blog
+title: "Frontiers, fsGroups and frogs: the Kubernetes 1.23 release interview"
+date: 2022-04-29
+---
+
+**Author**: Craig Box (Google)
+
+One of the highlights of hosting the weekly [Kubernetes Podcast from Google](https://kubernetespodcast.com/) is talking to the release managers for each new Kubernetes version. The release team is constantly refreshing. Many working their way from small documentation fixes, step up to shadow roles, and then eventually lead a release.
+
+As we prepare for the 1.24 release next week, [in accordance with long-standing tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the story of 1.23. The release was led by [Rey Lejano](https://twitter.com/reylejano), a Field Engineer at SUSE. [I spoke to Rey](https://kubernetespodcast.com/episode/167-kubernetes-1.23/) in December, as he was awaiting the birth of his first child.
+
+Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/), so you hear all our stories from the Cloud Native community, including the story of 1.24 next week.
+
+*This transcript has been lightly edited and condensed for clarity.*
+
+---
+
+**CRAIG BOX: I'd like to start with what is, of course, on top of everyone's mind at the moment. Let's talk African clawed frogs!**
+
+REY LEJANO: [CHUCKLES] Oh, you mean [Xenopus lavis](https://en.wikipedia.org/wiki/African_clawed_frog), the scientific name for the African clawed frog?
+
+**CRAIG BOX: Of course.**
+
+REY LEJANO: Not many people know, but my background and my degree is actually in microbiology, from the University of California Davis. I did some research for about four years in biochemistry, in a biochemistry lab, and I [do have a research paper published](https://www.sciencedirect.com/science/article/pii/). It's actually on glycoproteins, particularly something called "cortical granule lectin". We used frogs, because they generate lots and lots of eggs, from which we can extract the protein. That protein prevents polyspermy. When the sperm goes into the egg, the egg releases a glycoprotein, cortical granule lectin, to the membrane, and prevents any other sperm from going inside the egg.
+
+**CRAIG BOX: Were you able to take anything from the testing that we did on frogs and generalize that to higher-order mammals, perhaps?**
+
+REY LEJANO: Yes. Since mammals also have cortical granule lectin, we were able to analyze both the convergence and the evolutionary pattern, not just from multiple species of frogs, but also into mammals as well.
+
+**CRAIG BOX: Now, there's a couple of different threads to unravel here. When you were young, what led you into the fields of biology, and perhaps more the technical side of it?**
+
+REY LEJANO: I think it was mostly from family, since I do have a family history in the medical field that goes back generations. So I kind of felt like that was the natural path going into college.
+
+**CRAIG BOX: Now, of course, you're working in a more abstract tech field. What led you out of microbiology?**
+
+REY LEJANO: [CHUCKLES] Well, I've always been interested in tech. Taught myself a little programming when I was younger, before high school, did some web dev stuff. Just kind of got burnt out being in a lab. I was literally in the basement. I had a great opportunity to join a consultancy that specialized in [ITIL](https://www.axelos.com/certifications/itil-service-management/what-is-itil). I actually started off with application performance management, went into monitoring, went into operation management and also ITIL, which is aligning your IT asset management and service managements with business services. Did that for a good number of years, actually.
+
+**CRAIG BOX: It's very interesting, as people describe the things that they went through and perhaps the technologies that they worked on, you can pretty much pinpoint how old they might be. There's a lot of people who come into tech these days that have never heard of ITIL. They have no idea what it is. It's basically just SRE with more process.**
+
+REY LEJANO: Yes, absolutely. It's not very cloud native. [CHUCKLES]
+
+**CRAIG BOX: Not at all.**
+
+REY LEJANO: You don't really hear about it in the cloud native landscape. Definitely, you can tell someone's been in the field for a little bit, if they specialize or have worked with ITIL before.
+
+**CRAIG BOX: You mentioned that you wanted to get out of the basement. That is quite often where people put the programmers. Did they just give you a bit of light in the new basement?**
+
+REY LEJANO: [LAUGHS] They did give us much better lighting. Able to get some vitamin D sometimes, as well.
+
+**CRAIG BOX: To wrap up the discussion about your previous career — over the course of the last year, with all of the things that have happened in the world, I could imagine that microbiology skills may be more in demand than perhaps they were when you studied them?**
+
+REY LEJANO: Oh, absolutely. I could definitely see a big increase of numbers of people going into the field. Also, reading what's going on with the world currently kind of brings back all the education I've learned in the past, as well.
+
+**CRAIG BOX: Do you keep in touch with people you went through school with?**
+
+REY LEJANO: Just some close friends, but not in the microbiology field.
+
+**CRAIG BOX: One thing that I think will probably happen as a result of the pandemic is a renewed interest in some of these STEM fields. It will be interesting to see what impact that has on society at large.**
+
+REY LEJANO: Yeah. I think that'll be great.
+
+**CRAIG BOX: You mentioned working at a consultancy doing IT management, application performance monitoring, and so on. When did Kubernetes come into your professional life?**
+
+REY LEJANO: One of my good friends at the company I worked at, left in mid-2015. He went on to a company that was pretty heavily into Docker. He taught me a little bit. I did my first "docker run" around 2015, maybe 2016. Then, one of the applications we were using for the ITIL framework was containerized around 2018 or so, also in Kubernetes. At that time, it was pretty buggy. That was my initial introduction to Kubernetes and containerised applications.
+
+Then I left that company, and I actually joined my friend over at [RX-M](https://rx-m.com/), which is a cloud native consultancy and training firm. They specialize in Docker and Kubernetes. I was able to get my feet wet. I got my CKD, got my CKA as well. And they were really, really great at encouraging us to learn more about Kubernetes and also to be involved in the community.
+
+**CRAIG BOX: You will have seen, then, the life cycle of people adopting Kubernetes and containerization at large, through your own initial journey and then through helping customers. How would you characterize how that journey has changed from the early days to perhaps today?**
+
+REY LEJANO: I think the early days, there was a lot of questions of, why do I have to containerize? Why can't I just stay with virtual machines?
+
+**CRAIG BOX: It's a line item on your CV.**
+
+REY LEJANO: [CHUCKLES] It is. And nowadays, I think people know the value of using containers, of orchestrating containers with Kubernetes. I don't want to say "jumping on the bandwagon", but it's become the de-facto standard to orchestrate containers.
+
+**CRAIG BOX: It's not something that a consultancy needs to go out and pitch to customers that they should be doing. They're just taking it as, that will happen, and starting a bit further down the path, perhaps.**
+
+REY LEJANO: Absolutely.
+
+**CRAIG BOX: Working at a consultancy like that, how much time do you get to work on improving process, perhaps for multiple customers, and then looking at how you can upstream that work, versus paid work that you do for just an individual customer at a time?**
+
+REY LEJANO: Back then, it would vary. They helped me introduce myself, and I learned a lot about the cloud native landscape and Kubernetes itself. They helped educate me as to how the cloud native landscape, and the tools around it, can be used together. My boss at that company, Randy, he actually encouraged us to start contributing upstream, and encouraged me to join the release team. He just said, this is a great opportunity. Definitely helped me with starting with the contributions early on.
+
+**CRAIG BOX: Was the release team the way that you got involved with upstream Kubernetes contribution?**
+
+REY LEJANO: Actually, no. My first contribution was  with SIG Docs. I met Taylor Dolezal — he was the release team lead for 1.19, but he is involved with SIG Docs as well. I met him at KubeCon 2019, I sat at his table during a luncheon. I remember Paris Pittman was hosting this luncheon at the Marriott. Taylor says he was involved with SIG Docs. He encouraged me to join. I started joining into meetings, started doing a few drive-by PRs. That's what we call them — drive-by — little typo fixes. Then did a little bit more, started to send better or higher quality pull requests, and also reviewing PRs.
+
+**CRAIG BOX: When did you first formally take your release team role?**
+
+REY LEJANO: That was in [1.18](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.18/release_team.md), in December. My boss at the time encouraged me to apply. I did, was lucky enough to get accepted for the release notes shadow. Then from there, stayed in with release notes for a few cycles, then went into Docs, naturally then led Docs, then went to Enhancements, and now I'm the release lead for 1.23.
+
+**CRAIG BOX: I don't know that a lot of people think about what goes into a good release note. What would you say does?**
+
+REY LEJANO: [CHUCKLES] You have to tell the end user what has changed or what effect that they might see in the release notes. It doesn't have to be highly technical. It could just be a few lines, and just saying what has changed, what they have to do if they have to do anything as well.
+
+**CRAIG BOX: As you moved through the process of shadowing, how did you learn from the people who were leading those roles?**
+
+REY LEJANO: I said this a few times when I was the release lead for this cycle. You get out of the release team as much as you put in, or it directly aligns to how much you put in. I learned a lot. I went into the release team having that mindset of learning from the role leads, learning from the other shadows, as well. That's actually a saying that my first role lead told me. I still carry it to heart, and that was back in 1.18. That was Eddie, in the very first meeting we had, and I still carry it to heart.
+
+**CRAIG BOX: You, of course, were [the release lead for 1.23](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.23). First of all, congratulations on the release.**
+
+REY LEJANO: Thank you very much.
+
+**CRAIG BOX: The theme for this release is [The Next Frontier](https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/). Tell me the story of how we came to the theme and then the logo.**
+
+REY LEJANO: The Next Frontier represents a few things. It not only represents the next enhancements in this release, but Kubernetes itself also has a history of Star Trek references. The original codename for Kubernetes was Project Seven, a reference to Seven of Nine, originally from Star Trek Voyager. Also the seven spokes in the helm in the logo of Kubernetes as well. And, of course, Borg, the predecessor to Kubernetes.
+
+The Next Frontier continues that Star Trek reference. It's a fusion of two titles in the Star Trek universe. One is [Star Trek V, the Final Frontier](https://en.wikipedia.org/wiki/Star_Trek_V:_The_Final_Frontier), and the Star Trek: The Next Generation.
+
+**CRAIG BOX: Do you have any opinion on the fact that Star Trek V was an odd-numbered movie, and they are [canonically referred to as being lesser than the even-numbered ones](https://screenrant.com/star-trek-movies-odd-number-curse-explained/)?**
+
+REY LEJANO: I can't say, because I am such a sci-fi nerd that I love all of them even though they're bad. Even the post-Next Generation movies, after the series, I still liked all of them, even though I know some weren't that great.
+
+**CRAIG BOX: Am I right in remembering that Star Trek V was the one directed by William Shatner?**
+
+REY LEJANO: Yes, that is correct.
+
+**CRAIG BOX: I think that says it all.**
+
+REY LEJANO: [CHUCKLES] Yes.
+
+**CRAIG BOX: Now, I understand that the theme comes from a part of the [SIG Release charter](https://github.com/kubernetes/community/blob/master/sig-release/charter.md)?**
+
+REY LEJANO: Yes. There's a line in the SIG Release charter, "ensure there is a consistent group of community members in place to support the release process across time." With the release team, we have new shadows that join every single release cycle. With this, we're growing with this community. We're growing the release team members. We're growing SIG Release. We're growing the Kubernetes community itself. For a lot of people, this is their first time contributing to open source, so that's why I say it's their new open source frontier.
+
+**CRAIG BOX: And the logo is obviously very Star Trek-inspired. It sort of surprised me that it took that long for someone to go this route.**
+
+REY LEJANO: I was very surprised as well. I had to relearn Adobe Illustrator to create the logo.
+
+**CRAIG BOX: This your own work, is it?**
+
+REY LEJANO: This is my own work.
+
+**CRAIG BOX: It's very nice.**
+
+REY LEJANO: Thank you very much. Funny, the galaxy actually took me the longest time versus the ship. Took me a few days to get that correct. I'm always fine-tuning it, so there might be a final change when this is actually released.
+
+**CRAIG BOX: No frontier is ever truly final.**
+
+REY LEJANO: True, very true.
+
+**CRAIG BOX: Moving now from the theme of the release to the substance, perhaps, what is new in 1.23?**
+
+REY LEJANO: We have 47 enhancements. I'm going to run through most of the stable ones, if not all of them, some of the key Beta ones, and a few of the Alpha enhancements for 1.23.
+
+One of the key enhancements is [dual-stack IPv4/IPv6](https://github.com/kubernetes/enhancements/issues/563), which went GA in 1.23.
+
+Some background info: dual-stack was introduced as Alpha in 1.15. You probably saw a keynote at KubeCon 2019. Back then, the way dual-stack worked was that you needed two services — you needed a service per IP family. You would need a service for IPv4 and a service for IPv6. It was refactored in 1.20. In 1.21, it was in Beta; clusters were enabled to be dual-stack by default.
+
+And then in 1.23 we did remove the IPv6 dual-stack feature flag. It's not mandatory to use dual-stack. It's actually not "default" still. The pods, the services still default to single-stack. There are some requirements to be able to use dual-stack. The nodes have to be routable on IPv4 and IPv6 network interfaces. You need a CNI plugin that supports dual-stack. The pods themselves have to be configured to be dual-stack. And the services need the ipFamilyPolicy field to specify prefer dual-stack, or require dual-stack.
+
+**CRAIG BOX: This sounds like there's an implication in this that v4 is still required. Do you see a world where we can actually move to v6-only clusters?**
+
+REY LEJANO: I think we'll be talking about IPv4 and IPv6 for many, many years to come. I remember a long time ago, they kept saying "it's going to be all IPv6", and that was decades ago.
+
+**CRAIG BOX: I think I may have mentioned on the show before, but there was [a meeting in London that Vint Cerf attended](https://www.youtube.com/watch?v=AEaJtZVimqs), and he gave a public presentation at the time to say, now is the time of v6. And that was 10 years ago at least. It's still not the time of v6, and my desktop still doesn't have Linux on it. One day.**
+
+REY LEJANO: [LAUGHS] In my opinion, that's one of the big key features that went stable for 1.23.
+
+One of the other highlights of 1.23 is [pod security admission going to Beta](/blog/2021/12/09/pod-security-admission-beta/). I know this feature is going to Beta, but I highlight this because as some people might know, PodSecurityPolicy, which was deprecated in 1.21, is targeted to be removed in 1.25. Pod security admission replaces pod security policy. It's an admission controller. It evaluates the pods against a predefined set of pod security standards to either admit or deny the pod for running.
+
+There's three levels of pod security standards. Privileged, that's totally open. Baseline, known privileges escalations are minimized. Or Restricted, which is hardened. And you could set pod security standards either to run in three modes, which is enforce: reject any pods that are in violation; to audit: pods are allowed to be created, but the violations are recorded; or warn: it will send a warning message to the user, and the pod is allowed.
+
+**CRAIG BOX: You mentioned there that PodSecurityPolicy is due to be deprecated in two releases' time. Are we lining up these features so that pod security admission will be GA at that time?**
+
+REY LEJANO: Yes. Absolutely. I'll talk about that for another feature in a little bit as well. There's also another feature that went to GA. It was an API that went to GA, and therefore the Beta API is now deprecated. I'll talk about that a little bit.
+
+**CRAIG BOX: All right. Let's talk about what's next on the list.**
+
+REY LEJANO: Let's move on to more stable enhancements. One is the [TTL controller](https://github.com/kubernetes/enhancements/issues/592). This cleans up jobs and pods after the jobs are finished. There is a TTL timer that starts when the job or pod is finished. This TTL controller watches all the jobs, and ttlSecondsAfterFinished needs to be set. The controller will see if the ttlSecondsAfterFinished, combined with the last transition time, if it's greater than now. If it is, then it will delete the job and the pods of that job.
+
+**CRAIG BOX: Loosely, it could be called a garbage collector?**
+
+REY LEJANO: Yes. Garbage collector for pods and jobs, or jobs and pods.
+
+**CRAIG BOX: If Kubernetes is truly becoming a programming language, it of course has to have a garbage collector implemented.**
+
+REY LEJANO: Yeah. There's another one, too, coming in Alpha. [CHUCKLES]
+
+**CRAIG BOX: Tell me about that.**
+
+REY LEJANO: That one is coming in in Alpha. It's actually one of my favorite features, because there's only a few that I'm going to highlight today. [PVCs for StafeulSet will be cleaned up](https://github.com/kubernetes/enhancements/issues/1847). It will auto-delete PVCs created by StatefulSets, when you delete that StatefulSet.
+
+**CRAIG BOX: What's next on our tour of stable features?**
+
+REY LEJANO: Next one is, [skip volume ownership change goes to stable](https://github.com/kubernetes/enhancements/issues/695). This is from SIG Storage. There are times when you're running a stateful application, like many databases, they're sensitive to permission bits changing underneath. Currently, when a volume is bind mounted inside the container, the permissions of that volume will change recursively. It might take a really long time.
+
+Now, there's a field, the fsGroupChangePolicy, which allows you, as a user, to tell Kubernetes how you want the permission and ownership change for that volume to happen. You can set it to always, to always change permissions, or just on mismatch, to only do it when the permission ownership changes at the top level is different from what is expected.
+
+**CRAIG BOX: It does feel like a lot of these enhancements came from a very particular use case where someone said, "hey, this didn't work for me and I've plumbed in a feature that works with exactly the thing I need to have".**
+
+REY LEJANO: Absolutely. People create issues for these, then create Kubernetes enhancement proposals, and then get targeted for releases.
+
+**CRAIG BOX: Another GA feature in this release — ephemeral volumes.**
+
+REY LEJANO: We've always been able to use empty dir for ephemeral volumes, but now we could actually have [ephemeral inline volumes](https://github.com/kubernetes/enhancements/issues/1698), meaning that you could take your standard CSI driver and be able to use ephemeral volumes with it.
+
+**CRAIG BOX: And, a long time coming, [CronJobs](https://github.com/kubernetes/enhancements/issues/19).**
+
+REY LEJANO: CronJobs is a funny one, because it was stable before 1.23. For 1.23, it was still tracked,but it was just cleaning up some of the old controller. With CronJobs, there's a v2 controller. What was cleaned up in 1.23 is just the old v1 controller.
+
+**CRAIG BOX: Were there any other duplications or major cleanups of note in this release?**
+
+REY LEJANO: Yeah. There were a few you might see in the major themes. One's a little tricky, around FlexVolumes. This is one of the efforts from SIG Storage. They have an effort to migrate in-tree plugins to CSI drivers. This is a little tricky, because FlexVolumes were actually deprecated in November 2020. We're [formally announcing it in 1.23](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors).
+
+**CRAIG BOX: FlexVolumes, in my mind, predate CSI as a concept. So it's about time to get rid of them.**
+
+REY LEJANO: Yes, it is. There's another deprecation, just some [klog specific flags](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog), but other than that, there are no other big deprecations in 1.23.
+
+**CRAIG BOX: The buzzword of the last KubeCon, and in some ways the theme of the last 12 months, has been secure software supply chain. What work is Kubernetes doing to improve in this area?**
+
+REY LEJANO: For 1.23, Kubernetes is now SLSA compliant at Level 1, which means that provenance attestation files that describe the staging and release phases of the release process are satisfactory for the SLSA framework.
+
+**CRAIG BOX: What needs to happen to step up to further levels?**
+
+REY LEJANO: Level 1 means a few things — that the build is scripted; that the provenance is available, meaning that the artifacts are verified and they're handed over from one phase to the next; and describes how the artifact is produced. Level 2 means that the source is version-controlled, which it is, provenance is authenticated, provenance is service-generated, and there is a build service. There are four levels of SLSA compliance.
+
+**CRAIG BOX: It does seem like the levels were largely influenced by what it takes to build a big, secure project like this. It doesn't seem like it will take a lot of extra work to move up to verifiable provenance, for example. There's probably just a few lines of script required to meet many of those requirements.**
+
+REY LEJANO: Absolutely. I feel like we're almost there; we'll see what will come out of 1.24. And I do want to give a big shout-out to SIG Release and Release Engineering, primarily to Adolfo García Veytia, who is aka Puerco on GitHub and on Slack. He's been driving this forward.
+
+**CRAIG BOX: You've mentioned some APIs that are being graduated in time to replace their deprecated version. Tell me about the new HPA API.**
+
+REY LEJANO: The [horizontal pod autoscaler v2 API](https://github.com/kubernetes/enhancements/issues/2702), is now stable, which means that the v2beta2 API is deprecated. Just for everyone's knowledge, the v1 API is not being deprecated. The difference is that v2 adds support for multiple and custom metrics to be used for HPA.
+
+**CRAIG BOX: There's also now a facility to validate my CRDs with an expression language.**
+
+REY LEJANO: Yeah. You can use the [Common Expression Language, or CEL](https://github.com/google/cel-spec), to validate your CRDs, so you no longer need to use webhooks. This also makes the CRDs more self-contained and declarative, because the rules are now kept within the CRD object definition.
+
+**CRAIG BOX: What new features, perhaps coming in Alpha or Beta, have taken your interest?**
+
+REY LEJANO: Aside from pod security policies, I really love [ephemeral containers](https://github.com/kubernetes/enhancements/issues/277) supporting kubectl debug. It launches an ephemeral container and a running pod, shares those pod namespaces, and you can do all your troubleshooting with just running kubectl debug.
+
+**CRAIG BOX: There's also been some interesting changes in the way that events are handled with kubectl.**
+
+REY LEJANO: Yeah. kubectl events has always had some issues, like how things weren't sorted. [kubectl events improved](https://github.com/kubernetes/enhancements/issues/1440) that so now you can do `--watch`, and it will also sort with the `--watch` option as well. That is something new. You can actually combine fields and custom columns. And also, you can list events in the timeline with doing the last N number of minutes. And you can also sort events using other criteria as well.
+
+**CRAIG BOX: You are a field engineer at SUSE. Are there any things that are coming in that your individual customers that you deal with are looking out for?**
+
+REY LEJANO: More of what I look out for to help the customers.
+
+**CRAIG BOX: Right.**
+
+REY LEJANO: I really love kubectl events. Really love the PVCs being cleaned up with StatefulSets. Most of it's for selfish reasons that it will improve troubleshooting efforts. [CHUCKLES]
+
+**CRAIG BOX: I have always hoped that a release team lead would say to me, "yes, I have selfish reasons. And I finally got something I wanted in."**
+
+REY LEJANO: [LAUGHS]
+
+**CRAIG BOX: Perhaps I should run to be release team lead, just so I can finally get init containers fixed once and for all.**
+
+REY LEJANO: Oh, init containers, I've been looking for that for a while. I've actually created animated GIFs on how init containers will be run with that Kubernetes enhancement proposal, but it's halted currently.
+
+**CRAIG BOX: One day.**
+
+REY LEJANO: One day. Maybe I shouldn't stay halted.
+
+**CRAIG BOX: You mentioned there are obviously the things you look out for. Are there any things that are coming down the line, perhaps Alpha features or maybe even just proposals you've seen lately, that you're personally really looking forward to seeing which way they go?**
+
+REY LEJANO: Yeah. Oone is a very interesting one, it affects the whole community, so it's not just for personal reasons. As you may have known, Dockershim is deprecated. And we did release a blog that it will be removed in 1.24.
+
+**CRAIG BOX: Scared a bunch of people.**
+
+REY LEJANO: Scared a bunch of people. From a survey, we saw that a lot of people are still using Docker and Dockershim. One of the enhancements for 1.23 is, [kubelet CRI goes to Beta](https://github.com/kubernetes/enhancements/issues/2040). This promotes the CRI API, which is required. This had to be in Beta for Dockershim to be removed in 1.24.
+
+**CRAIG BOX: Now, in the last release team lead interview, [we spoke with Savitha Raghunathan](https://kubernetespodcast.com/episode/157-kubernetes-1.22/), and she talked about what she would advise you as her successor. It was to look out for the mental health of the team members. How were you able to take that advice on board?**
+
+REY LEJANO: That was great advice from Savitha. A few things I've made note of with each release team meeting. After each release team meeting, I stop the recording, because we do record all the meetings and post them on YouTube. And I open up the floor to anyone who wants to say anything that's not recorded, that's not going to be on the agenda. Also, I tell people not to work on weekends. I broke this rule once, but other than that, I told people it could wait. Just be mindful of your mental health.
+
+**CRAIG BOX: It's just been announced that [James Laverack from Jetstack](https://twitter.com/JamesLaverack/status/1466834312993644551) will be the release team lead for 1.24. James and I shared an interesting Mexican dinner at the last KubeCon in San Diego.**
+
+REY LEJANO: Oh, nice. I didn't know you knew James.
+
+**CRAIG BOX: The British tech scene. We're a very small world. What will your advice to James be?**
+
+REY LEJANO: What I would tell James for 1.24 is use teachable moments in the release team meetings. When you're a shadow for the first time, it's very daunting. It's very difficult, because you don't know the repos. You don't know the release process. Everyone around you seems like they know the release process, and very familiar with what the release process is. But as a first-time shadow, you don't know all the vernacular for the community. I just advise to use teachable moments. Take a few minutes in the release team meetings to make it a little easier for new shadows to ramp up and to be familiar with the release process.
+
+**CRAIG BOX: Has there been major evolution in the process in the time that you've been involved? Or do you think that it's effectively doing what it needs to do?**
+
+REY LEJANO: It's always evolving. I remember my first time in release notes, 1.18, we said that our goal was to automate and program our way out so that we don't have a release notes team anymore. That's changed [CHUCKLES] quite a bit. Although there's been significant advancements in the release notes process by Adolfo and also James, they've created a subcommand in krel to generate release notes.
+
+But nowadays, all their release notes are richer. Still not there at the automation process yet. Every release cycle, there is something a little bit different. For this release cycle, we had a production readiness review deadline. It was a soft deadline. A production readiness review is a review by several people in the community. It's actually been required since 1.21, and it ensures that the enhancements are observable, scalable, supportable, and it's safe to operate in production, and could also be disabled or rolled back. In 1.23, we had a deadline to have the production readiness review completed by a specific date.
+
+**CRAIG BOX: How have you found the change of schedule to three releases per year rather than four?**
+
+REY LEJANO: Moving to three releases a year from four, in my opinion, has been an improvement, because we support the last three releases, and now we can actually support the last releases in a calendar year instead of having 9 months out of 12 months of the year.
+
+**CRAIG BOX: The next event on the calendar is a [Kubernetes contributor celebration](https://www.kubernetes.dev/events/kcc2021/) starting next Monday. What can we expect from that event?**
+
+REY LEJANO: This is our second time running this virtual event. It's a virtual celebration to recognize the whole community and all of our accomplishments of the year, and also contributors. There's a number of events during this week of celebration. It starts the week of December 13.
+
+There's events like the Kubernetes Contributor Awards, where SIGs honor and recognize the hard work of the community and contributors. There's also a DevOps party game as well. There is a cloud native bake-off. I do highly suggest people to go to [kubernetes.dev/celebration](https://www.kubernetes.dev/events/past-events/2021/kcc2021/) to learn more.
+
+**CRAIG BOX: How exactly does one judge a virtual bake-off?**
+
+REY LEJANO: That I don't know. [CHUCKLES]
+
+**CRAIG BOX: I tasted my scones. I think they're the best. I rate them 10 out of 10.**
+
+REY LEJANO: Yeah. That is very difficult to do virtually. I would have to say, probably what the dish is, how closely it is tied with Kubernetes or open source or to CNCF. There's a few judges. I know Josh Berkus and Rin Oliver are a few of the judges running the bake-off.
+
+**CRAIG BOX: Yes. We spoke with Josh about his love of the kitchen, and so he seems like a perfect fit for that role.**
+
+REY LEJANO: He is.
+
+**CRAIG BOX: Finally, your wife and yourself are expecting your first child in January. Have you had a production readiness review for that?**
+
+REY LEJANO: I think we failed that review. [CHUCKLES]
+
+**CRAIG BOX: There's still time.**
+
+REY LEJANO: We are working on refactoring. We're going to refactor a little bit in December, and `--apply` again.
+
+---
+
+_[Rey Lejano](https://twitter.com/reylejano) is a field engineer at SUSE, by way of Rancher Labs, and was the release team lead for Kubernetes 1.23. He is now also a co-chair for SIG Docs. His son Liam is now 3 and a half months old._
+
+_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._
@@ -0,0 +1,79 @@
+---
+layout: blog
+title: "Storage Capacity Tracking reaches GA in Kubernetes 1.24"
+date: 2022-05-06
+slug: storage-capacity-ga
+---
+
+ **Authors:** Patrick Ohly (Intel)
+
+The v1.24 release of Kubernetes brings [storage capacity](/docs/concepts/storage/storage-capacity/)
+tracking as a generally available feature.
+
+## Problems we have solved
+
+As explained in more detail in the [previous blog post about this
+feature](/blog/2021/04/14/local-storage-features-go-beta/), storage capacity
+tracking allows a CSI driver to publish information about remaining
+capacity. The kube-scheduler then uses that information to pick suitable nodes
+for a Pod when that Pod has volumes that still need to be provisioned.
+
+Without this information, a Pod may get stuck without ever being scheduled onto
+a suitable node because kube-scheduler has to choose blindly and always ends up
+picking a node for which the volume cannot be provisioned because the
+underlying storage system managed by the CSI driver does not have sufficient
+capacity left.
+
+Because CSI drivers publish storage capacity information that gets used at a
+later time when it might not be up-to-date anymore, it can still happen that a
+node is picked that doesn't work out after all. Volume provisioning recovers
+from that by informing the scheduler that it needs to try again with a
+different node.
+
+[Load
+tests](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/docs/storage-capacity-tracking.md)
+that were done again for promotion to GA confirmed that all storage in a
+cluster can be consumed by Pods with storage capacity tracking whereas Pods got
+stuck without it.
+
+## Problems we have *not* solved
+
+Recovery from a failed volume provisioning attempt has one known limitation: if a Pod
+uses two volumes and only one of them could be provisioned, then all future
+scheduling decisions are limited by the already provisioned volume. If that
+volume is local to a node and the other volume cannot be provisioned there, the
+Pod is stuck. This problem pre-dates storage capacity tracking and while the
+additional information makes it less likely to occur, it cannot be avoided in
+all cases, except of course by only using one volume per Pod.
+
+An idea for solving this was proposed in a [KEP
+draft](https://github.com/kubernetes/enhancements/pull/1703): volumes that were
+provisioned and haven't been used yet cannot have any valuable data and
+therefore could be freed and provisioned again elsewhere. SIG Storage is
+looking for interested developers who want to continue working on this.
+
+Also not solved is support in Cluster Autoscaler for Pods with volumes. For CSI
+drivers with storage capacity tracking, a prototype was developed and discussed
+in [a PR](https://github.com/kubernetes/autoscaler/pull/3887). It was meant to
+work with arbitrary CSI drivers, but that flexibility made it hard to configure
+and slowed down scale up operations: because autoscaler was unable to simulate
+volume provisioning, it only scaled the cluster by one node at a time, which
+was seen as insufficient.
+
+Therefore that PR was not merged and a different approach with tighter coupling
+between autoscaler and CSI driver will be needed. For this a better
+understanding is needed about which local storage CSI drivers are used in
+combination with cluster autoscaling. Should this lead to a new KEP, then users
+will have to try out an implementation in practice before it can move to beta
+or GA. So please reach out to SIG Storage if you have an interest in this
+topic.
+
+## Acknowledgements
+
+Thanks a lot to the members of the community who have contributed to this
+feature or given feedback including members of [SIG
+Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling),
+[SIG
+Autoscaling](https://github.com/kubernetes/community/tree/master/sig-autoscaling),
+and of course [SIG
+Storage](https://github.com/kubernetes/community/tree/master/sig-storage)!
@@ -0,0 +1,162 @@
+---
+layout: blog
+title: "Kubernetes 1.24: Volume Populators Graduate to Beta"
+date: 2022-05-16
+slug: volume-populators-beta
+---
+
+**Author:**
+Ben Swartzlander (NetApp)
+
+The volume populators feature is now two releases old and entering beta! The `AnyVolumeDataSouce` feature
+gate defaults to enabled in Kubernetes v1.24, which means that users can specify any custom resource
+as the data source of a PVC.
+
+An [earlier blog article](/blog/2021/08/30-volume-populators-redesigned/) detailed how the
+volume populators feature works. In short, a cluster administrator can install a CRD and
+associated populator controller in the cluster, and any user who can create instances of 
+the CR can create pre-populated volumes by taking advantage of the populator.
+
+Multiple populators can be installed side by side for different purposes. The SIG storage
+community is already seeing some implementations in public, and more prototypes should
+appear soon.
+
+Cluster administrations are **strongly encouraged** to install the
+volume-data-source-validator controller and associated `VolumePopulator` CRD before installing
+any populators so that users can get feedback about invalid PVC data sources.
+
+## New Features
+
+The [lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator) library
+on which populators are built now includes metrics to help operators monitor and detect
+problems. This library is now beta and latest release is v1.0.1.
+
+The [volume data source validator](https://github.com/kubernetes-csi/volume-data-source-validator)
+controller also has metrics support added, and is in beta. The `VolumePopulator` CRD is
+beta and the latest release is v1.0.1.
+
+## Trying it out
+
+To see how this works, you can install the sample "hello" populator and try it
+out.
+
+First install the volume-data-source-validator controller.
+
+```shell
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/client/config/crd/populator.storage.k8s.io_volumepopulators.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/rbac-data-source-validator.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/setup-data-source-validator.yaml
+```
+
+Next install the example populator.
+
+```shell
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/v1.0.1/example/hello-populator/crd.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/87a47467b86052819e9ad13d15036d65b9a32fbb/example/hello-populator/deploy.yaml
+```
+
+Your cluster now has a new CustomResourceDefinition that provides a test API named Hello.
+Create an instance of the `Hello` custom resource, with some text:
+
+```yaml
+apiVersion: hello.example.com/v1alpha1
+kind: Hello
+metadata:
+  name: example-hello
+spec:
+  fileName: example.txt
+  fileContents: Hello, world!
+```
+
+Create a PVC that refers to that CR as its data source.
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: example-pvc
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Mi
+  dataSourceRef:
+    apiGroup: hello.example.com
+    kind: Hello
+    name: example-hello
+  volumeMode: Filesystem
+```
+
+Next, run a Job that reads the file in the PVC.
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: example-job
+spec:
+  template:
+    spec:
+      containers:
+        - name: example-container
+          image: busybox:latest
+          command:
+            - cat
+            - /mnt/example.txt
+          volumeMounts:
+            - name: vol
+              mountPath: /mnt
+      restartPolicy: Never
+      volumes:
+        - name: vol
+          persistentVolumeClaim:
+            claimName: example-pvc
+```
+
+Wait for the job to complete (including all of its dependencies).
+
+```shell
+kubectl wait --for=condition=Complete job/example-job
+```
+
+And last examine the log from the job.
+
+```shell
+kubectl logs job/example-job
+```
+
+The output should be:
+
+```terminal
+Hello, world!
+```
+
+Note that the volume already contained a text file with the string contents from
+the CR. This is only the simplest example. Actual populators can set up the volume
+to contain arbitrary contents.
+
+## How to write your own volume populator
+
+Developers interested in writing new poplators are encouraged to use the
+[lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator) library
+and to only supply a small controller wrapper around the library, and a pod image
+capable of attaching to volumes and writing the appropriate data to the volume.
+
+Individual populators can be extremely generic such that they work with every type
+of PVC, or they can do vendor specific things to rapidly fill a volume with data
+if the volume was provisioned by a specific CSI driver from the same vendor, for
+example, by communicating directly with the storage for that volume.
+
+## How can I learn more?
+
+The enhancement proposal,
+[Volume Populators](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators), includes lots of detail about the history and technical implementation
+of this feature.
+
+[Volume populators and data sources](/docs/concepts/storage/persistent-volumes/#volume-populators-and-data-sources), within the documentation topic about persistent volumes,
+explains how to use this feature in your cluster.
+
+Please get involved by joining the Kubernetes storage SIG to help us enhance this
+feature. There are a lot of good ideas already and we'd be thrilled to have more!
+
@@ -59,7 +59,7 @@ Before choosing a guide, here are some considerations:

 * [Using Sysctls in a Kubernetes Cluster](/docs/tasks/administer-cluster/sysctl-cluster/) describes to an administrator how to use the `sysctl` command-line tool to set kernel parameters .

-* [Auditing](/docs/tasks/debug-application-cluster/audit/) describes how to interact with Kubernetes' audit logs.
+* [Auditing](/docs/tasks/debug/debug-cluster/audit/) describes how to interact with Kubernetes' audit logs.

 ### Securing the kubelet
  * [Control Plane-Node communication](/docs/concepts/architecture/control-plane-node-communication/)
@@ -47,10 +47,9 @@ or by enforcement (the system prevents the container from ever exceeding the lim
 runtimes can have different ways to implement the same restrictions.

 {{< note >}}
-If a container specifies its own memory limit, but does not specify a memory request, Kubernetes
-automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own
-CPU limit, but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches
-the limit.
+If you specify a limit for a resource, but do not specify any request, and no admission-time
+mechanism has applied a default request for that resource, then Kubernetes copies the limit
+you specified and uses it as the requested value for the resource.
 {{< /note >}}

 ## Resource types
@@ -229,9 +228,9 @@ see the [Troubleshooting](#troubleshooting) section.
 The kubelet reports the resource usage of a Pod as part of the Pod
 [`status`](/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status).

-If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
+If optional [tools for monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)
 are available in your cluster, then Pod resource usage can be retrieved either
-from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-api)
+from the [Metrics API](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api)
 directly or from your monitoring tools.

 ## Local ephemeral storage
@@ -114,7 +114,7 @@ Containers started by Kubernetes automatically include this DNS server in their

 ### Container Resource Monitoring

-[Container Resource Monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) records generic time-series metrics
+[Container Resource Monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) records generic time-series metrics
 about containers in a central database, and provides a UI for browsing that data.

 ### Cluster-level Logging
@@ -134,7 +134,7 @@ for the corresponding API object, and then written to the object store (shown as
 Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster.
 The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.

-For more information, see [Auditing](/docs/tasks/debug-application-cluster/audit/).
+For more information, see [Auditing](/docs/tasks/debug/debug-cluster/audit/).

 ## API server ports and IPs

@@ -86,7 +86,7 @@ takes if a potential violation is detected:
 Mode | Description
 :---------|:------------
 **enforce** | Policy violations will cause the pod to be rejected.
-**audit** | Policy violations will trigger the addition of an audit annotation to the event recorded in the [audit log](/docs/tasks/debug-application-cluster/audit/), but are otherwise allowed.
+**audit** | Policy violations will trigger the addition of an audit annotation to the event recorded in the [audit log](/docs/tasks/debug/debug-cluster/audit/), but are otherwise allowed.
 **warn** | Policy violations will trigger a user-facing warning, but are otherwise allowed.
 {{< /table >}}

@@ -658,8 +658,7 @@ added. Capabilities listed in `RequiredDropCapabilities` must not be included in

 **DefaultAddCapabilities** - The capabilities which are added to containers by
 default, in addition to the runtime defaults. See the
-[Docker documentation](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities)
-for the default list of capabilities when using the Docker runtime.
+the documentation for your container runtime for information on working with Linux capabilities.

 ### SELinux

@@ -49,7 +49,7 @@ metadata:
  name: standard
 provisioner: kubernetes.io/aws-ebs
 parameters:
-  type: gp3
+  type: gp2
 reclaimPolicy: Retain
 allowVolumeExpansion: true
 mountOptions:
@@ -271,9 +271,9 @@ parameters:
  fsType: ext4
 ```

-* `type`: `io1`, `gp2`, `gp3`, `sc1`, `st1`. See
+* `type`: `io1`, `gp2`, `sc1`, `st1`. See
  [AWS docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)
-  for details. Default: `gp3`.
+  for details. Default: `gp2`.
 * `zone` (Deprecated): AWS zone. If neither `zone` nor `zones` is specified, volumes are
  generally round-robin-ed across all active zones where Kubernetes cluster
  has a node. `zone` and `zones` parameters must not be used at the same time.
@@ -70,5 +70,5 @@ you can view processes in other containers.

 ## {{% heading "whatsnext" %}}

-* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container).
+* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container).

@@ -332,5 +332,6 @@ Kubernetes, consult the documentation for the version you are using.
 ## {{% heading "whatsnext" %}}

 * Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container)
-* Learn how to [debug init containers](/docs/tasks/debug-application-cluster/debug-init-containers/)
+* Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/)
+

@@ -46,7 +46,7 @@ To run the link checker:
 2. Run the following command:

  ```
-  make docker-internal-linkcheck
+  make container-internal-linkcheck
  ```

 ## Understanding the output
@@ -1396,7 +1396,7 @@ monitoring mechanisms help cluster admins to answer questions like:
 Sometimes it's useful to know which mutating webhook mutated the object in a API request, and what change did the
 webhook apply.

-In v1.16+, kube-apiserver performs [auditing](/docs/tasks/debug-application-cluster/audit/) on each mutating webhook
+In v1.16+, kube-apiserver performs [auditing](/docs/tasks/debug/debug-cluster/audit/) on each mutating webhook
 invocation. Each invocation generates an auditing annotation
 capturing if a request object is mutated by the invocation, and optionally generates an annotation capturing the applied
 patch from the webhook admission response. The annotations are set in the audit event for given request on given stage of
@@ -607,7 +607,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
  Docker Engine; no longer available. See
  [Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) for
  an alternative.
- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug-application-cluster/audit/#advanced-audit)
+- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug/debug-cluster/audit/#advanced-audit)
 - `AffinityInAnnotations`: Enable setting
  [Pod affinity or anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity).
 - `AllowExtTrafficLocalEndpoints`: Enable a service to route external requests to node local endpoints.
@@ -0,0 +1,18 @@
+---
+title: Dockershim
+id: dockershim
+date: 2022-04-15
+full_link: /dockershim
+short_description: >
+   A component of Kubernetes v1.23 and earlier, which allows Kubernetes system components to communicate with Docker Engine.
+
+aka:
+tags:
+- fundamental
+---
+The dockershim is a component of Kubernetes version 1.23 and earlier. It allows the kubelet
+to communicate with {{< glossary_tooltip text="Docker Engine" term_id="docker" >}}.
+
+<!--more-->
+
+Starting with version 1.24, dockershim has been removed from Kubernetes. For more information, see [Dockershim FAQ](/dockershim).
@@ -21,5 +21,5 @@ or the continued existence of events with that reason.

 Events should be treated as informative, best-effort, supplemental data.

-In Kubernetes, [auditing](/docs/tasks/debug-application-cluster/audit/) generates a different kind of
+In Kubernetes, [auditing](/docs/tasks/debug/debug-cluster/audit/) generates a different kind of
 Event record (API group `audit.k8s.io`).
@@ -187,7 +187,7 @@ kubectl exec -ti nginx-app-5jyvm -- /bin/sh
 # exit
 ```

-For more information, see [Get a Shell to a Running Container](/docs/tasks/debug-application-cluster/get-shell-running-container/).
+For more information, see [Get a Shell to a Running Container](/docs/tasks/debug/debug-application/get-shell-running-container/).

 ## docker logs

@@ -11,7 +11,7 @@ namespace. These annotations apply to `Event` object from API group

 {{< note >}}
 The following annotations are not used within the Kubernetes API. When you
-[enable auditing](/docs/tasks/debug-application-cluster/audit/) in your cluster,
+[enable auditing](/docs/tasks/debug/debug-cluster/audit/) in your cluster,
 audit event data is written using `Event` from API group `audit.k8s.io`.
 The annotations apply to audit events. Audit events are different from objects in the
 [Event API](/docs/reference/kubernetes-api/cluster-resources/event-v1/) (API group
@@ -64,15 +64,15 @@ Example: `authorization.k8s.io/decision: "forbid"`

 This annotation indicates whether or not a request was authorized in Kubernetes audit logs.

-See [Auditing](/docs/tasks/debug-application-cluster/audit/) for more information.
+See [Auditing](/docs/tasks/debug/debug-cluster/audit/) for more information.

 ## authorization.k8s.io/reason

-Example: `authorization.k8s.io/decision: "Human-readable reason for the decision"`
+Example: `authorization.k8s.io/reason: "Human-readable reason for the decision"`

 This annotation gives reason for the [decision](#authorization-k8s-io-decision) in Kubernetes audit logs.

-See [Auditing](/docs/tasks/debug-application-cluster/audit/) for more information.
+See [Auditing](/docs/tasks/debug/debug-cluster/audit/) for more information.

 ## missing-san.invalid-cert.kubernetes.io/$hostname

@@ -30,9 +30,6 @@ resources itself.
 [Helm](https://helm.sh/) is a tool for managing packages of pre-configured
 Kubernetes resources. These packages are known as _Helm charts_.

-Helm is a third party managed tool for managing packages of pre-configured
-Kubernetes resources, aka Kubernetes charts.
-
 Use Helm to:

 * Find and use popular software packaged as Kubernetes charts
@@ -286,7 +286,7 @@ behavior get removed.
 Starting in Kubernetes v1.19, making an API request to a deprecated REST API endpoint:

 1. Returns a `Warning` header (as defined in [RFC7234, Section 5.5](https://tools.ietf.org/html/rfc7234#section-5.5)) in the API response.
-2. Adds a `"k8s.io/deprecated":"true"` annotation to the [audit event](/docs/tasks/debug-application-cluster/audit/) recorded for the request.
+2. Adds a `"k8s.io/deprecated":"true"` annotation to the [audit event](/docs/tasks/debug/debug-cluster/audit/) recorded for the request.
 3. Sets an `apiserver_requested_deprecated_apis` gauge metric to `1` in the `kube-apiserver`
   process. The metric has labels for `group`, `version`, `resource`, `subresource` that can be joined
   to the `apiserver_request_total` metric, and a `removed_release` label that indicates the
@@ -197,7 +197,7 @@ are some virtualization platforms that can be scripted to spin up new nodes
 based on demand.
 - *Set up node health checks*: For important workloads, you want to make sure
 that the nodes and pods running on those nodes are healthy. Using the
-[Node Problem Detector](/docs/tasks/debug-application-cluster/monitor-node-health/)
+[Node Problem Detector](/docs/tasks/debug/debug-cluster/monitor-node-health/)
 daemon, you can ensure your nodes are healthy.

 ## Production user management
@@ -166,18 +166,14 @@ Use the following commands to install Containerd on your system:
 1. Install containerd:

   Visit
-   [Getting started with containerd](https://containerd.io/docs/getting-started/#starting-containerd)
+   [Getting started with containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md)
   and follow the instructions there, up to the point where you have a valid
-   configuration file (on Linux: `/etc/containerd/config.toml`).
+   configuration file, config.toml.
+   On Linux, you can find this file under the path `/etc/containerd/config.toml`.
+   On Windows, you can find this file under the path `C:\Program Files\containerd\config.toml`.

-   If you are running Windows, you might want to exclude containerd from Windows Defender Scans
-   ```powershell
-   # If excluding containerd from Windows Defender scans, consider how else
-   # you will make sure that the executable is genuine.
-   Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe"
-   ```
-
-For containerd, the CRI socket is `/run/containerd/containerd.sock` by default.
+On Linux the default CRI socket for containerd is `/run/containerd/containerd.sock`.
+On Windows the default CRI endpoint is `npipe://./pipe/containerd-containerd`.

 #### Configuring the `systemd` cgroup driver {#containerd-systemd}

@@ -93,7 +93,7 @@ This may be caused by a number of problems. The most common are:
 configure it properly see [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).
 - control plane containers are crashlooping or hanging. You can check this by running `docker ps`
 and investigating each container by running `docker logs`. For other container runtime see
-[Debugging Kubernetes nodes with crictl](/docs/tasks/debug-application-cluster/crictl/).
+[Debugging Kubernetes nodes with crictl](/docs/tasks/debug/debug-cluster/crictl/).

 ## kubeadm blocks when removing managed containers

@@ -151,7 +151,7 @@ services](/docs/concepts/services-networking/service/#type-nodeport) or use `Hos

 ## Pods are not accessible via their Service IP

- Many network add-ons do not yet enable [hairpin mode](/docs/tasks/debug-application-cluster/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip)
+- Many network add-ons do not yet enable [hairpin mode](/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip)
  which allows pods to access themselves via their Service IP. This is an issue related to
  [CNI](https://github.com/containernetworking/cni/issues/476). Please contact the network
  add-on provider to get the latest status of their support for hairpin mode.
@@ -547,7 +547,7 @@ None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-reso
 ### Node problem detector

 The node problem detector (see
-[Monitor Node Health](/docs/tasks/debug-application-cluster/monitor-node-health/))
+[Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/))
 is not compatible with Windows.

 ### Pause container
@@ -625,7 +625,7 @@ The Kubernetes [version-skew policy](/docs/setup/release/version-skew-policy/) a
 ## Getting help and troubleshooting {#troubleshooting}

 Your main source of help for troubleshooting your Kubernetes cluster should start
-with the [Troubleshooting](/docs/tasks/debug-application-cluster/troubleshooting/)
+with the [Troubleshooting](/docs/tasks/debug/debug-cluster/)
 page.

 Some additional, Windows-specific troubleshooting help is included
@@ -176,7 +176,7 @@ The service name is `kube-dns` for both CoreDNS and kube-dns deployments.

 If you have created the Service or in the case it should be created by default
 but it does not appear, see
-[debugging Services](/docs/tasks/debug-application-cluster/debug-service/) for
+[debugging Services](/docs/tasks/debug/debug-application/debug-service/) for
 more information.

 ### Are DNS endpoints exposed?
@@ -193,7 +193,7 @@ kube-dns   10.180.3.17:53,10.180.3.17:53    1h
 ```

 If you do not see the endpoints, see the endpoints section in the
-[debugging Services](/docs/tasks/debug-application-cluster/debug-service/) documentation.
+[debugging Services](/docs/tasks/debug/debug-application/debug-service/) documentation.

 For additional Kubernetes DNS examples, see the
 [cluster-dns examples](https://github.com/kubernetes/examples/tree/master/staging/cluster-dns)
@@ -11,7 +11,7 @@ weight: 20
 This page explains how to upgrade a Kubernetes cluster created with kubeadm from version
 {{< skew currentVersionAddMinor -1 >}}.x to version {{< skew currentVersion >}}.x, and from version
 {{< skew currentVersion >}}.x to {{< skew currentVersion >}}.y (where `y > x`). Skipping MINOR versions
-when upgrading is unsupported.
+when upgrading is unsupported. For more details, please visit [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/).

 To see information about upgrading clusters created using older versions of kubeadm,
 please refer to following pages instead:
@@ -13,7 +13,7 @@ Kubernetes' support for direct integration with Docker Engine is deprecated, and
 ## Telemetry and security agents

 Within a Kubernetes cluster there are a few different ways to run telemetry or security agents.
-Some agents have a direct  dependency on Docker Engine when they as DaemonSets or
+Some agents have a direct dependency on Docker Engine when they run as DaemonSets or
 directly on nodes.

 ### Why do some telemetry agents communicate with Docker Engine?
@@ -210,7 +210,7 @@ access to a subset of the keyspace is strongly recommended.

 ### Enable audit logging

-The [audit logger](/docs/tasks/debug-application-cluster/audit/) is a beta feature that records actions taken by the
+The [audit logger](/docs/tasks/debug/debug-cluster/audit/) is a beta feature that records actions taken by the
 API for later analysis in the event of a compromise. It is recommended to enable audit logging
 and archive the audit file on a secure server.

@@ -59,7 +59,7 @@ and restarts it.
 When the container starts, it executes this command:

 ```shell
-/bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600"
+/bin/sh -c "touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600"
 ```

 For the first 30 seconds of the container's life, there is a `/tmp/healthy` file.
@@ -85,7 +85,7 @@ The output shows that nginx is serving the web page that was written by the init
 [communicating between Containers running in the same Pod](/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/).
 * Learn more about [Init Containers](/docs/concepts/workloads/pods/init-containers/).
 * Learn more about [Volumes](/docs/concepts/storage/volumes/).
-* Learn more about [Debugging Init Containers](/docs/tasks/debug-application-cluster/debug-init-containers/)
+* Learn more about [Debugging Init Containers](/docs/tasks/debug/debug-application/debug-init-containers/)



@@ -399,7 +399,7 @@ in the `securityContext` section of your Pod or Container manifest. The
 [SeccompProfile](/docs/reference/generated/kubernetes-api/{{< param "version"
 >}}/#seccompprofile-v1-core) object consisting of `type` and `localhostProfile`.
 Valid options for `type` include `RuntimeDefault`, `Unconfined`, and
-`Localhost`. `localhostProfile` must only be set set if `type: Localhost`. It
+`Localhost`. `localhostProfile` must only be set if `type: Localhost`. It
 indicates the path of the pre-configured profile on the node, relative to the
 kubelet's configured Seccomp profile location (configured with the `--root-dir`
 flag).
@@ -7,7 +7,7 @@ content_type: task

 {{% thirdparty-content %}}

-Kubernetes applications usually consist of multiple, separate services, each running in its own container. Developing and debugging these services on a remote Kubernetes cluster can be cumbersome, requiring you to [get a shell on a running container](/docs/tasks/debug-application-cluster/get-shell-running-container/) in order to run debugging tools.
+Kubernetes applications usually consist of multiple, separate services, each running in its own container. Developing and debugging these services on a remote Kubernetes cluster can be cumbersome, requiring you to [get a shell on a running container](/docs/tasks/debug/debug-application/get-shell-running-container/) in order to run debugging tools.
 
 `telepresence` is a tool to ease the process of developing and debugging services locally while proxying the service to a remote Kubernetes cluster. Using `telepresence` allows you to use custom tools, such as a debugger and IDE, for a local service and provides the service full access to ConfigMap, secrets, and the services running on the remote cluster.
 
@@ -28,7 +28,7 @@ The Metrics API, and the metrics pipeline that it enables, only offers the minim
 CPU and memory metrics to enable automatic scaling using HPA and / or VPA.
 If you would like to provide a more complete set of metrics, you can complement
 the simpler Metrics API by deploying a second
-[metrics pipeline](/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipeline)
+[metrics pipeline](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/#full-metrics-pipeline)
 that uses the _Custom Metrics API_.
 {{< /note >}}

@@ -65,8 +65,8 @@ Full metrics pipeline projects that are not part of the CNCF are outside the sco
 Learn about additional debugging tools, including:

 * [Logging](/docs/concepts/cluster-administration/logging/)
-* [Monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
-* [Getting into containers via `exec`](/docs/tasks/debug-application-cluster/applications/get-shell-running-container/)
+* [Monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)
+* [Getting into containers via `exec`](/docs/tasks/debug/debug-application/get-shell-running-container/)
 * [Connecting to containers via proxies](/docs/tasks/extend-kubernetes/http-proxy-access-api/)
 * [Connecting to containers via port forwarding](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
-* [Inspect Kubernetes node with crictl](/docs/tasks/debug-application-cluster/monitoring/crictl/)
+* [Inspect Kubernetes node with crictl](/docs/tasks/debug/debug-cluster/crictl/)
@@ -114,5 +114,5 @@ args: ["-c", "while true; do echo hello; sleep 10;done"]


 * Learn more about [configuring pods and containers](/docs/tasks/).
-* Learn more about [running commands in a container](/docs/tasks/debug-application-cluster/get-shell-running-container/).
+* Learn more about [running commands in a container](/docs/tasks/debug/debug-application/get-shell-running-container/).
 * See [Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core).
@@ -90,6 +90,6 @@ Always perform force deletion of StatefulSet Pods carefully and with complete kn
 ## {{% heading "whatsnext" %}}


-Learn more about [debugging a StatefulSet](/docs/tasks/debug-application-cluster/debug-stateful-set/).
+Learn more about [debugging a StatefulSet](/docs/tasks/debug/debug-application/debug-statefulset/).


@@ -90,7 +90,7 @@ The common use for HorizontalPodAutoscaler is to configure it to fetch metrics f
 (`metrics.k8s.io`, `custom.metrics.k8s.io`, or `external.metrics.k8s.io`).  The `metrics.k8s.io` API is
 usually provided by an add-on named Metrics Server, which needs to be launched separately.
 For more information about resource metrics, see
-[Metrics Server](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server).
+[Metrics Server](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server).

 [Support for metrics APIs](#support-for-metrics-apis) explains the stability guarantees and support status for these
 different APIs.
@@ -532,7 +532,7 @@ kubectl delete pvc data-mysql-4
 ## {{% heading "whatsnext" %}}

 * Learn more about [scaling a StatefulSet](/docs/tasks/run-application/scale-stateful-set/).
-* Learn more about [debugging a StatefulSet](/docs/tasks/debug-application-cluster/debug-stateful-set/).
+* Learn more about [debugging a StatefulSet](/docs/tasks/debug/debug-application/debug-statefulset/).
 * Learn more about [deleting a StatefulSet](/docs/tasks/run-application/delete-stateful-set/).
 * Learn more about [force deleting StatefulSet Pods](/docs/tasks/run-application/force-delete-stateful-set-pod/).
 * Look in the [Helm Charts repository](https://artifacthub.io/)
@@ -143,7 +143,7 @@ Configurations with a single API server will experience unavailability while the

   1. Validate the logs from control plane components, along with the kubelet and the
      kube-proxy are not throwing any tls errors, see
-      [looking at the logs](/docs/tasks/debug-application-cluster/debug-cluster/#looking-at-logs).
+      [looking at the logs](/docs/tasks/debug/debug-cluster/#looking-at-logs).

   1. Validate logs from any aggregated api servers and pods using in-cluster config.

@@ -93,7 +93,7 @@ cassandra   ClusterIP   None         <none>        9042/TCP   45s
 ```

 If you don't see a Service named `cassandra`, that means creation failed. Read
-[Debug Services](/docs/tasks/debug-application-cluster/debug-service/)
+[Debug Services](/docs/tasks/debug/debug-application/debug-service/)
 for help troubleshooting common issues.

 ## Using a StatefulSet to create a Cassandra ring
@@ -236,10 +236,10 @@ Do not leave your WordPress installation on this page. If another user finds it,
 ## {{% heading "whatsnext" %}}


-* Learn more about [Introspection and Debugging](/docs/tasks/debug-application-cluster/debug-application-introspection/)
+* Learn more about [Introspection and Debugging](/docs/tasks/debug/debug-application)
 * Learn more about [Jobs](/docs/concepts/workloads/controllers/job/)
 * Learn more about [Port Forwarding](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
-* Learn how to [Get a Shell to a Container](/docs/tasks/debug-application-cluster/get-shell-running-container/)
+* Learn how to [Get a Shell to a Container](/docs/tasks/debug/debug-application/get-shell-running-container/)



@@ -11,7 +11,7 @@ spec:
    args:
    - /bin/sh
    - -c
-    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
+    - touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
@@ -3,6 +3,7 @@ kind: PodSecurityPolicy
 metadata:
  name: restricted
  annotations:
+    # docker/default identifies a profile for seccomp, but it is not particularly tied to the Docker runtime
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'