Merge remote-tracking branch 'upstream/main' into dev-1.24

2022-05-02 10:29:49 -07:00
parent 7f13ad092c a8b640bd6f
commit 5ead53b3e8
141 changed files with 5085 additions and 666 deletions
@@ -4,6 +4,9 @@

 このリポジトリには、[KubernetesのWebサイトとドキュメント](https://kubernetes.io/)をビルドするために必要な全アセットが格納されています。貢献に興味を持っていただきありがとうございます！

+- [ドキュメントに貢献する](#contributing-to-the-docs)
+- [翻訳された`README.md`一覧](#localization-readmemds)
+
 # リポジトリの使い方

 Hugo(Extended version)を使用してWebサイトをローカルで実行することも、コンテナランタイムで実行することもできます。コンテナランタイムを使用することを強くお勧めします。これにより、本番Webサイトとのデプロイメントの一貫性が得られます。
@@ -56,6 +59,43 @@ make serve

 これで、Hugoのサーバーが1313番ポートを使って開始します。お使いのブラウザにて http://localhost:1313 にアクセスしてください。リポジトリ内のソースファイルに変更を加えると、HugoがWebサイトの内容を更新してブラウザに反映します。

+## API reference pagesをビルドする
+
+`content/en/docs/reference/kubernetes-api`に配置されているAPIリファレンスページは<https://github.com/kubernetes-sigs/reference-docs/tree/master/gen-resourcesdocs>を使ってSwagger仕様書からビルドされています。
+
+新しいKubernetesリリースのためにリファレンスページをアップデートするには、次の手順を実行します:
+
+1. `api-ref-generator`サブモジュールをプルする:
+
+   ```bash
+   git submodule update --init --recursive --depth 1
+   ```
+
+2. Swagger仕様書を更新する:
+
+   ```bash
+   curl 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json' > api-ref-assets/api/swagger.json
+   ```
+
+3. 新しいリリースの変更を反映するため、`api-ref-assets/config/`で`toc.yaml`と`fields.yaml`を適用する。
+
+4. 次に、ページをビルドする:
+
+   ```bash
+   make api-reference
+   ```
+
+   コンテナイメージからサイトを作成・サーブする事でローカルで結果をテストすることができます:
+
+   ```bash
+   make container-image
+   make container-serve
+   ```
+
+   APIリファレンスを見るために、ブラウザで<http://localhost:1313/docs/reference/kubernetes-api/>を開いてください。
+
+5. 新しいコントラクトのすべての変更が設定ファイル`toc.yaml`と`fields.yaml`に反映されたら、新しく生成されたAPIリファレンスページとともにPull Requestを作成します。
+
 ## トラブルシューティング

 ### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version
@@ -107,7 +147,7 @@ sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist
 - [Slack](https://kubernetes.slack.com/messages/kubernetes-docs-ja)
 - [メーリングリスト](https://groups.google.com/forum/#!forum/kubernetes-sig-docs)

-## ドキュメントに貢献する
+## ドキュメントに貢献する {#contributing-to-the-docs}

 GitHubの画面右上にある**Fork**ボタンをクリックすると、お使いのGitHubアカウントに紐付いた本リポジトリのコピーが作成され、このコピーのことを*フォーク*と呼びます。フォークリポジトリの中ではお好きなように変更を加えていただいて構いません。加えた変更をこのリポジトリに追加したい任意のタイミングにて、フォークリポジトリからPull Reqeustを作成してください。

@@ -124,7 +164,15 @@ Kubernetesのドキュメントへの貢献に関する詳細については以
 * [ドキュメントのスタイルガイド](https://kubernetes.io/docs/contribute/style/style-guide/)
 * [Kubernetesドキュメントの翻訳方法](https://kubernetes.io/docs/contribute/localization/)

-## 翻訳された`README.md`一覧
+### New Contributor Ambassadors
+
+コントリビュートする時に何か助けが必要なら、[New Contributor Ambassadors](https://kubernetes.io/docs/contribute/advanced/#serve-as-a-new-contributor-ambassador)に聞いてみると良いでしょう。彼らはSIG Docsのapproverで、最初の数回のPull Requestを通して新しいコントリビューターを指導し助けることを責務としています。New Contributors Ambassadorsにコンタクトするには、[Kubernetes Slack](https://slack.k8s.io)が最適な場所です。現在のSIG DocsのNew Contributor Ambassadorは次の通りです:
+
+| 名前                       | Slack                      | GitHub                     |                   
+| -------------------------- | -------------------------- | -------------------------- |
+| Arsh Sharma                | @arsh                      | @RinkiyaKeDad              |
+
+## 翻訳された`README.md`一覧 {#localization-readmemds}

 | Language  | Language |
 |---|---|
@@ -19,7 +19,7 @@ Security:
 - [Node authorizer](/docs/reference/access-authn-authz/node/) and admission control plugin are new additions that restrict kubelet’s access to secrets, pods and other objects based on its node.
 - [Encryption for Secrets](/docs/tasks/administer-cluster/encrypt-data/), and other resources in etcd, is now available as alpha.&nbsp;
 - [Kubelet TLS bootstrapping](/docs/admin/kubelet-tls-bootstrapping/) now supports client and server certificate rotation.
- [Audit logs](/docs/tasks/debug-application-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit.
+- [Audit logs](/docs/tasks/debug/debug-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit.

 Stateful workloads:  

@@ -117,7 +117,7 @@ To achieve the best possible isolation, each function call would have to happen
 By using Landlock, we could isolate function calls from each other within the same container, making a temporary file created by one function call inaccessible to the next function call, for example. Integration between Landlock and technologies like Kubernetes-based serverless frameworks would be a ripe area for further exploration.  

 ## Auditing kubectl-exec with eBPF
-In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug-application-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster.   
+In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug/debug-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster.   

 While these events log Kubernetes events, they don't currently provide the level of visibility that some may require. For example, while we can see that someone has used `kubectl exec` to enter a container, we are not able to see what commands were executed in that session. With eBPF one can attach a BPF program that would record any commands executed in the `kubectl exec` session and pass those commands to a user-space program that logs those events. We could then play that session back and know the exact sequence of events that took place.
 ## Learn more about eBPF
@@ -66,7 +66,7 @@ There are plenty of [good examples](https://docs.bitnami.com/kubernetes/how-to/c

 Incorrect or excessively permissive RBAC policies are a security threat in case of a compromised pod. Maintaining least privilege, and continuously reviewing and improving RBAC rules, should be considered part of the "technical debt hygiene" that teams build into their development lifecycle.

-[Audit Logging](/docs/tasks/debug-application-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation&#39;s security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started.
+[Audit Logging](/docs/tasks/debug/debug-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation&#39;s security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started.

 For read requests such as get, list, and watch, only the request object is saved in the audit logs; the response object is not. For requests involving sensitive data such as Secret and ConfigMap, only the metadata is exported. For all other requests, both request and response objects are saved in audit logs.

@@ -174,7 +174,7 @@ Cluster-distributed stateful services (e.g., Cassandra) can benefit from splitti

 ## Other considerations

-[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location.
+[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location.

 Your production deployment should utilize an automated installation, configuration and update tool (e.g., [Ansible](https://github.com/kubernetes-incubator/kubespray), [BOSH](https://github.com/cloudfoundry-incubator/kubo-deployment), [Chef](https://github.com/chef-cookbooks/kubernetes), [Juju](/docs/getting-started-guides/ubuntu/installation/), [kubeadm](/docs/reference/setup-tools/kubeadm/), [Puppet](https://forge.puppet.com/puppetlabs/kubernetes), etc.). A manual process will have repeatability issues, be labor intensive, error prone, and difficult to scale. [Certified distributions](https://www.cncf.io/certification/software-conformance/#logos) are likely to include a facility for retaining configuration settings across updates, but if you implement your own install and config toolchain, then retention, backup and recovery of the configuration artifacts is essential. Consider keeping your deployment components and settings under a version control system such as Git.

@@ -177,7 +177,7 @@ group_right() apiserver_request_total

 Metrics are a fast way to check whether deprecated APIs are being used, and at what rate,
 but they don't include enough information to identify particular clients or API objects.
-Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug-application-cluster/audit/)
+Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug/debug-cluster/audit/)
 for requests to deprecated APIs include an audit annotation of `"k8s.io/deprecated":"true"`.
 Administrators can use those audit events to identify specific clients or objects that need to be updated.

@@ -20,7 +20,7 @@ The paper attempts to _not_ focus on any specific [cloud native project](https:/
 When using Kubernetes as a workload orchestrator, some of the security controls this version of the whitepaper recommends are:
 * [Pod Security Policies](/docs/concepts/security/pod-security-policy/): Implement a single source of truth for “least privilege” workloads across the entire cluster
 * [Resource requests and limits](/docs/concepts/configuration/manage-resources-containers/#requests-and-limits): Apply requests (soft constraint) and limits (hard constraint) for shared resources such as memory and CPU
-* [Audit log analysis](/docs/tasks/debug-application-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events
+* [Audit log analysis](/docs/tasks/debug/debug-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events
 * [Control plane authentication and certificate root of trust](/docs/concepts/architecture/control-plane-node-communication/): Enable mutual TLS authentication with a trusted CA for communication within the cluster
 * [Secrets management](/docs/concepts/configuration/secret/): Integrate with a built-in or external secrets store

@@ -155,7 +155,7 @@ runtime where possible.

 Another thing to look out for is anything expecting to run for system maintenance
 or nested inside a container when building images will no longer work. For the
-former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
+former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
 latter you can use newer container build options like [img], [buildah],
 [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker.

@@ -32,7 +32,7 @@ The `kubectl alpha debug` features graduates to beta in 1.20, becoming `kubectl

 Note that as a new built-in command, `kubectl debug` takes priority over any kubectl plugin named “debug”. You must rename the affected plugin.

-Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/).
+Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/).

 ### Beta: API Priority and Fairness

@@ -317,7 +317,7 @@ RequestResponse's including metadata and request / response bodies. While helpfu

 Each organization needs to evaluate their
 own threat model and build an audit policy that complements or helps troubleshooting incident response. Think
-about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug-application-cluster/audit/#audit-policy).
+about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug/debug-cluster/audit/#audit-policy).
 It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point. 

 Audit logging configurations can also be tested with
@@ -0,0 +1,155 @@
+---
+layout: blog
+title: 'Increasing the security bar in Ingress-NGINX v1.2.0'
+date: 2022-04-28
+slug: ingress-nginx-1-2-0
+---
+
+**Authors:** Ricardo Katz (VMware), James Strong (Chainguard)
+
+The [Ingress](/docs/concepts/services-networking/ingress/) may be one of the most targeted components
+of Kubernetes. An Ingress typically defines an HTTP reverse proxy, exposed to the Internet, containing
+multiple websites, and with some privileged access to Kubernetes API (such as to read Secrets relating to
+TLS certificates and their private keys).
+
+While it is a risky component in your architecture, it is still the most popular way to properly expose your services.
+
+Ingress-NGINX has been part of security assessments that figured out we have a big problem: we don't
+do all proper sanitization before turning the configuration into an `nginx.conf` file, which may lead to information
+disclosure risks.
+
+While we understand this risk and the real need to fix this, it's not an easy process to do, so we took another approach to reduce (but not remove!) this risk in the current (v1.2.0) release.
+
+## Meet Ingress NGINX v1.2.0 and the chrooted NGINX process
+
+One of the main challenges is that Ingress-NGINX runs the web proxy server (NGINX) alongside the Ingress
+controller (the component that has access to Kubernetes API that and that creates the `nginx.conf` file).
+
+So, NGINX does have the same access to the filesystem of the controller (and Kubernetes service account token, and other configurations from the container). While splitting those components is our end goal, the project needed a fast response; that lead us to the idea of using `chroot()`.
+
+Let's take a look into what an Ingress-NGINX container looked like before this change:
+
+![Ingress NGINX pre chroot](ingress-pre-chroot.png)
+
+As we can see, the same container (not the Pod, the container!) that provides HTTP Proxy is the one that watches Ingress objects and writes the Container Volume
+
+Now, meet the new architecture:
+
+![Ingress NGINX post chroot](ingress-post-chroot.png)
+
+What does all of this mean? A basic summary is: that we are isolating the NGINX service as a container inside the
+controller container.
+
+While this is not strictly true, to understand what was done here, it's good to understand how
+Linux containers (and underlying mechanisms such as kernel namespaces) work.
+You can read about cgroups in the Kubernetes glossary: [`cgroup`](https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-cgroup) and learn more about cgroups interact with namespaces in the NGINX project article
+[What Are Namespaces and cgroups, and How Do They Work?](https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/).
+(As you read that, bear in mind that Linux kernel namespaces are a different thing from
+[Kubernetes namespaces](/docs/concepts/overview/working-with-objects/namespaces/)).
+
+## Skip the talk, what do I need to use this new approach?
+
+While this increases the security, we made this feature an opt-in in this release so you can have
+time to make the right adjustments in your environment(s). This new feature is only available from
+release v1.2.0 of the Ingress-NGINX controller.
+
+There are two required changes in your deployments to use this feature:
+* Append the suffix "-chroot" to the container image name. For example: `gcr.io/k8s-staging-ingress-nginx/controller-chroot:v1.2.0`
+* In your Pod template for the Ingress controller, find where you add the capability `NET_BIND_SERVICE` and add the capability `SYS_CHROOT`. After you edit the manifest, you'll see a snippet like:
+
+```yaml
+capabilities:
+  drop:
+  - ALL
+  add:
+  - NET_BIND_SERVICE
+  - SYS_CHROOT
+```
+
+If you deploy the controller using the official Helm chart then change the following setting in
+`values.yaml`:
+
+```yaml
+controller:
+  image:
+    chroot: true
+```
+
+Ingress controllers are normally set up cluster-wide (the IngressClass API is cluster scoped). If you manage the
+Ingress-NGINX controller but you're not the overall cluster operator, then check with your cluster admin about
+whether you can use the `SYS_CHROOT` capability, **before** you enable it in your deployment.
+
+## OK, but how does this increase the security of my Ingress controller?
+
+Take the following configuration snippet and imagine, for some reason it was added to your `nginx.conf`:
+```
+location /randomthing/ {
+      alias /;
+      autoindex on;
+}
+```
+
+If you deploy this configuration, someone can call `http://website.example/randomthing` and get some listing (and access) to the whole filesystem of the Ingress controller.
+
+Now, can you spot the difference between chrooted and non chrooted Nginx on the listings below?
+
+| Without extra `chroot()`             | With extra `chroot()` |
+|----------------------------|--------|
+| `bin`                      | `bin`  |
+| `dev`                      | `dev`  |
+| `etc`                      | `etc`  |
+| `home`                     |        |
+| `lib`                      | `lib`  |
+| `media`                    |        |
+| `mnt`                      |        |
+| `opt`                      | `opt`  |
+| `proc`                     | `proc` |
+| `root`                     |        |
+| `run`                      | `run`  |
+| `sbin`                     |        |
+| `srv`                      |        |
+| `sys`                      |        |
+| `tmp`                      | `tmp`  |
+| `usr`                      | `usr`  |
+| `var`                      | `var`  |
+| `dbg`                      |        |
+| `nginx-ingress-controller` |        |
+| `wait-shutdown`            |        |
+
+The one in left side is not chrooted. So NGINX has full access to the filesystem. The one in right side is chrooted, so a new filesystem with only the required files to make NGINX work is created.
+
+## What about other security improvements in this release?
+
+We know that the new `chroot()` mechanism helps address some portion of the risk, but still, someone
+can try to inject commands to read, for example, the `nginx.conf` file and extract sensitive information.
+
+So, another change in this release (this is opt-out!) is the _deep inspector_.
+We know that some directives or regular expressions may be dangerous to NGINX, so the deep inspector
+checks all fields from an Ingress object (during its reconciliation, and also with a
+[validating admission webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook))
+to verify if any fields contains these dangerous directives.
+
+The ingress controller already does this for annotations, and our goal is to move this existing validation to happen inside
+deep inspection as part of a future release.
+
+You can take a look into the existing rules in [https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go](https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go).
+
+Due to the nature of inspecting and matching all strings within relevant Ingress objects, this new feature may consume a bit more CPU. You can disable it by running the ingress controller with the command line argument `--deep-inspect=false`.
+
+## What's next?
+
+This is not our final goal. Our final goal is to split the control plane and the data plane processes.
+In fact, doing so will help us also achieve a [Gateway](https://gateway-api.sigs.k8s.io/) API implementation,
+as we may have a different controller as soon as it "knows" what to provide to the data plane
+(we need some help here!!)
+
+Some other projects in Kubernetes already take this approach
+(like [KPNG](https://github.com/kubernetes-sigs/kpng), the proposed replacement for `kube-proxy`),
+and we plan to align with them and get the same experience for Ingress-NGINX.
+
+## Further reading
+
+If you want to take a look into how chrooting was done in Ingress NGINX, take a look 
+into [https://github.com/kubernetes/ingress-nginx/pull/8337](https://github.com/kubernetes/ingress-nginx/pull/8337)
+The release v1.2.0 containing all the changes can be found at 
+[https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0](https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.2.0)
@@ -0,0 +1,319 @@
+---
+layout: blog
+title: "Frontiers, fsGroups and frogs: the Kubernetes 1.23 release interview"
+date: 2022-04-29
+---
+
+**Author**: Craig Box (Google)
+
+One of the highlights of hosting the weekly [Kubernetes Podcast from Google](https://kubernetespodcast.com/) is talking to the release managers for each new Kubernetes version. The release team is constantly refreshing. Many working their way from small documentation fixes, step up to shadow roles, and then eventually lead a release.
+
+As we prepare for the 1.24 release next week, [in accordance with long-standing tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the story of 1.23. The release was led by [Rey Lejano](https://twitter.com/reylejano), a Field Engineer at SUSE. [I spoke to Rey](https://kubernetespodcast.com/episode/167-kubernetes-1.23/) in December, as he was awaiting the birth of his first child.
+
+Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/), so you hear all our stories from the Cloud Native community, including the story of 1.24 next week.
+
+*This transcript has been lightly edited and condensed for clarity.*
+
+---
+
+**CRAIG BOX: I'd like to start with what is, of course, on top of everyone's mind at the moment. Let's talk African clawed frogs!**
+
+REY LEJANO: [CHUCKLES] Oh, you mean [Xenopus lavis](https://en.wikipedia.org/wiki/African_clawed_frog), the scientific name for the African clawed frog?
+
+**CRAIG BOX: Of course.**
+
+REY LEJANO: Not many people know, but my background and my degree is actually in microbiology, from the University of California Davis. I did some research for about four years in biochemistry, in a biochemistry lab, and I [do have a research paper published](https://www.sciencedirect.com/science/article/pii/). It's actually on glycoproteins, particularly something called "cortical granule lectin". We used frogs, because they generate lots and lots of eggs, from which we can extract the protein. That protein prevents polyspermy. When the sperm goes into the egg, the egg releases a glycoprotein, cortical granule lectin, to the membrane, and prevents any other sperm from going inside the egg.
+
+**CRAIG BOX: Were you able to take anything from the testing that we did on frogs and generalize that to higher-order mammals, perhaps?**
+
+REY LEJANO: Yes. Since mammals also have cortical granule lectin, we were able to analyze both the convergence and the evolutionary pattern, not just from multiple species of frogs, but also into mammals as well.
+
+**CRAIG BOX: Now, there's a couple of different threads to unravel here. When you were young, what led you into the fields of biology, and perhaps more the technical side of it?**
+
+REY LEJANO: I think it was mostly from family, since I do have a family history in the medical field that goes back generations. So I kind of felt like that was the natural path going into college.
+
+**CRAIG BOX: Now, of course, you're working in a more abstract tech field. What led you out of microbiology?**
+
+REY LEJANO: [CHUCKLES] Well, I've always been interested in tech. Taught myself a little programming when I was younger, before high school, did some web dev stuff. Just kind of got burnt out being in a lab. I was literally in the basement. I had a great opportunity to join a consultancy that specialized in [ITIL](https://www.axelos.com/certifications/itil-service-management/what-is-itil). I actually started off with application performance management, went into monitoring, went into operation management and also ITIL, which is aligning your IT asset management and service managements with business services. Did that for a good number of years, actually.
+
+**CRAIG BOX: It's very interesting, as people describe the things that they went through and perhaps the technologies that they worked on, you can pretty much pinpoint how old they might be. There's a lot of people who come into tech these days that have never heard of ITIL. They have no idea what it is. It's basically just SRE with more process.**
+
+REY LEJANO: Yes, absolutely. It's not very cloud native. [CHUCKLES]
+
+**CRAIG BOX: Not at all.**
+
+REY LEJANO: You don't really hear about it in the cloud native landscape. Definitely, you can tell someone's been in the field for a little bit, if they specialize or have worked with ITIL before.
+
+**CRAIG BOX: You mentioned that you wanted to get out of the basement. That is quite often where people put the programmers. Did they just give you a bit of light in the new basement?**
+
+REY LEJANO: [LAUGHS] They did give us much better lighting. Able to get some vitamin D sometimes, as well.
+
+**CRAIG BOX: To wrap up the discussion about your previous career — over the course of the last year, with all of the things that have happened in the world, I could imagine that microbiology skills may be more in demand than perhaps they were when you studied them?**
+
+REY LEJANO: Oh, absolutely. I could definitely see a big increase of numbers of people going into the field. Also, reading what's going on with the world currently kind of brings back all the education I've learned in the past, as well.
+
+**CRAIG BOX: Do you keep in touch with people you went through school with?**
+
+REY LEJANO: Just some close friends, but not in the microbiology field.
+
+**CRAIG BOX: One thing that I think will probably happen as a result of the pandemic is a renewed interest in some of these STEM fields. It will be interesting to see what impact that has on society at large.**
+
+REY LEJANO: Yeah. I think that'll be great.
+
+**CRAIG BOX: You mentioned working at a consultancy doing IT management, application performance monitoring, and so on. When did Kubernetes come into your professional life?**
+
+REY LEJANO: One of my good friends at the company I worked at, left in mid-2015. He went on to a company that was pretty heavily into Docker. He taught me a little bit. I did my first "docker run" around 2015, maybe 2016. Then, one of the applications we were using for the ITIL framework was containerized around 2018 or so, also in Kubernetes. At that time, it was pretty buggy. That was my initial introduction to Kubernetes and containerised applications.
+
+Then I left that company, and I actually joined my friend over at [RX-M](https://rx-m.com/), which is a cloud native consultancy and training firm. They specialize in Docker and Kubernetes. I was able to get my feet wet. I got my CKD, got my CKA as well. And they were really, really great at encouraging us to learn more about Kubernetes and also to be involved in the community.
+
+**CRAIG BOX: You will have seen, then, the life cycle of people adopting Kubernetes and containerization at large, through your own initial journey and then through helping customers. How would you characterize how that journey has changed from the early days to perhaps today?**
+
+REY LEJANO: I think the early days, there was a lot of questions of, why do I have to containerize? Why can't I just stay with virtual machines?
+
+**CRAIG BOX: It's a line item on your CV.**
+
+REY LEJANO: [CHUCKLES] It is. And nowadays, I think people know the value of using containers, of orchestrating containers with Kubernetes. I don't want to say "jumping on the bandwagon", but it's become the de-facto standard to orchestrate containers.
+
+**CRAIG BOX: It's not something that a consultancy needs to go out and pitch to customers that they should be doing. They're just taking it as, that will happen, and starting a bit further down the path, perhaps.**
+
+REY LEJANO: Absolutely.
+
+**CRAIG BOX: Working at a consultancy like that, how much time do you get to work on improving process, perhaps for multiple customers, and then looking at how you can upstream that work, versus paid work that you do for just an individual customer at a time?**
+
+REY LEJANO: Back then, it would vary. They helped me introduce myself, and I learned a lot about the cloud native landscape and Kubernetes itself. They helped educate me as to how the cloud native landscape, and the tools around it, can be used together. My boss at that company, Randy, he actually encouraged us to start contributing upstream, and encouraged me to join the release team. He just said, this is a great opportunity. Definitely helped me with starting with the contributions early on.
+
+**CRAIG BOX: Was the release team the way that you got involved with upstream Kubernetes contribution?**
+
+REY LEJANO: Actually, no. My first contribution was  with SIG Docs. I met Taylor Dolezal — he was the release team lead for 1.19, but he is involved with SIG Docs as well. I met him at KubeCon 2019, I sat at his table during a luncheon. I remember Paris Pittman was hosting this luncheon at the Marriott. Taylor says he was involved with SIG Docs. He encouraged me to join. I started joining into meetings, started doing a few drive-by PRs. That's what we call them — drive-by — little typo fixes. Then did a little bit more, started to send better or higher quality pull requests, and also reviewing PRs.
+
+**CRAIG BOX: When did you first formally take your release team role?**
+
+REY LEJANO: That was in [1.18](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.18/release_team.md), in December. My boss at the time encouraged me to apply. I did, was lucky enough to get accepted for the release notes shadow. Then from there, stayed in with release notes for a few cycles, then went into Docs, naturally then led Docs, then went to Enhancements, and now I'm the release lead for 1.23.
+
+**CRAIG BOX: I don't know that a lot of people think about what goes into a good release note. What would you say does?**
+
+REY LEJANO: [CHUCKLES] You have to tell the end user what has changed or what effect that they might see in the release notes. It doesn't have to be highly technical. It could just be a few lines, and just saying what has changed, what they have to do if they have to do anything as well.
+
+**CRAIG BOX: As you moved through the process of shadowing, how did you learn from the people who were leading those roles?**
+
+REY LEJANO: I said this a few times when I was the release lead for this cycle. You get out of the release team as much as you put in, or it directly aligns to how much you put in. I learned a lot. I went into the release team having that mindset of learning from the role leads, learning from the other shadows, as well. That's actually a saying that my first role lead told me. I still carry it to heart, and that was back in 1.18. That was Eddie, in the very first meeting we had, and I still carry it to heart.
+
+**CRAIG BOX: You, of course, were [the release lead for 1.23](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.23). First of all, congratulations on the release.**
+
+REY LEJANO: Thank you very much.
+
+**CRAIG BOX: The theme for this release is [The Next Frontier](https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/). Tell me the story of how we came to the theme and then the logo.**
+
+REY LEJANO: The Next Frontier represents a few things. It not only represents the next enhancements in this release, but Kubernetes itself also has a history of Star Trek references. The original codename for Kubernetes was Project Seven, a reference to Seven of Nine, originally from Star Trek Voyager. Also the seven spokes in the helm in the logo of Kubernetes as well. And, of course, Borg, the predecessor to Kubernetes.
+
+The Next Frontier continues that Star Trek reference. It's a fusion of two titles in the Star Trek universe. One is [Star Trek V, the Final Frontier](https://en.wikipedia.org/wiki/Star_Trek_V:_The_Final_Frontier), and the Star Trek: The Next Generation.
+
+**CRAIG BOX: Do you have any opinion on the fact that Star Trek V was an odd-numbered movie, and they are [canonically referred to as being lesser than the even-numbered ones](https://screenrant.com/star-trek-movies-odd-number-curse-explained/)?**
+
+REY LEJANO: I can't say, because I am such a sci-fi nerd that I love all of them even though they're bad. Even the post-Next Generation movies, after the series, I still liked all of them, even though I know some weren't that great.
+
+**CRAIG BOX: Am I right in remembering that Star Trek V was the one directed by William Shatner?**
+
+REY LEJANO: Yes, that is correct.
+
+**CRAIG BOX: I think that says it all.**
+
+REY LEJANO: [CHUCKLES] Yes.
+
+**CRAIG BOX: Now, I understand that the theme comes from a part of the [SIG Release charter](https://github.com/kubernetes/community/blob/master/sig-release/charter.md)?**
+
+REY LEJANO: Yes. There's a line in the SIG Release charter, "ensure there is a consistent group of community members in place to support the release process across time." With the release team, we have new shadows that join every single release cycle. With this, we're growing with this community. We're growing the release team members. We're growing SIG Release. We're growing the Kubernetes community itself. For a lot of people, this is their first time contributing to open source, so that's why I say it's their new open source frontier.
+
+**CRAIG BOX: And the logo is obviously very Star Trek-inspired. It sort of surprised me that it took that long for someone to go this route.**
+
+REY LEJANO: I was very surprised as well. I had to relearn Adobe Illustrator to create the logo.
+
+**CRAIG BOX: This your own work, is it?**
+
+REY LEJANO: This is my own work.
+
+**CRAIG BOX: It's very nice.**
+
+REY LEJANO: Thank you very much. Funny, the galaxy actually took me the longest time versus the ship. Took me a few days to get that correct. I'm always fine-tuning it, so there might be a final change when this is actually released.
+
+**CRAIG BOX: No frontier is ever truly final.**
+
+REY LEJANO: True, very true.
+
+**CRAIG BOX: Moving now from the theme of the release to the substance, perhaps, what is new in 1.23?**
+
+REY LEJANO: We have 47 enhancements. I'm going to run through most of the stable ones, if not all of them, some of the key Beta ones, and a few of the Alpha enhancements for 1.23.
+
+One of the key enhancements is [dual-stack IPv4/IPv6](https://github.com/kubernetes/enhancements/issues/563), which went GA in 1.23.
+
+Some background info: dual-stack was introduced as Alpha in 1.15. You probably saw a keynote at KubeCon 2019. Back then, the way dual-stack worked was that you needed two services — you needed a service per IP family. You would need a service for IPv4 and a service for IPv6. It was refactored in 1.20. In 1.21, it was in Beta; clusters were enabled to be dual-stack by default.
+
+And then in 1.23 we did remove the IPv6 dual-stack feature flag. It's not mandatory to use dual-stack. It's actually not "default" still. The pods, the services still default to single-stack. There are some requirements to be able to use dual-stack. The nodes have to be routable on IPv4 and IPv6 network interfaces. You need a CNI plugin that supports dual-stack. The pods themselves have to be configured to be dual-stack. And the services need the ipFamilyPolicy field to specify prefer dual-stack, or require dual-stack.
+
+**CRAIG BOX: This sounds like there's an implication in this that v4 is still required. Do you see a world where we can actually move to v6-only clusters?**
+
+REY LEJANO: I think we'll be talking about IPv4 and IPv6 for many, many years to come. I remember a long time ago, they kept saying "it's going to be all IPv6", and that was decades ago.
+
+**CRAIG BOX: I think I may have mentioned on the show before, but there was [a meeting in London that Vint Cerf attended](https://www.youtube.com/watch?v=AEaJtZVimqs), and he gave a public presentation at the time to say, now is the time of v6. And that was 10 years ago at least. It's still not the time of v6, and my desktop still doesn't have Linux on it. One day.**
+
+REY LEJANO: [LAUGHS] In my opinion, that's one of the big key features that went stable for 1.23.
+
+One of the other highlights of 1.23 is [pod security admission going to Beta](/blog/2021/12/09/pod-security-admission-beta/). I know this feature is going to Beta, but I highlight this because as some people might know, PodSecurityPolicy, which was deprecated in 1.21, is targeted to be removed in 1.25. Pod security admission replaces pod security policy. It's an admission controller. It evaluates the pods against a predefined set of pod security standards to either admit or deny the pod for running.
+
+There's three levels of pod security standards. Privileged, that's totally open. Baseline, known privileges escalations are minimized. Or Restricted, which is hardened. And you could set pod security standards either to run in three modes, which is enforce: reject any pods that are in violation; to audit: pods are allowed to be created, but the violations are recorded; or warn: it will send a warning message to the user, and the pod is allowed.
+
+**CRAIG BOX: You mentioned there that PodSecurityPolicy is due to be deprecated in two releases' time. Are we lining up these features so that pod security admission will be GA at that time?**
+
+REY LEJANO: Yes. Absolutely. I'll talk about that for another feature in a little bit as well. There's also another feature that went to GA. It was an API that went to GA, and therefore the Beta API is now deprecated. I'll talk about that a little bit.
+
+**CRAIG BOX: All right. Let's talk about what's next on the list.**
+
+REY LEJANO: Let's move on to more stable enhancements. One is the [TTL controller](https://github.com/kubernetes/enhancements/issues/592). This cleans up jobs and pods after the jobs are finished. There is a TTL timer that starts when the job or pod is finished. This TTL controller watches all the jobs, and ttlSecondsAfterFinished needs to be set. The controller will see if the ttlSecondsAfterFinished, combined with the last transition time, if it's greater than now. If it is, then it will delete the job and the pods of that job.
+
+**CRAIG BOX: Loosely, it could be called a garbage collector?**
+
+REY LEJANO: Yes. Garbage collector for pods and jobs, or jobs and pods.
+
+**CRAIG BOX: If Kubernetes is truly becoming a programming language, it of course has to have a garbage collector implemented.**
+
+REY LEJANO: Yeah. There's another one, too, coming in Alpha. [CHUCKLES]
+
+**CRAIG BOX: Tell me about that.**
+
+REY LEJANO: That one is coming in in Alpha. It's actually one of my favorite features, because there's only a few that I'm going to highlight today. [PVCs for StafeulSet will be cleaned up](https://github.com/kubernetes/enhancements/issues/1847). It will auto-delete PVCs created by StatefulSets, when you delete that StatefulSet.
+
+**CRAIG BOX: What's next on our tour of stable features?**
+
+REY LEJANO: Next one is, [skip volume ownership change goes to stable](https://github.com/kubernetes/enhancements/issues/695). This is from SIG Storage. There are times when you're running a stateful application, like many databases, they're sensitive to permission bits changing underneath. Currently, when a volume is bind mounted inside the container, the permissions of that volume will change recursively. It might take a really long time.
+
+Now, there's a field, the fsGroupChangePolicy, which allows you, as a user, to tell Kubernetes how you want the permission and ownership change for that volume to happen. You can set it to always, to always change permissions, or just on mismatch, to only do it when the permission ownership changes at the top level is different from what is expected.
+
+**CRAIG BOX: It does feel like a lot of these enhancements came from a very particular use case where someone said, "hey, this didn't work for me and I've plumbed in a feature that works with exactly the thing I need to have".**
+
+REY LEJANO: Absolutely. People create issues for these, then create Kubernetes enhancement proposals, and then get targeted for releases.
+
+**CRAIG BOX: Another GA feature in this release — ephemeral volumes.**
+
+REY LEJANO: We've always been able to use empty dir for ephemeral volumes, but now we could actually have [ephemeral inline volumes](https://github.com/kubernetes/enhancements/issues/1698), meaning that you could take your standard CSI driver and be able to use ephemeral volumes with it.
+
+**CRAIG BOX: And, a long time coming, [CronJobs](https://github.com/kubernetes/enhancements/issues/19).**
+
+REY LEJANO: CronJobs is a funny one, because it was stable before 1.23. For 1.23, it was still tracked,but it was just cleaning up some of the old controller. With CronJobs, there's a v2 controller. What was cleaned up in 1.23 is just the old v1 controller.
+
+**CRAIG BOX: Were there any other duplications or major cleanups of note in this release?**
+
+REY LEJANO: Yeah. There were a few you might see in the major themes. One's a little tricky, around FlexVolumes. This is one of the efforts from SIG Storage. They have an effort to migrate in-tree plugins to CSI drivers. This is a little tricky, because FlexVolumes were actually deprecated in November 2020. We're [formally announcing it in 1.23](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors).
+
+**CRAIG BOX: FlexVolumes, in my mind, predate CSI as a concept. So it's about time to get rid of them.**
+
+REY LEJANO: Yes, it is. There's another deprecation, just some [klog specific flags](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog), but other than that, there are no other big deprecations in 1.23.
+
+**CRAIG BOX: The buzzword of the last KubeCon, and in some ways the theme of the last 12 months, has been secure software supply chain. What work is Kubernetes doing to improve in this area?**
+
+REY LEJANO: For 1.23, Kubernetes is now SLSA compliant at Level 1, which means that provenance attestation files that describe the staging and release phases of the release process are satisfactory for the SLSA framework.
+
+**CRAIG BOX: What needs to happen to step up to further levels?**
+
+REY LEJANO: Level 1 means a few things — that the build is scripted; that the provenance is available, meaning that the artifacts are verified and they're handed over from one phase to the next; and describes how the artifact is produced. Level 2 means that the source is version-controlled, which it is, provenance is authenticated, provenance is service-generated, and there is a build service. There are four levels of SLSA compliance.
+
+**CRAIG BOX: It does seem like the levels were largely influenced by what it takes to build a big, secure project like this. It doesn't seem like it will take a lot of extra work to move up to verifiable provenance, for example. There's probably just a few lines of script required to meet many of those requirements.**
+
+REY LEJANO: Absolutely. I feel like we're almost there; we'll see what will come out of 1.24. And I do want to give a big shout-out to SIG Release and Release Engineering, primarily to Adolfo García Veytia, who is aka Puerco on GitHub and on Slack. He's been driving this forward.
+
+**CRAIG BOX: You've mentioned some APIs that are being graduated in time to replace their deprecated version. Tell me about the new HPA API.**
+
+REY LEJANO: The [horizontal pod autoscaler v2 API](https://github.com/kubernetes/enhancements/issues/2702), is now stable, which means that the v2beta2 API is deprecated. Just for everyone's knowledge, the v1 API is not being deprecated. The difference is that v2 adds support for multiple and custom metrics to be used for HPA.
+
+**CRAIG BOX: There's also now a facility to validate my CRDs with an expression language.**
+
+REY LEJANO: Yeah. You can use the [Common Expression Language, or CEL](https://github.com/google/cel-spec), to validate your CRDs, so you no longer need to use webhooks. This also makes the CRDs more self-contained and declarative, because the rules are now kept within the CRD object definition.
+
+**CRAIG BOX: What new features, perhaps coming in Alpha or Beta, have taken your interest?**
+
+REY LEJANO: Aside from pod security policies, I really love [ephemeral containers](https://github.com/kubernetes/enhancements/issues/277) supporting kubectl debug. It launches an ephemeral container and a running pod, shares those pod namespaces, and you can do all your troubleshooting with just running kubectl debug.
+
+**CRAIG BOX: There's also been some interesting changes in the way that events are handled with kubectl.**
+
+REY LEJANO: Yeah. kubectl events has always had some issues, like how things weren't sorted. [kubectl events improved](https://github.com/kubernetes/enhancements/issues/1440) that so now you can do `--watch`, and it will also sort with the `--watch` option as well. That is something new. You can actually combine fields and custom columns. And also, you can list events in the timeline with doing the last N number of minutes. And you can also sort events using other criteria as well.
+
+**CRAIG BOX: You are a field engineer at SUSE. Are there any things that are coming in that your individual customers that you deal with are looking out for?**
+
+REY LEJANO: More of what I look out for to help the customers.
+
+**CRAIG BOX: Right.**
+
+REY LEJANO: I really love kubectl events. Really love the PVCs being cleaned up with StatefulSets. Most of it's for selfish reasons that it will improve troubleshooting efforts. [CHUCKLES]
+
+**CRAIG BOX: I have always hoped that a release team lead would say to me, "yes, I have selfish reasons. And I finally got something I wanted in."**
+
+REY LEJANO: [LAUGHS]
+
+**CRAIG BOX: Perhaps I should run to be release team lead, just so I can finally get init containers fixed once and for all.**
+
+REY LEJANO: Oh, init containers, I've been looking for that for a while. I've actually created animated GIFs on how init containers will be run with that Kubernetes enhancement proposal, but it's halted currently.
+
+**CRAIG BOX: One day.**
+
+REY LEJANO: One day. Maybe I shouldn't stay halted.
+
+**CRAIG BOX: You mentioned there are obviously the things you look out for. Are there any things that are coming down the line, perhaps Alpha features or maybe even just proposals you've seen lately, that you're personally really looking forward to seeing which way they go?**
+
+REY LEJANO: Yeah. Oone is a very interesting one, it affects the whole community, so it's not just for personal reasons. As you may have known, Dockershim is deprecated. And we did release a blog that it will be removed in 1.24.
+
+**CRAIG BOX: Scared a bunch of people.**
+
+REY LEJANO: Scared a bunch of people. From a survey, we saw that a lot of people are still using Docker and Dockershim. One of the enhancements for 1.23 is, [kubelet CRI goes to Beta](https://github.com/kubernetes/enhancements/issues/2040). This promotes the CRI API, which is required. This had to be in Beta for Dockershim to be removed in 1.24.
+
+**CRAIG BOX: Now, in the last release team lead interview, [we spoke with Savitha Raghunathan](https://kubernetespodcast.com/episode/157-kubernetes-1.22/), and she talked about what she would advise you as her successor. It was to look out for the mental health of the team members. How were you able to take that advice on board?**
+
+REY LEJANO: That was great advice from Savitha. A few things I've made note of with each release team meeting. After each release team meeting, I stop the recording, because we do record all the meetings and post them on YouTube. And I open up the floor to anyone who wants to say anything that's not recorded, that's not going to be on the agenda. Also, I tell people not to work on weekends. I broke this rule once, but other than that, I told people it could wait. Just be mindful of your mental health.
+
+**CRAIG BOX: It's just been announced that [James Laverack from Jetstack](https://twitter.com/JamesLaverack/status/1466834312993644551) will be the release team lead for 1.24. James and I shared an interesting Mexican dinner at the last KubeCon in San Diego.**
+
+REY LEJANO: Oh, nice. I didn't know you knew James.
+
+**CRAIG BOX: The British tech scene. We're a very small world. What will your advice to James be?**
+
+REY LEJANO: What I would tell James for 1.24 is use teachable moments in the release team meetings. When you're a shadow for the first time, it's very daunting. It's very difficult, because you don't know the repos. You don't know the release process. Everyone around you seems like they know the release process, and very familiar with what the release process is. But as a first-time shadow, you don't know all the vernacular for the community. I just advise to use teachable moments. Take a few minutes in the release team meetings to make it a little easier for new shadows to ramp up and to be familiar with the release process.
+
+**CRAIG BOX: Has there been major evolution in the process in the time that you've been involved? Or do you think that it's effectively doing what it needs to do?**
+
+REY LEJANO: It's always evolving. I remember my first time in release notes, 1.18, we said that our goal was to automate and program our way out so that we don't have a release notes team anymore. That's changed [CHUCKLES] quite a bit. Although there's been significant advancements in the release notes process by Adolfo and also James, they've created a subcommand in krel to generate release notes.
+
+But nowadays, all their release notes are richer. Still not there at the automation process yet. Every release cycle, there is something a little bit different. For this release cycle, we had a production readiness review deadline. It was a soft deadline. A production readiness review is a review by several people in the community. It's actually been required since 1.21, and it ensures that the enhancements are observable, scalable, supportable, and it's safe to operate in production, and could also be disabled or rolled back. In 1.23, we had a deadline to have the production readiness review completed by a specific date.
+
+**CRAIG BOX: How have you found the change of schedule to three releases per year rather than four?**
+
+REY LEJANO: Moving to three releases a year from four, in my opinion, has been an improvement, because we support the last three releases, and now we can actually support the last releases in a calendar year instead of having 9 months out of 12 months of the year.
+
+**CRAIG BOX: The next event on the calendar is a [Kubernetes contributor celebration](https://www.kubernetes.dev/events/kcc2021/) starting next Monday. What can we expect from that event?**
+
+REY LEJANO: This is our second time running this virtual event. It's a virtual celebration to recognize the whole community and all of our accomplishments of the year, and also contributors. There's a number of events during this week of celebration. It starts the week of December 13.
+
+There's events like the Kubernetes Contributor Awards, where SIGs honor and recognize the hard work of the community and contributors. There's also a DevOps party game as well. There is a cloud native bake-off. I do highly suggest people to go to [kubernetes.dev/celebration](https://www.kubernetes.dev/events/past-events/2021/kcc2021/) to learn more.
+
+**CRAIG BOX: How exactly does one judge a virtual bake-off?**
+
+REY LEJANO: That I don't know. [CHUCKLES]
+
+**CRAIG BOX: I tasted my scones. I think they're the best. I rate them 10 out of 10.**
+
+REY LEJANO: Yeah. That is very difficult to do virtually. I would have to say, probably what the dish is, how closely it is tied with Kubernetes or open source or to CNCF. There's a few judges. I know Josh Berkus and Rin Oliver are a few of the judges running the bake-off.
+
+**CRAIG BOX: Yes. We spoke with Josh about his love of the kitchen, and so he seems like a perfect fit for that role.**
+
+REY LEJANO: He is.
+
+**CRAIG BOX: Finally, your wife and yourself are expecting your first child in January. Have you had a production readiness review for that?**
+
+REY LEJANO: I think we failed that review. [CHUCKLES]
+
+**CRAIG BOX: There's still time.**
+
+REY LEJANO: We are working on refactoring. We're going to refactor a little bit in December, and `--apply` again.
+
+---
+
+_[Rey Lejano](https://twitter.com/reylejano) is a field engineer at SUSE, by way of Rancher Labs, and was the release team lead for Kubernetes 1.23. He is now also a co-chair for SIG Docs. His son Liam is now 3 and a half months old._
+
+_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._
@@ -0,0 +1,79 @@
+---
+layout: blog
+title: "Storage Capacity Tracking reaches GA in Kubernetes 1.24"
+date: 2022-05-06
+slug: storage-capacity-ga
+---
+
+ **Authors:** Patrick Ohly (Intel)
+
+The v1.24 release of Kubernetes brings [storage capacity](/docs/concepts/storage/storage-capacity/)
+tracking as a generally available feature.
+
+## Problems we have solved
+
+As explained in more detail in the [previous blog post about this
+feature](/blog/2021/04/14/local-storage-features-go-beta/), storage capacity
+tracking allows a CSI driver to publish information about remaining
+capacity. The kube-scheduler then uses that information to pick suitable nodes
+for a Pod when that Pod has volumes that still need to be provisioned.
+
+Without this information, a Pod may get stuck without ever being scheduled onto
+a suitable node because kube-scheduler has to choose blindly and always ends up
+picking a node for which the volume cannot be provisioned because the
+underlying storage system managed by the CSI driver does not have sufficient
+capacity left.
+
+Because CSI drivers publish storage capacity information that gets used at a
+later time when it might not be up-to-date anymore, it can still happen that a
+node is picked that doesn't work out after all. Volume provisioning recovers
+from that by informing the scheduler that it needs to try again with a
+different node.
+
+[Load
+tests](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/docs/storage-capacity-tracking.md)
+that were done again for promotion to GA confirmed that all storage in a
+cluster can be consumed by Pods with storage capacity tracking whereas Pods got
+stuck without it.
+
+## Problems we have *not* solved
+
+Recovery from a failed volume provisioning attempt has one known limitation: if a Pod
+uses two volumes and only one of them could be provisioned, then all future
+scheduling decisions are limited by the already provisioned volume. If that
+volume is local to a node and the other volume cannot be provisioned there, the
+Pod is stuck. This problem pre-dates storage capacity tracking and while the
+additional information makes it less likely to occur, it cannot be avoided in
+all cases, except of course by only using one volume per Pod.
+
+An idea for solving this was proposed in a [KEP
+draft](https://github.com/kubernetes/enhancements/pull/1703): volumes that were
+provisioned and haven't been used yet cannot have any valuable data and
+therefore could be freed and provisioned again elsewhere. SIG Storage is
+looking for interested developers who want to continue working on this.
+
+Also not solved is support in Cluster Autoscaler for Pods with volumes. For CSI
+drivers with storage capacity tracking, a prototype was developed and discussed
+in [a PR](https://github.com/kubernetes/autoscaler/pull/3887). It was meant to
+work with arbitrary CSI drivers, but that flexibility made it hard to configure
+and slowed down scale up operations: because autoscaler was unable to simulate
+volume provisioning, it only scaled the cluster by one node at a time, which
+was seen as insufficient.
+
+Therefore that PR was not merged and a different approach with tighter coupling
+between autoscaler and CSI driver will be needed. For this a better
+understanding is needed about which local storage CSI drivers are used in
+combination with cluster autoscaling. Should this lead to a new KEP, then users
+will have to try out an implementation in practice before it can move to beta
+or GA. So please reach out to SIG Storage if you have an interest in this
+topic.
+
+## Acknowledgements
+
+Thanks a lot to the members of the community who have contributed to this
+feature or given feedback including members of [SIG
+Scheduling](https://github.com/kubernetes/community/tree/master/sig-scheduling),
+[SIG
+Autoscaling](https://github.com/kubernetes/community/tree/master/sig-autoscaling),
+and of course [SIG
+Storage](https://github.com/kubernetes/community/tree/master/sig-storage)!
@@ -0,0 +1,162 @@
+---
+layout: blog
+title: "Kubernetes 1.24: Volume Populators Graduate to Beta"
+date: 2022-05-16
+slug: volume-populators-beta
+---
+
+**Author:**
+Ben Swartzlander (NetApp)
+
+The volume populators feature is now two releases old and entering beta! The `AnyVolumeDataSouce` feature
+gate defaults to enabled in Kubernetes v1.24, which means that users can specify any custom resource
+as the data source of a PVC.
+
+An [earlier blog article](/blog/2021/08/30-volume-populators-redesigned/) detailed how the
+volume populators feature works. In short, a cluster administrator can install a CRD and
+associated populator controller in the cluster, and any user who can create instances of 
+the CR can create pre-populated volumes by taking advantage of the populator.
+
+Multiple populators can be installed side by side for different purposes. The SIG storage
+community is already seeing some implementations in public, and more prototypes should
+appear soon.
+
+Cluster administrations are **strongly encouraged** to install the
+volume-data-source-validator controller and associated `VolumePopulator` CRD before installing
+any populators so that users can get feedback about invalid PVC data sources.
+
+## New Features
+
+The [lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator) library
+on which populators are built now includes metrics to help operators monitor and detect
+problems. This library is now beta and latest release is v1.0.1.
+
+The [volume data source validator](https://github.com/kubernetes-csi/volume-data-source-validator)
+controller also has metrics support added, and is in beta. The `VolumePopulator` CRD is
+beta and the latest release is v1.0.1.
+
+## Trying it out
+
+To see how this works, you can install the sample "hello" populator and try it
+out.
+
+First install the volume-data-source-validator controller.
+
+```shell
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/client/config/crd/populator.storage.k8s.io_volumepopulators.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/rbac-data-source-validator.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/volume-data-source-validator/v1.0.1/deploy/kubernetes/setup-data-source-validator.yaml
+```
+
+Next install the example populator.
+
+```shell
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/v1.0.1/example/hello-populator/crd.yaml
+kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/lib-volume-populator/87a47467b86052819e9ad13d15036d65b9a32fbb/example/hello-populator/deploy.yaml
+```
+
+Your cluster now has a new CustomResourceDefinition that provides a test API named Hello.
+Create an instance of the `Hello` custom resource, with some text:
+
+```yaml
+apiVersion: hello.example.com/v1alpha1
+kind: Hello
+metadata:
+  name: example-hello
+spec:
+  fileName: example.txt
+  fileContents: Hello, world!
+```
+
+Create a PVC that refers to that CR as its data source.
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: example-pvc
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Mi
+  dataSourceRef:
+    apiGroup: hello.example.com
+    kind: Hello
+    name: example-hello
+  volumeMode: Filesystem
+```
+
+Next, run a Job that reads the file in the PVC.
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: example-job
+spec:
+  template:
+    spec:
+      containers:
+        - name: example-container
+          image: busybox:latest
+          command:
+            - cat
+            - /mnt/example.txt
+          volumeMounts:
+            - name: vol
+              mountPath: /mnt
+      restartPolicy: Never
+      volumes:
+        - name: vol
+          persistentVolumeClaim:
+            claimName: example-pvc
+```
+
+Wait for the job to complete (including all of its dependencies).
+
+```shell
+kubectl wait --for=condition=Complete job/example-job
+```
+
+And last examine the log from the job.
+
+```shell
+kubectl logs job/example-job
+```
+
+The output should be:
+
+```terminal
+Hello, world!
+```
+
+Note that the volume already contained a text file with the string contents from
+the CR. This is only the simplest example. Actual populators can set up the volume
+to contain arbitrary contents.
+
+## How to write your own volume populator
+
+Developers interested in writing new poplators are encouraged to use the
+[lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator) library
+and to only supply a small controller wrapper around the library, and a pod image
+capable of attaching to volumes and writing the appropriate data to the volume.
+
+Individual populators can be extremely generic such that they work with every type
+of PVC, or they can do vendor specific things to rapidly fill a volume with data
+if the volume was provisioned by a specific CSI driver from the same vendor, for
+example, by communicating directly with the storage for that volume.
+
+## How can I learn more?
+
+The enhancement proposal,
+[Volume Populators](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators), includes lots of detail about the history and technical implementation
+of this feature.
+
+[Volume populators and data sources](/docs/concepts/storage/persistent-volumes/#volume-populators-and-data-sources), within the documentation topic about persistent volumes,
+explains how to use this feature in your cluster.
+
+Please get involved by joining the Kubernetes storage SIG to help us enhance this
+feature. There are a lot of good ideas already and we'd be thrilled to have more!
+
@@ -59,7 +59,7 @@ Before choosing a guide, here are some considerations:

 * [Using Sysctls in a Kubernetes Cluster](/docs/tasks/administer-cluster/sysctl-cluster/) describes to an administrator how to use the `sysctl` command-line tool to set kernel parameters .

-* [Auditing](/docs/tasks/debug-application-cluster/audit/) describes how to interact with Kubernetes' audit logs.
+* [Auditing](/docs/tasks/debug/debug-cluster/audit/) describes how to interact with Kubernetes' audit logs.

 ### Securing the kubelet
  * [Control Plane-Node communication](/docs/concepts/architecture/control-plane-node-communication/)
@@ -47,10 +47,9 @@ or by enforcement (the system prevents the container from ever exceeding the lim
 runtimes can have different ways to implement the same restrictions.

 {{< note >}}
-If a container specifies its own memory limit, but does not specify a memory request, Kubernetes
-automatically assigns a memory request that matches the limit. Similarly, if a container specifies its own
-CPU limit, but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches
-the limit.
+If you specify a limit for a resource, but do not specify any request, and no admission-time
+mechanism has applied a default request for that resource, then Kubernetes copies the limit
+you specified and uses it as the requested value for the resource.
 {{< /note >}}

 ## Resource types
@@ -229,9 +228,9 @@ see the [Troubleshooting](#troubleshooting) section.
 The kubelet reports the resource usage of a Pod as part of the Pod
 [`status`](/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status).

-If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
+If optional [tools for monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)
 are available in your cluster, then Pod resource usage can be retrieved either
-from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-api)
+from the [Metrics API](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api)
 directly or from your monitoring tools.

 ## Local ephemeral storage
@@ -114,7 +114,7 @@ Containers started by Kubernetes automatically include this DNS server in their

 ### Container Resource Monitoring

-[Container Resource Monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) records generic time-series metrics
+[Container Resource Monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) records generic time-series metrics
 about containers in a central database, and provides a UI for browsing that data.

 ### Cluster-level Logging
@@ -134,7 +134,7 @@ for the corresponding API object, and then written to the object store (shown as
 Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster.
 The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.

-For more information, see [Auditing](/docs/tasks/debug-application-cluster/audit/).
+For more information, see [Auditing](/docs/tasks/debug/debug-cluster/audit/).

 ## API server ports and IPs

@@ -86,7 +86,7 @@ takes if a potential violation is detected:
 Mode | Description
 :---------|:------------
 **enforce** | Policy violations will cause the pod to be rejected.
-**audit** | Policy violations will trigger the addition of an audit annotation to the event recorded in the [audit log](/docs/tasks/debug-application-cluster/audit/), but are otherwise allowed.
+**audit** | Policy violations will trigger the addition of an audit annotation to the event recorded in the [audit log](/docs/tasks/debug/debug-cluster/audit/), but are otherwise allowed.
 **warn** | Policy violations will trigger a user-facing warning, but are otherwise allowed.
 {{< /table >}}

@@ -658,8 +658,7 @@ added. Capabilities listed in `RequiredDropCapabilities` must not be included in

 **DefaultAddCapabilities** - The capabilities which are added to containers by
 default, in addition to the runtime defaults. See the
-[Docker documentation](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities)
-for the default list of capabilities when using the Docker runtime.
+the documentation for your container runtime for information on working with Linux capabilities.

 ### SELinux

@@ -49,7 +49,7 @@ metadata:
  name: standard
 provisioner: kubernetes.io/aws-ebs
 parameters:
-  type: gp3
+  type: gp2
 reclaimPolicy: Retain
 allowVolumeExpansion: true
 mountOptions:
@@ -271,9 +271,9 @@ parameters:
  fsType: ext4
 ```

-* `type`: `io1`, `gp2`, `gp3`, `sc1`, `st1`. See
+* `type`: `io1`, `gp2`, `sc1`, `st1`. See
  [AWS docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)
-  for details. Default: `gp3`.
+  for details. Default: `gp2`.
 * `zone` (Deprecated): AWS zone. If neither `zone` nor `zones` is specified, volumes are
  generally round-robin-ed across all active zones where Kubernetes cluster
  has a node. `zone` and `zones` parameters must not be used at the same time.
@@ -70,5 +70,5 @@ you can view processes in other containers.

 ## {{% heading "whatsnext" %}}

-* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container).
+* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container).

@@ -332,5 +332,6 @@ Kubernetes, consult the documentation for the version you are using.
 ## {{% heading "whatsnext" %}}

 * Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container)
-* Learn how to [debug init containers](/docs/tasks/debug-application-cluster/debug-init-containers/)
+* Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/)
+

@@ -46,7 +46,7 @@ To run the link checker:
 2. Run the following command:

  ```
-  make docker-internal-linkcheck
+  make container-internal-linkcheck
  ```

 ## Understanding the output
@@ -1396,7 +1396,7 @@ monitoring mechanisms help cluster admins to answer questions like:
 Sometimes it's useful to know which mutating webhook mutated the object in a API request, and what change did the
 webhook apply.

-In v1.16+, kube-apiserver performs [auditing](/docs/tasks/debug-application-cluster/audit/) on each mutating webhook
+In v1.16+, kube-apiserver performs [auditing](/docs/tasks/debug/debug-cluster/audit/) on each mutating webhook
 invocation. Each invocation generates an auditing annotation
 capturing if a request object is mutated by the invocation, and optionally generates an annotation capturing the applied
 patch from the webhook admission response. The annotations are set in the audit event for given request on given stage of
@@ -607,7 +607,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
  Docker Engine; no longer available. See
  [Device Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) for
  an alternative.
- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug-application-cluster/audit/#advanced-audit)
+- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug/debug-cluster/audit/#advanced-audit)
 - `AffinityInAnnotations`: Enable setting
  [Pod affinity or anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity).
 - `AllowExtTrafficLocalEndpoints`: Enable a service to route external requests to node local endpoints.
@@ -0,0 +1,18 @@
+---
+title: Dockershim
+id: dockershim
+date: 2022-04-15
+full_link: /dockershim
+short_description: >
+   A component of Kubernetes v1.23 and earlier, which allows Kubernetes system components to communicate with Docker Engine.
+
+aka:
+tags:
+- fundamental
+---
+The dockershim is a component of Kubernetes version 1.23 and earlier. It allows the kubelet
+to communicate with {{< glossary_tooltip text="Docker Engine" term_id="docker" >}}.
+
+<!--more-->
+
+Starting with version 1.24, dockershim has been removed from Kubernetes. For more information, see [Dockershim FAQ](/dockershim).
@@ -21,5 +21,5 @@ or the continued existence of events with that reason.

 Events should be treated as informative, best-effort, supplemental data.

-In Kubernetes, [auditing](/docs/tasks/debug-application-cluster/audit/) generates a different kind of
+In Kubernetes, [auditing](/docs/tasks/debug/debug-cluster/audit/) generates a different kind of
 Event record (API group `audit.k8s.io`).
@@ -187,7 +187,7 @@ kubectl exec -ti nginx-app-5jyvm -- /bin/sh
 # exit
 ```

-For more information, see [Get a Shell to a Running Container](/docs/tasks/debug-application-cluster/get-shell-running-container/).
+For more information, see [Get a Shell to a Running Container](/docs/tasks/debug/debug-application/get-shell-running-container/).

 ## docker logs

@@ -11,7 +11,7 @@ namespace. These annotations apply to `Event` object from API group

 {{< note >}}
 The following annotations are not used within the Kubernetes API. When you
-[enable auditing](/docs/tasks/debug-application-cluster/audit/) in your cluster,
+[enable auditing](/docs/tasks/debug/debug-cluster/audit/) in your cluster,
 audit event data is written using `Event` from API group `audit.k8s.io`.
 The annotations apply to audit events. Audit events are different from objects in the
 [Event API](/docs/reference/kubernetes-api/cluster-resources/event-v1/) (API group
@@ -64,15 +64,15 @@ Example: `authorization.k8s.io/decision: "forbid"`

 This annotation indicates whether or not a request was authorized in Kubernetes audit logs.

-See [Auditing](/docs/tasks/debug-application-cluster/audit/) for more information.
+See [Auditing](/docs/tasks/debug/debug-cluster/audit/) for more information.

 ## authorization.k8s.io/reason

-Example: `authorization.k8s.io/decision: "Human-readable reason for the decision"`
+Example: `authorization.k8s.io/reason: "Human-readable reason for the decision"`

 This annotation gives reason for the [decision](#authorization-k8s-io-decision) in Kubernetes audit logs.

-See [Auditing](/docs/tasks/debug-application-cluster/audit/) for more information.
+See [Auditing](/docs/tasks/debug/debug-cluster/audit/) for more information.

 ## missing-san.invalid-cert.kubernetes.io/$hostname

@@ -30,9 +30,6 @@ resources itself.
 [Helm](https://helm.sh/) is a tool for managing packages of pre-configured
 Kubernetes resources. These packages are known as _Helm charts_.

-Helm is a third party managed tool for managing packages of pre-configured
-Kubernetes resources, aka Kubernetes charts.
-
 Use Helm to:

 * Find and use popular software packaged as Kubernetes charts
@@ -286,7 +286,7 @@ behavior get removed.
 Starting in Kubernetes v1.19, making an API request to a deprecated REST API endpoint:

 1. Returns a `Warning` header (as defined in [RFC7234, Section 5.5](https://tools.ietf.org/html/rfc7234#section-5.5)) in the API response.
-2. Adds a `"k8s.io/deprecated":"true"` annotation to the [audit event](/docs/tasks/debug-application-cluster/audit/) recorded for the request.
+2. Adds a `"k8s.io/deprecated":"true"` annotation to the [audit event](/docs/tasks/debug/debug-cluster/audit/) recorded for the request.
 3. Sets an `apiserver_requested_deprecated_apis` gauge metric to `1` in the `kube-apiserver`
   process. The metric has labels for `group`, `version`, `resource`, `subresource` that can be joined
   to the `apiserver_request_total` metric, and a `removed_release` label that indicates the
@@ -197,7 +197,7 @@ are some virtualization platforms that can be scripted to spin up new nodes
 based on demand.
 - *Set up node health checks*: For important workloads, you want to make sure
 that the nodes and pods running on those nodes are healthy. Using the
-[Node Problem Detector](/docs/tasks/debug-application-cluster/monitor-node-health/)
+[Node Problem Detector](/docs/tasks/debug/debug-cluster/monitor-node-health/)
 daemon, you can ensure your nodes are healthy.

 ## Production user management
@@ -166,18 +166,14 @@ Use the following commands to install Containerd on your system:
 1. Install containerd:

   Visit
-   [Getting started with containerd](https://containerd.io/docs/getting-started/#starting-containerd)
+   [Getting started with containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md)
   and follow the instructions there, up to the point where you have a valid
-   configuration file (on Linux: `/etc/containerd/config.toml`).
+   configuration file, config.toml.
+   On Linux, you can find this file under the path `/etc/containerd/config.toml`.
+   On Windows, you can find this file under the path `C:\Program Files\containerd\config.toml`.

-   If you are running Windows, you might want to exclude containerd from Windows Defender Scans
-   ```powershell
-   # If excluding containerd from Windows Defender scans, consider how else
-   # you will make sure that the executable is genuine.
-   Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe"
-   ```
-
-For containerd, the CRI socket is `/run/containerd/containerd.sock` by default.
+On Linux the default CRI socket for containerd is `/run/containerd/containerd.sock`.
+On Windows the default CRI endpoint is `npipe://./pipe/containerd-containerd`.

 #### Configuring the `systemd` cgroup driver {#containerd-systemd}

@@ -93,7 +93,7 @@ This may be caused by a number of problems. The most common are:
 configure it properly see [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).
 - control plane containers are crashlooping or hanging. You can check this by running `docker ps`
 and investigating each container by running `docker logs`. For other container runtime see
-[Debugging Kubernetes nodes with crictl](/docs/tasks/debug-application-cluster/crictl/).
+[Debugging Kubernetes nodes with crictl](/docs/tasks/debug/debug-cluster/crictl/).

 ## kubeadm blocks when removing managed containers

@@ -151,7 +151,7 @@ services](/docs/concepts/services-networking/service/#type-nodeport) or use `Hos

 ## Pods are not accessible via their Service IP

- Many network add-ons do not yet enable [hairpin mode](/docs/tasks/debug-application-cluster/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip)
+- Many network add-ons do not yet enable [hairpin mode](/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip)
  which allows pods to access themselves via their Service IP. This is an issue related to
  [CNI](https://github.com/containernetworking/cni/issues/476). Please contact the network
  add-on provider to get the latest status of their support for hairpin mode.
@@ -547,7 +547,7 @@ None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-reso
 ### Node problem detector

 The node problem detector (see
-[Monitor Node Health](/docs/tasks/debug-application-cluster/monitor-node-health/))
+[Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/))
 is not compatible with Windows.

 ### Pause container
@@ -625,7 +625,7 @@ The Kubernetes [version-skew policy](/docs/setup/release/version-skew-policy/) a
 ## Getting help and troubleshooting {#troubleshooting}

 Your main source of help for troubleshooting your Kubernetes cluster should start
-with the [Troubleshooting](/docs/tasks/debug-application-cluster/troubleshooting/)
+with the [Troubleshooting](/docs/tasks/debug/debug-cluster/)
 page.

 Some additional, Windows-specific troubleshooting help is included
@@ -176,7 +176,7 @@ The service name is `kube-dns` for both CoreDNS and kube-dns deployments.

 If you have created the Service or in the case it should be created by default
 but it does not appear, see
-[debugging Services](/docs/tasks/debug-application-cluster/debug-service/) for
+[debugging Services](/docs/tasks/debug/debug-application/debug-service/) for
 more information.

 ### Are DNS endpoints exposed?
@@ -193,7 +193,7 @@ kube-dns   10.180.3.17:53,10.180.3.17:53    1h
 ```

 If you do not see the endpoints, see the endpoints section in the
-[debugging Services](/docs/tasks/debug-application-cluster/debug-service/) documentation.
+[debugging Services](/docs/tasks/debug/debug-application/debug-service/) documentation.

 For additional Kubernetes DNS examples, see the
 [cluster-dns examples](https://github.com/kubernetes/examples/tree/master/staging/cluster-dns)
@@ -11,7 +11,7 @@ weight: 20
 This page explains how to upgrade a Kubernetes cluster created with kubeadm from version
 {{< skew currentVersionAddMinor -1 >}}.x to version {{< skew currentVersion >}}.x, and from version
 {{< skew currentVersion >}}.x to {{< skew currentVersion >}}.y (where `y > x`). Skipping MINOR versions
-when upgrading is unsupported.
+when upgrading is unsupported. For more details, please visit [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/).

 To see information about upgrading clusters created using older versions of kubeadm,
 please refer to following pages instead:
@@ -13,7 +13,7 @@ Kubernetes' support for direct integration with Docker Engine is deprecated, and
 ## Telemetry and security agents

 Within a Kubernetes cluster there are a few different ways to run telemetry or security agents.
-Some agents have a direct  dependency on Docker Engine when they as DaemonSets or
+Some agents have a direct dependency on Docker Engine when they run as DaemonSets or
 directly on nodes.

 ### Why do some telemetry agents communicate with Docker Engine?
@@ -210,7 +210,7 @@ access to a subset of the keyspace is strongly recommended.

 ### Enable audit logging

-The [audit logger](/docs/tasks/debug-application-cluster/audit/) is a beta feature that records actions taken by the
+The [audit logger](/docs/tasks/debug/debug-cluster/audit/) is a beta feature that records actions taken by the
 API for later analysis in the event of a compromise. It is recommended to enable audit logging
 and archive the audit file on a secure server.

@@ -59,7 +59,7 @@ and restarts it.
 When the container starts, it executes this command:

 ```shell
-/bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600"
+/bin/sh -c "touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600"
 ```

 For the first 30 seconds of the container's life, there is a `/tmp/healthy` file.
@@ -85,7 +85,7 @@ The output shows that nginx is serving the web page that was written by the init
 [communicating between Containers running in the same Pod](/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/).
 * Learn more about [Init Containers](/docs/concepts/workloads/pods/init-containers/).
 * Learn more about [Volumes](/docs/concepts/storage/volumes/).
-* Learn more about [Debugging Init Containers](/docs/tasks/debug-application-cluster/debug-init-containers/)
+* Learn more about [Debugging Init Containers](/docs/tasks/debug/debug-application/debug-init-containers/)



@@ -399,7 +399,7 @@ in the `securityContext` section of your Pod or Container manifest. The
 [SeccompProfile](/docs/reference/generated/kubernetes-api/{{< param "version"
 >}}/#seccompprofile-v1-core) object consisting of `type` and `localhostProfile`.
 Valid options for `type` include `RuntimeDefault`, `Unconfined`, and
-`Localhost`. `localhostProfile` must only be set set if `type: Localhost`. It
+`Localhost`. `localhostProfile` must only be set if `type: Localhost`. It
 indicates the path of the pre-configured profile on the node, relative to the
 kubelet's configured Seccomp profile location (configured with the `--root-dir`
 flag).
@@ -7,7 +7,7 @@ content_type: task

 {{% thirdparty-content %}}

-Kubernetes applications usually consist of multiple, separate services, each running in its own container. Developing and debugging these services on a remote Kubernetes cluster can be cumbersome, requiring you to [get a shell on a running container](/docs/tasks/debug-application-cluster/get-shell-running-container/) in order to run debugging tools.
+Kubernetes applications usually consist of multiple, separate services, each running in its own container. Developing and debugging these services on a remote Kubernetes cluster can be cumbersome, requiring you to [get a shell on a running container](/docs/tasks/debug/debug-application/get-shell-running-container/) in order to run debugging tools.
 
 `telepresence` is a tool to ease the process of developing and debugging services locally while proxying the service to a remote Kubernetes cluster. Using `telepresence` allows you to use custom tools, such as a debugger and IDE, for a local service and provides the service full access to ConfigMap, secrets, and the services running on the remote cluster.
 
@@ -28,7 +28,7 @@ The Metrics API, and the metrics pipeline that it enables, only offers the minim
 CPU and memory metrics to enable automatic scaling using HPA and / or VPA.
 If you would like to provide a more complete set of metrics, you can complement
 the simpler Metrics API by deploying a second
-[metrics pipeline](/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipeline)
+[metrics pipeline](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/#full-metrics-pipeline)
 that uses the _Custom Metrics API_.
 {{< /note >}}

@@ -65,8 +65,8 @@ Full metrics pipeline projects that are not part of the CNCF are outside the sco
 Learn about additional debugging tools, including:

 * [Logging](/docs/concepts/cluster-administration/logging/)
-* [Monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
-* [Getting into containers via `exec`](/docs/tasks/debug-application-cluster/applications/get-shell-running-container/)
+* [Monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)
+* [Getting into containers via `exec`](/docs/tasks/debug/debug-application/get-shell-running-container/)
 * [Connecting to containers via proxies](/docs/tasks/extend-kubernetes/http-proxy-access-api/)
 * [Connecting to containers via port forwarding](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
-* [Inspect Kubernetes node with crictl](/docs/tasks/debug-application-cluster/monitoring/crictl/)
+* [Inspect Kubernetes node with crictl](/docs/tasks/debug/debug-cluster/crictl/)
@@ -114,5 +114,5 @@ args: ["-c", "while true; do echo hello; sleep 10;done"]


 * Learn more about [configuring pods and containers](/docs/tasks/).
-* Learn more about [running commands in a container](/docs/tasks/debug-application-cluster/get-shell-running-container/).
+* Learn more about [running commands in a container](/docs/tasks/debug/debug-application/get-shell-running-container/).
 * See [Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core).
@@ -90,6 +90,6 @@ Always perform force deletion of StatefulSet Pods carefully and with complete kn
 ## {{% heading "whatsnext" %}}


-Learn more about [debugging a StatefulSet](/docs/tasks/debug-application-cluster/debug-stateful-set/).
+Learn more about [debugging a StatefulSet](/docs/tasks/debug/debug-application/debug-statefulset/).


@@ -90,7 +90,7 @@ The common use for HorizontalPodAutoscaler is to configure it to fetch metrics f
 (`metrics.k8s.io`, `custom.metrics.k8s.io`, or `external.metrics.k8s.io`).  The `metrics.k8s.io` API is
 usually provided by an add-on named Metrics Server, which needs to be launched separately.
 For more information about resource metrics, see
-[Metrics Server](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server).
+[Metrics Server](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-server).

 [Support for metrics APIs](#support-for-metrics-apis) explains the stability guarantees and support status for these
 different APIs.
@@ -532,7 +532,7 @@ kubectl delete pvc data-mysql-4
 ## {{% heading "whatsnext" %}}

 * Learn more about [scaling a StatefulSet](/docs/tasks/run-application/scale-stateful-set/).
-* Learn more about [debugging a StatefulSet](/docs/tasks/debug-application-cluster/debug-stateful-set/).
+* Learn more about [debugging a StatefulSet](/docs/tasks/debug/debug-application/debug-statefulset/).
 * Learn more about [deleting a StatefulSet](/docs/tasks/run-application/delete-stateful-set/).
 * Learn more about [force deleting StatefulSet Pods](/docs/tasks/run-application/force-delete-stateful-set-pod/).
 * Look in the [Helm Charts repository](https://artifacthub.io/)
@@ -143,7 +143,7 @@ Configurations with a single API server will experience unavailability while the

   1. Validate the logs from control plane components, along with the kubelet and the
      kube-proxy are not throwing any tls errors, see
-      [looking at the logs](/docs/tasks/debug-application-cluster/debug-cluster/#looking-at-logs).
+      [looking at the logs](/docs/tasks/debug/debug-cluster/#looking-at-logs).

   1. Validate logs from any aggregated api servers and pods using in-cluster config.

@@ -93,7 +93,7 @@ cassandra   ClusterIP   None         <none>        9042/TCP   45s
 ```

 If you don't see a Service named `cassandra`, that means creation failed. Read
-[Debug Services](/docs/tasks/debug-application-cluster/debug-service/)
+[Debug Services](/docs/tasks/debug/debug-application/debug-service/)
 for help troubleshooting common issues.

 ## Using a StatefulSet to create a Cassandra ring
@@ -236,10 +236,10 @@ Do not leave your WordPress installation on this page. If another user finds it,
 ## {{% heading "whatsnext" %}}


-* Learn more about [Introspection and Debugging](/docs/tasks/debug-application-cluster/debug-application-introspection/)
+* Learn more about [Introspection and Debugging](/docs/tasks/debug/debug-application)
 * Learn more about [Jobs](/docs/concepts/workloads/controllers/job/)
 * Learn more about [Port Forwarding](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
-* Learn how to [Get a Shell to a Container](/docs/tasks/debug-application-cluster/get-shell-running-container/)
+* Learn how to [Get a Shell to a Container](/docs/tasks/debug/debug-application/get-shell-running-container/)



@@ -11,7 +11,7 @@ spec:
    args:
    - /bin/sh
    - -c
-    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
+    - touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
@@ -3,6 +3,7 @@ kind: PodSecurityPolicy
 metadata:
  name: restricted
  annotations:
+    # docker/default identifies a profile for seccomp, but it is not particularly tied to the Docker runtime
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'
@@ -205,6 +205,7 @@ apiVersion: kubescheduler.config.k8s.io/v1alpha2
 kind: KubeSchedulerConfiguration

 profiles:
+  - schedulerName: default-scheduler
  - pluginConfig:
      - name: PodTopologySpread
        args:
@@ -89,7 +89,7 @@ Kamu tidak dibatasi untuk hanya menggunakan _provisioner_ internal yang disediak
 pada list yang tersedia (yang memiliki nama dengan prefix "kubernetes.io" dan
 didistribusikan bersamaan dengan Kubernetes). Kamu juga dapat menjalankan dan
 mendefinisikan _provisioner_ eksternal yang merupakan program independen selama
-program tersebut menerapkan [spesifikasi](https://git.k8s.io/community/contributors/design-proposals/storage/volume-provisioning.md)
+program tersebut menerapkan [spesifikasi](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/volume-provisioning.md)
 yang didefinisikan oleh Kubernetes. Penulis dari _provisioner_ eksternal Kubernetes
 memiliki kuasa penuh akan tempat dimana kode sumber yang mereka tulis, bagaimana
 mekanisme penyediaan (_provisioning_) dilakukan, serta bagaimana hal tersebut dapat dijalankan,
@@ -61,7 +61,6 @@ kube-system       Active   1d
   * `default` 다른 네임스페이스가 없는 오브젝트를 위한 기본 네임스페이스
   * `kube-system` 쿠버네티스 시스템에서 생성한 오브젝트를 위한 네임스페이스
   * `kube-public` 이 네임스페이스는 자동으로 생성되며 모든 사용자(인증되지 않은 사용자 포함)가 읽기 권한으로 접근할 수 있다. 이 네임스페이스는 주로 전체 클러스터 중에 공개적으로 드러나서 읽을 수 있는 리소스를 위해 예약되어 있다. 이 네임스페이스의 공개적인 성격은 단지 관례이지 요구 사항은 아니다.
-   * `kube-node-lease` 클러스터가 스케일링될 때 노드 하트비트의 성능을 향상시키는 각 노드와 관련된 리스(lease) 오브젝트에 대한 네임스페이스
   * `kube-node-lease` 이 네임스페이스는 각 노드와 연관된 [리스](/docs/reference/kubernetes-api/cluster-resources/lease-v1/)
      오브젝트를 갖는다. 노드 리스는 kubelet이 [하트비트](/ko/docs/concepts/architecture/nodes/#하트비트)를
      보내서 컨트롤 플레인이 노드의 장애를 탐지할 수 있게 한다.
@@ -59,7 +59,7 @@ O cloud-controller-manager executa apenas controladores que são específicos pa
 Se você estiver executando o Kubernetes em suas próprias instalações ou em um ambiente de aprendizagem dentro de seu
 próprio PC, o cluster não possui um gerenciador de controlador de nuvem.

-Tal como acontece com o kube-controller-manager, o cloud-controller-manager combina vários ciclos de controle logicamente independentes em um binário único que você executa como um processo único. Você pode escalar horizontalmente (exectuar mais de uma cópia) para melhorar o desempenho ou para auxiliar na tolerância a falhas.
+Tal como acontece com o kube-controller-manager, o cloud-controller-manager combina vários ciclos de controle logicamente independentes em um binário único que você executa como um processo único. Você pode escalar horizontalmente (executar mais de uma cópia) para melhorar o desempenho ou para auxiliar na tolerância a falhas.

 Os seguintes controladores podem ter dependências de provedor de nuvem:

@@ -0,0 +1,18 @@
+---
+title: Anotação
+id: annotation
+date: 2018-04-12
+full_link: /docs/concepts/overview/working-with-objects/annotations
+short_description: >
+  Um par de chave-valor (key-value) é usado para anexar metadados arbitrários não identificáveis a objetos.
+
+aka: 
+tags:
+- fundamental
+---
+ Um par de chave-valor (key-value) é usado para anexar metadados arbitrários não identificáveis a objetos.
+
+<!--more--> 
+
+Os metadados em uma anotação podem ser pequenos ou grandes, estruturados ou não estruturados, e podem incluir caracteres não permitidos pelos {{< glossary_tooltip text="rótulos" term_id="label" >}}. Clientes como ferramentas e bibliotecas podem recuperar esses metadados.
+
@@ -29,7 +29,7 @@ content_type: concept
 * [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) - это подключаемый модуль контроллера CNI на основе OVN для обеспечения облачной цепочки сервисных функций (SFC), несколько наложенных сетей OVN, динамического создания подсети, динамического создания виртуальных сетей, сети поставщика VLAN, сети прямого поставщика и подключаемого к другим Multi Сетевые плагины, идеально подходящие для облачных рабочих нагрузок на периферии в сети с несколькими кластерами.
 * [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) плагин для контейнера (NCP) обеспечивающий интеграцию между VMware NSX-T и контейнерами оркестраторов, таких как Kubernetes, а так же интеграцию между NSX-T и контейнеров на основе платформы CaaS/PaaS, таких как Pivotal Container Service (PKS) и OpenShift.
 * [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) - эта платформа SDN, которая обеспечивает сетевое взаимодействие на основе политик между Kubernetes подами и не Kubernetes окружением, с отображением и мониторингом безопасности.
-* [Romana](https://romana.io) - это сетевое решение уровня 3 для сетей подов, которое также поддерживает [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Подробности установки Kubeadm доступны [здесь](https://github.com/romana/romana/tree/master/containerize).
+* [Romana](https://github.com/romana/romana) - это сетевое решение уровня 3 для сетей подов, которое также поддерживает [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Подробности установки Kubeadm доступны [здесь](https://github.com/romana/romana/tree/master/containerize).
 * [Weave Net](https://www.weave.works/docs/net/latest/kubernetes/kube-addon/) предоставляет сеть и обеспечивает сетевую политику, будет работать на обеих сторонах сетевого раздела и не требует внешней базы данных.

 ## Обнаружение служб
@@ -16,6 +16,15 @@ date: 2018-10-01
 --->
 **作者**： [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google)

+<!-- 
+**Update (December 2021):** _Kubernetes now has built-in gRPC health probes starting in v1.23.
+To learn more, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe).
+This article was originally written about an external tool to achieve the same task._
+-->
+**更新（2021 年 12 月）：** “Kubernetes 从 v1.23 开始具有内置 gRPC 健康探测。
+了解更多信息，请参阅[配置存活探针、就绪探针和启动探针](/zh/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)。
+本文最初是为有关实现相同任务的外部工具所写。”
+
 <!--
 [gRPC](https://grpc.io) is on its way to becoming the lingua franca for
 communication between cloud-native microservices. If you are deploying gRPC
@@ -3,7 +3,7 @@ layout: blog
 title: "弃用 Dockershim 的常见问题"
 date: 2020-12-02
 slug: dockershim-faq
-aliases: [ '/dockershim' ]
+aliases: [ '/zh/dockershim' ]
 ---
 <!-- 
 layout: blog
@@ -55,8 +55,8 @@ evergreen: true
 <!-- Note that as a new built-in command, `kubectl debug` takes priority over any kubectl plugin named “debug”. You must rename the affected plugin. -->
 请注意，作为新的内置命令，`kubectl debug` 优先于任何名为 “debug” 的 kubectl 插件。你必须重命名受影响的插件。

-<!-- Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/). -->
-`kubectl alpha debug` 现在不推荐使用，并将在后续版本中删除。更新你的脚本以使用 `kubectl debug`。 有关更多信息 `kubectl debug`，请参阅[调试正在运行的 Pod]((https://kubernetes.io/zh/docs/tasks/debug-application-cluster/debug-running-pod/)。
+<!-- Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/). -->
+`kubectl alpha debug` 现在不推荐使用，并将在后续版本中删除。更新你的脚本以使用 `kubectl debug`。 有关更多信息 `kubectl debug`，请参阅[调试正在运行的 Pod]((https://kubernetes.io/zh/docs/tasks/debug/debug-application/debug-running-pod/)。

 <!-- ### Beta: API Priority and Fairness -->
 ### 测试版：API 优先级和公平性 {#beta-api-priority-and-fairness)
@@ -1,12 +1,14 @@
 ---
 layout: blog
 title: "更新：弃用 Dockershim 的常见问题"
+linkTitle: "弃用 Dockershim 的常见问题"
 date: 2022-02-17
 slug: dockershim-faq
 ---
 <!-- 
 layout: blog
 title: "Updated: Dockershim Removal FAQ"
+linkTitle: "Dockershim Removal FAQ"
 date: 2022-02-17
 slug: dockershim-faq
 aliases: [ '/dockershim' ]
@@ -312,13 +314,13 @@ runtime where possible.
 <!--
 Another thing to look out for is anything expecting to run for system maintenance
 or nested inside a container when building images will no longer work. For the
-former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
+former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
 latter you can use newer container build options like [img], [buildah],
 [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker.
 -->
 另外还有一个需要关注的点，那就是当创建镜像时，系统维护或嵌入容器方面的任务将无法工作。
 对于前者，可以用 [`crictl`](https://github.com/kubernetes-sigs/cri-tools) 工具作为临时替代方案
-(参阅[从 docker cli 到 crictl 的映射](/zh/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl))。
+(参阅[从 docker cli 到 crictl 的映射](/zh/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl))。
 对于后者，可以用新的容器创建选项，例如
 [img](https://github.com/genuinetools/img)、
 [buildah](https://github.com/containers/buildah)、
@@ -335,10 +337,10 @@ options are available as you migrate things over.

 <!--
 For instructions on how to use containerd and CRI-O with Kubernetes, see the
-Kubernetes documentation on [Container Runtimes]
+Kubernetes documentation on [Container Runtimes].
 -->
 有关如何在 Kubernetes 中使用 containerd 和 CRI-O 的说明，
-请参阅 [Kubernetes 相关文档](/docs/setup/production-environment/container-runtimes/)
+请参阅 [Kubernetes 相关文档](/docs/setup/production-environment/container-runtimes/)。

 <!--
 ### What if I have more questions?
@@ -353,6 +355,13 @@ to our end user community forum: https://discuss.kubernetes.io/.
 如果你使用了供应商支持的 Kubernetes 发行版，你可以咨询供应商他们产品的升级计划。
 对于最终用户的问题，请把问题发到我们的最终用户社区的论坛：https://discuss.kubernetes.io/。

+<!--
+You can discuss the decision to remove dockershim via a dedicated
+[GitHub issue](https://github.com/kubernetes/kubernetes/issues/106917).
+-->
+你可以通过专用 [GitHub 问题](https://github.com/kubernetes/kubernetes/issues/106917) 
+讨论删除 dockershim 的决定。
+
 <!--
 You can also check out the excellent blog post
 [Wait, Docker is deprecated in Kubernetes now?][dep] a more in-depth technical
@@ -361,6 +370,24 @@ discussion of the changes.
 你也可以看看这篇优秀的博客文章：[等等，Docker 被 Kubernetes 弃用了?](https://dev.to/inductor/wait-docker-is-deprecated-in-kubernetes-now-what-do-i-do-e4m)
 对这些变化进行更深入的技术讨论。

+<!--
+### Is there any tooling that can help me find dockershim in use
+-->
+### 是否有任何工具可以帮助我找到正在使用的 dockershim
+
+<!--
+Yes! The [Detector for Docker Socket (DDS)][dds] is a kubectl plugin that you can
+install and then use to check your cluster. DDS can detect if active Kubernetes workloads
+are mounting the Docker Engine socket (`docker.sock`) as a volume.
+Find more details and usage patterns in the DDS project's [README][dds].
+-->
+是的！ [Docker Socket 检测器 (DDS)][dds] 是一个 kubectl 插件，
+你可以安装它用于检查你的集群。 DDS 可以检测运行中的 Kubernetes
+工作负载是否将 Docker 引擎套接字 (`docker.sock`) 作为卷挂载。
+在 DDS 项目的 [README][dds] 中查找更多详细信息和使用方法。
+
+[dds]: https://github.com/aws-containers/kubectl-detector-for-docker-socket
+
 <!--
 ### Can I have a hug?
 -->
@@ -0,0 +1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 215 127"><defs><style>.cls-1{fill:transparent;}.cls-2{fill:#040606;}</style></defs><title>kubernetes.io-54664</title><rect class="cls-1" x="-4.55738" y="-4.48481" width="223.25536" height="134.51136"/><path class="cls-2" d="M169.05853,71.307c.23549,6.07483,5.42615,10.38072,14.09569,10.38072,7.0783,0,12.91766-3.06693,12.91766-9.85009,0-4.71846-2.65436-7.49024-8.78885-8.66954l-4.77685-.94382c-3.06756-.59028-5.19066-1.17992-5.19066-3.0079,0-2.00568,2.06471-2.89047,4.65943-2.89047,3.77525,0,5.3081,1.887,5.42678,4.1288H194.951c-.41258-5.89838-5.1304-9.8501-12.73994-9.8501-7.845,0-12.50382,4.30588-12.50382,9.90912,0,6.84157,5.54483,7.96247,10.3217,8.84662l3.95171.70834c2.83082.53062,4.06977,1.35638,4.06977,3.0079,0,1.47444-1.41541,2.94887-4.77748,2.94887-4.89553,0-6.488-2.53631-6.54705-4.71845Zm-27.4259-5.13164a8.58413,8.58413,0,0,1,8.557-8.61113q.02706-.00009.05409,0a8.58415,8.58415,0,0,1,8.61115,8.557q.00009.02706,0,.0541a8.58413,8.58413,0,0,1-8.55641,8.61177q-.02738.00009-.05473,0a8.58413,8.58413,0,0,1-8.61113-8.557q-.00009-.02738,0-.05473M158.79589,81.098h7.49149V50.95811h-7.49149v2.41825a15.58033,15.58033,0,1,0-9.01133,28.37034q.05285.00018.10567,0a15.47693,15.47693,0,0,0,8.90566-2.77179ZM106.59844,66.17532a8.58413,8.58413,0,0,1,8.557-8.61113q.02706-.00009.05409,0a8.58414,8.58414,0,0,1,8.61114,8.557q.00009.02706,0,.05409a8.58413,8.58413,0,0,1-8.55641,8.61177q-.02738.00009-.05473,0a8.58413,8.58413,0,0,1-8.61113-8.557q-.00009-.02738,0-.05473m17.16387-25.47987V53.37636a15.58048,15.58048,0,1,0-9.01195,28.37034q.05251.00018.105,0a15.48233,15.48233,0,0,0,8.90691-2.77179V81.098h7.49023V40.69545ZM96.21772,50.95811H88.72811V81.098h7.49023Zm0-10.26266H88.72811v7.49023h7.49023ZM60.41805,66.17532a8.58414,8.58414,0,0,1,8.557-8.61113q.02673-.00009.05346,0a8.58414,8.58414,0,0,1,8.61114,8.557q.00009.02706,0,.05409a8.58414,8.58414,0,0,1-8.55642,8.61177q-.02736.00009-.05472,0a8.58412,8.58412,0,0,1-8.61113-8.557q-.00009-.02738,0-.05473M77.58067,40.69545V53.37636A15.58033,15.58033,0,1,0,68.56935,81.7467q.05283.00018.10567,0a15.4769,15.4769,0,0,0,8.90565-2.77179V81.098h7.4915V40.69545ZM25.38259,66.176a8.58414,8.58414,0,0,1,8.557-8.61114q.02736-.00009.05472,0a8.58414,8.58414,0,0,1,8.61114,8.557q.00009.027,0,.05409a8.58414,8.58414,0,0,1-8.55642,8.61176q-.02736.00009-.05472,0a8.58413,8.58413,0,0,1-8.61177-8.55641q-.00009-.02768,0-.05535m17.16388,14.9227H50.038V50.95872H42.54647V53.377a15.58048,15.58048,0,1,0-9.01069,28.37035q.0522.00016.10441,0a15.48032,15.48032,0,0,0,8.90628-2.77179Z"/></svg>
@@ -0,0 +1,78 @@
+---
+title: adidas Case Study
+linkTitle: adidas
+case_study_styles: true
+cid: caseStudies
+featured: false
+
+new_case_study_styles: true
+heading_background: /images/case-studies/adidas/banner1.png
+heading_title_text: adidas
+use_gradient_overlay: true
+subheading: >
+  Staying True to Its Culture, adidas Got 40% of Its Most Impactful Systems Running on Kubernetes in a Year
+case_study_details:
+  - Company: adidas
+  - Location: Herzogenaurach, Germany
+  - Industry: Fashion
+---
+
+<h2>Challenge</h2>
+
+<p>In recent years, the adidas team was happy with its software choices from a technology perspective—but accessing all of the tools was a problem. For instance, "just to get a developer VM, you had to send a request form, give the purpose, give the title of the project, who's responsible, give the internal cost center a call so that they can do recharges," says Daniel Eichten, Senior Director of Platform Engineering. "The best case is you got your machine in half an hour. Worst case is half a week or sometimes even a week."</p>
+
+<h2>Solution</h2>
+
+<p>To improve the process, "we started from the developer point of view," and looked for ways to shorten the time it took to get a project up and running and into the adidas infrastructure, says Senior Director of Platform Engineering Fernando Cornago. They found the solution with containerization, agile development, continuous delivery, and a cloud native platform that includes Kubernetes and Prometheus.</p>
+
+<h2>Impact</h2>
+
+<p>Just six months after the project began, 100% of the adidas e-commerce site was running on Kubernetes. Load time for the e-commerce site was reduced by half. Releases went from every 4-6 weeks to 3-4 times a day. With 4,000 pods, 200 nodes, and 80,000 builds per month, adidas is now running 40% of its most critical, impactful systems on its cloud native platform.</p>
+
+{{< case-studies/quote 
+  image="/images/case-studies/adidas/banner2.png"
+  author="FERNANDO CORNAGO, SENIOR DIRECTOR OF PLATFORM ENGINEERING AT ADIDAS"
+>}}
+"For me, Kubernetes is a platform made by engineers for engineers. It's relieving the development team from tasks that they don't want to do, but at the same time giving the visibility of what is behind the curtain, so they can also control it."
+{{< /case-studies/quote >}}
+
+{{< case-studies/lead >}}
+In recent years, the adidas team was happy with its software choices from a technology perspective—but accessing all of the tools was a problem.
+{{< /case-studies/lead >}}
+
+<p>For engineers at adidas, says Daniel Eichten, Senior Director of Platform Engineering, "it felt like being an artist with your hands tied behind your back, and you're supposed to paint something."</p>
+
+<p>For instance, "just to get a developer VM, you had to send a request form, give the purpose, give the title of the project, who's responsible, give the internal cost center a call so that they can do recharges," says Eichten. "Eventually, after a ton of approvals, then the provisioning of the machine happened within minutes, and then the best case is you got your machine in half an hour. Worst case is half a week or sometimes even a week."</p>
+
+<p>To improve the process, "we started from the developer point of view," and looked for ways to shorten the time it took to get a project up and running and into the adidas infrastructure, says Senior Director of Platform Engineering Fernando Cornago.</p>
+
+{{< case-studies/quote author="DANIEL EICHTEN, SENIOR DIRECTOR OF PLATFORM ENGINEERING AT ADIDAS" >}}
+"I call our cloud native platform the field of dreams. We built it, and we never anticipated that people would come and just love it."
+{{< /case-studies/quote >}}
+
+<p>"We were engineers before," adds Eichten. "We know what a typical engineer needs, is craving for, what he or she doesn't want to take care of. For us it was pretty clear. We filled the gaps that no one wants to take care of, and we make the stuff that is usually painful as painless as possible." The goals: to improve speed, operability, and observability.</p>
+
+<p>Cornago and Eichten found the solution with containerization, agile development, continuous delivery, and a cloud native platform that includes Kubernetes and Prometheus. "Choosing Kubernetes was pretty clear," says Eichten. "Day zero, deciding, easy. Day one, installing, configuring, easy. Day two, keeping it up and running even with small workloads, if something goes wrong, you don't know how these things work in detail, you're lost. For day two problems, we needed a partner who's helping us."</p>
+
+<p>In early 2017, adidas chose Giant Swarm to consult, install, configure, and run all of its Kubernetes clusters in AWS and on premise. "There is no competitive edge over our competitors like Puma or Nike in running and operating a Kubernetes cluster," says Eichten. "Our competitive edge is that we teach our internal engineers how to build cool e-comm stores that are fast, that are resilient, that are running perfectly."</p>
+
+{{< case-studies/quote 
+  image="/images/case-studies/adidas/banner3.png"
+  author="DANIEL EICHTEN, SENIOR DIRECTOR OF PLATFORM ENGINEERING AT ADIDAS"
+>}}
+"There is no competitive edge over our competitors like Puma or Nike in running and operating a Kubernetes cluster. Our competitive edge is that we teach our internal engineers how to build cool e-comm stores that are fast, that are resilient, that are running perfectly."
+{{< /case-studies/quote >}}
+
+<p>Adds Cornago: "For me, our Kubernetes platform is made by engineers for engineers. It's relieving the development team from tasks that they don't want to do, but at the same time giving the visibility of what is behind the curtain, so they can also control it."</p>
+
+<p>Case in point: For Cyber Week, the team has to create a lot of custom metrics. In November 2017, "because we used the same Prometheus that we use for monitoring the cluster, we really filled the Prometheus database, and we were not able to reduce the retention period [enough]," says Cornago. So during the freeze period before the peak shopping week, five engineers from the platform team worked with five engineers from the e-comm team to figure out a federated solution that was implemented in two days.</p>
+
+<p>In addition to being ready for Cyber Week—100% of the adidas e-commerce site was running on Kubernetes then, just six months after the project began—the cloud native stack has had other impressive results. Load time for the e-commerce site was reduced by half. Releases went from every 4-6 weeks to 3-4 times a day. With 4,000 pods, 200 nodes, and 80,000 builds per month, adidas is now running 40% of its most critical, impactful systems on its cloud native platform.</p>
+
+<p>And adoption has spread quickly among adidas's 300-strong engineering corps. "I call our cloud native platform the field of dreams," says Eichten. "We built it, and we never anticipated that people would come and just love it."</p>
+
+<p>For one thing, "everybody who can touch a line of code" has spent one full week onboarding and learning the platform with members of the 35-person platform engineering team, says Cornago. "We try to spend 50% of our time sitting with the teams, because this is the only way to understand how our platform is being used. And this is how the teams will feel safe that there is someone on the other side of the wall, also feeling the pain."</p>
+
+<p>Additionally, Cornago and Eichten took advantage of the fact that as a fashion athletic wear brand, adidas has sports and competition in its DNA. "Top-down mandates don't work at adidas, but gamification works," says Cornago. "So this year we had a DevOps Cup competition. Every team created new technical capabilities and had a hypothesis of how this affected business value. We announced the winner at a big internal tech summit with more than 600 people. It's been really, really useful for the teams."</p>
+
+<p>So if they had any advice for other companies looking to start a cloud native journey, it would be this: "There is no one-size-fits-all for all companies," says Cornago. "Apply your company's culture to everything that you do."</p>
@@ -0,0 +1,84 @@
+---
+title: Babylon Case Study
+linkTitle: Babylon
+case_study_styles: true
+cid: caseStudies
+logo: babylon_featured_logo.svg
+featured: true
+weight: 1
+quote: >
+   Kubernetes is a great platform for machine learning because it comes with all the scheduling and scalability that you need.
+
+new_case_study_styles: true
+heading_background: /images/case-studies/babylon/banner4.jpg
+heading_title_text: Babylon
+use_gradient_overlay: true
+subheading: >
+  AppDirect: How Cloud Native Is Enabling Babylon's Medical AI Innovations
+case_study_details:
+  - Company: Babylon
+  - Location: United Kingdom
+  - Industry: AI, Healthcare
+---
+
+<h2>Challenge</h2>
+
+<p>A large number of Babylon's products leverage machine learning and artificial intelligence, and in 2019, there wasn't enough computing power in-house to run a particular experiment. The company was also growing (from 100 to 1,600 in three years) and planning expansion into other countries.</p>
+
+<h2>Solution</h2>
+
+<p>Babylon had migrated its user-facing applications to a Kubernetes platform in 2018, so the infrastructure team turned to Kubeflow, a toolkit for machine learning on Kubernetes. "We tried to create a Kubernetes core server, we deployed Kubeflow, and we orchestrated the whole experiment, which ended up being a really good success," says AI Infrastructure Lead Jérémie Vallée. The team began building a self-service AI training platform on top of Kubernetes.</p>
+
+<h2>Impact</h2>
+
+<p>Instead of waiting hours or days to be able to compute, teams can get access instantaneously. Clinical validations used to take 10 hours; now they are done in under 20 minutes. The portability of the cloud native platform has also enabled Babylon to expand into other countries.</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/babylon/banner1.jpg"
+  author="JÉRÉMIE VALLÉE, AI INFRASTRUCTURE LEAD AT BABYLON"
+>}}
+"Kubernetes is a great platform for machine learning because it comes with all the scheduling and scalability that you need."
+{{< /case-studies/quote >}}
+
+{{< case-studies/lead >}}
+Babylon's mission is to put accessible and affordable healthcare services in the hands of every person on earth.
+{{< /case-studies/lead >}}
+
+<p>Since its launch in the U.K. in 2013, the startup has facilitated millions of digital consultations around the world. In the U.K., patients were typically waiting a week or two for a doctor's appointment. Through Babylon's NHS service, GP at Hand—which has more than 75,000 registered patients—39% get an appointment through their phone within 30 minutes, and 89% within 6 hours.</p>
+
+<p>That's just the start. "We try to combine different types of technology with the medical expertise that we have in-house to build products that will help patients manage and understand their health, and also help doctors be more efficient at what they do," says Jérémie Vallée, AI Infrastructure Lead at Babylon. </p>
+
+<p>A large number of these products leverage machine learning and artificial intelligence, and in 2019, researchers hit a pain point. "We have some servers in-house where our researchers were doing a lot of AI experiments and some training of models, and we came to a point where we didn't have enough compute in-house to run a particular experiment," says Vallée. </p>
+
+<p>Babylon had migrated its user-facing applications to a Kubernetes platform in 2018, "and we had a lot of Kubernetes knowledge thanks to the migration," he adds. To optimize some of the models that had been created, the team turned to Kubeflow, a toolkit for machine learning on Kubernetes. "We tried to create a Kubernetes core server, we deployed Kubeflow, and we orchestrated the whole experiment, which ended up being a really good success," he says.</p>
+
+<p>Based on that experience, Vallée's team was tasked with building a self-service platform to help Babylon's AI teams become more efficient, and by extension help get products to market faster. The main requirements: (1) the ability to give researchers and engineers access to the compute they needed, regardless of the size of the experiments they may need to run; (2) a way to provide teams with the best tools that they needed to do their work, on demand and in a centralized way; and (3) the training platform had to be close to the data that was being managed, because of the company's expansion into different countries.</p>
+
+{{< case-studies/quote author="CAROLINE HARGROVE, CHIEF TECHNOLOGY OFFICER AT BABYLON" >}}
+"Delivering a self-service platform where users are empowered to run their own workload has enabled our data scientist community to do hyper parameter tuning and general algorithm development without any cloud skill and without the help of platform engineers, thus accelerating our innovation."
+{{< /case-studies/quote >}}
+
+<p>Kubernetes was an enabler on every count. "Kubernetes is a great platform for machine learning because it comes with all the scheduling and scalability that you need," says Vallée. The need to keep data in every country in which Babylon operates requires a multi-region, multi-cloud strategy, and some countries might not even have a public cloud provider at all. "We wanted to make this platform portable so that we can run training jobs anywhere," he says. "Kubernetes offered a base layer that allows you to deploy the platform outside of the cloud provider, and then deploy whatever tooling you need. That was a very good selling point for us."</p>
+
+<p>Once the team decided to build the Babylon AI Research platform on top of Kubernetes, they referred to the Cloud Native Landscape to build out the stack: Prometheus and Grafana for monitoring; an Istio service mesh to control the network on the training platform and control what access all of the workflows would have; Helm to deploy the stack; and Flux to manage the GitOps part of the pipeline.</p>
+
+<p>The cloud native AI platform has had a huge impact at Babylon. The first research projects run on the platform mostly involved machine learning and natural language processing. These experiments required a huge amount of compute—1600 CPU, 3.2 TB RAM—which was much more than Babylon had in-house. Plus, access to compute used to take hours, or sometimes even days, depending on how busy the platform team was. "Now, with Kubernetes and the self-service platform that we provide, it's pretty much instantaneous," says Vallée.</p>
+
+<p>Another important type of work that's done on the platform is clinical validation for new applications such as Babylon's Symptom Checker, which calculates the probability of a disease given the evidence input by the user. "Being in healthcare, we want all of our models to be safe before they're going to hit production," says Vallée. Using Argo for GitOps "enabled us to scale the process massively."</p>
+
+{{< case-studies/quote 
+  image="/images/case-studies/babylon/banner2.jpg"
+  author="JEAN MARIE FERDEGUE, DIRECTOR OF PLATFORM OPERATIONS AT BABYLON"
+>}}
+"Giving a Kubernetes-based platform to our data scientists has meant increased security, increased innovation through empowerment, and a more affordable health service as our cloud engineers are building an experience that is used by hundreds on a daily basis, rather than supporting specific bespoke use cases."
+{{< /case-studies/quote >}}
+
+<p>Researchers used to have to wait up to 10 hours to get results on new versions of their models. With Kubernetes, that time is now down to under 20 minutes. Plus, previously they could only run one clinical validation at a time, now they can run many parallel ones if they need to—a huge benefit considering that in the past three years, Babylon has grown from 100 to 1,600 employees.</p>
+
+<p>"Delivering a self-service platform where users are empowered to run their own workload has enabled our data scientist community to do hyper parameter tuning and general algorithm development without any cloud skill and without the help of platform engineers, thus accelerating our innovation," says Chief Technology Officer Caroline Hargrove.</p>
+
+<p>Adds Director of Platform Operations Jean Marie Ferdegue: "Giving a Kubernetes-based platform to our data scientists has meant increased security, increased innovation through empowerment, and a more affordable health service as our cloud engineers are building an experience that is used by hundreds on a daily basis, rather than supporting specific bespoke use cases."</p>
+
+<p>Plus, as Babylon continues to expand, "it will be very easy to onboard new countries," says Vallée. "Fifteen months ago when we deployed this platform, we had one big environment in the U.K., but now we have one in Canada, we have one in Asia, and we have one coming in the U.S. This is one of the things that Kubernetes and the other cloud native projects have enabled for us."</p>
+
+<p>Babylon's road map for cloud native involves onboarding all of the company's AI efforts to the platform. Increasingly, that includes AI services of care. "I think this is going to be an interesting field where AI and healthcare meet," Vallée says. "It's kind of a complex problem and there's a lot of issues around this. So with our platform, we want to say, 'What can we do to make this less painful for our developers and machine learning engineers?'"</p>
@@ -0,0 +1,86 @@
+---
+title: Booking.com Case Study
+linkTitle: Booking.com
+case_study_styles: true
+cid: caseStudies
+logo: booking.com_featured_logo.png
+featured: true
+weight: 3
+quote: >
+  We realized that we needed to learn Kubernetes better in order to fully use the potential of it. At that point, we made the shift to build our own Kubernetes platform.
+
+new_case_study_styles: true
+heading_background: /images/case-studies/booking/banner1.jpg
+heading_title_text: Booking.com
+use_gradient_overlay: true
+subheading: >
+  After Learning the Ropes with a Kubernetes Distribution, Booking.com Built a Platform of Its Own
+case_study_details:
+  - Company: Booking.com
+  - Location: Netherlands
+  - Industry: Travel
+---
+
+<h2>Challenge</h2>
+
+<p>In 2016, Booking.com migrated to an OpenShift platform, which gave product developers faster access to infrastructure. But because Kubernetes was abstracted away from the developers, the infrastructure team became a "knowledge bottleneck" when challenges arose. Trying to scale that support wasn't sustainable.</p>
+
+<h2>Solution</h2>
+
+<p>After a year operating OpenShift, the platform team decided to build its own vanilla Kubernetes platform—and ask developers to learn some Kubernetes in order to use it. "This is not a magical platform," says Ben Tyler, Principal Developer, B Platform Track. "We're not claiming that you can just use it with your eyes closed. Developers need to do some learning, and we're going to do everything we can to make sure they have access to that knowledge."</p>
+
+<h2>Impact</h2>
+
+<p>Despite the learning curve, there's been a great uptick in adoption of the new Kubernetes platform. Before containers, creating a new service could take a couple of days if the developers understood Puppet, or weeks if they didn't. On the new platform, it can take as few as 10 minutes. About 500 new services were built on the platform in the first 8 months.</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/booking/banner2.jpg"
+  author="BEN TYLER, PRINCIPAL DEVELOPER, B PLATFORM TRACK AT BOOKING.COM"
+>}}
+"As our users learn Kubernetes and become more sophisticated Kubernetes users, they put pressure on us to provide a better, more native Kubernetes experience, which is great. It's a super healthy dynamic."
+{{< /case-studies/quote >}}
+
+{{< case-studies/lead >}}
+Booking.com has a long history with Kubernetes: In 2015, a team at the travel platform prototyped a container platform based on Mesos and Marathon.
+{{< /case-studies/lead >}}
+
+<p>Impressed by what the technology offered, but in need of enterprise features at its scale—the site handles more than 1.5 million room-night reservations a day on average—the team decided to adopt an OpenShift platform.</p>
+
+<p>This platform, which was wrapped in a Heroku-style, high-level CLI interface, "was definitely popular with our product developers," says Ben Tyler, Principal Developer, B Platform Track. "We gave them faster access to infrastructure."</p>
+
+<p>But, he adds, "anytime something went slightly off the rails, developers didn't have any of the knowledge required to support themselves."</p>
+
+<p>And after a year of operating this platform, the infrastructure team found that it had become "a knowledge bottleneck," he says. "Most of the developers who used it did not know it was Kubernetes underneath. An application failure and a platform failure both looked like failures of that Heroku-style tool."</p>
+
+<p>Scaling the necessary support did not seem feasible or sustainable, so the platform team needed a new solution. The understanding of Kubernetes that they had gained operating the OpenShift platform gave them confidence to build a vanilla Kubernetes platform of their own and customize it to suit the company's needs.</p>
+
+{{< case-studies/quote author="EDUARD IACOBOAIA, SENIOR SYSTEM ADMINISTRATOR, B PLATFORM TRACK AT BOOKING.COM" >}}
+"For entering the landscape, OpenShift was definitely very helpful. It shows you what the technology can do, and it makes it easy for you to use it. After we spent some time on it, we realized that we needed to learn Kubernetes better in order to fully use the potential of it. At that point, we made the shift to build our own Kubernetes platform. We definitely benefit in the long term for taking that step and investing the time in gaining that knowledge."
+{{< /case-studies/quote >}}
+
+<p>"For entering the landscape, OpenShift was definitely very helpful," says Eduard Iacoboaia, Senior System Administrator, B Platform Track. "It shows you what the technology can do, and it makes it easy for you to use it. After we spent some time on it, we realized that we needed to learn Kubernetes better in order to fully use the potential of it. At that point, we made the shift to build our own Kubernetes platform. We definitely benefit in the long term for taking that step and investing the time in gaining that knowledge."</p>
+
+<p>Iacoboaia's team had customized a lot of OpenShift tools to make them work at Booking.com, and "those integrations points were kind of fragile," he says. "We spent much more time understanding all the components of Kubernetes, how they work, how they interact with each other." That research led the team to switch from OpenShift's built-in Ansible playbooks to Puppet deployments, which are used for the rest of Booking's infrastructure. The control plane was also moved from inside the cluster onto bare metal, as the company runs tens of thousands of bare-metal servers and a large infrastructure for running applications on bare metal. (Booking runs Kubernetes in multiple clusters in multiple data centers across the various regions where it has compute.) "We decided to keep it as simple as possible and to also use the tools that we know best," says Iacoboaia.</p>
+
+<p>The other big change was that product engineers would have to learn Kubernetes in order to onboard. "This is not a magical platform," says Tyler. "We're not claiming that you can just use it with your eyes closed. Developers need to do some learning, and we're going to do everything we can to make sure they have access to that knowledge." That includes trainings, blog posts, videos, and Udemy courses.</p>
+
+<p>Despite the learning curve, there's been a great uptick in adoption of the new Kubernetes platform. "I think the reason we've been able to strike this bargain successfully is that we're not asking them to learn a proprietary app system," says Tyler. "We're asking them to learn something that's open source, where the knowledge is transferable. They're investing in their own careers by learning Kubernetes."</p>
+
+<p>One clear sign that this strategy has been a success is that in the support channel, when users have questions, other product engineers are jumping in to respond. "I haven't seen that kind of community engagement around a particular platform product internally before," says Tyler. "It helps a lot that it's visibly an ecosystem standard outside of the company, so people feel value in investing in that knowledge and sharing it with others, which is really, really powerful."</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/booking/banner3.jpg"
+  author="BEN TYLER, PRINCIPAL DEVELOPER, B PLATFORM TRACK AT BOOKING.COM"
+>}}
+"We have a tutorial. You follow the tutorial. Your code is running. Then, it's business-logic time. The time to gain access to resources is decreased enormously."
+{{< /case-studies/quote >}}
+
+<p>There's other quantifiable evidence too: Before containers, creating a new service could take a couple of days if the developers understood Puppet, or weeks if they didn't. On the new platform, it takes 10 minutes. "We have a tutorial. You follow the tutorial. Your code is running. Then, it's business-logic time," says Tyler. "The time to gain access to resources is decreased enormously." About 500 new services were built in the first 8 months on the platform, with hundreds of releases per day.</p>
+
+<p>The platform offers different "layers of contracts, so to speak," says Tyler. "At the very base, it's just Kubernetes. If you're a pro Kubernetes user, here's a Kubernetes API, just like you get from GKE or AKS. We're trying to be a provider on that same level. But our whole job inside the company is to be a bigger value add than just vanilla infrastructure, so we provide a set of base images for our main stacks, Perl and Java."</p>
+
+<p>And "as our users learn Kubernetes and become more sophisticated Kubernetes users, they put pressure on us to provide a better more native Kubernetes experience, which is great," says Tyler. "It's a super healthy dynamic."</p>
+
+<p>The platform also includes other CNCF technologies, such as Envoy, Helm, and Prometheus. Most of the critical service traffic for Booking.com is routed through Envoy, and Prometheus is used primarily to monitor infrastructure components. Helm is consumed as a packaging standard. The team also developed and open sourced Shipper, an extension for Kubernetes to add more complex rollout strategies and multi-cluster orchestration.</p>
+
+<p>To be sure, there have been internal discussions about the wisdom of building a Kubernetes platform from the ground up. "This is not really our core competency—Kubernetes and travel, they're kind of far apart, right?" says Tyler. "But we've made a couple of bets on CNCF components that have worked out really well for us. Envoy and Kubernetes, in particular, have been really beneficial to our organization. We were able to customize them, either because we could look at the source code or because they had extension points, and we were able to get value out of them very quickly without having to change any paradigms internally."</p>
@@ -0,0 +1,80 @@
+---
+title: Booz Allen Case Study
+linkTitle: Booz Allen Hamilton
+case_study_styles: true
+cid: caseStudies
+logo: booz-allen-featured-logo.svg
+featured: true
+weight: 2
+quote: >
+  Kubernetes is a great solution for us. It allows us to rapidly iterate on our clients' demands.
+
+new_case_study_styles: true
+heading_background: /images/case-studies/booz-allen/banner4.jpg
+heading_title_text: Booz Allen Hamilton
+use_gradient_overlay: true
+subheading: >
+  How Booz Allen Hamilton Is Helping Modernize the Federal Government with Kubernetes
+case_study_details:
+  - Company: Booz Allen Hamilton
+  - Location: United States
+  - Industry: Government
+---
+
+<h2>Challenge</h2>
+
+<p>In 2017, Booz Allen Hamilton's Strategic Innovation Group worked with the federal government to relaunch the decade-old recreation.gov website, which provides information and real-time booking for more than 100,000 campsites and facilities on federal lands across the country. The infrastructure needed to be agile, reliable, and scalable—as well as repeatable for the other federal agencies that are among Booz Allen Hamilton's customers.</p>
+
+<h2>Solution</h2>
+
+<p>"The only way that we thought we could be successful with this problem across all the different agencies is to create a microservice architecture and containers, so that we could be very dynamic and very agile to any given agency for whatever requirements that they may have," says Booz Allen Hamilton Senior Lead Technologist Martin Folkoff. To meet those requirements, Folkoff's team looked to Kubernetes for orchestration.</p>
+
+<h2>Impact</h2>
+
+<p>With the recreation.gov Kubernetes platform, changes can be implemented in about 30 minutes, compared to the multiple hours or even days legacy government applications require to review the code, get approval, and deploy the fix. Recreation.gov deploys to production on average 10 times a day. With monitoring, security, and logging built in, developers can create and publish new services to production within a week. Additionally, Folkoff says, "supporting the large, existing monoliths in the government is extremely expensive," and migrating into a more modern platform has resulted in perhaps 50% cost savings.</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/booz-allen/banner2.jpg"
+  author="JOSH BOYD, CHIEF TECHNOLOGIST AT BOOZ ALLEN HAMILTON"
+>}}
+"When there's a regulatory change in an agency, or a legislative change in Congress, or an executive order that changes the way you do business, how do I deploy that and get that out to the people who need it rapidly? At the end of the day, that's the problem we're trying to help the government solve with tools like Kubernetes."
+{{< /case-studies/quote >}}
+
+{{< case-studies/lead >}}
+The White House launched an IT modernization effort in 2017, and in addition to improving cybersecurity and shifting to the public cloud and a consolidated IT model, "the federal government is looking to provide a better experience to citizens in every way that we interact with the government through every channel," says Booz Allen Hamilton Senior Lead Technologist Martin Folkoff.
+{{< /case-studies/lead >}}
+
+<p>To that end, Folkoff's Strategic Innovation Group worked with the federal government last year to relaunch the decade-old recreation.gov website, which provides information and real-time booking for more than 100,000 campsites and facilities on federal lands across the country.</p>
+
+<p>The infrastructure needed to be agile, reliable, and scalable—as well as repeatable for the other federal agencies that are among Booz Allen Hamilton's customers. "The only way that we thought we could be successful with this problem across all the different agencies is to create a microservice architecture, so that we could be very dynamic and very agile to any given agency for whatever requirements that they may have," says Folkoff.</p>
+
+{{< case-studies/quote author="MARTIN FOLKOFF, SENIOR LEAD TECHNOLOGIST AT BOOZ ALLEN HAMILTON" >}}
+"With CNCF, there's a lot of focus on scale, and so there's a lot of comfort knowing that as the project grows, we're going to be comfortable using that tool set."
+{{< /case-studies/quote >}}
+
+<p>Booz Allen Hamilton, which has provided consulting services to the federal government for more than a century, introduced microservices, Docker containers, and AWS to its federal agency clients about five years ago. The next logical step was Kubernetes for orchestration. "Knowing that we had to be really agile and really reliable and scalable, we felt that the only technology that we know that can enable those kinds of things are the ones the CNCF provides," Folkoff says. "One of the things that is always important for the government is to make sure that the things that we build really endure. Using technology that is supported across multiple different companies and has strong governance gives people a lot of confidence."</p>
+
+<p>Kubernetes was also aligned with the government's open source and IT modernization initiatives, so there has been an uptick in its usage at federal agencies over the past two years. "Now that Kubernetes is becoming offered as a service by the cloud providers like AWS and Microsoft, we're starting to see even more interest," says Chief Technologist Josh Boyd. Adds Folkoff: "With CNCF, there's a lot of focus on scale, and so there's a lot of comfort knowing that as the project grows, we're going to be comfortable using that tool set."</p>
+
+<p>The greenfield recreation.gov project allowed the team to build a new Kubernetes-enabled site running on AWS, and the migration lasted only a week, when the old site didn't take bookings. "For the actual transition, we just swapped a DNS server, and it only took about 35 seconds between the old site being down and our new site being up and available," Folkoff adds. </p>
+
+{{< case-studies/quote
+  image="/images/case-studies/booz-allen/banner1.png"
+  author="MARTIN FOLKOFF, SENIOR LEAD TECHNOLOGIST AT BOOZ ALLEN HAMILTON"
+>}}
+"Kubernetes alone enables a dramatic reduction in cost as resources are prioritized to the day's event"
+{{< /case-studies/quote >}}
+
+<p>In addition to its work with the Department of Interior for recreation.gov, Booz Allen Hamilton has brought Kubernetes to various Defense, Intelligence, and civilian agencies. Says Boyd: "When there's a regulatory change in an agency, or a legislative change in Congress, or an executive order that changes the way you do business, how do I deploy that and get that out to the people who need it rapidly? At the end of the day, that's the problem we're trying to help the government solve with tools like Kubernetes."</p>
+
+<p>For recreation.gov, the impact was clear and immediate. With the Kubernetes platform, Folkoff says, "if a new requirement for a permit comes out, we have the ability to design and develop and implement that completely independently of reserving a campsite. It provides a much better experience to users." Today, changes can be implemented in about 30 minutes, compared to the multiple hours or even days legacy government applications require to review the code, get approval, and deploy the fix. Recreation.gov deploys to production on average 10 times a day.</p>
+
+<p>Developer velocity has been improved. "When I want to do monitoring or security or logging, I don't have to do anything to my services or my application to enable that anymore," says Boyd. "I get all of this magic just by being on the Kubernetes platform." With all of those things built in, developers can create and publish new services to production within one week. </p>
+
+<p>Additionally, Folkoff says, "supporting the large, existing monoliths in the government is extremely expensive," and migrating into a more modern platform has resulted in perhaps 50% cost savings. "Kubernetes alone enables a dramatic reduction in cost as resources are prioritized to the day's event," he says. "For example, during a popular campsite release, camping-related services are scaled out while permit services are scaled down."</p>
+
+<p>So far, "Kubernetes is a great solution for us," says Folkoff. "It allows us to rapidly iterate on our clients' demands." Looking ahead, the team sees further adoption of the Kubernetes platform across federal agencies. Says Boyd: "You get the ability for the rapid delivery of business value for your customers. You now have observability into everything that you're doing. You don't have these onesies and twosies unicorn servers anymore. Now everything that you deploy is deployed in the same way, it's all instrumented the same way, and it's all built and deployed the same way through our CI/CD processes."</p>
+
+<p>They also see a push toward re-platforming. "There's still a lot of legacy workloads out there," says Boyd. "We've got the new challenges of greenfield development and integration with legacy systems, but also that brown field of 'Hey, how do I take this legacy monolith and get it onto a platform where now it's instrumented with all the magic of the Kubernetes platform without having to do a whole lot to my application?' I think re-platforming is a pretty big use case for the government right now."</p>
+
+<p>And given the success that they've had with Kubernetes so far, Boyd says, "I think at this point that technology is becoming pretty easy to sell." Adds Folkoff: "People are really excited about being able to deploy, scale, be reliable, and do cheaper maintenance of all of this."</p>
@@ -0,0 +1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 215 127"><defs><style>.cls-1{fill:none;}.cls-2{fill:transparent;}.cls-3{clip-path:url(#clip-path);}.cls-4{fill:#ee3248;fill-rule:evenodd;}</style><clipPath id="clip-path"><rect class="cls-1" x="4.70501" y="-8.81268" width="206.90403" height="145.3255"/></clipPath></defs><rect class="cls-2" x="-3.37342" y="-3.34411" width="223.25536" height="134.51135"/><g class="cls-3"><g class="cls-3"><path class="cls-4" d="M28.02166,50.76936l7.69058-.0617c6.9525.12305,7.75228,4.1837,4.61435,13.41275-2.64567,7.87531-6.76768,12.67414-14.58165,12.61279h-6.091ZM169.22325,61.044c4.55274-11.01316,10.52065-15.19686,22.76431-15.50454,10.9518-.24609,15.81224,4.79908,11.813,17.41187C199.49376,76.67151,191.67987,82.209,179.86693,82.209c-12.98175.06134-16.24263-7.87514-10.64368-21.165m8.429.73843c2.83006-7.01411,6.95215-11.87463,13.04345-11.93633,5.47555-.06125,6.46007,3.938,3.99905,12.36706-3.13794,10.82842-7.69068,15.5659-13.90466,15.44286-6.89106-.06135-6.768-6.82946-3.13784-15.87359m-15.07382-8.67536,2.09172-6.02932a34.76316,34.76316,0,0,0-10.95146-1.66134c-7.62924-.0616-13.35114,2.33806-15.6892,7.69066-3.69162,8.183,1.4766,10.70564,7.69084,13.59749,9.10583,4.245,3.876,11.56684-4.86069,10.82842-3.50688-.3077-6.58311-1.90724-9.65961-3.38384l-1.96859,6.15264A33.79646,33.79646,0,0,0,142.64393,82.209c9.0444-.55369,14.64308-4.184,16.91972-9.90595,2.584-6.52159-1.41525-10.52064-7.38324-12.42789-12.42831-3.99939-4.98338-15.75088,10.398-6.76811M95.57659,46.15492a19.153,19.153,0,0,1,2.215,3.62993L87.76306,81.47059H93.854l8.36766-26.45618,9.59791,26.45618h8.61375l-.36939-.98451,10.89012-34.33116h-6.15273l-7.99827,25.34837-9.22879-25.34837c-3.999,0-7.99836.06135-11.99767,0m-34.39259,0a14.395,14.395,0,0,1,2.21468,3.62993L53.43173,81.47059H79.826L81.05656,77.533H63.21437L67.152,65.16635H78.96492l1.23051-3.87627H68.3825L71.95081,49.9695H89.17822L90.347,46.15492c-9.72121,0-19.44208.06135-29.163,0m-42.26817,0a16.4482,16.4482,0,0,1,2.21468,3.62993L11.102,81.47059h8.61366l5.04517-.06135c14.766,0,21.77988-7.32179,24.73333-16.85828,4.245-13.5973-.92316-18.33469-13.47418-18.33469Z"/></g></g></svg>
@@ -0,0 +1,82 @@
+---
+title: Denso Case Study
+linkTitle: Denso
+case_study_styles: true
+cid: caseStudies
+logo: denso_featured_logo.svg
+featured: true
+weight: 4
+quote: >
+  We got Kubernetes experts involved on our team, and it dramatically accelerated development speed.
+
+new_case_study_styles: true
+heading_background: /images/case-studies/denso/banner2.jpg
+heading_title_text: Denso
+use_gradient_overlay: true
+subheading: >
+  How DENSO Is Fueling Development on the Vehicle Edge with Kubernetes
+case_study_details:
+  - Company: Denso
+  - Location: Japan
+  - Industry: Automotive, Edge
+---
+
+<h2>Challenge</h2>
+
+<p>DENSO Corporation is one of the biggest automotive components suppliers in the world. With the advent of connected cars, the company launched a Digital Innovation Department to expand into software, working on vehicle edge and vehicle cloud products. But there were several technical challenges to creating an integrated vehicle edge/cloud platform: "the amount of computing resources, the occasional lack of mobile signal, and an enormous number of distributed vehicles," says R&D Product Manager Seiichi Koizumi.</p>
+
+<h2>Solution</h2>
+
+<p>Koizumi's team realized that because mobility services evolve every day, they needed the flexibility of the cloud native ecosystem for their platform. After considering other orchestrators, DENSO went with Kubernetes for orchestration and added Prometheus, Fluentd, Envoy, Istio, and Helm to the platform. Today, DENSO is using a vehicle edge computer, a private Kubernetes cloud, and managed Kubernetes (GKE, EKS, AKS).</p>
+
+<h2>Impact</h2>
+
+<p>Critical layer features can take 2-3 years to implement in the traditional, waterfall model of development at DENSO. With the Kubernetes platform and agile methods, there's a 2-month development cycle for non-critical software. Now, ten new applications are released a year, and a new prototype is introduced every week. "By utilizing Kubernetes managed services, such as GKE/EKS/AKS, we can unify the environment and simplify our maintenance operation," says Koizumi.</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/denso/banner1.png"
+  author="SEIICHI KOIZUMI, R&D PRODUCT MANAGER, DIGITAL INNOVATION DEPARTMENT AT DENSO"
+>}}
+"Another disruptive innovation is coming, so to survive in this situation, we need to change our culture."
+{{< /case-studies/quote >}}
+
+{{< case-studies/lead >}}
+Spun off from Toyota in 1949, DENSO Corporation is one of the top automotive suppliers in the world today, with consolidated net revenue of $48.3 billion.
+{{< /case-studies/lead >}}
+
+<p>The company's mission is "contributing to a better world by creating value together with a vision for the future"—and part of that vision in recent years has been development on the vehicle edge and vehicle cloud.</p>
+
+<p>With the advent of connected cars, DENSO established a Digital Innovation Department to expand its business beyond the critical layer of the engine, braking systems, and other automotive parts into the non-critical analytics and entertainment layer. Comparing connected cars to smartphones, R&D Product Manager Seiichi Koizumi says DENSO wants the ability to quickly and easily develop and install apps for the "blank slate" of the car, and iterate them based on the driver's preferences. Thus "we need a flexible application platform," he says.</p>
+
+<p>But working on vehicle edge and vehicle cloud products meant there were several technical challenges: "the amount of computing resources, the occasional lack of mobile signal, and an enormous number of distributed vehicles," says Koizumi. "We are tackling these challenges to create an integrated vehicle edge/cloud platform."</p>
+
+{{< case-studies/quote author="SEIICHI KOIZUMI, R&D PRODUCT MANAGER, DIGITAL INNOVATION DEPARTMENT AT DENSO" >}}
+"We got Kubernetes experts involved on our team, and it dramatically accelerated development speed."
+{{< /case-studies/quote >}}
+
+<p>Koizumi's team realized that because mobility services evolve every day, they needed the flexibility of the cloud native ecosystem for their platform. As they evaluated technologies, they were led by these criteria: Because their service-enabler business needed to support multiple cloud and on-premise environments, the solution needed to be cloud agnostic, with no vendor lock-in and open governance. It also had to support an edge-cloud integrated environment.</p>
+
+<p>After considering other orchestrators, DENSO went with Kubernetes for orchestration and added Prometheus, Fluentd, Envoy, Istio, and Helm to the platform. During implementation, the team used "design thinking to clarify use cases and their value proposition," says Koizumi. Next, an agile development team worked on a POC, then an MVP, in DevOps style. "Even in the development phase, we are keeping a channel to end users," he adds.</p>
+
+<p>One lesson learned during this process was the value of bringing in experts. "We tried to learn Kubernetes and cloud native technologies from scratch, but it took more time than expected," says Koizumi. "We got Kubernetes experts involved on our team, and it dramatically accelerated development speed."</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/denso/banner4.jpg"
+  author="SEIICHI KOIZUMI, R&D PRODUCT MANAGER, DIGITAL INNOVATION DEPARTMENT AT DENSO"
+>}}
+"By utilizing Kubernetes managed services, such as GKE/EKS/AKS, we can unify the environment and simplify our maintenance operation."
+{{< /case-studies/quote >}}
+
+<p>Today, DENSO is using a vehicle edge computer, a private Kubernetes cloud, and managed Kubernetes on GKE, EKS, and AKS. "We are developing a vehicle edge/cloud integrated platform based on a microservice and service mesh architecture," says Koizumi. "We extend cloud into multiple vehicle edges and manage it as a unified platform."</p>
+
+<p>Cloud native has enabled DENSO to deliver applications via its new dash cam, which has a secure connection that collects data to the cloud. "It's like a smartphone," he says. "We are installing new applications and getting the data through the cloud, and we can keep updating new applications all through the dash cam."</p>
+
+<p>The unified cloud native platform, combined with agile development, has had a positive impact on productivity. Critical layer features—those involving engines or braking systems, for example—can take 2-3 years to implement at DENSO, because of the time needed to test safety, but also because of the traditional, waterfall model of development. With the Kubernetes platform and agile methods, there's a 2-month development cycle for non-critical software. Now, ten new applications are released a year, and with the department's scrum-style development, a new prototype is introduced every week.</p>
+
+<p>Application portability has also led to greater developer efficiency. "There's no need to care about differences in the multi-cloud platform anymore," says Koizumi. Now, "we are also trying to have the same portability between vehicle edge and cloud platform."</p>
+
+<p>Another improvement: Automotive Tier-1 suppliers like DENSO always have multiple Tier-2 suppliers. "To provide automotive-grade high-availability services, we tried to do the same thing on a multi-cloud platform," says Koizumi. Before Kubernetes, maintaining two different systems simultaneously was difficult. "By utilizing Kubernetes managed services, such as GKE/EKS/AKS, we can unify the environment and simplify our maintenance operation," he says.</p>
+
+<p>Cloud native has also profoundly changed the culture at DENSO. The Digital Innovation Department is known as "Noah's Ark," and it has grown from 2 members to 70—with plans to more than double in the next year. The way they operate is completely different from the traditional Japanese automotive culture. But just as the company embraced change brought by hybrid cars in the past decade, Koizumi says, they're doing it again now, as technology companies have moved into the connected car space. "Another disruptive innovation is coming," he says, "so to survive in this situation, we need to change our culture."</p>
+
+<p>Looking ahead, Koizumi and his team are expecting serverless and zero-trust security architecture to be important enhancements of Kubernetes. They are glad DENSO has come along for the ride. "Mobility service businesses require agility and flexibility," he says. "DENSO is trying to bring cloud native flexibility into the vehicle infrastructure."</p>
@@ -0,0 +1,78 @@
+---
+title: ING Case Study
+linkTitle: ING
+case_study_styles: true
+cid: caseStudies
+weight: 50
+featured: true
+quote: >
+  The big cloud native promise to our business is the ability to go from idea to production within 48 hours. We are some years away from this, but that's quite feasible to us.
+
+new_case_study_styles: true
+heading_background: /images/case-studies/ing/banner1.jpg
+heading_title_logo: /images/ing_logo.png
+subheading: >
+  Driving Banking Innovation with Cloud Native
+case_study_details:
+  - Company: ING
+  - Location: Amsterdam, Netherlands
+  - Industry: Finance
+---
+
+<h2>Challenge</h2>
+
+<p>After undergoing an agile transformation, <a href="https://www.ing.com/">ING</a> realized it needed a standardized platform to support the work their developers were doing. "Our DevOps teams got empowered to be autonomous," says Infrastructure Architect Thijs Ebbers. "It has benefits; you get all kinds of ideas. But a lot of teams are going to devise the same wheel. Teams started tinkering with <a href="https://www.docker.com/">Docker</a>, Docker Swarm, <a href="https://kubernetes.io/">Kubernetes</a>, <a href="https://mesosphere.com/">Mesos</a>. Well, it's not really useful for a company to have one hundred wheels, instead of one good wheel.</p>
+
+<h2>Solution</h2>
+
+<p>Using Kubernetes for container orchestration and Docker for containerization, the ING team began building an internal public cloud for its CI/CD pipeline and green-field applications. The pipeline, which has been built on Mesos Marathon, will be migrated onto Kubernetes. The bank-account management app <a href="https://www.yolt.com/">Yolt</a> in the U.K. (and soon France and Italy) market already is live hosted on a Kubernetes framework. At least two greenfield projects currently on the Kubernetes framework will be going into production later this year. By the end of 2018, the company plans to have converted a number of APIs used in the banking customer experience to cloud native APIs and host these on the Kubernetes-based platform.</p>
+
+<h2>Impact</h2>
+
+<p>"Cloud native technologies are helping our speed, from getting an application to test to acceptance to production," says Infrastructure Architect Onno Van der Voort. "If you walk around ING now, you see all these DevOps teams, doing stand-ups, demoing. They try to get new functionality out there really fast. We held a hackathon for one of our existing components and basically converted it to cloud native within 2.5 days, though of course the tail takes more time before code is fully production ready."</p>
+
+{{< case-studies/quote author="Thijs Ebbers, Infrastructure Architect, ING">}}
+"The big cloud native promise to our business is the ability to go from idea to production within 48 hours. We are some years away from this, but that's quite feasible to us."
+{{< /case-studies/quote >}}
+
+{{< case-studies/lead >}}
+ING has long embraced innovation in banking, launching the internet-based ING Direct in 1997.
+{{< /case-studies/lead >}}
+
+<p>In that same spirit, the company underwent an agile transformation a few years ago. "Our DevOps teams got empowered to be autonomous," says Infrastructure Architect Thijs Ebbers. "It has benefits; you get all kinds of ideas. But a lot of teams are going to devise the same wheel. Teams started tinkering with Docker, Docker Swarm, Kubernetes, Mesos. Well, it's not really useful for a company to have one hundred wheels, instead of one good wheel."</p>
+
+<p>Looking to standardize the deployment process within the company's strict security guidelines, the team looked at several solutions and found that in the past year, "Kubernetes won the container management framework wars," says Ebbers. "We decided to standardize ING on a Kubernetes framework." Everything is run on premise due to banking regulations, he adds, but "we will be building an internal public cloud. We are trying to get on par with what public clouds are doing. That's one of the reasons we got Kubernetes."</p>
+
+<p>They also embraced Docker to address a major pain point in ING's CI/CD pipeline. Before containerization, "Every development team had to order a VM, and it was quite a heavy delivery model for them," says Infrastructure Architect Onno Van der Voort. "Another use case for containerization is when the application travels through the pipeline, they fire up Docker containers to do test work against the applications and after they've done the work, the containers get killed again."</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/ing/banner3.jpg"
+  author="Thijs Ebbers, Infrastructure Architect, ING"
+>}}
+"We decided to standardize ING on a Kubernetes framework." Everything is run on premise due to banking regulations, he adds, but "we will be building an internal public cloud. We are trying to get on par with what public clouds are doing. That's one of the reasons we got Kubernetes."
+{{< /case-studies/quote >}}
+
+<p>Because of industry regulations, applications are only allowed to go through the pipeline, where compliance is enforced, rather than be deployed directly into a container. "We have to run the complete platform of services we need, many routing from different places," says Van der Voort. "We need this Kubernetes framework for deploying the containers, with all those components, monitoring, logging. It's complex." For that reason, ING has chosen to start on the <a href="https://www.openshift.org/">OpenShift Origin</a> Kubernetes distribution.</p>
+
+<p>Already, "cloud native technologies are helping our speed, from getting an application to test to acceptance to production," says Van der Voort. "If you walk around ING now, you see all these DevOps teams, doing stand-ups, demoing. They try to get new functionality out there really fast. We held a hackathon for one of our existing components and basically converted it to cloud native within 2.5 days, though of course the tail takes more time before code is fully production ready."</p>
+
+<p>The pipeline, which has been built on Mesos Marathon, will be migrated onto Kubernetes. Some legacy applications are also being rewritten as cloud native in order to run on the framework. At least two smaller greenfield projects built on Kubernetes will go into production this year. By the end of 2018, the company plans to have converted a number of APIs used in the banking customer experience to cloud native APIs and host these on the Kubernetes-based platform.</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/ing/banner4.jpg"
+  author="Onno Van der Voort,  Infrastructure Architect, ING"
+>}}
+"We have to run the complete platform of services we need, many routing from different places. We need this Kubernetes framework for deploying the containers, with all those components, monitoring, logging. It's complex."
+{{< /case-studies/quote >}}
+
+<p>The team, however, doesn't see the bank's back-end systems going onto the Kubernetes platform. "Our philosophy is it only makes sense to move things to cloud if they are cloud native," says Van der Voort. "If you have traditional architecture, build traditional patterns, it doesn't hold any value to go to the cloud." Adds Cloud Platform Architect Alfonso Fernandez-Barandiaran: "ING has a strategy about where we will go, in order to improve our agility. So it's not about how cool this technology is, it's about finding the right technology and the right approach."</p>
+
+<p>The Kubernetes framework will be hosting some greenfield projects that are high priority for ING: applications the company is developing in response to <a href="https://ec.europa.eu/info/law/payment-services-psd-2-directive-eu-2015-2366_en">PSD2</a>, the European Commission directive requiring more innovative online and mobile payments that went into effect at the beginning of 2018. For example, a bank-account management app called <a href="https://www.yolt.com/">Yolt</a>, serving the U.K. market (and soon France and Italy), was built on a Kubernetes platform and has gone into production. ING is also developing blockchain-enabled applications that will live on the Kubernetes platform. "We've been contacted by a lot of development teams that have ideas with what they want to do with containers," says Ebbers.</p>
+
+{{< case-studies/quote author="Alfonso Fernandez-Barandiaran, Cloud Platform Architect, ING" >}}
+Even with the particular requirements that come in banking, ING has managed to take a lead in technology and innovation. "Every time we have constraints, we look for maybe a better way that we can use this technology."
+{{< /case-studies/quote >}}
+
+<p>Even with the particular requirements that come in banking, ING has managed to take a lead in technology and innovation. "Every time we have constraints, we look for maybe a better way that we can use this technology," says Fernandez-Barandiaran.</p>
+
+<p>The results, after all, are worth the effort. "The big cloud native promise to our business is the ability to go from idea to production within 48 hours," says Ebbers. "That would require all these projects to be mature. We are some years away from this, but that's quite feasible to us."</p>
@@ -0,0 +1,79 @@
+---
+title: JD.com Case Study
+linkTitle: jd-com
+case_study_styles: true
+cid: caseStudies
+featured: false
+
+new_case_study_styles: true
+heading_background: /images/case-studies/jdcom/banner1.jpg
+heading_title_logo: /images/jdcom_logo.png
+subheading: >
+  JD.com: How JD.com Pioneered Kubernetes for E-Commerce at Hyperscale
+case_study_details:
+  - Company: JD.com
+  - Location: Beijing, China
+  - Industry: eCommerce
+---
+
+<h2>Challenge</h2>
+
+<p>With more than 300 million active users and total 2017 revenue of more than $55 billion, <a href="https://corporate.JD.com/home">JD.com</a> is China's largest retailer, and its operations are the epitome of hyperscale. For example, there are more than a trillion images in JD.com's product databases—with 100 million being added daily—and this enormous amount of data needs to be instantly accessible. In 2014, JD.com moved its applications to containers running on bare metal machines using OpenStack and Docker to "speed up the delivery of our computing resources and make the operations much simpler," says Haifeng Liu, JD.com's Chief Architect. But by the end of 2015, with tens of thousands of nodes running in multiple data centers, "we encountered a lot of problems because our platform was not strong enough, and we suffered from bottlenecks and scalability issues," says Liu. "We needed infrastructure for the next five years of development, now."</p>
+
+<h2>Solution</h2>
+
+<p>JD.com turned to Kubernetes to accommodate its clusters. At the beginning of 2016, the company began to transition from OpenStack to Kubernetes, and today, JD.com runs the world's largest Kubernetes cluster. "Kubernetes has provided a strong foundation on top of which we have customized the solution to suit our needs as China's largest retailer."</p>
+
+<h2>Impact</h2>
+
+<p>"We have greater data center efficiency, better managed resources, and smarter deployment with the Kubernetes platform," says Liu. Deployment time went from several hours to tens of seconds. Efficiency has improved by 20-30%, measured in IT costs. With the further optimizations the team is working on, Liu believes there is the potential to save hundreds of millions of dollars a year. But perhaps the best indication of success was the annual Singles Day shopping event, which ran on the Kubernetes platform for the first time in 2018. Over 11 days, transaction volume on JD.com was $23 billion, and "our e-commerce platforms did great," says Liu. "Infrastructure led the way to prep for 11.11. We took the approach of predicting volume, emulating the behavior of customers to prepare beforehand, and drilled for malfunctions. Because of Kubernetes's scalability, we were able to handle an extremely high level of demand."</p>
+
+{{< case-studies/quote author="HAIFENG LIU, CHIEF ARCHITECT, JD.com" >}}
+"Kubernetes helped us reduce the complexity of operations to make distributed systems stable and scalable. Most importantly, we can leverage Kubernetes for scheduling resources to reduce hardware costs. That's the big win."
+{{< /case-studies/quote >}}
+
+{{< case-studies/lead >}}
+With more than 300 million active users and $55.7 billion in annual revenues last year, JD.com is China's largest retailer, and its operations are the epitome of hyperscale.
+{{< /case-studies/lead >}}
+
+<p>For example, there are more than a trillion images in JD.com's product databases for customers, with 100 million being added daily. And this enormous amount of data needs to be instantly accessible to enable a smooth online customer experience.</p>
+
+<p>In 2014, JD.com moved its applications to containers running on bare metal machines using OpenStack and Docker to "speed up the delivery of our computing resources and make the operations much simpler," says Haifeng Liu, JD.com's Chief Architect. But by the end of 2015, with hundreds of thousands of nodes in multiple data centers, "we encountered a lot of problems because our platform was not strong enough, and we suffered from bottlenecks and scalability issues," Liu adds. "We needed infrastructure for the next five years of development, now."</p>
+
+<p>After considering a number of orchestration technologies, JD.com decided to adopt Kubernetes to accommodate its ever-growing clusters. "The main reason is because Kubernetes can give us more efficient, scalable and much simpler application deployments, plus we can leverage it to do flexible platform scheduling," says Liu.</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/jdcom/banner3.jpg"
+  author="HAIFENG LIU, CHIEF ARCHITECT, JD.com"
+>}}
+"We customized Kubernetes and built a modern system on top of it. This entire ecosystem of Kubernetes plus our own optimizations have helped us save costs and time."
+{{< /case-studies/quote >}}
+
+<p>The fact that Kubernetes is based on Google's Borg also gave the company confidence. The team liked that Kubernetes has a clear and simple architecture, and that it's developed mostly in Go, which is a popular language within JD.com.  Though he felt that at the time Kubernetes "was not mature enough," Liu says, "we adopted it anyway."</p>
+
+<p>The team spent a year developing the new container engine platform based on Kubernetes, and at the end of 2016, began promoting it within the company. "We wanted the cluster to be the default way for creating services, so scalability is easier," says Liu. "We talked to developers, interest grew, and we solved problems together." Some of these problems included networking performance and etcd scalability. "But during the past two years, Kubernetes has become more mature and very stable," he adds.</p>
+
+<p>Today, the company runs the world's largest Kubernetes cluster. "We customized Kubernetes and built a modern system on top of it," says Liu. "This entire ecosystem of Kubernetes plus our own optimizations have helped us save costs and time. We have greater data center efficiency, better managed resources, and smarter deployment with the Kubernetes platform."</p>
+
+{{< case-studies/quote
+  image="/images/case-studies/jdcom/banner4.jpg"
+  author="HAIFENG LIU, CHIEF ARCHITECT, JD.com"
+>}}
+"My advice is first you need to combine this technology with your own businesses, and the second is you need clear goals. You cannot just use the technology because others are using it. You need to consider your own objectives."
+{{< /case-studies/quote >}}
+
+<p>The results are clear: Deployment time went from several hours to tens of seconds. Efficiency has improved by 20-30%, measured in IT costs. But perhaps the best indication of success was the annual <a href="https://JD.comcorporateblog.com/shoppers-snap-up-quality-and-imported-products-on-JD.com-com-for-record-breaking-singles-day-festival/">Singles Day</a> shopping event, which ran on the Kubernetes platform for the first time in 2018. Over 11 days, transaction volume on JD.com was $23 billion, and "our e-commerce platforms did great," says Liu. "Infrastructure led the way to prep for 11.11. We took the approach of predicting volume, emulating the behavior of customers to prepare beforehand, and drilled for malfunctions. Because of Kubernetes's scalability, we were able to handle an extremely high level of demand."</p>
+
+<p>JD.com is now in its second stage with Kubernetes: The platform is already stable, scalable, and flexible, so the focus is on how to run things much more efficiently to further reduce costs. With the optimizations the team is working on with resource management, Liu believes there is the potential to save hundreds of millions of dollars a year.</p>
+
+<p>"We run Kubernetes and container clusters on roughly tens of thousands of physical bare metal nodes," he says. "Using Kubernetes and leveraging our own machine learning pipeline to predict how many resources we need for each application we use, and our own intelligent scaling algorithm, we can improve our resource usage. If we boost the resource usage, for example, by several percent, that means we can reduce huge hardware costs. Then we don't need that many servers to get that same amount of workload. That can save us a lot of resources."</p>
+
+{{< case-studies/quote author="HAIFENG LIU, CHIEF ARCHITECT, JD.com" >}}
+"We can share our successful experience with the community, and we also receive good feedback from others. So it's mutually beneficial."
+{{< /case-studies/quote >}}
+
+<p>JD.com, which won CNCF's 2018 End User Award, is also using <a href="https://helm.sh/">Helm</a>, <a href="https://github.com/containernetworking">CNI</a>, <a href="https://goharbor.io/">Harbor</a>, and <a href="https://vitess.io/">Vitess</a> on its platform. JD.com developers have made considerable contributions to Vitess, the CNCF project for scalable MySQL cluster management, and the company hopes to donate its own project to CNCF in the near future. Community participation is a priority for JD.com. "We have a good partnership with this community," says Liu. "We can share our successful experience with the community, and we also receive good feedback from others. So it's mutually beneficial."</p>
+
+<p>To that end, Liu offers this advice for other companies considering adopting cloud native technology. "First you need to combine this technology with your own businesses, and the second is you need clear goals," he says. "You cannot just use the technology because others are using it. You need to consider your own objectives."</p>
+
+<p>For JD.com's objectives, these cloud native technologies have been an ideal fit with the company's own homegrown innovation. "Kubernetes helped us reduce the complexity of operations to make distributed systems stable and scalable," says Liu. "Most importantly, we can leverage Kubernetes for scheduling resources to reduce hardware costs. That's the big win."</p>
@@ -30,9 +30,9 @@ One or more forms of [authorization](/docs/reference/access-authn-authz/authoriz
 ## 节点到控制面

 Kubernetes 采用的是中心辐射型（Hub-and-Spoke）API 模式。
-所有从集群（或所运行的 Pods）发出的 API 调用都终止于 apiserver。
+所有从集群（或所运行的 Pods）发出的 API 调用都终止于 API 服务器。
 其它控制面组件都没有被设计为可暴露远程服务。
-apiserver 被配置为在一个安全的 HTTPS 端口（通常为 443）上监听远程连接请求，
+API 服务器被配置为在一个安全的 HTTPS 端口（通常为 443）上监听远程连接请求，
 并启用一种或多种形式的客户端[身份认证](/zh/docs/reference/access-authn-authz/authentication/)机制。
 一种或多种客户端[鉴权机制](/zh/docs/reference/access-authn-authz/authorization/)应该被启用，
 特别是在允许使用[匿名请求](/zh/docs/reference/access-authn-authz/authentication/#anonymous-requests)
@@ -41,7 +41,7 @@ apiserver 被配置为在一个安全的 HTTPS 端口（通常为 443）上监
 <!--
 Nodes should be provisioned with the public root certificate for the cluster such that they can connect securely to the apiserver along with valid client credentials. A good approach is that the client credentials provided to the kubelet are in the form of a client certificate. See [kubelet TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) for automated provisioning of kubelet client certificates.
 -->
-应该使用集群的公共根证书开通节点，这样它们就能够基于有效的客户端凭据安全地连接 apiserver。
+应该使用集群的公共根证书开通节点，这样它们就能够基于有效的客户端凭据安全地连接 API 服务器。
 一种好的方法是以客户端证书的形式将客户端凭据提供给 kubelet。
 请查看 [kubelet TLS 启动引导](/zh/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/)
 以了解如何自动提供 kubelet 客户端证书。
@@ -52,12 +52,12 @@ The `kubernetes` service (in `default` namespace) is configured with a virtual I

 The control plane components also communicate with the cluster apiserver over the secure port.
 -->
-想要连接到 apiserver 的 Pod 可以使用服务账号安全地进行连接。
+想要连接到 API 服务器的 Pod 可以使用服务账号安全地进行连接。
 当 Pod 被实例化时，Kubernetes 自动把公共根证书和一个有效的持有者令牌注入到 Pod 里。
 `kubernetes` 服务（位于 `default` 名字空间中）配置了一个虚拟 IP 地址，用于（通过 kube-proxy）转发
-请求到 apiserver 的 HTTPS 末端。
+请求到 API 服务器的 HTTPS 末端。

-控制面组件也通过安全端口与集群的 apiserver 通信。
+控制面组件也通过安全端口与集群的 API 服务器通信。

 <!--
 As a result, the default operating mode for connections from the nodes and pods running on the nodes to the control plane is secured by default and can run over untrusted and/or public networks.
@@ -72,9 +72,9 @@ There are two primary communication paths from the control plane (apiserver) to
 -->
 ## 控制面到节点

-从控制面（apiserver）到节点有两种主要的通信路径。
-第一种是从 apiserver 到集群中每个节点上运行的 kubelet 进程。
-第二种是从 apiserver 通过它的代理功能连接到任何节点、Pod 或者服务。
+从控制面（API 服务器）到节点有两种主要的通信路径。
+第一种是从 API 服务器到集群中每个节点上运行的 kubelet 进程。
+第二种是从 API 服务器通过它的代理功能连接到任何节点、Pod 或者服务。

 <!--
 ### apiserver to kubelet
@@ -89,14 +89,14 @@ These connections terminate at the kubelet's HTTPS endpoint. By default, the api
 -->
 ### API 服务器到 kubelet

-从 apiserver 到 kubelet 的连接用于：
+从 API 服务器到 kubelet 的连接用于：

 * 获取 Pod 日志
 * 挂接（通过 kubectl）到运行中的 Pod
 * 提供 kubelet 的端口转发功能。

 这些连接终止于 kubelet 的 HTTPS 末端。
-默认情况下，apiserver 不检查 kubelet 的服务证书。这使得此类连接容易受到中间人攻击，
+默认情况下，API 服务器不检查 kubelet 的服务证书。这使得此类连接容易受到中间人攻击，
 在非受信网络或公开网络上运行也是 **不安全的**。

 <!--
@@ -107,10 +107,10 @@ untrusted or public network.

 Finally, [Kubelet authentication and/or authorization](/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/) should be enabled to secure the kubelet API.
 -->
-为了对这个连接进行认证，使用 `--kubelet-certificate-authority` 标志给 apiserver 
-提供一个根证书包，用于 kubelet 的服务证书。
+为了对这个连接进行认证，使用 `--kubelet-certificate-authority` 标志给 API
+服务器提供一个根证书包，用于 kubelet 的服务证书。

-如果无法实现这点，又要求避免在非受信网络或公共网络上进行连接，可在 apiserver 和
+如果无法实现这点，又要求避免在非受信网络或公共网络上进行连接，可在 API 服务器和
 kubelet 之间使用 [SSH 隧道](#ssh-tunnels)。

 最后，应该启用
@@ -122,9 +122,9 @@ kubelet 之间使用 [SSH 隧道](#ssh-tunnels)。

 The connections from the apiserver to a node, pod, or service default to plain HTTP connections and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS connection by prefixing `https:` to the node, pod, or service name in the API URL, but they will not validate the certificate provided by the HTTPS endpoint nor provide client credentials so while the connection will be encrypted, it will not provide any guarantees of integrity. These connections **are not currently safe** to run over untrusted and/or public networks.
 -->
-### apiserver 到节点、Pod 和服务
+### API 服务器到节点、Pod 和服务

-从 apiserver 到节点、Pod 或服务的连接默认为纯 HTTP 方式，因此既没有认证，也没有加密。
+从 API 服务器到节点、Pod 或服务的连接默认为纯 HTTP 方式，因此既没有认证，也没有加密。
 这些连接可通过给 API URL 中的节点、Pod 或服务名称添加前缀 `https:` 来运行在安全的 HTTPS 连接上。
 不过这些连接既不会验证 HTTPS 末端提供的证书，也不会提供客户端证书。
 因此，虽然连接是加密的，仍无法提供任何完整性保证。
@@ -140,8 +140,8 @@ SSH tunnels are currently deprecated so you shouldn't opt to use them unless you
 -->
 ### SSH 隧道 {#ssh-tunnels}

-Kubernetes 支持使用 SSH 隧道来保护从控制面到节点的通信路径。在这种配置下，apiserver
-建立一个到集群中各节点的 SSH 隧道（连接到在 22 端口监听的 SSH 服务）
+Kubernetes 支持使用 SSH 隧道来保护从控制面到节点的通信路径。在这种配置下，API
+服务器建立一个到集群中各节点的 SSH 隧道（连接到在 22 端口监听的 SSH 服务）
 并通过这个隧道传输所有到 kubelet、节点、Pod 或服务的请求。
 这一隧道保证通信不会被暴露到集群节点所运行的网络之外。

@@ -63,7 +63,7 @@ valid. For example, if you try to create a Node from the following JSON manifest
 1. 节点上的 `kubelet` 向控制面执行自注册；
 2. 你，或者别的什么人，手动添加一个 Node 对象。

-在你创建了 Node {{< glossary_tooltip text="object" term_id="object" >}}或者节点上的
+在你创建了 Node {{< glossary_tooltip text="对象" term_id="object" >}}或者节点上的
 `kubelet` 执行了自注册操作之后，控制面会检查新的 Node 对象是否合法。
 例如，如果你尝试使用下面的 JSON 对象来创建 Node 对象：

@@ -100,10 +100,9 @@ it becomes healthy.
 You, or a {{< glossary_tooltip term_id="controller" text="controller">}}, must explicitly
 delete the Node object to stop that health checking.
 -->
-Kubernetes 会一直保存着非法节点对应的对象，并持续检查该节点是否已经
-变得健康。
-你，或者某个{{< glossary_tooltip term_id="controller" text="控制器">}}必需显式地
-删除该 Node 对象以停止健康检查操作。
+Kubernetes 会一直保存着非法节点对应的对象，并持续检查该节点是否已经变得健康。
+你，或者某个{{< glossary_tooltip term_id="controller" text="控制器">}}必须显式地删除该
+Node 对象以停止健康检查操作。
 {{< /note >}}

 <!--
@@ -127,7 +126,7 @@ first and re-added after the update.
 -->
 ### 节点名称唯一性     {#node-name-uniqueness}

-节点的[名称](/docs/concepts/overview/working-with-objects/names#names)用来标识 Node 对象。
+节点的[名称](/zh/docs/concepts/overview/working-with-objects/names#names)用来标识 Node 对象。
 没有两个 Node 可以同时使用相同的名称。 Kubernetes 还假定名字相同的资源是同一个对象。
 就 Node 而言，隐式假定使用相同名称的实例会具有相同的状态（例如网络配置、根磁盘内容）
 和类似节点标签这类属性。这可能在节点被更改但其名称未变时导致系统状态不一致。
@@ -177,8 +176,8 @@ When the [Node authorization mode](/docs/reference/access-authn-authz/node/) and
 kubelets are only authorized to create/modify their own Node resource.
 -->
 启用[Node 鉴权模式](/zh/docs/reference/access-authn-authz/node/)和
-[NodeRestriction 准入插件](/zh/docs/reference/access-authn-authz/admission-controllers/#noderestriction)
-时，仅授权 `kubelet` 创建或修改其自己的节点资源。
+[NodeRestriction 准入插件](/zh/docs/reference/access-authn-authz/admission-controllers/#noderestriction)时，
+仅授权 `kubelet` 创建或修改其自己的节点资源。

 {{< note >}}
 <!--
@@ -257,7 +256,7 @@ kubectl cordon $NODENAME
 See [Safely Drain a Node](/docs/tasks/administer-cluster/safely-drain-node/)
 for more details.
 -->
-更多细节参考[安全腾空节点](/zh/docs/tasks/administer-cluster/safely-drain-node/)。
+更多细节参考[安全地腾空节点](/zh/docs/tasks/administer-cluster/safely-drain-node/)。

 {{< note >}}
 <!--
@@ -285,10 +284,10 @@ A node's status contains the following information:

 一个节点的状态包含以下信息:

-* [地址](#addresses)
-* [状况](#condition)
-* [容量与可分配](#capacity)
-* [信息](#info)
+* [地址（Addresses）](#addresses)
+* [状况（Condition）](#condition)
+* [容量与可分配（Capacity）](#capacity)
+* [信息（Info）](#info)

 <!--
 You can use `kubectl` to view a Node's status and other details:
@@ -327,7 +326,7 @@ The `conditions` field describes the status of all `Running` nodes. Examples of
 -->
 ### 状况 {#condition}

-`conditions` 字段描述了所有 `Running` 节点的状态。状况的示例包括：
+`conditions` 字段描述了所有 `Running` 节点的状况。状况的示例包括：

 <!--
 {{< table caption = "Node conditions, and a description of when each condition applies." >}}
@@ -393,7 +392,7 @@ for all Pods assigned to that node. The default eviction timeout duration is
 如果 Ready 状况的 `status` 处于 `Unknown` 或者 `False` 状态的时间超过了
 `pod-eviction-timeout` 值（一个传递给
 {{< glossary_tooltip text="kube-controller-manager" term_id="kube-controller-manager" >}}
-的参数），[节点控制器](#node-controller) 会对节点上的所有 Pod 触发
+的参数），[节点控制器](#node-controller)会对节点上的所有 Pod 触发
 {{< glossary_tooltip text="API-发起的驱逐" term_id="api-eviction" >}}。
 默认的逐出超时时长为 **5 分钟**。

@@ -417,7 +416,7 @@ all the Pod objects running on the node to be deleted from the API server, and f
 names.
 -->
 节点控制器在确认 Pod 在集群中已经停止运行前，不会强制删除它们。
-你可以看到这些可能在无法访问的节点上运行的 Pod 处于 `Terminating` 或者 `Unknown` 状态。
+你可以看到可能在这些无法访问的节点上运行的 Pod 处于 `Terminating` 或者 `Unknown` 状态。
 如果 kubernetes 不能基于下层基础设施推断出某节点是否已经永久离开了集群，
 集群管理员可能需要手动删除该节点对象。
 从 Kubernetes 删除节点对象将导致 API 服务器删除节点上所有运行的 Pod 对象并释放它们的名字。
@@ -480,7 +479,7 @@ the Kubernetes API.
 ### 信息（Info） {#info}

 Info 指的是节点的一般信息，如内核版本、Kubernetes 版本（`kubelet` 和 `kube-proxy` 版本）、
-容器运行时详细信息，以及 节点使用的操作系统。
+容器运行时详细信息，以及节点使用的操作系统。
 `kubelet` 从节点收集这些信息并将其发布到 Kubernetes API。

 <!--
@@ -492,6 +491,7 @@ availability of each node, and to take action when failures are detected.
 For nodes there are two forms of heartbeats:
 -->
 ## 心跳  {#heartbeats}
+
 Kubernetes 节点发送的心跳帮助你的集群确定每个节点的可用性，并在检测到故障时采取行动。

 对于节点，有两种形式的心跳:
@@ -504,7 +504,7 @@ Kubernetes 节点发送的心跳帮助你的集群确定每个节点的可用性
  Each Node has an associated Lease object.
 -->
 * 更新节点的 `.status`
-* `kube-node-lease` {{<glossary_tooltip term_id="namespace" text="命名空间">}}中的
+* `kube-node-lease` {{<glossary_tooltip term_id="namespace" text="名字空间">}}中的
  [Lease（租约）](/docs/reference/kubernetes-api/cluster-resources/lease-v1/)对象。
  每个节点都有一个关联的 Lease 对象。

@@ -570,26 +570,29 @@ controller deletes the node from its list of nodes.
 The third is monitoring the nodes' health. The node controller is
 responsible for:

- In the case that a node becomes unreachable, updating the NodeReady condition
-  of within the Node's `.status`. In this case the node controller sets the
-  NodeReady condition to `ConditionUnknown`.
+- In the case that a node becomes unreachable, updating the `Ready` condition
+  in the Node's `.status` field. In this case the node controller sets the
+  `Ready` condition to `Unknown`.
 - If a node remains unreachable: triggering
  [API-initiated eviction](/docs/concepts/scheduling-eviction/api-eviction/)
  for all of the Pods on the unreachable node. By default, the node controller
-  waits 5 minutes between marking the node as `ConditionUnknown` and submitting
+  waits 5 minutes between marking the node as `Unknown` and submitting
  the first eviction request.

-The node controller checks the state of each node every `-node-monitor-period` seconds.
+By default, the node controller checks the state of each node every 5 seconds.
+This period can be configured using the `--node-monitor-period` flag on the
+`kube-controller-manager` component.
 -->
 第三个是监控节点的健康状况。节点控制器负责：

- 在节点不可达的情况下，在 Node 的 `.status` 中更新 NodeReady 状况。
+- 在节点不可达的情况下，在 Node 的 `.status` 中更新 `Ready` 状况。
  在这种情况下，节点控制器将 NodeReady 状况更新为 `Unknown` 。
 - 如果节点仍然无法访问：对于不可达节点上的所有 Pod 触发
-  [API-发起的逐出](/zh/docs/concepts/scheduling-eviction/api-eviction/)。
+  [API 发起的逐出](/zh/docs/concepts/scheduling-eviction/api-eviction/)操作。
  默认情况下，节点控制器在将节点标记为 `Unknown` 后等待 5 分钟提交第一个驱逐请求。

-节点控制器每隔 `--node-monitor-period` 秒检查每个节点的状态。
+默认情况下，节点控制器每 5 秒检查一次节点状态，可以使用 `kube-controller-manager`
+组件上的 `--node-monitor-period` 参数来配置周期。

 <!--
 ### Rate limits on eviction
@@ -606,11 +609,11 @@ from more than 1 node per 10 seconds.
 <!--
 The node eviction behavior changes when a node in a given availability zone
 becomes unhealthy. The node controller checks what percentage of nodes in the zone
-are unhealthy (NodeReady condition is `ConditionUnknown` or `ConditionFalse`) at
+are unhealthy (the `Ready` condition is `Unknown` or `False`) at
 the same time:
 -->
 当一个可用区域（Availability Zone）中的节点变为不健康时，节点的驱逐行为将发生改变。
-节点控制器会同时检查可用区域中不健康（NodeReady 状况为 `Unknown` 或 `False`）
+节点控制器会同时检查可用区域中不健康（`Ready` 状况为 `Unknown` 或 `False`）
 的节点的百分比：

 <!--
@@ -713,7 +716,7 @@ If you want to explicitly reserve resources for non-Pod processes, follow this t
 -->
 ## 节点拓扑  {#node-topology}

-{{< feature-state state="alpha" for_k8s_version="v1.16" >}}
+{{< feature-state state="beta" for_k8s_version="v1.18" >}}

 <!--
 If you have enabled the `TopologyManager`
@@ -766,11 +769,11 @@ enabled by default in 1.21.
 <!--
 Note that by default, both configuration options described below,
 `ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are set to zero,
-thus not activating Graceful node shutdown functionality.
+thus not activating the graceful node shutdown functionality.
 To activate the feature, the two kubelet config settings should be configured appropriately and set to non-zero values.
 -->
-注意，默认情况下，下面描述的两个配置选项，`ShutdownGracePeriod` 和
-`ShutdownGracePeriodCriticalPods` 都是被设置为 0 的，因此不会激活体面节点关闭功能。
+注意，默认情况下，下面描述的两个配置选项，`shutdownGracePeriod` 和
+`shutdownGracePeriodCriticalPods` 都是被设置为 0 的，因此不会激活体面节点关闭功能。
 要激活此功能特性，这两个 kubelet 配置选项要适当配置，并设置为非零值。

 <!-- 
@@ -794,12 +797,12 @@ Graceful Node Shutdown feature is configured with two [`KubeletConfiguration`](/
 节点体面关闭的特性对应两个
 [`KubeletConfiguration`](/zh/docs/tasks/administer-cluster/kubelet-config-file/) 选项：

-* `ShutdownGracePeriod`：
+* `shutdownGracePeriod`：
  * 指定节点应延迟关闭的总持续时间。此时间是 Pod 体面终止的时间总和，不区分常规 Pod
    还是[关键 Pod](/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical)。
-* `ShutdownGracePeriodCriticalPods`：
+* `shutdownGracePeriodCriticalPods`：
  * 在节点关闭期间指定用于终止[关键 Pod](/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical)
-    的持续时间。该值应小于 `ShutdownGracePeriod`。
+    的持续时间。该值应小于 `shutdownGracePeriod`。

 <!--  
 For example, if `ShutdownGracePeriod=30s`, and
@@ -808,7 +811,7 @@ For example, if `ShutdownGracePeriod=30s`, and
 for gracefully terminating normal pods, and the last 10 seconds would be
 reserved for terminating [critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical).
 -->
-例如，如果设置了 `ShutdownGracePeriod=30s` 和 `ShutdownGracePeriodCriticalPods=10s`，
+例如，如果设置了 `shutdownGracePeriod=30s` 和 `shutdownGracePeriodCriticalPods=10s`，
 则 kubelet 将延迟 30 秒关闭节点。
 在关闭期间，将保留前 20（30 - 10）秒用于体面终止常规 Pod，
 而保留最后 10 秒用于终止[关键 Pod](/zh/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical)。
@@ -817,11 +820,6 @@ reserved for terminating [critical pods](/docs/tasks/administer-cluster/guarante
 When pods were evicted during the graceful node shutdown, they are marked as failed.
 Running `kubectl get pods` shows the status of the the evicted pods as `Shutdown`.
 And `kubectl describe pod` indicates that the pod was evicted because of node shutdown:
-
-```
-Reason:         Terminated
-Message:        Pod was terminated in response to imminent node shutdown.
-```
 -->
 {{< note >}}
 当 Pod 在正常节点关闭期间被驱逐时，它们会被标记为已经失败（Failed）。
@@ -970,8 +968,8 @@ containing the pod priority class values and their respective shutdown periods.
 -->
 如果此功能特性被启用，但没有提供配置数据，则不会出现排序操作。

-使用此功能特性需要启用 `GracefulNodeShutdownBasedOnPodPriority` 功能特性，
-并将 kubelet 配置中的 `ShutdownGracePeriodByPodPriority` 设置为期望的配置，
+使用此功能特性需要启用 `GracefulNodeShutdownBasedOnPodPriority` 特性门控，
+并将 kubelet 配置中的 `shutdownGracePeriodByPodPriority` 设置为期望的配置，
 其中包含 Pod 的优先级类数值以及对应的关闭期限。

 <!--
@@ -1005,8 +1003,7 @@ must be set to false.
 -->
 要在节点上启用交换内存，必须启用kubelet 的 `NodeSwap` 特性门控，
 同时使用 `--fail-swap-on` 命令行参数或者将 `failSwapOn`
-[配置](/zh/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
-设置为 false。
+[配置](/zh/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)设置为 false。

 <!--
 A user can also optionally configure `memorySwap.swapBehavior` in order to
@@ -98,7 +98,7 @@ Before choosing a guide, here are some considerations:
 * [Authorization](/docs/reference/access-authn-authz/authorization/) is separate from authentication, and controls how HTTP calls are handled.
 * [Using Admission Controllers](/docs/reference/access-authn-authz/admission-controllers/) explains plug-ins which intercepts requests to the Kubernetes API server after authentication and authorization.
 * [Using Sysctls in a Kubernetes Cluster](/docs/concepts/cluster-administration/sysctl-cluster/) describes to an administrator how to use the `sysctl` command-line tool to set kernel parameters .
-* [Auditing](/docs/tasks/debug-application-cluster/audit/) describes how to interact with Kubernetes' audit logs.
+* [Auditing](/docs/tasks/debug/debug-cluster/audit/) describes how to interact with Kubernetes' audit logs.
 -->
 ## 保护集群  {#securing-a-cluster}

@@ -116,7 +116,7 @@ Before choosing a guide, here are some considerations:
  阐述了在认证和授权之后拦截到 Kubernetes API 服务的请求的插件。
 * [在 Kubernetes 集群中使用 Sysctls](/zh/docs/tasks/administer-cluster/sysctl-cluster/)
  描述了管理员如何使用 `sysctl` 命令行工具来设置内核参数。
-* [审计](/zh/docs/tasks/debug-application-cluster/audit/)
+* [审计](/zh/docs/tasks/debug/debug-cluster/audit/)
  描述了如何与 Kubernetes 的审计日志交互。

 <!--
@@ -411,9 +411,9 @@ see the [Troubleshooting](#troubleshooting) section.
 The kubelet reports the resource usage of a Pod as part of the Pod
 [`status`](/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status).

-If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
+If optional [tools for monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)
 are available in your cluster, then Pod resource usage can be retrieved either
-from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#the-metrics-api)
+from the [Metrics API](/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api)
 directly or from your monitoring tools.
 -->
 ## 监控计算和内存资源用量
@@ -422,8 +422,8 @@ kubelet 会将 Pod 的资源使用情况作为 Pod
 [`status`](/zh/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status)
 的一部分来报告的。

-如果为集群配置了可选的[监控工具](/zh/docs/tasks/debug-application-cluster/resource-usage-monitoring/)，
-则可以直接从[指标 API](/zh/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#the-metrics-api) 
+如果为集群配置了可选的[监控工具](/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)，
+则可以直接从[指标 API](/zh/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#metrics-api) 
 或者监控工具获得 Pod 的资源使用情况。

 <!--
@@ -210,12 +210,12 @@ Kubernetes 启动的容器自动将此 DNS 服务器包含在其 DNS 搜索列
 <!--
 ### Container Resource Monitoring

-[Container Resource Monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) records generic time-series metrics
+[Container Resource Monitoring](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) records generic time-series metrics
 about containers in a central database, and provides a UI for browsing that data.
 -->
 ### 容器资源监控

-[容器资源监控](/zh/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
+[容器资源监控](/zh/docs/tasks/debug/debug-cluster/resource-usage-monitoring/)
 将关于容器的一些常见的时间序列度量值保存到一个集中的数据库中，并提供用于浏览这些数据的界面。

 <!--
@@ -241,4 +241,3 @@ saving container logs to a central log store with search/browsing interface.
 * 进一步了解[控制器](/zh/docs/concepts/architecture/controller/)
 * 进一步了解 [kube-scheduler](/zh/docs/concepts/scheduling-eviction/kube-scheduler/)
 * 阅读 etcd 官方[文档](https://etcd.io/docs/)
-
@@ -83,7 +83,7 @@ metadata:
  name: standard
 provisioner: kubernetes.io/aws-ebs
 parameters:
-  type: gp3
+  type: gp2
 reclaimPolicy: Retain
 allowVolumeExpansion: true
 mountOptions:
@@ -143,7 +143,7 @@ the specification. Some external provisioners are listed under the repository
 -->
 你不限于指定此处列出的 "内置" 制备器（其名称前缀为 "kubernetes.io" 并打包在 Kubernetes 中）。
 你还可以运行和指定外部制备器，这些独立的程序遵循由 Kubernetes 定义的
-[规范](https://git.k8s.io/community/contributors/design-proposals/storage/volume-provisioning.md)。
+[规范](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/volume-provisioning.md)。
 外部供应商的作者完全可以自由决定他们的代码保存于何处、打包方式、运行方式、使用的插件（包括 Flex）等。
 代码仓库 [kubernetes-sigs/sig-storage-lib-external-provisioner](https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner)
 包含一个用于为外部制备器编写功能实现的类库。你可以访问代码仓库
@@ -431,8 +431,9 @@ parameters:

 <!--
 * `type`: `io1`, `gp2`, `gp2`, `sc1`, `st1`. See
+* `type`: `io1`, `gp2`, `sc1`, `st1`. See
  [AWS docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)
-  for details. Default: `gp3`.
+  for details. Default: `gp2`.
 * `zone` (Deprecated): AWS zone. If neither `zone` nor `zones` is specified, volumes are
  generally round-robin-ed across all active zones where Kubernetes cluster
  has a node. `zone` and `zones` parameters must not be used at the same time.
@@ -453,8 +454,8 @@ parameters:
  encrypting the volume. If none is supplied but `encrypted` is true, a key is
  generated by AWS. See AWS docs for valid ARN value.
 -->
-* `type`：`io1`，`gp2`，`gp3`，`sc1`，`st1`。详细信息参见
-  [AWS 文档](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)。默认值：`gp3`。
+* `type`：`io1`，`gp2`，`sc1`，`st1`。详细信息参见
+  [AWS 文档](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)。默认值：`gp2`。
 * `zone`(弃用)：AWS 区域。如果没有指定 `zone` 和 `zones`，
  通常卷会在 Kubernetes 集群节点所在的活动区域中轮询调度分配。
  `zone` 和 `zones` 参数不能同时使用。
@@ -92,6 +92,37 @@ Pod 的共享上下文包括一组 Linux 名字空间、控制组（cgroup）和
 <!--
 ## Using Pods

+The following is an example of a Pod which consists of a container running the image `nginx:1.14.2`.
+
+{{< codenew file="pods/simple-pod.yaml" >}}
+
+To create the Pod shown above, run the following command:
+-->
+## 使用 Pod   {#using-pods}
+
+下面是一个 Pod 示例，它由一个运行镜像 `nginx:1.14.2` 的容器组成。
+
+{{< codenew file="pods/simple-pod.yaml" >}}
+
+要创建上面显示的 Pod，请运行以下命令：
+
+```shell
+kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml
+```
+
+<!--
+Pods are generally not created directly and are created using workload resources.
+See [Working with Pods](#working-with-pods) for more information on how Pods are used
+with workload resources.
+
+### Workload resources for managing pods
+-->
+Pod 通常不是直接创建的，而是使用工作负载资源创建的。
+有关如何将 Pod 用于工作负载资源的更多信息，请参阅 [使用 Pod](#working-with-pods)。
+
+### 用于管理 pod 的工作负载资源
+
+<!--
 Usually you don't need to create Pods directly, even singleton Pods. 
 Instead, create them using workload resources such as {{< glossary_tooltip text="Deployment"
 term_id="deployment" >}} or {{< glossary_tooltip text="Job" term_id="job" >}}.
@@ -100,8 +131,6 @@ If your Pods need to track state, consider the

 Pods in a Kubernetes cluster are used in two main ways:
 -->
-## 使用 Pod   {#using-pods}
-
 通常你不需要直接创建 Pod，甚至单实例 Pod。
 相反，你会使用诸如
 {{< glossary_tooltip text="Deployment" term_id="deployment" >}} 或
@@ -187,9 +216,9 @@ that updates those files from a remote source, as in the following diagram:
 -->

 例如，你可能有一个容器，为共享卷中的文件提供 Web 服务器支持，以及一个单独的
-“sidecar（挂斗）”容器负责从远端更新这些文件，如下图所示：
+"边车 (sidercar)" 容器负责从远端更新这些文件，如下图所示：

-{{< figure src="/images/docs/pod.svg" alt="example pod diagram" width="50%" >}}
+{{< figure src="/images/docs/pod.svg" alt="Pod creation diagram" class="diagram-medium" >}}

 <!--
 Some Pods have {{< glossary_tooltip text="init containers" term_id="init-container" >}}
@@ -287,7 +316,7 @@ _Pod 模板（Pod Template）_ 来替你创建 Pod 并管理它们。
 Pod 模板是包含在工作负载对象中的规范，用来创建 Pod。这类负载资源包括
 [Deployment](/zh/docs/concepts/workloads/controllers/deployment/)、
 [Job](/zh/docs/concepts/workloads/controllers/job/) 和
-[DaemonSets](/zh/docs/concepts/workloads/controllers/daemonset/)等。
+[DaemonSets](/zh/docs/concepts/workloads/controllers/daemonset/) 等。

 <!--
 Each controller for a workload resource uses the `PodTemplate` inside the workload
@@ -314,7 +343,7 @@ spec:
    spec:
      containers:
      - name: hello
-        image: busybox
+        image: busybox:1.28
        command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
      restartPolicy: OnFailure
    # 以上为 Pod 模版
@@ -345,7 +374,7 @@ details are abstracted away. That abstraction and separation of concerns simplif
 system semantics, and makes it feasible to extend the cluster's behavior without
 changing existing code.
 -->
-在节点上，{{< glossary_tooltip term_id="kubelet" text="kubelet" >}}并不直接监测
+在节点上，{{< glossary_tooltip term_id="kubelet" text="kubelet" >}} 并不直接监测
 或管理与 Pod 模版相关的细节或模版的更新，这些细节都被抽象出来。
 这种抽象和关注点分离简化了整个系统的语义，并且使得用户可以在不改变现有代码的
 前提下就能扩展集群的行为。
@@ -412,7 +441,7 @@ Kubernetes 并不禁止你直接管理 Pod。对运行中的 Pod 的某些字段
 Pods enable data sharing and communication among their constituent
 containters.
 -->
-### 资源共享和通信    {#resource-sharing-and-communication}
+### 资源共享和通信 {#resource-sharing-and-communication}

 Pod 使它的成员容器间能够进行数据共享和通信。

@@ -459,14 +488,13 @@ can find each other via `localhost`. The containers in a Pod can also communicat
 with each other using standard inter-process communications like SystemV semaphores
 or POSIX shared memory.  Containers in different Pods have distinct IP addresses
 and can not communicate by IPC without
-[special configuration](/docs/concepts/policy/pod-security-policy/).
+and can not communicate by OS-level IPC without special configuration.
 Containers that want to interact with a container running in a different Pod can
 use IP networking to communicate.
 -->
 在同一个 Pod 内，所有容器共享一个 IP 地址和端口空间，并且可以通过 `localhost` 发现对方。
 他们也能通过如 SystemV 信号量或 POSIX 共享内存这类标准的进程间通信方式互相通信。
-不同 Pod 中的容器的 IP 地址互不相同，没有
-[特殊配置](/zh/docs/concepts/policy/pod-security-policy/) 就不能使用 IPC 进行通信。
+不同 Pod 中的容器的 IP 地址互不相同，没有特殊配置，无法通过 OS 级 IPC 进行通信就不能使用 IPC 进行通信。
 如果某容器希望与运行于其他 Pod 中的容器通信，可以通过 IP 联网的方式实现。

 <!--
@@ -129,6 +129,6 @@ you can view processes in other containers.

 {{% heading "whatsnext" %}}
 <!--
-* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container).
+* Learn how to [debug pods using ephemeral containers](/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container).
 -->
-* 了解如何[使用临时调试容器来进行调试](/zh/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container)
+* 了解如何[使用临时调试容器来进行调试](/zh/docs/tasks/debug/debug-application/debug-running-pod/#ephemeral-container)
@@ -860,7 +860,7 @@ K8s components to start a container.

 图 8 给出的是一个 Mermaid 时序图，展示启动容器时 K8s 组件间的控制流。

-{{< figure src="/docs/images/diagram-guide-example-3.svg" alt="K8s system flow diagram" class="diagram-large" caption="Figure 8. K8s system flow diagram" link="https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiJSV7aW5pdDp7XCJ0aGVtZVwiOlwibmV1dHJhbFwifX0lJVxuc2VxdWVuY2VEaWFncmFtXG4gICAgYWN0b3IgbWVcbiAgICBwYXJ0aWNpcGFudCBhcGlTcnYgYXMgY29udHJvbCBwbGFuZTxicj48YnI-YXBpLXNlcnZlclxuICAgIHBhcnRpY2lwYW50IGV0Y2QgYXMgY29udHJvbCBwbGFuZTxicj48YnI-ZXRjZCBkYXRhc3RvcmVcbiAgICBwYXJ0aWNpcGFudCBjbnRybE1nciBhcyBjb250cm9sIHBsYW5lPGJyPjxicj5jb250cm9sbGVyPGJyPm1hbmFnZXJcbiAgICBwYXJ0aWNpcGFudCBzY2hlZCBhcyBjb250cm9sIHBsYW5lPGJyPjxicj5zY2hlZHVsZXJcbiAgICBwYXJ0aWNpcGFudCBrdWJlbGV0IGFzIG5vZGU8YnI-PGJyPmt1YmVsZXRcbiAgICBwYXJ0aWNpcGFudCBjb250YWluZXIgYXMgbm9kZTxicj48YnI-Y29udGFpbmVyPGJyPnJ1bnRpbWVcbiAgICBtZS0-PmFwaVNydjogMS4ga3ViZWN0bCBjcmVhdGUgLWYgcG9kLnlhbWxcbiAgICBhcGlTcnYtLT4-ZXRjZDogMi4gc2F2ZSBuZXcgc3RhdGVcbiAgICBjbnRybE1nci0-PmFwaVNydjogMy4gY2hlY2sgZm9yIGNoYW5nZXNcbiAgICBzY2hlZC0-PmFwaVNydjogNC4gd2F0Y2ggZm9yIHVuYXNzaWduZWQgcG9kcyhzKVxuICAgIGFwaVNydi0-PnNjaGVkOiA1LiBub3RpZnkgYWJvdXQgcG9kIHcgbm9kZW5hbWU9XCIgXCJcbiAgICBzY2hlZC0-PmFwaVNydjogNi4gYXNzaWduIHBvZCB0byBub2RlXG4gICAgYXBpU3J2LS0-PmV0Y2Q6IDcuIHNhdmUgbmV3IHN0YXRlXG4gICAga3ViZWxldC0-PmFwaVNydjogOC4gbG9vayBmb3IgbmV3bHkgYXNzaWduZWQgcG9kKHMpXG4gICAgYXBpU3J2LT4-a3ViZWxldDogOS4gYmluZCBwb2QgdG8gbm9kZVxuICAgIGt1YmVsZXQtPj5jb250YWluZXI6IDEwLiBzdGFydCBjb250YWluZXJcbiAgICBrdWJlbGV0LT4-YXBpU3J2OiAxMS4gdXBkYXRlIHBvZCBzdGF0dXNcbiAgICBhcGlTcnYtLT4-ZXRjZDogMTIuIHNhdmUgbmV3IHN0YXRlIiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjp0cnVlfQ" >}}
+{{< figure src="/zh/docs/images/diagram-guide-example-3.svg" alt="K8s system flow diagram" class="diagram-large" caption="Figure 8. K8s system flow diagram" link="https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiJSV7aW5pdDp7XCJ0aGVtZVwiOlwibmV1dHJhbFwifX0lJVxuc2VxdWVuY2VEaWFncmFtXG4gICAgYWN0b3IgbWVcbiAgICBwYXJ0aWNpcGFudCBhcGlTcnYgYXMgY29udHJvbCBwbGFuZTxicj48YnI-YXBpLXNlcnZlclxuICAgIHBhcnRpY2lwYW50IGV0Y2QgYXMgY29udHJvbCBwbGFuZTxicj48YnI-ZXRjZCBkYXRhc3RvcmVcbiAgICBwYXJ0aWNpcGFudCBjbnRybE1nciBhcyBjb250cm9sIHBsYW5lPGJyPjxicj5jb250cm9sbGVyPGJyPm1hbmFnZXJcbiAgICBwYXJ0aWNpcGFudCBzY2hlZCBhcyBjb250cm9sIHBsYW5lPGJyPjxicj5zY2hlZHVsZXJcbiAgICBwYXJ0aWNpcGFudCBrdWJlbGV0IGFzIG5vZGU8YnI-PGJyPmt1YmVsZXRcbiAgICBwYXJ0aWNpcGFudCBjb250YWluZXIgYXMgbm9kZTxicj48YnI-Y29udGFpbmVyPGJyPnJ1bnRpbWVcbiAgICBtZS0-PmFwaVNydjogMS4ga3ViZWN0bCBjcmVhdGUgLWYgcG9kLnlhbWxcbiAgICBhcGlTcnYtLT4-ZXRjZDogMi4gc2F2ZSBuZXcgc3RhdGVcbiAgICBjbnRybE1nci0-PmFwaVNydjogMy4gY2hlY2sgZm9yIGNoYW5nZXNcbiAgICBzY2hlZC0-PmFwaVNydjogNC4gd2F0Y2ggZm9yIHVuYXNzaWduZWQgcG9kcyhzKVxuICAgIGFwaVNydi0-PnNjaGVkOiA1LiBub3RpZnkgYWJvdXQgcG9kIHcgbm9kZW5hbWU9XCIgXCJcbiAgICBzY2hlZC0-PmFwaVNydjogNi4gYXNzaWduIHBvZCB0byBub2RlXG4gICAgYXBpU3J2LS0-PmV0Y2Q6IDcuIHNhdmUgbmV3IHN0YXRlXG4gICAga3ViZWxldC0-PmFwaVNydjogOC4gbG9vayBmb3IgbmV3bHkgYXNzaWduZWQgcG9kKHMpXG4gICAgYXBpU3J2LT4-a3ViZWxldDogOS4gYmluZCBwb2QgdG8gbm9kZVxuICAgIGt1YmVsZXQtPj5jb250YWluZXI6IDEwLiBzdGFydCBjb250YWluZXJcbiAgICBrdWJlbGV0LT4-YXBpU3J2OiAxMS4gdXBkYXRlIHBvZCBzdGF0dXNcbiAgICBhcGlTcnYtLT4-ZXRjZDogMTIuIHNhdmUgbmV3IHN0YXRlIiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjp0cnVlfQ" >}}


 <!--
@@ -106,7 +106,7 @@ To run the link checker:
 2. 执行如下命令：

  ```
-  make docker-internal-linkcheck
+  make container-internal-linkcheck
  ```

 <!-- 
@@ -2,7 +2,7 @@
 title: Pod 安全策略
 id: pod-security-policy
 date: 2018-04-12
-full_link: /zh/docs/concepts/policy/pod-security-policy/
+full_link: /zh/docs/concepts/security/pod-security-policy/
 short_description: >
  为 Pod 的创建和更新操作启用细粒度的授权。

@@ -17,7 +17,7 @@ tags:
 title: Pod Security Policy
 id: pod-security-policy
 date: 2018-04-12
-full_link: /docs/concepts/policy/pod-security-policy/
+full_link: /docs/concepts/security/pod-security-policy/
 short_description: >
  Enables fine-grained authorization of pod creation and updates.

@@ -43,4 +43,9 @@ A cluster-level resource that controls security sensitive aspects of the Pod spe
 Pod 安全策略是集群级别的资源，它控制着 Pod 规约中的安全性敏感的内容。
 `PodSecurityPolicy`对象定义了一组条件以及相关字段的默认值，Pod 运行时必须满足这些条件。Pod 安全策略控制实现上体现为一个可选的准入控制器。

+<!--
+PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25. We recommend migrating to [Pod Security Admission](/docs/concepts/security/pod-security-admission/), or a 3rd party admission plugin.
+-->
+PodSecurityPolicy 自 Kubernetes v1.21 起已弃用，并将在 v1.25 中删除。 
+我们建议迁移到 [Pod 安全准入](/zh/docs/concepts/security/pod-security-admission/)或第三方准入插件。

@@ -88,7 +88,7 @@ that was transgressed as well as the specific policies on the fields that were
 violated from the PodSecurity enforcement.

 See [Pod Security Standards](/docs/concepts/security/pod-security-standards/)
-for more information
+for more information.
 -->
 ## pod-security.kubernetes.io/audit-violations {#pod-security-kubernetes-io-audit-violations}

@@ -99,4 +99,38 @@ PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container
 注解值给出审计策略违规的详细说明，它包含所违反的 [Pod 安全标准](/zh/docs/concepts/security/pod-security-standards/)级别以及
 PodSecurity 执行中违反的特定策略及对应字段。

-有关详细信息，请参阅 [Pod 安全标准](/zh/docs/concepts/security/pod-security-standards/)。
+有关详细信息，请参阅 [Pod 安全标准](/zh/docs/concepts/security/pod-security-standards/)。
+
+<!--
+## authorization.k8s.io/decision
+
+Example: `authorization.k8s.io/decision: "forbid"`
+
+This annotation indicates whether or not a request was authorized in Kubernetes audit logs.
+
+See [Auditing](/docs/tasks/debug-application-cluster/audit/) for more information.
+-->
+## authorization.k8s.io/decision {#authorization-k8s-io-decision}
+
+例子：`authorization.k8s.io/decision: "forbid"`
+
+此注解在 Kubernetes 审计日志中表示请求是否获得授权。
+
+有关详细信息，请参阅[审计](/zh/docs/tasks/debug-application-cluster/audit/)。
+
+<!--
+## authorization.k8s.io/reason
+
+Example: `authorization.k8s.io/reason: "Human-readable reason for the decision"`
+
+This annotation gives reason for the [decision](#authorization-k8s-io-decision) in Kubernetes audit logs.
+
+See [Auditing](/docs/tasks/debug-application-cluster/audit/) for more information.
+-->
+## authorization.k8s.io/reason {#authorization-k8s-io-reason}
+
+例子：`authorization.k8s.io/reason: "Human-readable reason for the decision"`
+
+此注解给出了 Kubernetes 审计日志中 [decision](#authorization-k8s-io-decision) 的原因。
+
+有关详细信息，请参阅[审计](/zh/docs/tasks/debug-application-cluster/audit/)。
@@ -1,8 +1,11 @@

 <!-- 
+kubeadm: easily bootstrap a secure Kubernetes cluster
+
 ### Synopsis 
 -->

+kubeadm: 轻松创建一个安全的 Kubernetes 集群
 ### 摘要

 <!--
@@ -90,9 +93,9 @@ Example usage:
 <tr>
 <td></td><td style="line-height: 130%; word-wrap: break-word;">
 <!--
-help for kubeadm
+<p>help for kubeadm</p>
 -->
-kubeadm 操作的帮助信息 
+<p>kubeadm 操作的帮助信息<p>
 </td>
 </tr>

@@ -102,9 +105,9 @@ kubeadm 操作的帮助信息
 <tr>
 <td></td><td style="line-height: 130%; word-wrap: break-word;">
 <!--
-[EXPERIMENTAL] The path to the 'real' host root filesystem.
+<p>[EXPERIMENTAL] The path to the 'real' host root filesystem.</p>
 -->
-[实验] 指向 '真实' 宿主机根文件系统的路径。
+<p>[实验] 指向 '真实' 宿主机根文件系统的路径。<p>
 </td>
 </tr>

--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 215 127"><defs><style>.cls-1{fill:transparent;}.cls-2{fill:#040606;}</style></defs><title>kubernetes.io-54664</title><rect class="cls-1" x="-4.55738" y="-4.48481" width="223.25536" height="134.51136"/><path class="cls-2" d="M169.05853,71.307c.23549,6.07483,5.42615,10.38072,14.09569,10.38072,7.0783,0,12.91766-3.06693,12.91766-9.85009,0-4.71846-2.65436-7.49024-8.78885-8.66954l-4.77685-.94382c-3.06756-.59028-5.19066-1.17992-5.19066-3.0079,0-2.00568,2.06471-2.89047,4.65943-2.89047,3.77525,0,5.3081,1.887,5.42678,4.1288H194.951c-.41258-5.89838-5.1304-9.8501-12.73994-9.8501-7.845,0-12.50382,4.30588-12.50382,9.90912,0,6.84157,5.54483,7.96247,10.3217,8.84662l3.95171.70834c2.83082.53062,4.06977,1.35638,4.06977,3.0079,0,1.47444-1.41541,2.94887-4.77748,2.94887-4.89553,0-6.488-2.53631-6.54705-4.71845Zm-27.4259-5.13164a8.58413,8.58413,0,0,1,8.557-8.61113q.02706-.00009.05409,0a8.58415,8.58415,0,0,1,8.61115,8.557q.00009.02706,0,.0541a8.58413,8.58413,0,0,1-8.55641,8.61177q-.02738.00009-.05473,0a8.58413,8.58413,0,0,1-8.61113-8.557q-.00009-.02738,0-.05473M158.79589,81.098h7.49149V50.95811h-7.49149v2.41825a15.58033,15.58033,0,1,0-9.01133,28.37034q.05285.00018.10567,0a15.47693,15.47693,0,0,0,8.90566-2.77179ZM106.59844,66.17532a8.58413,8.58413,0,0,1,8.557-8.61113q.02706-.00009.05409,0a8.58414,8.58414,0,0,1,8.61114,8.557q.00009.02706,0,.05409a8.58413,8.58413,0,0,1-8.55641,8.61177q-.02738.00009-.05473,0a8.58413,8.58413,0,0,1-8.61113-8.557q-.00009-.02738,0-.05473m17.16387-25.47987V53.37636a15.58048,15.58048,0,1,0-9.01195,28.37034q.05251.00018.105,0a15.48233,15.48233,0,0,0,8.90691-2.77179V81.098h7.49023V40.69545ZM96.21772,50.95811H88.72811V81.098h7.49023Zm0-10.26266H88.72811v7.49023h7.49023ZM60.41805,66.17532a8.58414,8.58414,0,0,1,8.557-8.61113q.02673-.00009.05346,0a8.58414,8.58414,0,0,1,8.61114,8.557q.00009.02706,0,.05409a8.58414,8.58414,0,0,1-8.55642,8.61177q-.02736.00009-.05472,0a8.58412,8.58412,0,0,1-8.61113-8.557q-.00009-.02738,0-.05473M77.58067,40.69545V53.37636A15.58033,15.58033,0,1,0,68.56935,81.7467q.05283.00018.10567,0a15.4769,15.4769,0,0,0,8.90565-2.77179V81.098h7.4915V40.69545ZM25.38259,66.176a8.58414,8.58414,0,0,1,8.557-8.61114q.02736-.00009.05472,0a8.58414,8.58414,0,0,1,8.61114,8.557q.00009.027,0,.05409a8.58414,8.58414,0,0,1-8.55642,8.61176q-.02736.00009-.05472,0a8.58413,8.58413,0,0,1-8.61177-8.55641q-.00009-.02768,0-.05535m17.16388,14.9227H50.038V50.95872H42.54647V53.377a15.58048,15.58048,0,1,0-9.01069,28.37035q.0522.00016.10441,0a15.48032,15.48032,0,0,0,8.90628-2.77179Z"/></svg>