* Added link to Version Skew Policy in Upgrading Kubeadm Cluster
* Update content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
* reorg the monitoring task section
Signed-off-by: Paul S. Schweigert <paulschw@us.ibm.com>
* reorg from review comments
Signed-off-by: Paul S. Schweigert <paulschw@us.ibm.com>
* review comments
Signed-off-by: Paul S. Schweigert <paulschw@us.ibm.com>
* review fixes
Signed-off-by: Paul S. Schweigert <paulschw@us.ibm.com>
Fix a minor bug in the codenew shortcode.
When serving a previous release, link to the example for that specific
release (and not to the example for the current release).
* [bugfix/fix-cncf-hyperlink-on-home] fix🐛: Explicitly set the margin-top property
* [bugfix/fix-cncf-hyperlink-on-home] fix🐛: remove important
* [bugfix/fix-cncf-hyperlink-on-home] format🥚: Add comments to indicate the role of css
* Adding more information to Installing kubeadm step.
* Adding more information to installing kubeadm step.
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
* Remove unnecessary quotes.
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
* Update 2022-02-17-updated-dockershim-faq.md
Adding the new Detector for Docker Socket plugin to the FAQ.
The tool will help folks find Dockershim usage in files on disk as well as running clusters.
* Apply suggestions from code review
Co-authored-by: Nate W. <4453979+nate-double-u@users.noreply.github.com>
* Update 2022-02-17-updated-dockershim-faq.md
Adding period
* Update content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md
Co-authored-by: Rey Lejano <rlejano@gmail.com>
* Update content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: Nate W. <4453979+nate-double-u@users.noreply.github.com>
Co-authored-by: Rey Lejano <rlejano@gmail.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
- Link to k/kubernetes issue 106917
(various places)
- Related rewording to make that extra link work in context
and also:
- Replace alias for dockershim FAQ with a Netlify redirect
Co-authored-by: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com>
The new task page outlines steps for reconfiguring
a kubeadm cluster and persisting reconfiguration.
Link the new page from the existing guides for
"customizing components", "creating a cluster",
and "kubeadm upgrade".
Co-authored-by: Paco Xu <paco.xu@daocloud.io>
Co-authored-by: Qiming Teng <tengqm@outlook.com>
While reading through the documentation, I noticed weird asterisk characters that seemed like they should be bullet points. I kept reading and saw the formatting for `loadBalancer.ingress.ports.protocol`. I liked how it organized the values into an easy to read list.
This proposed change formats the possible values for `rules.http.paths.pathType` to be a bullet point list, in (imo) a format that is easier to read.
The `containerd/cri` repo was merged into the `containerd/containerd` repo.
The `containerd/cri` repo is now archived.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
* Minor edit for clarity
The previous phrasing didn't emphasize the point that the reason you define a Service with no selectors is to point to a backend that's not a Pod, and the emphasis on the external nature of the backend
* Add note that Services w/o selectors, but +Endpoints is the technique to use for abstracting external backends
Co-authored-by: Rey Lejano <rlejano@gmail.com>
Co-authored-by: Rey Lejano <rlejano@gmail.com>
The logical navigation definitely works better if Pod Security admission
and PodSecurityPolicy are pages in the same section. Make It So.
Co-authored-by: Rey Lejano <rlejano@gmail.com>
Either we correct the code from pods/pod-with-node-affinity.yaml to match the description about the rules that apply or we correct the description in order to match the pods/pod-with-node-affinity.yaml
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
Translate /blog/2022/02/16/sig-node-ci-subproject-celebrates/ page into Chinese
* sample outputs created without docker working with kubernetes v1.24
* updated
* cleaned and updated
* sample outputs created without docker working with kubernetes v1.24
* updated
* cleaned and updated
The feature has been promoted to Beta in 1.22, according to
content/en/docs/tasks/administer-cluster/controller-manager-leader-migration.md
and upstream source.
Telnet is a command that really should not be used, as there is too great a chance it could be misused. NetCat, nc, is a better and newer tool for testing single open ports.
* Translate node-selector-requirement.md into Chinese
* Modify as suggested
* Modify according to the review results
* 删除文档中注释的部分
* Translate node-selector-requirement.md into Chinese
Modify according to the review results
* Translate docs/reference/labels-annotations-taints to zh language #32244
* Translate docs/reference/labels-annotations-taints to zh language #32244
* Translate docs/reference/labels-annotations-taints to zh language #32244
* Translate docs/reference/labels-annotations-taints to zh language #32244
* Translate docs/reference/labels-annotations-taints to zh language #32244
The acess-cluster-service page, has nothing to do with administering a cluster.
A better home for this page should be the `/docs/tasks/access-application-cluster` subdirectory instead.
This PR drops some outdated information about very old Kubernetes versions.
The last section is improved so that readers get a better idea why those pointers are provided.
* Update replicationcontroller.md
Synchronize English documentation about replicationcontroller
* Update replicationcontroller.md
Adjustment format
* Update replicationcontroller.md
I rechecked the changes and translated，thanks for reviews
* Update replicationcontroller.md
* Update replicationcontroller.md
Modified as suggested
* Update replicationcontroller.md
There are some italics font style which doesn't render well in Chinese, we should avoid using them when possible.
There are some inappropriate line wraps that are creating extra spaces between Chinese characters, this PR tries to optimizes that as well.
The subsection title 'configuration' was not translated, also fixed in this PR.
Double quotes was giving error as below, may be need to update the code to accept the double quotes.
Error: invalid argument "\"GracefulNodeShutdown=true\"" for "--feature-gates" flag: invalid value of "GracefulNodeShutdown=true", err: strconv.ParseBool: parsing "true\"": invalid syntax
Zsh completion is native since v1.22. Native zsh completion
automatically deals with aliases without needing
any extra configuration from the user.
Signed-off-by: Marc Khouzam <marc.khouzam@gmail.com>
This PR updates the scheduler configuration YAML snippets to use the 'v1beta3' version API. 'v1beta1' is gone and not recommended. We don't have config API reference for 'v1beta1' now.
1. Sync the diagram property "diagram-class"
1. Avoid translating the stages of the scheduling context. These
translations are not necessary and they may cause confusion when read
along with the diagram.
As a beginner to kubectl, I must have typed (and re-typed due to typos) --all-namespaces many hundreds of times before I stumbled across the -A shorthand notation for this. I sincerely would be grateful on behalf of all kubectl beginners if this very useful cheat could be added to this cheatsheet. I have proposed in this PR a location quite near the beginning of the cheatsheet for this so that it is very hard to miss.
Some blog articles with custom URIs have been localized into Chinese
without removing the custom URI from the front matter.
Update these articles to use appropriate URIs.
* Improve mentions of CS CA in managing-tls-in-a-cluster
* Update content/en/docs/tasks/tls/managing-tls-in-a-cluster.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update managing-tls-in-a-cluster.md
* Update managing-tls-in-a-cluster.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update hyperlinks to point to main branch
* Revert changes to post older than a year ⏪
* Update link to point to localize document 📝
* Fix fragement in the link 📝
change "#the-kubernetes-network-model" to "#how-to-implement-the-kubernetes-networking-model"
* Revert changes in zh localization pages
* Remove changes in files of other localization
The guide generates some files on one of three ETCD
hosts. It then copies files from host 1 to 2 and 3.
Due to that some file paths differ.
Update step 7 to reflect that and to match step 6.
- `content/zh/docs/concepts/cluster-administration/system-metrics.md`
line 131 , '启用' should be '弃用'
- `content/zh/docs/reference/access-authn-authz/_index.md`
Title '访问 API' should be ‘API 访问控制'
This list appears to name binaries whose configuration flags are documented in this section: [Command Line Tools Reference](/docs/reference/command-line-tools-reference/). Since kube-proxy is an item in that reference, it seems helpful to give it an entry and link here.
- This page referenced the "CertificationSigningRequests API," but this should be "CertificateSigningRequests API" or "Certificates API."
- Added a link to the documentation for CertificateSigningRequests.
* Adding an example
This text mentions that you can view the logs from a specific container by appending the container name. I've created an example, based on the previous few lines.
Also, I've added a -c flag, to bring this in line with both the ```kubectl logs``` [documentation's examples](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs) and other kubectl commands that require the -c flag when selecting a specific container from a pod.
* Wrapping long line and marking `-c` as code.
Co-authored-by: Qiming Teng <tengqm@outlook.com>
Co-authored-by: Qiming Teng <tengqm@outlook.com>
* Replaced "master" with newer terms in two places
- Replacing the outdated "master" term with API server.
- Changed "master" to "control plane node(s)"
* control plane node -> control plane
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
The rest of this paragraph talks about nodes being "machines," whereas this sentence is more specific, referring to virtual machines ("VMs"). The article this sentence points to also uses "machines" and isn't specific to VMs.
Original link says
```
Design proposals have been archived.
To view the last version of this document, see the Design Proposals Archive Repo.
Please remove after 2022-04-01 or the release of Kubernetes 1.24, whichever comes first.
```
This change enables an opt-in switch to new styles for /community
It is opt-in to enable localizations to migrate to the new page on their
own timeline.
The `TracingConfiguration` struct we reference from the system traces
page only exists in the v1alpha1 version of the API server
configuration. This PR fixes the problem.
* Create updated dockershim FAQ
- The current dockershim FAQ is marked as outdated
- This new FAQ states clearly "removal"
- This new FAQ also has links to additional resources created by the community and others.
* deprecation -> removal scrub
- It's important this new FAQ says removal rather than deprecation as we're at that moment in time.
* Update content/en/blog/_posts/2022-02-16-updated-dockershim-faq.md
Did not know Hugo allowed for this.
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2022-02-16-updated-dockershim-faq.md
🤯
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2022-02-16-updated-dockershim-faq.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2022-02-16-updated-dockershim-faq.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2022-02-16-updated-dockershim-faq.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2022-02-16-updated-dockershim-faq.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Apply suggestions from code review
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Bump version to 1.23
Warning in 1.23
Removed in 1.24
* Update for today's date
Co-authored-by: Tim Bannister <tim@scalefactory.com>
The current example showing how to decode a secret
does it in several steps which is fine but if some level
of security is required will leak the encoded secret
value in the shell history thus making it retrievable.
This patch adds an example on how to retrieve that
value without exposing it.
Simplify commands to output paths to all elements.
Note: the jq filter "path(..)" is simplified to "paths," however, there is a trivial difference: the filter "paths" doesn't output the top path "[]."
The links to AppArmor etc are using redirection records. This is a bad
practice because when translating the page to a different language, the
link will become invalid. So we should use the actual target in the
English source.
This PR also fixes some incorrect syntax directives for code snippets.
* Update link to TECHNET article about Windows NAT
* Fix inconsistent capitalisation of win-overlay
No other instance of win-overlay is capitalised, and win-bridge in the same place on the previous row is not capitalised either.
* Docs to change Container runtime
* Updated header
* Updated header
* Few changes made according to the reviews
* Updated few headings
* Updated few markdown changes
* Reverted a unwanted changes
* Removed the double extension in the filename
* Updated
* Updated according to review
* Final Updates
* Added instructions to remove docker engine
* Minor changes
* Minor updates on heading
* Minor updates on lists
* Minor updates on line 106
* Minor updates on line 106
This PR fixes a link (at the bottom of the page) to reference.
It also fixes the indentation of the number lists, the incorrect syntax tag for code snippets.
This PR fixes two problems in the cascading deletion page:
- The indentation of list items should be 2 spaces for unordered lists,
or 3 spaces for ordered lists. We should avoid using 4 spaces as
indentation because 4-spaces indentation has special meaning in
markdown.
- The garbage-collection page was moved to a new place. With redirect
records for English site, there seems no obvious problems. However,
for localization teams, such redirection records don't exist. It is
super easy to create a dangling link in localized pages.
- sort external content alphabetically
- retitle to “Articles on dockershim Removal and on Using CRI-compatible Runtimes”
- style tweaks
- add some entries
* Update logging.md
Updated some sentences for clarity.
* Update content/en/docs/concepts/cluster-administration/logging.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update to SIG Multicluster Spotlight
* Implemented review comments
* Update content/en/blog/_posts/2022-01-14-SIG-Multicluster-Spotlight/index.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2022-01-14-SIG-Multicluster-Spotlight/index.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2022-01-14-SIG-Multicluster-Spotlight/index.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Renaming blog folder to match publication date
* Update publish date to match
* Fix path to article
Co-authored-by: Tim Bannister <tim@scalefactory.com>
This change is to update the documentation for reserved-memory flag.
Now, for specifying memory reservations across multiple NUMA nodes,
semicolon needs to be used as separator.
Signed-off-by: Ravindra Thakur <ravindra.nath.thakur@est.tech>
Trying to use this flag `--record` in the example leads to the warning:
```
Flag --record has been deprecated, --record will be removed in the future
```
Use white on blue. This color combination is on-brand and stands out,
whilst looking less alarming than the black on yellow for old
documentation versions.
The command "kubeadm kubeconfig user" is missing
some examples and more details on why it is needed.
- Add a new section "Generating kubeconfig files
for additional users" under the kubeadm-certs page.
- Link to this section from the kubeadm-kubeconfig reference page.
- Link to this section from the create-cluster-kubeadm page.
Credit to Tim Bannister <tim@scalefactory.com> for the same improvements
in load-balancer-nodeport-allocation above.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Looks like this was overlooked to be updated when the feature gate documentation was updated in #28770.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Sysbox is an open-source container runtime (similar to "runc") that supports
running VM-workloads such as Docker and Kubernetes inside unprivileged
containers or pods.
Sysbox containers always use the Linux user-namespace for isolation, plus
specially crafted proc and sys filesystems, some syscall interception,
filesystem ID-mapping, and more.
It's possible to run Kubernetes, K3s, K0s, inside containers or pods deployed
with Sysbox. This commit aims to make the Kubernetes community aware of this
option.
Signed-off-by: Cesar Talledo <ctalledo@nestybox.com>
Use the right URL for Server Fault queries tagged Kubernetes, not the
wrong URL for Stack Overflow (which is more about workload / app
development help).
Uppercase Service API object following documentation style guide
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Add mitigation propositions
Add RFC reference for the trailing dot in an absolute DNS name
Co-authored-by: Shannon Kularathna <ax3shannonkularathna@gmail.com>
- Highlight that we do not change the Kubernetes API and how to verify
the used seccomp profile
- Fix the kind configuration and enhance the example with a custom
workload.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
* Include link to Pod Security Admission in the PodSecurityPolicy deprecation notice
Add a link to the Pod Security Admission feature, which replaces PodSecurityPolicy. Currently users who read this page may not realize what the replacement is. The linked blog post talks about the KEP that created Pod Security Admission, but with no direct link.
* Remove link prefix
Co-authored-by: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com>
Co-authored-by: Jihoon Seo <46767780+jihoon-seo@users.noreply.github.com>
- Fixed brain flatuence around which runtime I was testing
- Outputs changed
- Noted added about crictl ps's default behavior of outputting sha256sum in image name
- Image name in outputs have been shortened per Slack convo https://kubernetes.slack.com/archives/C02PZMA3R6E/p1643026840047600
The reference for kubelet is manually maintained due to upstream code changes.
This PR updates the reference according to the help message from the 1.23 version of kubelet executable.
Primarily the change, in release 1.22, of how configuration objects
are maintained.
Also describe the new priority level.
Also move the sections on defaults and health check configuration to
follow the description of configuration objects.
The cloud-controller-manager is a Kubernetes control plane component" according to docs. That is, a singular manager. This PR fixes typo in docs where it was referenced to as "cloud controller managers".
If a user were to copy & paste from the docs into the feature-gate option they would receive the error `flag: unrecognized feature gate: OpenAPIv3` or `flag: unrecognized feature gate: OpenAPIEnum`. This change uses the correct case for `V` and uses the plural form for `Enums` as the command line parser expects.
1. Remove examples contrasting k8s networking with Docker networking
from concept document explaining connecting applications using services
2. Remove reference to dockerlinks in favor of English description
of the env vars autocreated for every Service.
The how-to docs to create a new namespace already specify this, but I think it's worth mentioning in the overview doc that a namespace name must be an RFC 1123 label, as I didn't initially think to look in the how-to doc for limitations.
The current entry under additional information about drain can
be confusing and users might jump into draining the nodes before
the upgrade steps (as a pre-req), following the
instructions in the external link. This is not accurate.
Adjust the entry text to explain that drain is a step as part of
the upgrade process and this additional information entry just
mentions why it is needed.
There were a couple of reported problems with this guide:
- The introductory paragraph talks about single control plane nodes
and does not mention the different options for HA etcd. Clear the
language to reduce the confusion and cross-link to the ha-topology
page.
- The hostname / IP detection in kubeadm can end
up with values not suitable for the certificates that kubeadm
generates for all etcd instances. Ensure that the hostnames / IPs
are pinned by the user in the example script.
Side cleanup related to the dockershim removal:
- Use containerd in the setup example and don't mention docker as
a requirement.
Added extra links to the guidance pertaining asking questions on Stack Overflow, to ensure users first read through guidance to ensure questions are both on-topic and well-written — both of which will improve these users' experience on Stack Overflow, and avoid having off-topic questions asked there.
This change updates how we run Hugo AND changes the logic for checking
whether a page should be indexable (copied with a backport from upstream
Docsy).
Some nits fixed in this PR:
- The typo of 'MaxContainers' and 'MaxPerPodContainer';
- The section name which should be 'Container garbage collection' rather
than 'Container image garbage collection'.
This commit reformats the memory manager page:
- convert the page from DOS to UNIX format;
- wrap long lines where appropriate for ease of change tracking;
- remove useless trailing spaces and empty lines.
This commit contains **NO** changes to the text itself.
This PR fixes the memory manager page by:
- removing links to non-existent sections;
- fixing links with bad anchors;
- fixing incorrect language tag for code snippets
Obviously the better approach. I was not familiar enough with how which repo corresponds to which site and just blindly followed the link in the placeholder.
Co-authored-by: Tim Bannister <tim@scalefactory.com>
The previous commit for configuration APIs has some nits to fix:
- The client-authentication API has both v1beta1 and v1 supported.
We need to include both.
- The kube-scheduler v1alpha1 is superceded by v1alpha3 which is new.
- The links to some external type definitions should point to the 1.23
API rather than old versions.
- Promote palnabarun and Verolop to release-engineering-approvers
- Remove hasheddan from release-engineering-approvers
Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>
Removing archaic requirements for SIG Docs co-chairs. For the record, that I did not fully meet when becoming a co-chair a couple years ago. The current set of requirements hurt more than they help.
The service account token one must have `serviceAccountName` specified
in order to be successfully validated.
The PV duplicate example can only refer to an existing volume.
* Add config API for kubelet v1alpha1
The CredentialProvider struct is defined in the v1alpha1 config API
rather than v1beta1. The upstream generator is updated to include this
version (https://github.com/kubernetes-sigs/reference-docs/pull/268).
This commit adds references to the API as well.
* Wrap long lines in the page
Revise tasks within “Manage Memory, CPU, and API Resources”:
- reword to avoid implying that API clients can create a container
(you can, but its done by creating a Pod)
- call a manifest a manifest
- use tooltips where relevant
- link to new API reference not the old one
- other improvements
*should* implies that an `extra` can be mixed case. but really it can't because a mixed case `extra` will mismatch on an RBAC `ClusterRole` once the header is canonicalized.
Applied some wordsmithing
Move attribution into the figure tag
Address comment
Add project open-source link and update content
Address review from sftim
Update future improvement section with current gaps to ideal K8s design
Set release date to 2021-12-21
This might be useful for custom styling, or for using
a pure-CSS fallback for some scripted behavior.
There is already a no-js class set on the <html> element;
this change removes it when JavaScript is running OK.
@@ -34,6 +34,6 @@ Note that code issues should be filed against the main kubernetes repository, wh
### Submitting Documentation Pull Requests
If you're fixing an issue in the existing documentation, you should submit a PR against the master branch. Follow [these instructions to create a documentation pull request against the kubernetes.io repository](http://kubernetes.io/docs/home/contribute/create-pull-request/).
If you're fixing an issue in the existing documentation, you should submit a PR against the main branch. Follow [these instructions to create a documentation pull request against the kubernetes.io repository](http://kubernetes.io/docs/home/contribute/create-pull-request/).
For more information, see [contributing to Kubernetes docs](https://kubernetes.io/docs/contribute/).
module-check:## Check if all of the required submodules are correctly initialized.
@git submodule status --recursive | awk '/^[+-]/ {err = 1; printf "\033[31mWARNING\033[0m Submodule not initialized: \033[34m%s\033[0m\n",$$2} END { if (err != 0) print "You need to run \033[32mmake module-init\033[0m to initialize missing modules first"; exit err }' 1>&2
all:build## Build site with production settings and put deliverables in ./public
build:module-check## Build site with production settings and put deliverables in ./public
hugo --minify
build:module-check## Build site with non-production settings and put deliverables in ./public
hugo --minify --environment development
build-preview:module-check## Build site with drafts and future posts enabled
hugo --buildDrafts --buildFuture
hugo --buildDrafts --buildFuture --environment preview
deploy-preview:## Deploy preview site via netlify
hugo --enableGitInfo --buildFuture -b $(DEPLOY_PRIME_URL)
hugo --enableGitInfo --buildFuture --environment preview -b $(DEPLOY_PRIME_URL)
functions-build:
$(NETLIFY_FUNC) build functions-src
@@ -43,13 +43,15 @@ functions-build:
check-headers-file:
scripts/check-headers-file.sh
production-build:buildcheck-headers-file## Build the production site and ensure that noindex headers aren't added
production-build:module-check## Build the production site and ensure that noindex headers aren't added
hugo --minify --environment production
HUGO_ENV=production $(MAKE) check-headers-file
non-production-build:## Build the non-production site, which adds noindex headers to prevent indexing
hugo --enableGitInfo
non-production-build:module-check## Build the non-production site, which adds noindex headers to prevent indexing
hugo --enableGitInfo --environment nonprod
serve:module-check## Boot the development server.
hugo server --buildFuture
hugo server --buildFuture --environment development
docker-image:
@echo -e "$(CCRED)**** The use of docker-image is deprecated. Use container-image instead. ****$(CCEND)"
@@ -70,10 +72,10 @@ container-image: ## Build a container image for the preview of the website
--build-arg HUGO_VERSION=$(HUGO_VERSION)
container-build:module-check
$(CONTAINER_RUN) --read-only --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777$(CONTAINER_IMAGE) sh -c "npm ci && hugo --minify"
$(CONTAINER_RUN) --read-only --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777$(CONTAINER_IMAGE) sh -c "npm ci && hugo --minify --environment development"
container-serve:module-check## Boot the development server using container. Run `make container-image` before this.
$(CONTAINER_RUN) --cap-drop=ALL --cap-add=AUDIT_WRITE --read-only --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 -p 1313:1313 $(CONTAINER_IMAGE) hugo server --buildFuture --bind 0.0.0.0 --destination /tmp/hugo --cleanDestinationDir
$(CONTAINER_RUN) --cap-drop=ALL --cap-add=AUDIT_WRITE --read-only --mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 -p 1313:1313 $(CONTAINER_IMAGE) hugo server --buildFuture --environment development --bind 0.0.0.0 --destination /tmp/hugo --cleanDestinationDir
Herzlich willkommen! Dieses Repository enthält alle Assets, die zur Erstellung der [Kubernetes-Website und Dokumentation](https://kubernetes.io/) erforderlich sind. Wir freuen uns sehr, dass Sie dazu beitragen wollen!
Herzlich Willkommen! Dieses Repository enthält alle Assets, die zur Erstellung der [Kubernetes-Website und Dokumentation](https://kubernetes.io/) erforderlich sind. Wir freuen uns sehr, dass Sie dazu beitragen wollen!
## Beiträge zur Dokumentation
Sie können auf die Schaltfläche **Fork** im oberen rechten Bereich des Bildschirms klicken, um eine Kopie dieses Repositorys in Ihrem GitHub-Konto zu erstellen. Diese Kopie wird als *Fork* bezeichnet. Nehmen Sie die gewünschten Änderungen an Ihrem Fork vor. Wenn Sie bereit sind, diese Änderungen an uns zu senden, gehen Sie zu Ihrem Fork und erstellen Sie eine neue Pull-Anforderung, um uns darüber zu informieren.
Sie können auf die Schaltfläche **Fork** im oberen rechten Bereich des Bildschirms klicken, um eine Kopie dieses Repositorys in Ihrem GitHub-Konto zu erstellen. Diese Kopie wird als *Fork* bezeichnet. Nehmen Sie die gewünschten Änderungen an Ihrem Fork vor. Wenn Sie bereit sind, diese Änderungen an uns zu senden, gehen Sie zu Ihrem Fork und erstellen Sie eine neue Pull-Request, um uns darüber zu informieren.
Sobald Ihre Pull-Anfrage erstellt wurde, übernimmt ein Rezensent von Kubernetes die Verantwortung für klares, umsetzbares Feedback. Als Eigentümer des Pull-Request **liegt es in Ihrer Verantwortung Ihren Pull-Reqest entsprechend des Feedbacks, dass Sie vom Kubernetes-Reviewer erhalten haben abzuändern.** Beachten Sie auch, dass Sie am Ende mehr als einen Rezensenten von Kubernetes erhalten, der Ihnen Feedback gibt, oder dass Sie Rückmeldungen von einem Rezensenten von Kubernetes erhalten, der sich von demjenigen unterscheidet, der ursprünglich für das Feedback zugewiesen wurde. In einigen Fällen kann es vorkommen, dass einer Ihrer Prüfer bei Bedarf eine technische Überprüfung von einem [Kubernetes Tech-Reviewer](https://github.com/kubernetes/website/wiki/tech-reviewers) anfordert. Reviewer geben ihr Bestes, um zeitnah Feedback zu geben, die Antwortzeiten können jedoch je nach den Umständen variieren.
Sobald Ihre Pull-Request erstellt wurde, übernimmt ein Rezensent von Kubernetes die Verantwortung für klares, umsetzbares Feedback. Als Eigentümer des Pull-Request **liegt es in Ihrer Verantwortung Ihre Pull-Request entsprechend des Feedbacks, welches Sie vom Kubernetes-Reviewer erhalten haben, abzuändern.** Beachten Sie auch, dass Sie am Ende mehr als einen Rezensenten von Kubernetes erhalten, der Ihnen Feedback gibt, oder dass Sie Rückmeldungen von einem Rezensenten von Kubernetes erhalten, der sich von demjenigen unterscheidet, der ursprünglich für das Feedback zugewiesen wurde. In einigen Fällen kann es vorkommen, dass einer Ihrer Prüfer bei Bedarf eine technische Überprüfung von einem [Kubernetes Tech-Reviewer](https://github.com/kubernetes/website/wiki/tech-reviewers) anfordert. Reviewer geben ihr Bestes, um zeitnah Feedback zu geben, die Antwortzeiten können jedoch je nach den Umständen variieren.
Weitere Informationen zum Beitrag zur Kubernetes-Dokumentation finden Sie unter:
@@ -28,14 +28,14 @@ Die Betreuer der deutschen Lokalisierung erreichen Sie unter:
* Max Körbächer ([@mkorbi](https://github.com/mkorbi))
Um die Kubernetes-Website lokal laufen zu lassen, empfiehlt es sich, ein spezielles [Docker](https://docker.com) Image auszuführen, das den statischen Site-Generator [Hugo](https://gohugo.io) enthält.
Um die Kubernetes-Website lokal laufen zu lassen, empfiehlt es sich, ein spezielles [Docker](https://docker.com) Image auszuführen, das den statischen Website-Generator [Hugo](https://gohugo.io) enthält.
> Unter Windows benötigen Sie einige weitere Tools, die Sie mit [Chocolatey](https://chocolatey.org) installieren können.
`choco install make`
> Wenn Sie die Website lieber lokal ohne Docker ausführen möchten, finden Sie weitere Informationen unter [Website lokal mit Hugo ausführen](#Die-Site-lokal-mit-Hugo-ausführen).
> Wenn Sie die Website lieber lokal ohne Docker ausführen möchten, finden Sie weitere Informationen unter [Website lokal mit Hugo ausführen](#Die-Website-lokal-mit-Hugo-ausführen).
Das benötigte [Docsy Hugo theme](https://github.com/google/docsy#readme) muss als git submodule installiert werden:
@@ -50,26 +50,26 @@ Wenn Sie Docker [installiert](https://www.docker.com/get-started) haben, erstell
make container-image
```
Nachdem das Image erstellt wurde, können Sie die Site lokal ausführen:
Nachdem das Image erstellt wurde, können Sie die Website lokal öffnen:
```bash
make container-serve
```
Öffnen Sie Ihren Browser unter http://localhost:1313, um die Site anzuzeigen. Wenn Sie Änderungen an den Quelldateien vornehmen, aktualisiert Hugo die Site und erzwingt eine Browseraktualisierung.
Öffnen Sie Ihren Browser unter http://localhost:1313, um die Website anzuzeigen. Wenn Sie Änderungen an den Quelldateien vornehmen, aktualisiert Hugo die Website und erzwingt eine Browseraktualisierung.
## Die Site lokal mit Hugo ausführen
## Die Website lokal mit Hugo ausführen
Hugo-Installationsanweisungen finden Sie in der [offiziellen Hugo-Dokumentation](https://gohugo.io/getting-started/installing/). Stellen Sie sicher, dass Sie die Hugo-Version installieren, die in der Umgebungsvariablen `HUGO_VERSION` in der Datei [`netlify.toml`](netlify.toml#L9) angegeben ist.
Das benötigte [Docsy Hugo theme](https://github.com/google/docsy#readme) muss als git submodule installiert werden:
Das benötigte [Docsy Hugo-Theme](https://github.com/google/docsy#readme) muss als git submodule installiert werden:
```
#Füge das Docsy submodule hinzu
git submodule update --init --recursive --depth 1
```
So führen Sie die Site lokal aus, wenn Sie Hugo installiert haben:
So führen Sie die Website lokal aus, wenn Sie Hugo installiert haben:
```bash
# Installieren der JavaScript Abhängigkeiten
@@ -77,7 +77,7 @@ npm ci
make serve
```
Dadurch wird der lokale Hugo-Server an Port 1313 gestartet. Öffnen Sie Ihren Browser unter http://localhost:1313, um die Site anzuzeigen. Wenn Sie Änderungen an den Quelldateien vornehmen, aktualisiert Hugo die Site und erzwingt eine Browseraktualisierung.
Dadurch wird der lokale Hugo-Server an Port 1313 gestartet. Öffnen Sie Ihren Browser unter http://localhost:1313, um die Website anzuzeigen. Wenn Sie Änderungen an den Quelldateien vornehmen, aktualisiert Hugo die Website und erzwingt eine Browseraktualisierung.
## Community, Diskussion, Beteiligung und Unterstützung
@@ -94,4 +94,4 @@ Die Teilnahme an der Kubernetes-Community unterliegt dem [Kubernetes-Verhaltensk
## Vielen Dank!
Kubernetes lebt vom CommunityEngagement und wir freuen uns sehr über Ihre Beiträge zu unserer Website und unserer Dokumentation!
Kubernetes lebt vom Community-Engagement und wir freuen uns sehr über Ihre Beiträge zu unserer Website und unserer Dokumentation!
### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version
W tym repozytorium znajdziesz wszystko, czego potrzebujesz do zbudowania [strony internetowej Kubernetesa wraz z dokumentacją](https://kubernetes.io/). Bardzo nam miło, że chcesz wziąć udział w jej współtworzeniu!
+ [Twój wkład w dokumentację](#twój-wkład-w-dokumentację)
+ [Informacje o wersjach językowych](#informacje-o-wersjach-językowych)
+ [Informacje o wersjach językowych](#różne-wersje-językowe-readmemd)
# Jak używać tego repozytorium
## Jak używać tego repozytorium
Możesz uruchomić serwis lokalnie poprzez Hugo (Extended version) lub ze środowiska kontenerowego. Zdecydowanie zalecamy korzystanie z kontenerów, bo dzięki temu lokalna wersja będzie spójna z tym, co jest na oficjalnej stronie.
@@ -22,14 +22,14 @@ Aby móc skorzystać z tego repozytorium, musisz lokalnie zainstalować:
Przed rozpoczęciem zainstaluj niezbędne zależności. Sklonuj repozytorium i przejdź do odpowiedniego katalogu:
Strona Kubernetesa używa [Docsy Hugo theme](https://github.com/google/docsy#readme). Nawet jeśli planujesz uruchomić serwis w środowisku kontenerowym, zalecamy pobranie podmodułów i innych zależności za pomocą polecenia:
Aby zbudować i uruchomić serwis wewnątrz środowiska kontenerowego, wykonaj następujące polecenia:
```
```bash
make container-image
make container-serve
```
Jeśli widzisz błędy, prawdopodobnie kontener z Hugo nie dysponuje wystarczającymi zasobami. Aby rozwiązać ten problem, zwiększ ilość dostępnych zasobów CPU i pamięci dla Dockera na Twojej maszynie ([MacOSX](https://docs.docker.com/docker-for-mac/#resources) i [Windows](https://docs.docker.com/docker-for-windows/#resources)).
Aby obejrzeć zawartość serwisu, otwórz w przeglądarce adres http://localhost:1313. Po każdej zmianie plików źródłowych, Hugo automatycznie aktualizuje stronę i odświeża jej widok w przeglądarce.
Aby obejrzeć zawartość serwisu, otwórz w przeglądarce adres <http://localhost:1313>. Po każdej zmianie plików źródłowych, Hugo automatycznie aktualizuje stronę i odświeża jej widok w przeglądarce.
## Jak uruchomić lokalną kopię strony przy pomocy Hugo?
@@ -59,13 +59,14 @@ npm ci
make serve
```
Zostanie uruchomiony lokalny serwer Hugo na porcie 1313. Otwórz w przeglądarce adres http://localhost:1313, aby obejrzeć zawartość serwisu. Po każdej zmianie plików źródłowych, Hugo automatycznie aktualizuje stronę i odświeża jej widok w przeglądarce.
Zostanie uruchomiony lokalny serwer Hugo na porcie 1313. Otwórz w przeglądarce adres <http://localhost:1313>, aby obejrzeć zawartość serwisu. Po każdej zmianie plików źródłowych, Hugo automatycznie aktualizuje stronę i odświeża jej widok w przeglądarce.
## Budowanie dokumentacji źródłowej API
Budowanie dokumentacji źródłowej API zostało opisane w [angielskiej wersji pliku README.md](README.md#building-the-api-reference-pages).
## Rozwiązywanie problemów
### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version
Z przyczyn technicznych, Hugo jest rozprowadzany w dwóch wersjach. Aktualny serwis używa tylko wersji **Hugo Extended**. Na stronie z [wydaniami](https://github.com/gohugoio/hugo/releases) poszukaj archiwum z `extended` w nazwie. Dla potwierdzenia, uruchom `hugo version` i poszukaj słowa `extended`.
@@ -74,7 +75,7 @@ Z przyczyn technicznych, Hugo jest rozprowadzany w dwóch wersjach. Aktualny ser
Jeśli po uruchomieniu `make serve` na macOS widzisz następujący błąd:
```
```bash
ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files
Możesz kliknąć w przycisk **Fork** w prawym górnym rogu ekranu, aby stworzyć kopię tego repozytorium na swoim koncie GitHub. Taki rodzaj kopii (odgałęzienia) nazywa się *fork*. Zmieniaj w nim, co chcesz, a kiedy będziesz już gotowy/a przesłać te zmiany do nas, przejdź do swojej kopii i stwórz nowy *pull request*, abyśmy zostali o tym poinformowani.
Po stworzeniu *pull request*, jeden z recenzentów projektu Kubernetes podejmie się przekazania jasnych wskazówek pozwalających podjąć następne działania. Na Tobie, jako właścicielu *pull requesta*, **spoczywa odpowiedzialność za wprowadzenie poprawek zgodnie z uwagami recenzenta.**
Po stworzeniu *pull request*, jeden z recenzentów projektu Kubernetes podejmie się przekazania jasnych wskazówek pozwalających podjąć następne działania. Na Tobie, jako właścicielu *pull requesta*, **spoczywa odpowiedzialność za wprowadzenie poprawek zgodnie z uwagami recenzenta.**
Może też się zdarzyć, że swoje uwagi zgłosi więcej niż jeden recenzent, lub że recenzję będzie robił ktoś inny, niż ten, kto został przydzielony na początku.
W niektórych przypadkach, jeśli zajdzie taka potrzeba, recenzent może poprosić dodatkowo o recenzję jednego z [recenzentów technicznych](https://github.com/kubernetes/website/wiki/Tech-reviewers). Recenzenci zrobią wszystko, aby odpowiedzieć sprawnie, ale konkretny czas odpowiedzi zależy od wielu czynników.
W niektórych przypadkach, jeśli zajdzie taka potrzeba, recenzent może poprosić dodatkowo o recenzję jednego z recenzentów technicznych. Recenzenci zrobią wszystko, aby odpowiedzieć sprawnie, ale konkretny czas odpowiedzi zależy od wielu czynników.
Więcej informacji na temat współpracy przy tworzeniu dokumentacji znajdziesz na stronach:
* [Udział w rozwijaniu dokumentacji](https://kubernetes.io/docs/contribute/)
Udział w działaniach społeczności Kubernetesa jest regulowany przez [Kodeks postępowania CNCF](https://github.com/cncf/foundation/blob/master/code-of-conduct-languages/pl.md).
# Dziękujemy!
## Dziękujemy!
Kubernetes rozkwita dzięki zaangażowaniu społeczności — doceniamy twój wkład w tworzenie naszego serwisu i dokumentacji!
@@ -65,9 +65,9 @@ This will start the local Hugo server on port 1313. Open up your browser to <htt
The API reference pages located in `content/en/docs/reference/kubernetes-api` are built from the Swagger specification, using <https://github.com/kubernetes-sigs/reference-docs/tree/master/gen-resourcesdocs>.
To update the reference pages for a new Kubernetes release (replace v1.20 in the following examples with the release to update to):
To update the reference pages for a new Kubernetes release follow these steps:
1. Pull the `kubernetes-resources-reference` submodule:
1. Pull in the `api-ref-generator` submodule:
```bash
git submodule update --init --recursive --depth 1
@@ -75,9 +75,9 @@ To update the reference pages for a new Kubernetes release (replace v1.20 in the
If you need help at any point when contributing, the [New Contributor Ambassadors](https://kubernetes.io/docs/contribute/advanced/#serve-as-a-new-contributor-ambassador) are a good point of contact. These are SIG Docs approvers whose responsibilities include mentoring new contributors and helping them through their first few pull requests. The best place to contact the New Contributors Ambassadors would be on the [Kubernetes Slack](https://slack.k8s.io/). Current New Contributors Ambassadors for SIG Docs:
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21"buttonid="desktopKCButton">Besuche die KubeCon North America vom 11. bis 15. Oktober 2021</a>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22"buttonid="desktopKCButton">Besuche die KubeCon Europe vom 16. bis 20. Mai 2022</a>
<br>
<br>
<br>
<br>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22"buttonid="desktopKCButton">Besuche die KubeCon Europe vom 17. bis 20. Mai 2022</a>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna22"buttonid="desktopKCButton">Besuchen die KubeCon North America vom 24. bis 28. Oktober 2022</a>
@@ -24,14 +24,14 @@ Die Add-Ons in den einzelnen Kategorien sind alphabetisch sortiert - Die Reihenf
* [Canal](https://github.com/tigera/canal/tree/master/k8s-install) vereint Flannel und Calico um Networking- und Network-Policies bereitzustellen.
* [Cilium](https://github.com/cilium/cilium) ist ein L3 Network- and Network-Policy-Plugin welches das transparent HTTP/API/L7-Policies durchsetzen kann. Sowohl Routing- als auch Overlay/Encapsulation-Modes werden uterstützt. Außerdem kann Cilium auf andere CNI-Plugins aufsetzen.
* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) ermöglicht das nahtlose Verbinden von Kubernetes mit einer Reihe an CNI-Plugins wie z.B. Calico, Canal, Flannel, Romana, oder Weave.
* [Contiv](http://contiv.github.io) bietet konfigurierbares Networking (Native L3 auf BGP, Overlay mit vxlan, Klassisches L2, Cisco-SDN/ACI) für verschiedene Anwendungszwecke und auch umfangreiches Policy-Framework. Das Contiv-Projekt ist vollständig [Open Source](http://github.com/contiv). Der [installer](http://github.com/contiv/install) bietet sowohl kubeadm als auch nicht-kubeadm basierte Installationen.
* [Contiv](https://contivpp.io/) bietet konfigurierbares Networking (Native L3 auf BGP, Overlay mit vxlan, Klassisches L2, Cisco-SDN/ACI) für verschiedene Anwendungszwecke und auch umfangreiches Policy-Framework. Das Contiv-Projekt ist vollständig [Open Source](http://github.com/contiv). Der [installer](http://github.com/contiv/install) bietet sowohl kubeadm als auch nicht-kubeadm basierte Installationen.
* [Contrail](http://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), basierend auf [Tungsten Fabric](https://tungsten.io), ist eine Open Source, multi-Cloud Netzwerkvirtualisierungs- und Policy-Management Plattform. Contrail und Tungsten Fabric sind mit Orechstratoren wie z.B. Kubernetes, OpenShift, OpenStack und Mesos integriert und bieten Isolationsmodi für Virtuelle Maschinen, Container (bzw. Pods) und Bare Metal workloads.
* [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) ist ein Overlay-Network-Provider der mit Kubernetes genutzt werden kann.
* [Knitter](https://github.com/ZTE/Knitter/) ist eine Network-Lösung die Mehrfach-Network in Kubernetes ermöglicht.
*[Multus](https://github.com/Intel-Corp/multus-cni) ist ein Multi-Plugin für Mehrfachnetzwerk-Unterstützung um alle CNI-Plugins (z.B. Calico, Cilium, Contiv, Flannel), zusätzlich zu SRIOV-, DPDK-, OVS-DPDK- und VPP-Basierten Workloads in Kubernetes zu unterstützen.
*Multus ist ein Multi-Plugin für Mehrfachnetzwerk-Unterstützung um alle CNI-Plugins (z.B. Calico, Cilium, Contiv, Flannel), zusätzlich zu SRIOV-, DPDK-, OVS-DPDK- und VPP-Basierten Workloads in Kubernetes zu unterstützen.
* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) bietet eine Integration zwischen VMware NSX-T und einem Orchestator wie z.B. Kubernetes. Außerdem bietet es eine Integration zwischen NSX-T und Containerbasierten CaaS/PaaS-Plattformen wie z.B. Pivotal Container Service (PKS) und OpenShift.
* [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) ist eine SDN-Plattform die Policy-Basiertes Networking zwischen Kubernetes Pods und nicht-Kubernetes Umgebungen inklusive Sichtbarkeit und Security-Monitoring bereitstellt.
* [Romana](http://romana.io) ist eine Layer 3 Network-Lösung für Pod-Netzwerke welche auch die [NetworkPolicy API](/docs/concepts/services-networking/network-policies/) unterstützt. Details zur Installation als kubeadm Add-On sind [hier](https://github.com/romana/romana/tree/master/containerize) verfügbar.
* [Romana](https://github.com/romana/romana) ist eine Layer 3 Network-Lösung für Pod-Netzwerke welche auch die [NetworkPolicy API](/docs/concepts/services-networking/network-policies/) unterstützt. Details zur Installation als kubeadm Add-On sind [hier](https://github.com/romana/romana/tree/master/containerize) verfügbar.
* [Weave Net](https://www.weave.works/docs/net/latest/kube-addon/) bietet Networking and Network-Policies und arbeitet auf beiden Seiten der Network-Partition ohne auf eine externe Datenbank angwiesen zu sein.
## Service-Discovery
@@ -52,5 +52,3 @@ Die Add-Ons in den einzelnen Kategorien sind alphabetisch sortiert - Die Reihenf
Es gibt einige weitere Add-Ons die in dem abgekündigten [cluster/addons](https://git.k8s.io/kubernetes/cluster/addons)-Verzeichnis dokumentiert sind.
Add-Ons die ordentlich gewartet werden dürfen gerne hier aufgezählt werden. Wir freuen uns auf PRs!
@@ -15,7 +15,7 @@ Diese Seite ist eine Übersicht über Kubernetes.
Kubernetes ist eine portable, erweiterbare Open-Source-Plattform zur Verwaltung von
containerisierten Arbeitslasten und Services, die sowohl die deklarative Konfiguration als auch die Automatisierung erleichtert.
Es hat einen großes, schnell wachsendes Ökosystem. Kubernetes Dienstleistungen, Support und Tools sind weit verbreitet.
Es hat ein großes, schnell wachsendes Ökosystem. Kubernetes Dienstleistungen, Support und Tools sind weit verbreitet.
Google hat das Kubernetes-Projekt 2014 als Open-Source-Projekt zur Verfügung gestellt. Kubernetes baut auf anderthalb Jahrzehnten
Erfahrung auf, die Google mit der Ausführung von Produktions-Workloads in großem Maßstab hat, kombiniert mit den besten Ideen und Praktiken der Community.
@@ -50,7 +50,7 @@ für Managementtools zu bieten, den Status von Kontrollpunkten zu ermitteln.
Darüber hinaus basiert die [Kubernetes-Steuerungsebene](/docs/concepts/overview/components/) auf den gleichen APIs,
die Entwicklern und Anwendern zur Verfügung stehen. Benutzer können ihre eigenen Controller, wie z.B.
[Scheduler](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/devel/scheduler.md), mit
[Scheduler](https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md), mit
ihren [eigenen APIs](/docs/concepts/api-extension/custom-resources/) schreiben, die von einem
universellen [Kommandozeilen-Tool](/docs/user-guide/kubectl-overview/) angesprochen werden können.
@@ -22,30 +22,30 @@ Da Mitwirkende nicht ihren eigenen Pull Request freigeben können, brauchst du m
Alle Lokalisierungsteams müssen sich mit ihren eigenen Ressourcen selbst tragen. Die Kubernetes-Website ist gerne bereit, deine Arbeit zu beherbergen, aber es liegt an dir, sie zu übersetzen.
### Finden deinen Zwei-Buchstaben-Sprachcode
### Ermittlung deines Zwei-Buchstaben-Sprachcodes
Rufe den [ISO 639-1 Standard](https://www.loc.gov/standards/iso639-2/php/code_list.php) auf und finde deinen Zwei-Buchstaben-Ländercode zur Lokalisierung. Zum Beispiel ist der Zwei-Buchstaben-Code für Korea `ko`.
### Duplizieren und klonen des Repositories
### Duplizieren und Klonen des Repositories
Als erstes [erstells du dir deine eigenes Duplikat](/docs/contribute/new-content/new-content/#fork-the-repo) vom [kubernetes/website] Repository.
Als erstes [erstellst du dir deine eigenes Duplikat](/docs/contribute/new-content/new-content/#fork-the-repo) vom [kubernetes/website] Repository.
Dann klonst du das Duplikat und `cd` hinein:
Dann klonst du das Duplikat und wechselst in das neu erstellte Verzeichnis:
```shell
git clone https://github.com/<username>/website
cd website
```
### Eröffne ein Pull Request
### Eröffnen eines Pull Requests
Als nächstes [eröffnest du einen Pull Request](/docs/contribute/new-content/open-a-pr/#open-a-pr) (PR) um eine Lokalisierung zum `kubernetes/website` Repository hinzuzufügen.
Der PR muss die [minimalen Inhaltsanforderungen](#mindestanforderungen) erfüllen bevor dieser genehmigt werden kann.
Der PR muss die [minimalen Inhaltsanforderungen](#mindestanforderungen) erfüllen, bevor dieser genehmigt werden kann.
Wie der PR für eine neue Lokalisierung aussieht kannst du dir an dem PR für die [Französische Dokumentation](https://github.com/kubernetes/website/pull/12548) ansehen.
Wie der PR für eine neue Lokalisierung aussieht, kannst du dir an dem PR für die [Französische Dokumentation](https://github.com/kubernetes/website/pull/12548) ansehen.
### Trete der Kubernetes GitHub Organisation bei
### Tritt der Kubernetes GitHub Organisation bei
Sobald du eine Lokalisierungs-PR eröffnet hast, kannst du Mitglied der Kubernetes GitHub Organisation werden. Jede Person im Team muss einen eigenen [Antrag auf Mitgliedschaft in der Organisation](https://github.com/kubernetes/org/issues/new/choose) im `kubernetes/org`-Repository erstellen.
@@ -94,9 +94,9 @@ contentDir = "content/de"
weight=3
```
Wenn du deinem Block einen Parameter `weight` zuweist, suche den Sprachblock mit dem höchsten Gewicht und addiere 1 zu diesem Wert.
Wenn du deinem Block einen Parameter `weight` zuweist, suche den Sprachblock mit dem höchsten Gewicht und addiere 1 zu diesem Wert.
Weitere Informationen zu Hugos Multilingualen Support findest du unter "[Multilingual Mode](https://gohugo.io/content-management/multilingual/)" auf in der Hugo Dokumentation.
Weitere Informationen zu Hugos multilingualem Support findest du unter "[Multilingual Mode](https://gohugo.io/content-management/multilingual/)" auf in der Hugo Dokumentation.
### Neuen Lokalisierungsordner erstellen
@@ -213,7 +213,7 @@ Die neueste Version ist {{< latest-version >}}, so dass der neueste Versionszwei
### Seitenverlinkung in der Internationalisierung
Lokalisierungen müssen den Inhalt von [`i18n/de.toml`](https://github.com/kubernetes/website/blob/master/i18n/en.toml) in einer neuen sprachspezifischen Datei enthalten. Als Beispiel: `i18n/de.toml`.
Lokalisierungen müssen den Inhalt von [`i18n/de.toml`](https://github.com/kubernetes/website/blob/main/i18n/en.toml) in einer neuen sprachspezifischen Datei enthalten. Als Beispiel: `i18n/de.toml`.
Füge eine neue Lokalisierungsdatei zu `i18n/` hinzu. Zum Beispiel mit Deutsch (`de`):
@@ -278,7 +278,7 @@ Die Teams müssen den lokalisierten Inhalt in demselben Versionszweig zusammenf
Ein Genehmiger muss einen Entwicklungszweig aufrechterhalten, indem er seinen Quellzweig auf dem aktuellen Stand hält und Merge-Konflikte auflöst. Je länger ein Entwicklungszweig geöffnet bleibt, desto mehr Wartung erfordert er in der Regel. Ziehe in Betracht, regelmäßig Entwicklungszweige zusammenzuführen und neue zu eröffnen, anstatt einen extrem lang laufenden Entwicklungszweig zu unterhalten.
Zu Beginn jedes Team-Meilensteins ist es hilfreich, ein Problem [Vergleich der Upstream-Änderungen](https://github.com/kubernetes/website/blob/master/scripts/upstream_changes.py) zwischen dem vorherigen Entwicklungszweig und dem aktuellen Entwicklungszweig zu öffnen.
Zu Beginn jedes Team-Meilensteins ist es hilfreich, ein Problem [Vergleich der Upstream-Änderungen](https://github.com/kubernetes/website/blob/main/scripts/upstream_changes.py) zwischen dem vorherigen Entwicklungszweig und dem aktuellen Entwicklungszweig zu öffnen.
Während nur Genehmiger einen neuen Entwicklungszweig eröffnen und Pull-Anfragen zusammenführen können, kann jeder eine Pull-Anfrage für einen neuen Entwicklungszweig eröffnen. Es sind keine besonderen Genehmigungen erforderlich.
@@ -301,5 +301,3 @@ Sobald eine Lokalisierung die Anforderungen an den Arbeitsablauf und die Mindest
- Die Sprachauswahl auf der Website aktivieren
- Die Verfügbarkeit der Lokalisierung über die Kanäle der [Cloud Native Computing Foundation](https://www.cncf.io/about/) (CNCF), einschließlich des [Kubernetes Blogs](https://kubernetes.io/blog/) veröffentlichen.
Die SIG Docs ist eine der [Special Interest Groups](https://github.com/kubernetes/community/blob/master/sig-list.md) (Fachspezifischen Interessengruppen) innerhalb des Kubernetes-Projekts, die sich auf das Schreiben, Aktualisieren und Pflegen der Dokumentation für Kubernetes als Ganzes konzentriert. Weitere Informationen über die SIG findest du unter SIG Docs im [GitHub Repository der Community](https://github.com/kubernetes/community/tree/master/sig-docs).
SIG Docs begrüßt Inhalte und Bewertungen von allen Mitwirkenden. Jeder kann einen
Pull Request (PR) eröffnen, und jeder ist willkommen, Fragen zum Inhalt zu stellen oder Kommentare
zu laufenden Pull Requests abzugeben.
Du kannst dich ausserdem als [Member](/de/docs/contribute/participate/roles-and-responsibilities/#member),
[Reviewer](/de/docs/contribute/participate/roles-and-responsibilities/#reviewer), oder
Diese Rollen erfordern einen erweiterten Zugriff und bringen bestimmte Verantwortlichkeiten zur Genehmigung und Bestätigung von Änderungen mit sich.
Unter [community-membership](https://github.com/kubernetes/community/blob/master/community-membership.md) findest du weitere Informationen darüber, wie die Mitgliedschaft in der Kubernetes-Community funktioniert.
Der Rest dieses Dokuments umreißt einige spezielle Vorgehensweisen dieser Rollen innerhalb von SIG Docs, die für die Pflege eines der öffentlichsten Aushängeschilder von Kubernetes verantwortlich ist - die Kubernetes-Website und die Dokumentation.
<!-- body -->
## SIG Docs Vorstand
Jede SIG, auch die SIG Docs, wählt ein oder mehrere SIG-Mitglieder, die als
Vorstand fungieren. Sie sind die Kontaktstellen zwischen der SIG Docs und anderen Teilen der
der Kubernetes-Organisation. Sie benötigen umfassende Kenntnisse über die Struktur
des Kubernetes-Projekts als Ganzes und wie SIG Docs darin arbeitet. Hier findest alle weiteren Informationen zu den aktuellen Vorsitzenden und der [Leitung](https://github.com/kubernetes/community/tree/master/sig-docs#leadership).
## SIG Docs-Teams und Automatisierung
Die Automatisierung in SIG Docs stützt sich auf zwei verschiedene Mechanismen:
GitHub-Teams und OWNERS-Dateien.
### GitHub Teams
Es gibt zwei Kategorien von SIG Docs [Teams](https://github.com/orgs/kubernetes/teams?query=sig-docs) auf GitHub:
-`@sig-docs-{language}-owners` sind Genehmiger und Verantwortliche
-`@sig-docs-{language}-reviewers` sind Reviewer
Jede Gruppe kann in GitHub-Kommentaren mit ihrem `@name` referenziert werden, um mit allen Mitgliedern dieser Gruppe zu kommunizieren.
Manchmal überschneiden sich Prow- und GitHub-Teams, ohne eine genaue Übereinstimmung. Für die Zuordnung von Issues, Pull-Requests und zur Unterstützung von PR-Genehmigungen verwendet die
Automatisierung die Informationen aus den `OWNERS`-Dateien.
### OWNERS Dateien und Front-Matter
Das Kubernetes-Projekt verwendet ein Automatisierungstool namens prow für die Automatisierung im Zusammenhang mit GitHub-Issues und Pull-Requests.
Das [Kubernetes-Website-Repository](https://github.com/kubernetes/website) verwendet zwei [prow-Plugins](https://github.com/kubernetes/test-infra/tree/master/prow/plugins):
- blunderbuss
- approve
Diese beiden Plugins nutzen die
[OWNERS](https://github.com/kubernetes/website/blob/main/OWNERS) und
Außerdem kann eine einzelne Markdown-Datei in ihrem Front-Matter (Vorspann) Reviewer und Genehmiger auflisten.
Entweder durch Auflistung einzelner GitHub-Benutzernamen oder GitHub-Gruppen.
Die Kombination aus OWNERS-Dateien und Front-Matter in Markdown-Dateien bestimmt, welche Empfehlungen PR-Eigentümer von automatisierten Systemen erhalten, und wen sie um eine technische und redaktionelle Überprüfung ihres PRs bitten sollen.
## So funktioniert das Zusammenführen
Wenn ein Pull Request mit der Branch (Ast) zusammengeführt wird, in dem der Inhalt bereitgestellt werden soll, wird dieser Inhalt auf http://kubernetes.io veröffentlicht. Um sicherzustellen, dass die Qualität der veröffentlichten Inhalte hoch ist, beschränken wir das Zusammenführen von Pull Requests auf
SIG Docs Freigabeberechtigte. So funktioniert es:
- Wenn eine Pull-Anfrage sowohl das `lgtm`- als auch das `approve`-Label hat, kein `hold`-Label hat,
und alle Tests bestanden sind, wird der Pull Request automatisch zusammengeführt.
- Jedes Kubernetes-Mitglied kann das `lgtm`-Label hinzufügen, indem es einen `/lgtm`-Kommentar hinzufügt.
- Mitglieder der Kubernetes-Organisation und SIG Docs-Genehmiger können kommentieren, um das automatische Zusammenführen eines Pull Requests zu verhindern (durch Hinzufügen eines `/hold`-Kommentars
kann ein vorheriger `/lgtm`-Kommentar zurückgehalten werden).
- Nur SIG Docs-Genehmiger können einen Pull Request zusammenführen indem sie einen `/approve` Kommentar hinzufügen.
Einige Genehmiger übernehmen auch weitere spezielle Rollen, wie zum Beispiel [PR Wrangler](/docs/contribute/participate/pr-wranglers/) oder [SIG Docs Vorsitzende](#sig-docs-chairperson).
## {{% heading "whatsnext" %}}
Weitere Informationen über die Mitarbeit an der Kubernetes-Dokumentation findest du unter:
SIG Docs [approvers](/docs/contribute/participate/roles-and-responsibilities/#approvers) übernehmen einwöchige Schichten um die [Pull Requests](https://github.com/kubernetes/website/wiki/PR-Wranglers) des Repositories zu verwalten.
Dieser Abschnitt behandelt die Aufgaben eines PR-Wranglers. Weitere Informationen über gute Reviews findest du unter [Überprüfen von Änderungen](/docs/contribute/review/).
<!-- body -->
## Aufgaben
Tägliche Aufgaben in einer einwöchigen Schicht als PR Wrangler:
- Sortiere und kennzeichne täglich eingehende Probleme. Siehe [Einstufung und Kategorisierung von Problemen](/docs/contribute/review/for-approvers/#triage-and-categorize-issues) für Richtlinien, wie SIG Docs Metadaten verwendet.
- Überprüfe [offene Pull Requests](https://github.com/kubernetes/website/pulls) auf Qualität und Einhaltung der [Style](/docs/contribute/style/style-guide/) und [Content](/docs/contribute/style/content-guide/) Leitfäden.
- Beginne mit den kleinsten PRs (`size/XS`) und ende mit den größten (`size/XXL`). Überprüfe so viele PRs, wie du kannst.
- Achte darauf, dass die PR-Autoren den [CLA](https://github.com/kubernetes/community/blob/master/CLA.md) unterschreiben.
- Verwende [dieses](https://github.com/zparnold/k8s-docs-pr-botherer) Skript, um diejenigen, die den CLA noch nicht unterschrieben haben, daran zu erinnern, dies zu tun.
- Gib Feedback zu den Änderungen und bitte die Mitglieder anderer SIGs um technische Überprüfung.
- Gib inline Vorschläge für die vorgeschlagenen inhaltlichen Änderungen in den PR ein.
- Wenn du den Inhalt überprüfen musst, kommentiere den PR und bitte um weitere Details.
- Vergebe das/die entsprechende(n) `sig/`-Label.
- Falls nötig, weise die Reviever aus dem Block `revievers:` im Vorspann der Datei zu.
- Benutze den Kommentar `/approve`, um einen PR zum Zusammenführen zu genehmigen. Führe den PR zusammen, wenn er inhaltlich und technisch einwandfrei ist.
- PRs sollten einen `/lgtm`-Kommentar von einem anderen Mitglied haben, bevor sie zusammengeführt werden.
- Erwäge, technisch korrekte Inhalte zu akzeptieren, die nicht den [Stilrichtlinien](/docs/contribute/style/style-guide/) entsprechen. Eröffne ein neues Thema mit dem Label `good first issue`, um Stilprobleme anzusprechen.
### Hilfreiche GitHub-Anfragen für Wranglers
Die folgenden Anfragen sind beim Wrangling hilfreich.
Wenn du diese Anfragen abgearbeitet hast, ist die verbleibende Liste der zu prüfenden PRs meist klein.
Diese Anfragen schließen Lokalisierungs-PRs aus. Alle Anfragen beziehen sich auf den `main`-Branch, außer der letzten.
- [Kein CLA, nicht zusammenfürbar](https://github.com/kubernetes/website/pulls?q=is%3Aopen+ist%3Apr+label%3A%22cncf-cla%3A+no%22+-label%3A%22do-not-merge%2Fwork-in-progress%22+-label%3A%22do-not-merge%2Fhold%22+label%3Alanguage%2Fen):
Erinnere den Beitragenden daran, den CLA zu unterschreiben. Wenn sowohl der Bot als auch ein Mensch sie daran erinnert haben, schließe
den PR und erinnere die Autoren daran, dass sie ihn erneut öffnen können, nachdem sie den CLA unterschrieben haben.
**Überprüfe keine PRs, deren Autoren den CLA nicht unterschrieben haben!**
Listet PRs auf, die ein LGTM von einem Mitglied benötigen. Wenn der PR eine technische Überprüfung benötigt, schalte einen der vom Bot vorgeschlagenen Reviewer ein. Wenn der Inhalt überarbeitet werden muss, füge Vorschläge und Feedback in-line hinzu.
- [Hat LGTM, braucht die Zustimmung von Docs](https://github.com/kubernetes/website/pulls?q=is%3Aopen+is%3Apr+-label%3Ado-not-merge%2Fwork-in-progress+-label%3Ado-not-merge%2Fhold+label%3Alanguage%2Fen+label%3Algtm+):
Listet PRs auf, die einen `/approve`-Kommentar benötigen, um zusammengeführt zu werden.
- [Quick Wins](https://github.com/kubernetes/website/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+base%3Amain+-label%3A%22do-not-merge%2Fwork-in-progress%22+-label%3A%22do-not-merge%2Fhold%22+label%3A%22cncf-cla%3A+yes%22+label%3A%22size%2FXS%22+label%3A%22language%2Fen%22): Listet PRs gegen den Hauptzweig auf, die nicht eindeutig blockiert sind. (ändere "XS" in der Größenbezeichnung, wenn du dich durch die PRs arbeitest [XS, S, M, L, XL, XXL]).
- [Nicht gegen den `main`-Branch](https://github.com/kubernetes/website/pulls?q=is%3Aopen+ist%3Apr+label%3Alanguage%2Fen+-base%3Amain): Wenn der PR gegen einen `dev-`Ast gerichtet ist, ist er für eine kommende Veröffentlichung. Weise diesen dem [Docs Release Manager](https://github.com/kubernetes/sig-release/tree/master/release-team#kubernetes-release-team-roles) zu: `/assign @<manager's_github-username>`. Wenn der PR gegen einen alten Ast gerichtet ist, hilf dem Autor herauszufinden, ob er auf den richtigen Ast gerichtet ist.
### Hilfreiche Prow-Befehle für Wranglers
```
# Englisches Label hinzufügen
/language en
# füge dem PR ein Squash-Label hinzu, wenn es mehr als einen Commit gibt
/label tide/merge-method-squash
# einen PR ueber Prow neu betiteln (z.B. als Work-in-Progress [WIP])
/retitle [WIP] <TITLE>
```
### Wann sind Pull Requests zu schließen
Reviews und Genehmigungen sind ein Mittel, um unsere PR-Warteschlange kurz und aktuell zu halten. Ein weiteres Mittel ist das Schließen.
PRs werden geschlossen, wenn:
- Der Autor den CLA seit zwei Wochen nicht unterschrieben hat.
Die Autoren können den PR wieder öffnen, nachdem sie den CLA unterschrieben haben. Dies ist ein risikoarmer Weg, um sicherzustellen, dass nichts zusammengeführt wird, ohne dass ein CLA unterzeichnet wurde.
- Der Autor hat seit Zwei oder mehr Wochen nicht auf Kommentare oder Feedback geantwortet.
Hab keine Angst, Pull Requests zu schließen. Mitwirkende können sie leicht wieder öffnen und die laufenden Arbeiten fortsetzen. Oft ist es die Nachricht über die Schließung, die einen Autor dazu anspornt, seinen Beitrag wieder aufzunehmen und zu beenden.
Um eine Pull-Anfrage zu schließen, hinterlasse einen `/close`-Kommentar zu dem PR.
{{< note >}}
Der [`fejta-bot`](https://github.com/fejta-bot) Bot markiert Themen nach 90 Tagen Inaktivität als veraltet. Nach weiteren 30 Tagen markiert er Issues als faul und schließt sie. PR-Beauftragte sollten Themen nach 14-30 Tagen Inaktivität schließen.
Jeder kann zu Kubernetes beitragen. Wenn deine Beiträge zu SIG Docs wachsen, kannst du dich für verschiedene Stufen der Mitgliedschaft in der Community bewerben.
Diese Rollen ermöglichen es dir, mehr Verantwortung innerhalb der Gemeinschaft zu übernehmen.
Jede Rolle erfordert mehr Zeit und Engagement. Die Rollen sind:
- Jeder: kann regelmäßig zur Kubernetes-Dokumentation beitragen
- Member: können Issues zuweisen und einstufen und Pull Requests unverbindlich prüfen
- Reviewer: können die Überprüfung von Dokumentations-Pull-Requests leiten und für die Qualität einer Änderung bürgen
- Approver: können die Überprüfung von Dokumentations- und Merge-Änderungen leiten
<!-- body -->
## Jeder
Jeder mit einem GitHub-Konto kann zu Kubernetes beitragen. SIG Docs heißt alle neuen Mitwirkenden willkommen!
Jeder kann:
- Ein Problem in einem beliebigen [Kubernetes](https://github.com/kubernetes/)
1. Finde zwei [Reviewer](#reviewers) oder [Approver](#approvers), die deine Mitgliedschaft [sponsern](/docs/contribute/advanced#sponsor-a-new-contributor).
Bitte um Sponsoring im [#sig-docs channel on Slack](https://kubernetes.slack.com) oder auf der
Die Ausführlichkeit von Kubectl wird mit den Flags `-v` oder `--v` gesteuert, gefolgt von einer Ganzzahl, die die Protokollebene darstellt. Allgemeine Protokollierungskonventionen für Kubernetes und die zugehörigen Protokollebenen werden [hier](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md) beschrieben.
Die Ausführlichkeit von Kubectl wird mit den Flags `-v` oder `--v` gesteuert, gefolgt von einer Ganzzahl, die die Protokollebene darstellt. Allgemeine Protokollierungskonventionen für Kubernetes und die zugehörigen Protokollebenen werden [hier](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md) beschrieben.
@@ -100,7 +100,7 @@ Das auflisten der Autoskalierer geschieht über `kubectl get hpa` und eine detai
Letzendlich können wir einen Autoskalierer mit `kubectl delete hpa` löschen.
Zusätzlich gibt es einen speziellen Befehl `kubectl autoscale` zur einfachen Erstellung eines Horizontal Pod Autoscalers.
Wenn du beispielsweise `kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80` ausführst, wird ein Autoskalierer für den Replication Set *foo* erstellt, wobei die Ziel-CPU-Auslastung auf `80%` und die Anzahl der Replikate zwischen 2 und 5 gesetzt wird.
Wenn du beispielsweise `kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80` ausführst, wird ein Autoskalierer für den ReplicaSet *foo* erstellt, wobei die Ziel-CPU-Auslastung auf `80%` und die Anzahl der Replikate zwischen 2 und 5 gesetzt wird.
Die Detaildokumentation von `kubectl autoscale` kann [hier](/docs/reference/generated/kubectl/kubectl-commands/#autoscale) gefunden werden.
@@ -48,7 +48,7 @@ Kubernetes is open source giving you the freedom to take advantage of on-premise
<br>
<br>
<br>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21"buttonid="desktopKCButton">Attend KubeCon North America on October 24-28, 2022</a>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna22"buttonid="desktopKCButton">Attend KubeCon North America on October 24-28, 2022</a>
- [Node authorizer](/docs/reference/access-authn-authz/node/) and admission control plugin are new additions that restrict kubelet’s access to secrets, pods and other objects based on its node.
- [Encryption for Secrets](/docs/tasks/administer-cluster/encrypt-data/), and other resources in etcd, is now available as alpha.
- [Kubelet TLS bootstrapping](/docs/admin/kubelet-tls-bootstrapping/) now supports client and server certificate rotation.
- [Audit logs](/docs/tasks/debug-application-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit.
- [Audit logs](/docs/tasks/debug/debug-cluster/audit/) stored by the API server are now more customizable and extensible with support for event filtering and webhooks. They also provide richer data for system audit.
**_Editor's note: Today's post is by Lantao Liu, Software Engineer at Google, and Mike Brown, Open Source Developer Advocate at IBM._**
**Authors:** Lantao Liu (Google), and Mike Brown (IBM)
_Update: Kubernetes support for Docker via `dockershim` is now deprecated.
For more information, read the [deprecation notice](/blog/2020/12/08/kubernetes-1-20-release-announcement/#dockershim-deprecation).
You can also discuss the deprecation via a dedicated [GitHub issue](https://github.com/kubernetes/kubernetes/issues/106917)._
A _container runtime_ is software that executes containers and manages container images on a node. Today, the most widely known container runtime is [Docker](https://www.docker.com/), but there are other container runtimes in the ecosystem, such as [rkt](https://coreos.com/rkt/), [containerd](https://containerd.io/), and [lxd](https://linuxcontainers.org/lxd/). Docker is by far the most common container runtime used in production Kubernetes environments, but Docker’s smaller offspring, containerd, may prove to be a better option. This post describes using containerd with Kubernetes.
@@ -117,7 +117,7 @@ To achieve the best possible isolation, each function call would have to happen
By using Landlock, we could isolate function calls from each other within the same container, making a temporary file created by one function call inaccessible to the next function call, for example. Integration between Landlock and technologies like Kubernetes-based serverless frameworks would be a ripe area for further exploration.
## Auditing kubectl-exec with eBPF
In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug-application-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster.
In Kubernetes 1.7 the [audit proposal](/docs/tasks/debug/debug-cluster/audit/) started making its way in. It's currently pre-stable with plans to be stable in the 1.10 release. As the name implies, it allows administrators to log and audit events that take place in a Kubernetes cluster.
While these events log Kubernetes events, they don't currently provide the level of visibility that some may require. For example, while we can see that someone has used `kubectl exec` to enter a container, we are not able to see what commands were executed in that session. With eBPF one can attach a BPF program that would record any commands executed in the `kubectl exec` session and pass those commands to a user-space program that logs those events. We could then play that session back and know the exact sequence of events that took place.
@@ -66,7 +66,7 @@ There are plenty of [good examples](https://docs.bitnami.com/kubernetes/how-to/c
Incorrect or excessively permissive RBAC policies are a security threat in case of a compromised pod. Maintaining least privilege, and continuously reviewing and improving RBAC rules, should be considered part of the "technical debt hygiene" that teams build into their development lifecycle.
[Audit Logging](/docs/tasks/debug-application-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation's security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started.
[Audit Logging](/docs/tasks/debug/debug-cluster/audit/) (beta in 1.10) provides customisable API logging at the payload (e.g. request and response), and also metadata levels. Log levels can be tuned to your organisation's security policy - [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging#audit_policy) provides sane defaults to get you started.
For read requests such as get, list, and watch, only the request object is saved in the audit logs; the response object is not. For requests involving sensitive data such as Secret and ConfigMap, only the metadata is exported. For all other requests, both request and response objects are saved in audit logs.
@@ -174,7 +174,7 @@ Cluster-distributed stateful services (e.g., Cassandra) can benefit from splitti
## Other considerations
[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location.
[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug/debug-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location.
Your production deployment should utilize an automated installation, configuration and update tool (e.g., [Ansible](https://github.com/kubernetes-incubator/kubespray), [BOSH](https://github.com/cloudfoundry-incubator/kubo-deployment), [Chef](https://github.com/chef-cookbooks/kubernetes), [Juju](/docs/getting-started-guides/ubuntu/installation/), [kubeadm](/docs/reference/setup-tools/kubeadm/), [Puppet](https://forge.puppet.com/puppetlabs/kubernetes), etc.). A manual process will have repeatability issues, be labor intensive, error prone, and difficult to scale. [Certified distributions](https://www.cncf.io/certification/software-conformance/#logos) are likely to include a facility for retaining configuration settings across updates, but if you implement your own install and config toolchain, then retention, backup and recovery of the configuration artifacts is essential. Consider keeping your deployment components and settings under a version control system such as Git.
_Built-in gRPC probes were introduced in Kubernetes 1.23. To learn more, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe)._
**Author**: [Ahmet Alp Balkan](https://twitter.com/ahmetb) (Google)
**Update (December 2021):** _Kubernetes now has built-in gRPC health probes starting in v1.23.
To learn more, see [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe).
This article was originally written about an external tool to achieve the same task._
[gRPC](https://grpc.io) is on its way to becoming the lingua franca for
communication between cloud-native microservices. If you are deploying gRPC
applications to Kubernetes today, you may be wondering about the best way to
This morning [a container escape vulnerability in runc was announced](https://www.openwall.com/lists/oss-security/2019/02/11/2). We wanted to provide some guidance to Kubernetes users to ensure everyone is safe and secure.
## What Is Runc?
## What is runc?
Very briefly, runc is the low-level tool which does the heavy lifting of spawning a Linux container. Other tools like Docker, Containerd, and CRI-O sit on top of runc to deal with things like data formatting and serialization, but runc is at the heart of all of these systems.
Kubernetes in turn sits on top of those tools, and so while no part of Kubernetes itself is vulnerable, most Kubernetes installations are using runc under the hood.
### What Is The Vulnerability?
### What is the vulnerability?
While full details are still embargoed to give people time to patch, the rough version is that when running a process as root (UID 0) inside a container, that process can exploit a bug in runc to gain root privileges on the host running the container. This then allows them unlimited access to the server as well as any other containers on that server.
@@ -19,13 +22,14 @@ If the process inside the container is either trusted (something you know is not
The most common source of risk is attacker-controller container images, such as unvetted images from public repositories.
### What Should I Do?
### What should i do?
As with all security issues, the two main options are to mitigate the vulnerability or upgrade your version of runc to one that includes the fix.
As the exploit requires UID 0 within the container, a direct mitigation is to ensure all your containers are running as a non-0 user. This can be set within the container image, or via your pod specification:
```yaml
---
apiVersion: v1
kind: Pod
metadata:
@@ -39,6 +43,7 @@ spec:
This can also be enforced globally using a PodSecurityPolicy:
```yaml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
@@ -89,7 +94,7 @@ We don't have specific confirmation that Docker for Mac and Docker for Windows a
If you are unable to upgrade Docker, the Rancher team has provided backports of the fix for many older versions at [github.com/rancher/runc-cve](https://github.com/rancher/runc-cve).
## Getting More Information
## Getting more information
If you have any further questions about how this vulnerability impacts Kubernetes, please join us at [discuss.kubernetes.io](https://discuss.kubernetes.io/).
Metrics are a fast way to check whether deprecated APIs are being used, and at what rate,
but they don't include enough information to identify particular clients or API objects.
Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug-application-cluster/audit/)
Starting in Kubernetes v1.19, [audit events](/docs/tasks/debug/debug-cluster/audit/)
for requests to deprecated APIs include an audit annotation of `"k8s.io/deprecated":"true"`.
Administrators can use those audit events to identify specific clients or objects that need to be updated.
@@ -327,7 +328,3 @@ A couple areas we're looking at next are warning about [known problematic values
we cannot reject outright for compatibility reasons, and warning about use of deprecated fields or field values
(like selectors using beta os/arch node labels, [deprecated in v1.14](/docs/reference/labels-annotations-taints/#beta-kubernetes-io-arch-deprecated)).
I'm excited to see progress in this area, continuing to make it easier to use Kubernetes.
---
_[Jordan Liggitt](https://twitter.com/liggitt) is a software engineer at Google, and helps lead Kubernetes authentication, authorization, and API efforts._
@@ -18,9 +18,9 @@ The paper attempts to _not_ focus on any specific [cloud native project](https:/
## Kubernetes native security controls
When using Kubernetes as a workload orchestrator, some of the security controls this version of the whitepaper recommends are:
* [Pod Security Policies](/docs/concepts/policy/pod-security-policy/): Implement a single source of truth for “least privilege” workloads across the entire cluster
* [Pod Security Policies](/docs/concepts/security/pod-security-policy/): Implement a single source of truth for “least privilege” workloads across the entire cluster
* [Resource requests and limits](/docs/concepts/configuration/manage-resources-containers/#requests-and-limits): Apply requests (soft constraint) and limits (hard constraint) for shared resources such as memory and CPU
* [Audit log analysis](/docs/tasks/debug-application-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events
* [Audit log analysis](/docs/tasks/debug/debug-cluster/audit/): Enable Kubernetes API auditing and filtering for security relevant events
* [Control plane authentication and certificate root of trust](/docs/concepts/architecture/control-plane-node-communication/): Enable mutual TLS authentication with a trusted CA for communication within the cluster
* [Secrets management](/docs/concepts/configuration/secret/): Integrate with a built-in or external secrets store
_**Update**: There is a [newer version](/blog/2022/02/17/dockershim-faq/) of this article available._
This document goes over some frequently asked questions regarding the Dockershim
deprecation announced as a part of the Kubernetes v1.20 release. For more detail
on the deprecation of Docker as a container runtime for Kubernetes kubelets, and
what that means, check out the blog post
[Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/).
Also, you can read [check whether Dockershim deprecation affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/) to check whether it does.
### Why is dockershim being deprecated?
Maintaining dockershim has become a heavy burden on the Kubernetes maintainers.
@@ -151,7 +155,7 @@ runtime where possible.
Another thing to look out for is anything expecting to run for system maintenance
or nested inside a container when building images will no longer work. For the
former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
latter you can use newer container build options like [img], [buildah],
[kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker.
@@ -31,7 +32,7 @@ The `kubectl alpha debug` features graduates to beta in 1.20, becoming `kubectl
Note that as a new built-in command, `kubectl debug` takes priority over any kubectl plugin named “debug”. You must rename the affected plugin.
Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/).
Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/).
@@ -21,7 +21,7 @@ Until then, PSP is still PSP. There will be at least a year during which the new
## What is PodSecurityPolicy?
[PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) is a built-in [admission controller](/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/) that allows a cluster administrator to control security-sensitive aspects of the Pod specification.
[PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) is a built-in [admission controller](/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/) that allows a cluster administrator to control security-sensitive aspects of the Pod specification.
First, one or more PodSecurityPolicy resources are created in a cluster to define the requirements Pods must meet. Then, RBAC rules are created to control which PodSecurityPolicy applies to a given pod. If a pod meets the requirements of its PSP, it will be admitted to the cluster as usual. In some cases, PSP can also modify Pod fields, effectively creating new defaults for those fields. If a Pod does not meet the PSP requirements, it is rejected, and cannot run.
@@ -30,15 +30,15 @@ This led to design principles that allow the Gateway API to improve upon Ingress
The Gateway API introduces a few new resource types:
- **[GatewayClasses](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.GatewayClass)** are cluster-scoped resources that act as templates to explicitly define behavior for Gateways derived from them. This is similar in concept to StorageClasses, but for networking data-planes.
- **[Gateways](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.Gateway)** are the deployed instances of GatewayClasses. They are the logical representation of the data-plane which performs routing, which may be in-cluster proxies, hardware LBs, or cloud LBs.
- **Routes** are not a single resource, but represent many different protocol-specific Route resources. The [HTTPRoute](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.HTTPRoute) has matching, filtering, and routing rules that get applied to Gateways that can process HTTP and HTTPS traffic. Similarly, there are [TCPRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.TCPRoute), [UDPRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.UDPRoute), and [TLSRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.TLSRoute) which also have protocol-specific semantics. This model also allows the Gateway API to incrementally expand its protocol support in the future.
- **[GatewayClasses](https://gateway-api.sigs.k8s.io/v1alpha1/references/spec/#networking.x-k8s.io/v1alpha1.GatewayClass)** are cluster-scoped resources that act as templates to explicitly define behavior for Gateways derived from them. This is similar in concept to StorageClasses, but for networking data-planes.
- **[Gateways](https://gateway-api.sigs.k8s.io/v1alpha1/references/spec/#networking.x-k8s.io/v1alpha1.Gateway)** are the deployed instances of GatewayClasses. They are the logical representation of the data-plane which performs routing, which may be in-cluster proxies, hardware LBs, or cloud LBs.
- **Routes** are not a single resource, but represent many different protocol-specific Route resources. The [HTTPRoute](https://gateway-api.sigs.k8s.io/v1alpha1/references/spec/#networking.x-k8s.io/v1alpha1.HTTPRoute) has matching, filtering, and routing rules that get applied to Gateways that can process HTTP and HTTPS traffic. Similarly, there are [TCPRoutes](https://gateway-api.sigs.k8s.io/v1alpha1/references/spec/#networking.x-k8s.io/v1alpha1.TCPRoute), [UDPRoutes](https://gateway-api.sigs.k8s.io/v1alpha1/references/spec/#networking.x-k8s.io/v1alpha1.UDPRoute), and [TLSRoutes](https://gateway-api.sigs.k8s.io/v1alpha1/references/spec/#networking.x-k8s.io/v1alpha1.TLSRoute) which also have protocol-specific semantics. This model also allows the Gateway API to incrementally expand its protocol support in the future.
### Gateway Controller Implementations
The good news is that although Gateway is in [Alpha](https://github.com/kubernetes-sigs/gateway-api/releases), there are already several [Gateway controller implementations](https://gateway-api.sigs.k8s.io/references/implementations/) that you can run. Since it’s a standardized spec, the following example could be run on any of them and should function the exact same way. Check out [getting started](https://gateway-api.sigs.k8s.io/guides/getting-started/) to see how to install and use one of these Gateway controllers.
The good news is that although Gateway is in [Alpha](https://github.com/kubernetes-sigs/gateway-api/releases), there are already several [Gateway controller implementations](https://gateway-api.sigs.k8s.io/implementations/) that you can run. Since it’s a standardized spec, the following example could be run on any of them and should function the exact same way. Check out [getting started](https://gateway-api.sigs.k8s.io/v1alpha1/guides/getting-started/) to see how to install and use one of these Gateway controllers.
## Getting Hands-on with the Gateway API
@@ -134,7 +134,7 @@ spec:
So we have two HTTPRoutes matching and routing traffic to different Services. You might be wondering, where are these Services accessible? Through which networks or IPs are they exposed?
How Routes are exposed to clients is governed by [Route binding](https://gateway-api.sigs.k8s.io/concepts/api-overview/#route-binding), which describes how Routes and Gateways create a bidirectional relationship between each other. When Routes are bound to a Gateway it means their collective routing rules are configured on the underlying load balancers or proxies and the Routes are accessible through the Gateway. Thus, a Gateway is a logical representation of a networking data plane that can be configured through Routes.
How Routes are exposed to clients is governed by [Route binding](https://gateway-api.sigs.k8s.io/concepts/api-overview/#route-resources), which describes how Routes and Gateways create a bidirectional relationship between each other. When Routes are bound to a Gateway it means their collective routing rules are configured on the underlying load balancers or proxies and the Routes are accessible through the Gateway. Thus, a Gateway is a logical representation of a networking data plane that can be configured through Routes.
@@ -192,6 +192,6 @@ When you put it all together, you have a single load balancing infrastructure th
There are many resources to check out to learn more.
* Check out the [user guides](https://gateway-api.sigs.k8s.io/guides/getting-started/) to see what use-cases can be addressed.
* Try out one of the [existing Gateway controllers ](https://gateway-api.sigs.k8s.io/references/implementations/)
* Check out the [user guides](https://gateway-api.sigs.k8s.io/v1alpha1/guides/getting-started/) to see what use-cases can be addressed.
* Try out one of the [existing Gateway controllers ](https://gateway-api.sigs.k8s.io/implementations/)
* Or [get involved](https://gateway-api.sigs.k8s.io/contributing/community/) and help design and influence the future of Kubernetes service networking!
To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=orphan option for `kubectl delete` to delete an object and orphan its children.
To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=orphan option for `kubectl delete` to delete an object and orphan its children. *Update: starting with kubectl v1.20, the default for cascade is `background`.*
In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=orphan, the parent is deleted but the child still exists:
@@ -175,7 +175,7 @@ require internet access to update the vulnerability database.
### Pod Security Policies
Since Kubernetes v1.21, the [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/)
Since Kubernetes v1.21, the [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/)
API and related features are [deprecated](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/),
but some of the guidance in this section will still apply for the next few years, until cluster operators
upgrade their clusters to newer Kubernetes versions.
@@ -317,7 +317,7 @@ RequestResponse's including metadata and request / response bodies. While helpfu
Each organization needs to evaluate their
own threat model and build an audit policy that complements or helps troubleshooting incident response. Think
about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug-application-cluster/audit/#audit-policy).
about how someone would attack your organization and what audit trail could identify it. Review more advanced options for tuning audit logs in the official [audit logging documentation](/docs/tasks/debug/debug-cluster/audit/#audit-policy).
It's crucial to tune your audit logs to only include events that meet your threat model. A minimal audit policy that logs everything at `metadata` level can also be a good starting point.
Audit logging configurations can also be tested with
@@ -138,4 +138,5 @@ Stay tuned for what comes next, and if you have any questions, comments or sugge
* Chat with us on the Kubernetes [Slack](http://slack.k8s.io/):[#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T)
* Join the SIG Cluster Lifecycle [Google Group](https://groups.google.com/g/kubernetes-sig-cluster-lifecycle) to receive calendar invites and gain access to documents
* Join our [Zoom meeting](https://zoom.us/j/861487554), every Wednesday at 10:00 Pacific Time
* Check out the [ClusterClass tutorial](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-classes.html) in the Cluster API book.
* Check out the [ClusterClass quick-start](https://cluster-api.sigs.k8s.io/user/quick-start.html) for the Docker provider (CAPD) in the Cluster API book.
* _UPDATE_: Check out the [ClusterClass experimental feature](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-class/index.html) documentation in the Cluster API book.
@@ -18,7 +18,7 @@ One of the key security principles for running containers in Kubernetes is the
principle of least privilege. The Pod/container `securityContext` specifies the config
options to set, e.g., Linux capabilities, MAC policies, and user/group ID values to achieve this.
Furthermore, the cluster admins are supported with tools like [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) (deprecated) or
Furthermore, the cluster admins are supported with tools like [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) (deprecated) or
[Pod Security Admission](/docs/concepts/security/pod-security-admission/) (alpha) to enforce the desired security settings for pods that are being deployed in
the cluster. These settings could, for instance, require that containers must be `runAsNonRoot` or
that they are forbidden from running with root's group ID in `runAsGroup` or `supplementalGroups`.
Last year we announced that Dockershim is being deprecated: [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/).
Our current plan is to remove dockershim from the Kubernetes codebase soon.
We are looking for feedback from you whether you are ready for dockershim
removal and to ensure that you are ready when the time comes.
**Please fill out this survey: https://forms.gle/svCJmhvTv78jGdSx8**.
<del>Please fill out this survey: https://forms.gle/svCJmhvTv78jGdSx8</del>
The dockershim component that enables Docker as a Kubernetes container runtime is
being deprecated in favor of runtimes that directly use the [Container Runtime Interface](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)
@@ -25,7 +37,7 @@ are still not ready: [migrating telemetry and security agents](/docs/tasks/admin
At this point, we believe that there is feature parity between Docker and the
other runtimes. Many end-users have used our [migration guide](/docs/tasks/administer-cluster/migrating-from-dockershim/)
and are running production workload using these different runtimes. The plan of
record today is that dockershim will be removed in version 1.24, slated for
record today is that dockershim will be removed in version 1.24, slated for
release around April of next year. For those developing or running alpha and
beta versions, dockershim will be removed in December at the beginning of the
1.24 release development cycle.
@@ -33,7 +45,7 @@ beta versions, dockershim will be removed in December at the beginning of the
There is only one month left to give us feedback. We want you to tell us how
ready you are.
**We are collecting opinions through this survey: [https://forms.gle/svCJmhvTv78jGdSx8](https://forms.gle/svCJmhvTv78jGdSx8)**
<del>We are collecting opinions through this survey: https://forms.gle/svCJmhvTv78jGdSx8</del>
To better understand preparedness for the dockershim removal, our survey is
asking the version of Kubernetes you are currently using, and an estimate of
when you think you will adopt Kubernetes 1.24. All the aggregated information
We’re pleased to announce the release of Kubernetes 1.23, the last release of 2021!
This release consists of 47 enhancements: 11 enhancements have graduated to stable, 17 enhancements are moving to beta, and 19 enhancements are entering alpha. Also, 1 feature has been deprecated.
## Major Themes
### Deprecation of FlexVolume
FlexVolume is deprecated. The out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes. See [this doc](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors) for more information. Maintainers of FlexVolume drivers should implement a CSI driver and move users of FlexVolume to CSI. Users of FlexVolume should move their workloads to the CSI driver.
### Deprecation of klog specific flags
To simplify the code base, several [logging flags were marked as deprecated](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog) in Kubernetes 1.23. The code which implements them will be removed in a future release, so users of those need to start replacing the deprecated flags with some alternative solutions.
### Software Supply Chain SLSA Level 1 Compliance in the Kubernetes Release Process
Kubernetes releases now generate provenance attestation files describing the staging and release phases of the release process. Artifacts are now verified as they are handed over from one phase to the next. This final piece completes the work needed to comply with Level 1 of the [SLSA security framework](https://slsa.dev/) (Supply-chain Levels for Software Artifacts).
### IPv4/IPv6 Dual-stack Networking graduates to GA
[IPv4/IPv6 dual-stack networking](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack) graduates to GA. Since 1.21, Kubernetes clusters have been enabled to support dual-stack networking by default. In 1.23, the `IPv6DualStack` feature gate is removed. The use of dual-stack networking is not mandatory. Although clusters are enabled to support dual-stack networking, Pods and Services continue to default to single-stack. To use dual-stack networking Kubernetes nodes must have routable IPv4/IPv6 network interfaces, a dual-stack capable CNI network plugin must be used, Pods must be configured to be dual-stack and Services must have their `.spec.ipFamilyPolicy` field set to either `PreferDualStack` or `RequireDualStack`.
### HorizontalPodAutoscaler v2 graduates to GA
The HorizontalPodAutscaler `autoscaling/v2` stable API moved to GA in 1.23. The HorizontalPodAutoscaler `autoscaling/v2beta2` API has been deprecated.
### Generic Ephemeral Volume feature graduates to GA
The generic ephemeral volume feature moved to GA in 1.23. This feature allows any existing storage driver that supports dynamic provisioning to be used as an ephemeral volume with the volume’s lifecycle bound to the Pod. All StorageClass parameters for volume provisioning and all features supported with PersistentVolumeClaims are supported.
### Skip Volume Ownership change graduates to GA
The feature to configure volume permission and ownership change policy for Pods moved to GA in 1.23. This allows users to skip recursive permission changes on mount and speeds up the pod start up time.
### Allow CSI drivers to opt-in to volume ownership and permission change graduates to GA
The feature to allow CSI Drivers to declare support for fsGroup based permissions graduates to GA in 1.23.
### PodSecurity graduates to Beta
[PodSecurity](https://kubernetes.io/docs/concepts/security/pod-security-admission/) moves to Beta. `PodSecurity` replaces the deprecated `PodSecurityPolicy` admission controller. `PodSecurity` is an admission controller that enforces Pod Security Standards on Pods in a Namespace based on specific namespace labels that set the enforcement level. In 1.23, the `PodSecurity` feature gate is enabled by default.
### Container Runtime Interface (CRI) v1 is default
The Kubelet now supports the CRI `v1` API, which is now the project-wide default.
If a container runtime does not support the `v1` API, Kubernetes will fall back to the `v1alpha2` implementation. There is no intermediate action required by end-users, because `v1` and `v1alpha2` do not differ in their implementation. It is likely that `v1alpha2` will be removed in one of the future Kubernetes releases to be able to develop `v1`.
### Structured logging graduate to Beta
Structured logging reached its Beta milestone. Most log messages from kubelet and kube-scheduler have been converted. Users are encouraged to try out JSON output or parsing of the structured text format and provide feedback on possible solutions for the open issues, such as handling of multi-line strings in log values.
### Simplified Multi-point plugin configuration for scheduler
The kube-scheduler is adding a new, simplified config field for Plugins to allow multiple extension points to be enabled in one spot. The new `multiPoint` plugin field is intended to simplify most scheduler setups for administrators. Plugins that are enabled via `multiPoint` will automatically be registered for each individual extension point that they implement. For example, a plugin that implements Score and Filter extensions can be simultaneously enabled for both. This means entire plugins can be enabled and disabled without having to manually edit individual extension point settings. These extension points can now be abstracted away due to their irrelevance for most users.
### CSI Migration updates
CSI Migration enables the replacement of existing in-tree storage plugins such as `kubernetes.io/gce-pd` or `kubernetes.io/aws-ebs` with a corresponding CSI driver.
If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference.
After migration, Kubernetes users may continue to rely on all the functionality of in-tree storage plugins using the existing interface.
- CSI Migration feature is turned on by default but stays in Beta for GCE PD, AWS EBS, and Azure Disk in 1.23.
- CSI Migration is introduced as an Alpha feature for Ceph RBD and Portworx in 1.23.
### Expression language validation for CRD is alpha
Expression language validation for CRD is in alpha starting in 1.23. If the `CustomResourceValidationExpressions` feature gate is enabled, custom resources will be validated by validation rules using the [Common Expression Language (CEL)](https://github.com/google/cel-spec).
### Server Side Field Validation is Alpha
If the `ServerSideFieldValidation` feature gate is enabled starting 1.23, users will receive warnings from the server when they send Kubernetes objects in the request that contain unknown or duplicate fields. Previously unknown fields and all but the last duplicate fields would be dropped by the server.
With the feature gate enabled, we also introduce the `fieldValidation` query parameter so that users can specify the desired behavior of the server on a per request basis. Valid values for the `fieldValidation` query parameter are:
- Ignore (default when feature gate is disabled, same as pre-1.23 behavior of dropping/ignoring unkonwn fields)
- Warn (default when feature gate is enabled).
- Strict (this will fail the request with an Invalid Request error)
### OpenAPI v3 is Alpha
If the `OpenAPIV3` feature gate is enabled starting 1.23, users will be able to request the OpenAPI v3.0 spec for all Kubernetes types. OpenAPI v3 aims to be fully transparent and includes support for a set of fields that are dropped when publishing OpenAPI v2: `default`, `nullable`, `oneOf`, `anyOf`. A separate spec is published per Kubernetes group version (at the `$cluster/openapi/v3/apis/<group>/<version>` endpoint) for improved performance and discovery, for all group versions can be found at the `$cluster/openapi/v3` path.
- [Graduate HPA API to GA](https://github.com/kubernetes/enhancements/issues/2702)
### Major Changes
- [Priority and Fairness for API Server Requests](https://github.com/kubernetes/enhancements/issues/1040)
### Release Notes
Check out the full details of the Kubernetes 1.23 release in our [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md).
### Availability
Kubernetes 1.23 is available for download on [GitHub](https://github.com/kubernetes/kubernetes/releases/tag/v1.23.0). To get started with Kubernetes, check out these [interactive tutorials](https://kubernetes.io/docs/tutorials/) or run local Kubernetes clusters using Docker container “nodes” with [kind](https://kind.sigs.k8s.io/). You can also easily install 1.23 using [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/).
### Release Team
This release was made possible by a very dedicated group of individuals, who came together as a team to deliver technical content, documentation, code, and a host of other components that go into every Kubernetes release.
A huge thank you to the release lead Rey Lejano for leading us through a successful release cycle, and to everyone else on the release team for supporting each other, and working so hard to deliver the 1.23 release for the community.
"The Next Frontier" theme represents the new and graduated enhancements in 1.23, Kubernetes' history of Star Trek references, and the growth of community members in the release team.
Kubernetes has a history of Star Trek references. The original codename for Kubernetes within Google is Project 7, a reference to Seven of Nine from Star Trek Voyager. And of course Borg was the name for the predecessor to Kubernetes. "The Next Frontier" theme continues the Star Trek references. "The Next Frontier" is a fusion of two Star Trek titles, Star Trek V: The Final Frontier and Star Trek the Next Generation.
"The Next Frontier" represents a line in the SIG Release charter, "Ensure there is a consistent group of community members in place to support the release process across time." With each release team, we grow the community with new release team members and for many it's their first contribution in their open source frontier.
The Kubernetes 1.23 release logo continues with the theme's Star Trek reference. Every star is a helm from the Kubernetes logo. The ship represents the collective teamwork of the release team.
Rey Lejano designed the logo.
### User Highlights
- [Findings of the latest CNCF End User Technology Radar](https://www.cncf.io/announcements/2021/09/22/cncf-end-user-technology-radar-provides-insights-into-devsecops/) were themed around DevSecOps. Check out the [Radar Page](https://radar.cncf.io/) for the full details and findings.
- Learn about how [end user Aegon Life India migrated core processes from its traditional monolith to a microservice-based architecture](https://www.cncf.io/case-studies/aegon-life-india/) in its effort to transform into a leading digital service company.
- Utilizing multiple cloud native projects, [Seagate engineered edgerX to run Real-time Analytics at the Edge](https://www.cncf.io/case-studies/seagate/).
- Check out how [Zambon worked with SparkFabrik to develop 16 websites, with cloud native technologies, to enable stakeholders to easily update content while maintaining a consistent brand identity](https://www.cncf.io/case-studies/zambon/).
- Using Kubernetes, [InfluxData was able to deliver on the promise of multi-cloud, multi-region service availability](https://www.cncf.io/case-studies/influxdata/) by creating a true cloud abstraction layer that allows for the seamless delivery of InfluxDB as a single application to multiple global clusters across three major cloud providers.
### Ecosystem Updates
- [KubeCon + CloudNativeCon NA 2021](https://www.cncf.io/events/kubecon-cloudnativecon-north-america-2021/) was held in October 2021, both online and in person. All talks are [now available on-demand](https://www.youtube.com/playlist?list=PLj6h78yzYM2Nd1U4RMhv7v88fdiFqeYAP) for anyone that would like to catch up!
- [Kubernetes and Cloud Native Essentials Training and KCNA Certification are now generally available for enrollment and scheduling](https://www.cncf.io/announcements/2021/11/18/kubernetes-and-cloud-native-essentials-training-and-kcna-certification-now-available/). Additionally, a new online training course, [Kubernetes and Cloud Native Essentials (LFS250)](https://www.cncf.io/announcements/2021/10/13/entry-level-kubernetes-certification-to-help-advance-cloud-careers/), has been released to both prepare individuals for entry-level cloud roles and to sit for the KCNA exam.
- [New resources are now available from the Inclusive Naming Initiative](https://www.cncf.io/announcements/2021/10/13/inclusive-naming-initiative-announces-new-community-resources-for-a-more-inclusive-future/), including an Inclusive Strategies for Open Source (LFC103) course, Language Evaluation Framework, and Implementation Path.
### Project Velocity
The [CNCF K8s DevStats](https://k8s.devstats.cncf.io/d/12/dashboards?orgId=1&refresh=15m) project aggregates a number of interesting data points related to the velocity of Kubernetes and various sub-projects. This includes everything from individual contributions to the number of companies that are contributing, and is an illustration of the depth and breadth of effort that goes into evolving this ecosystem.
In the v1.23 release cycle, which ran for 16 weeks (August 23 to December 7), we saw contributions from [1032 companies](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=v1.22.0%20-%20now&var-metric=contributions) and [1084 individuals](https://k8s.devstats.cncf.io/d/66/developer-activity-counts-by-companies?orgId=1&var-period_name=v1.22.0%20-%20now&var-metric=contributions&var-repogroup_name=Kubernetes&var-country_name=All&var-companies=All&var-repo_name=kubernetes%2Fkubernetes).
### Event Update
- [KubeCon + CloudNativeCon China 2021](https://www.lfasiallc.com/kubecon-cloudnativecon-open-source-summit-china/) is happening this month from December 9 - 11. After taking a break last year, the event will be virtual this year and includes 105 sessions. Check out the event schedule [here](https://www.lfasiallc.com/kubecon-cloudnativecon-open-source-summit-china/program/schedule/).
- KubeCon + CloudNativeCon Europe 2022 will take place in Valencia, Spain, May 4 – 7, 2022! You can find more information about the conference and registration on the [event site](https://events.linuxfoundation.org/archive/2021/kubecon-cloudnativecon-europe/).
- Kubernetes Community Days has upcoming events scheduled in Pakistan, Brazil, Chengdu, and in Australia.
### Upcoming Release Webinar
Join members of the Kubernetes 1.23 release team on January 4, 2022 to learn about the major features of this release, as well as deprecations and removals to help plan for upgrades. For more information and registration, visit the [event page](https://community.cncf.io/e/mrey9h/) on the CNCF Online Programs site.
### Get Involved
The simplest way to get involved with Kubernetes is by joining one of the many [Special Interest Groups](https://github.com/kubernetes/community/blob/master/sig-list.md) (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly [community meeting](https://github.com/kubernetes/community/tree/master/communication), and through the channels below:
- Find out more about contributing to Kubernetes at the [Kubernetes Contributors](https://www.kubernetes.dev/) website
- Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for the latest updates
- Join the community discussion on [Discuss](https://discuss.kubernetes.io/)
- Join the community on [Slack](http://slack.k8s.io/)
- Post questions (or answer questions) on [Stack Overflow](http://stackoverflow.com/questions/tagged/kubernetes)
- Share your Kubernetes [story](https://docs.google.com/a/linuxfoundation.org/forms/d/e/1FAIpQLScuI7Ye3VQHQTwBASrgkjQDSS5TP0g3AXfFhwSM9YpHgxRKFA/viewform)
- Read more about what’s happening with Kubernetes on the [blog](https://kubernetes.io/blog/)
- Learn more about the [Kubernetes Release Team](https://github.com/kubernetes/sig-release/tree/master/release-team)
"When will Kubernetes have IPv6?" This question has been asked with increasing frequency ever since alpha support for IPv6 was first added in k8s v1.9. While Kubernetes has supported IPv6-only clusters since v1.18, migration from IPv4 to IPv6 was not yet possible at that point. At long last, [dual-stack IPv4/IPv6 networking](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack/) has reached general availability (GA) in Kubernetes v1.23.
What does dual-stack networking mean for you? Let’s take a look…
## Service API updates
[Services](/docs/concepts/services-networking/service/) were single-stack before 1.20, so using both IP families meant creating one Service per IP family. The user experience was simplified in 1.20, when Services were re-implemented to allow both IP families, meaning a single Service can handle both IPv4 and IPv6 workloads. Dual-stack load balancing is possible between services running any combination of IPv4 and IPv6.
The Service API now has new fields to support dual-stack, replacing the single ipFamily field.
* You can select your choice of IP family by setting `ipFamilyPolicy` to one of three options: SingleStack, PreferDualStack, or RequireDualStack. A service can be changed between single-stack and dual-stack (within some limits).
* Setting `ipFamilies` to a list of families assigned allows you to set the order of families used.
* `clusterIPs` is inclusive of the previous `clusterIP` but allows for multiple entries, so it’s no longer necessary to run duplicate services, one in each of the two IP families. Instead, you can assign cluster IP addresses in both IP families.
Note that Pods are also dual-stack. For a given pod, there is no possibility of setting multiple IP addresses in the same family.
## Default behavior remains single-stack
Starting in 1.20 with the re-implementation of dual-stack services as alpha, the underlying networking for Kubernetes has included dual-stack whether or not a cluster was configured with the feature flag to enable dual-stack.
Kubernetes 1.23 removed that feature flag as part of graduating the feature to stable. Dual-stack networking is always available if you want to configure it. You can set your cluster network to operate as single-stack IPv4, as single-stack IPv6, or as dual-stack IPv4/IPv6.
While Services are set according to what you configure, Pods default to whatever the CNI plugin sets. If your CNI plugin assigns single-stack IPs, you will have single-stack unless `ipFamilyPolicy` specifies PreferDualStack or RequireDualStack. If your CNI plugin assigns dual-stack IPs, `pod.status.PodIPs` defaults to dual-stack.
Even though dual-stack is possible, it is not mandatory to use it. Examples in the documentation show the variety possible in [dual-stack service configurations](/docs/concepts/services-networking/dual-stack/#dual-stack-service-configuration-scenarios).
## Try dual-stack right now
While upstream Kubernetes now supports [dual-stack networking](/docs/concepts/services-networking/dual-stack/) as a GA or stable feature, each provider’s support of dual-stack Kubernetes may vary. Nodes need to be provisioned with routable IPv4/IPv6 network interfaces. Pods need to be dual-stack. The [network plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) is what assigns the IP addresses to the Pods, so it's the network plugin being used for the cluster that needs to support dual-stack. Some Container Network Interface (CNI) plugins support dual-stack, as does kubenet.
Ecosystem support of dual-stack is increasing; you can create [dual-stack clusters with kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/), try a [dual-stack cluster locally with KIND](https://kind.sigs.k8s.io/docs/user/configuration/#ip-family), and deploy dual-stack clusters in cloud providers (after checking docs for CNI or kubenet availability).
## Get involved with SIG Network
SIG-Network wants to learn from community experiences with dual-stack networking to find out more about evolving needs and your use cases. The [SIG-network update video from KubeCon NA 2021](https://www.youtube.com/watch?v=uZ0WLxpmBbY&list=PLj6h78yzYM2Nd1U4RMhv7v88fdiFqeYAP&index=4) summarizes the SIG’s recent updates, including dual-stack going to stable in 1.23.
The current SIG-Network [KEPs](https://github.com/orgs/kubernetes/projects/10) and [issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Fnetwork) on GitHub illustrate the SIG’s areas of emphasis. The [dual-stack API server](https://github.com/kubernetes/enhancements/issues/2438) is one place to consider contributing.
[SIG-Network meetings](https://github.com/kubernetes/community/tree/master/sig-network#meetings) are a friendly, welcoming venue for you to connect with the community and share your ideas. Looking forward to hearing from you!
## Acknowledgments
The dual-stack networking feature represents the work of many Kubernetes contributors. Thanks to all who contributed code, experience reports, documentation, code reviews, and everything in between. Bridget Kromhout details this community effort in [Dual-Stack Networking in Kubernetes](https://containerjournal.com/features/dual-stack-networking-in-kubernetes/). KubeCon keynotes by Tim Hockin & Khaled (Kal) Henidak in 2019 ([The Long Road to IPv4/IPv6 Dual-stack Kubernetes](https://www.youtube.com/watch?v=o-oMegdZcg4)) and by Lachlan Evenson in 2021 ([And Here We Go: Dual-stack Networking in Kubernetes](https://www.youtube.com/watch?v=lVrt8F2B9CM)) talk about the dual-stack journey, spanning five years and a great many lines of code.
title: 'Kubernetes 1.23: Pod Security Graduates to Beta'
date: 2021-12-09
slug: pod-security-admission-beta
---
**Authors:** Jim Angel (Google), Lachlan Evenson (Microsoft)
With the release of Kubernetes v1.23, [Pod Security admission](/docs/concepts/security/pod-security-admission/) has now entered beta. Pod Security is a [built-in](/docs/reference/access-authn-authz/admission-controllers/) admission controller that evaluates pod specifications against a predefined set of [Pod Security Standards](/docs/concepts/security/pod-security-standards/) and determines whether to `admit` or `deny` the pod from running.
Pod Security is the successor to [PodSecurityPolicy](/docs/concepts/security/pod-security-policy/) which was deprecated in the v1.21 release, and will be removed in Kubernetes v1.25. In this article, we cover the key concepts of Pod Security along with how to use it. We hope that cluster administrators and developers alike will use this new mechanism to enforce secure defaults for their workloads.
## Why Pod Security
The overall aim of Pod Security is to let you isolate workloads. You can run a cluster that runs different workloads and, without adding extra third-party tooling, implement controls that require Pods for a workload to restrict their own privileges to a defined bounding set.
Pod Security overcomes key shortcomings of Kubernetes' existing, but deprecated, PodSecurityPolicy (PSP) mechanism:
* Policy authorization model — challenging to deploy with controllers.
* Risks around switching — a lack of dry-run/audit capabilities made it hard to enable PodSecurityPolicy.
* Inconsistent and Unbounded API — the large configuration surface and evolving constraints led to a complex and confusing API.
The shortcomings of PSP made it very difficult to use which led the community to reevaluate whether or not a better implementation could achieve the same goals. One of those goals was to provide an out-of-the-box solution to apply security best practices. Pod Security ships with predefined Pod Security levels that a cluster administrator can configure to meet the desired security posture.
It's important to note that Pod Security doesn't have complete feature parity with the deprecated PodSecurityPolicy. Specifically, it doesn't have the ability to mutate or change Kubernetes resources to auto-remediate a policy violation on behalf of the user. Additionally, it doesn't provide fine-grained control over each allowed field and value within a pod specification or any other Kubernetes resource that you may wish to evaluate. If you need more fine-grained policy control then take a look at these [other](/docs/concepts/security/pod-security-standards/#faq) projects which support such use cases.
Pod Security also adheres to Kubernetes best practices of declarative object management by denying resources that violate the policy. This requires resources to be updated in source repositories, and tooling to be updated prior to being deployed to Kubernetes.
## How Does Pod Security Work?
Pod Security is a built-in [admission controller](/docs/reference/access-authn-authz/admission-controllers/) starting with Kubernetes v1.22, but can also be run as a standalone [webhook](/docs/concepts/security/pod-security-admission/#webhook). Admission controllers function by intercepting requests in the Kubernetes API server prior to persistence to storage. They can either `admit` or `deny` a request. In the case of Pod Security, pod specifications will be evaluated against a configured policy in the form of a Pod Security Standard. This means that security sensitive fields in a pod specification will only be allowed to have [specific](h/docs/concepts/security/pod-security-standards/#profile-details) values.
## Configuring Pod Security
### Pod Security Standards
In order to use Pod Security we first need to understand [Pod Security Standards](/docs/concepts/security/pod-security-standards/). These standards define three different policy levels that range from permissive to restrictive. These levels are as follows:
* `privileged` — open and unrestricted
* `baseline` — Covers known privilege escalations while minimizing restrictions
* `restricted` — Highly restricted, hardening against known and unknown privilege escalations. May cause compatibility issues
Each of these policy levels define which fields are restricted within a pod specification and the allowed values. Some of the fields restricted by these policies include:
* `spec.securityContext.sysctls`
* `spec.hostNetwork`
* `spec.volumes[*].hostPath`
* `spec.containers[*].securityContext.privileged`
Policy levels are applied via labels on Namespace resources, which allows for granular per-namespace policy selection. The AdmissionConfiguration in the API server can also be configured to set cluster-wide default levels and exemptions.
### Policy modes
Policies are applied in a specific mode. Multiple modes (with different policy levels) can be set on the same namespace. Here is a list of modes:
* `enforce` — Any Pods that violate the policy will be rejected
* `audit` — Violations will be recorded as an annotation in the audit logs, but don't affect whether the pod is allowed.
* `warn` — Violations will send a warning message back to the user, but don't affect whether the pod is allowed.
In addition to modes you can also pin the policy to a specific version (for example v1.22). Pinning to a specific version allows the behavior to remain consistent if the policy definition changes in future Kubernetes releases.
The best way to [confirm the API's default enabled plugins](/docs/reference/access-authn-authz/admission-controllers/#which-plugins-are-enabled-by-default) is to check the Kubernetes API container's help arguments.
# The following command does NOT create a workload (--dry-run=server)
kubectl -n verify-pod-security run test --dry-run=server --image=busybox --privileged
kubectl delete namespace verify-pod-security
```
The output is similar to this:
```
Error from server (Forbidden): pods "test" is forbidden: violates PodSecurity "restricted:latest": privileged (container "test" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
```
### Configure Pod Security
Policies are applied to a namespace via labels. These labels are as follows:
* `pod-security.kubernetes.io/<MODE>: <LEVEL>` (required to enable pod security)
* `pod-security.kubernetes.io/<MODE>-version: <VERSION>` (*optional*, defaults to latest)
A specific version can be supplied for each enforcement mode. The version pins the policy to the version that was shipped as part of the Kubernetes release. Pinning to a specific Kubernetes version allows for deterministic policy behavior while allowing flexibility for future updates to Pod Security Standards. The possible <MODE(S)> are `enforce`, `audit` and `warn`.
### When to use `warn`?
The typical uses for `warn` are to get ready for a future change where you want to enforce a different policy. The most two common cases would be:
* `warn` at the same level but a different version (e.g. pin `enforce` to *restricted+v1.23* and `warn` at *restricted+latest*)
* `warn` at a stricter level (e.g. `enforce` baseline, `warn` restricted)
It's not recommended to use `warn` for the exact same level+version of the policy as `enforce`. In the admission sequence, if `enforce` fails, the entire sequence fails before evaluating the `warn`.
First, create a namespace called `verify-pod-security` if not created earlier. For the demo, `--overwrite` is used when labeling to allow repurposing a single namespace for multiple examples.
```shell
kubectl create namespace verify-pod-security
```
### Deploy demo workloads
Each workload represents a higher level of security that would not pass the profile that comes after it.
For the following examples, use the `busybox` container runs a `sleep` command for 1 million seconds (≅11 days) or until deleted. Pod Security is not interested in which container image you chose, but rather the Pod level settings and their implications for security.
### Privileged level and workload
For the privileged pod, use the [privileged policy](/docs/concepts/security/pod-security-standards/#privileged). This allows the process inside a container to gain new processes (also known as "privilege escalation") and can be dangerous if untrusted.
First, let's apply a restricted Pod Security level for a test.
```shell
# enforces a "restricted" security policy and audits on restricted
Error from server (Forbidden): error when creating "STDIN": pods "busybox-privileged" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
```
Now let's apply the privileged Pod Security level and try again.
```shell
# enforces a "privileged" security policy and warns / audits on baseline
We can run `kubectl -n verify-pod-security get pods` to verify it is running. Clean up with:
```shell
kubectl -n verify-pod-security delete pod busybox-privileged
```
### Baseline level and workload
The [baseline policy](/docs/concepts/security/pod-security-standards/#baseline) demonstrates sensible defaults while preventing common container exploits.
Let's revert back to a restricted Pod Security level for a quick test.
```shell
# enforces a "restricted" security policy and audits on restricted
Error from server (Forbidden): error when creating "STDIN": pods "busybox-baseline" is forbidden: violates PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]; container "busybox" must not include "CHOWN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
```
Let's apply the baseline Pod Security level and try again.
```shell
# enforces a "baseline" security policy and warns / audits on restricted
The output is similar to the following. Note that the warnings match the error message from the test above, but the pod is still successfully created.
```
Warning: would violate PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]; container "busybox" must not include "CHOWN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/busybox-baseline created
```
Remember, we set the `verify-pod-security` namespace to `warn` based on the restricted profile. We can run `kubectl -n verify-pod-security get pods` to verify it is running. Clean up with:
```shell
kubectl -n verify-pod-security delete pod busybox-baseline
```
### Restricted level and workload
The [restricted policy](/docs/concepts/security/pod-security-standards/#restricted) requires rejection of all privileged parameters. It is the most secure with a trade-off for complexity.
The restricted policy allows containers to add the `NET_BIND_SERVICE` capability only.
While we've already tested restricted as a blocking function, let's try to get something running that meets all the criteria.
First we need to reapply the restricted profile, for the last time.
```shell
# enforces a "restricted" security policy and audits on restricted
Error from server (Forbidden): error when creating "STDIN": pods "busybox-restricted" is forbidden: violates PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
```
This is because the restricted profile explicitly requires that certain values are set to the most secure parameters.
By requiring explicit values, manifests become more declarative and your entire security model can shift left. With the `restricted` level of enforcement, a company could audit their cluster's compliance based on permitted manifests.
Let's fix each warning resulting in the following file:
Let's figure out why the container is not starting with `kubectl -n verify-pod-security describe pod busybox-restricted`. The output is similar to this:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 2m29s (x8 over 3m55s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "busybox-restricted_verify-pod-security(a4c6a62d-2166-41a9-b288-20df17cf5c90)", container: busybox)
```
To solve this, set the effective UID (`runAsUser`) to a non-zero (root) value or use the `nobody` UID (65534).
```shell
# delete the original pod
kubectl -n verify-pod-security delete pod busybox-restricted
Run `kubectl -n verify-pod-security get pods` to verify it is running. The output is similar to this:
```
NAME READY STATUS RESTARTS AGE
busybox-restricted 1/1 Running 0 25s
```
Clean up the demo (restricted pod and namespace) with:
```shell
kubectl delete namespace verify-pod-security
```
At this point, if you wanted to dive deeper into linux permissions or what is permitted for a certain container, exec into the control plane and play around with `containerd` and `crictl inspect`.
```shell
# if using docker, shell into the control plane
docker exec -it kind-control-plane bash
# list running containers
crictl ps
# inspect each one by container ID
crictl inspect <CONTAINER ID>
```
### Applying a cluster-wide policy
In addition to applying labels to namespaces to configure policy you can also configure cluster-wide policies and exemptions using the AdmissionConfiguration resource.
Using this resource, policy definitions are applied cluster-wide by default and any policy that is applied via namespace labels will take precedence.
There is no runtime configurable API for the `AdmissionConfiguration` configuration file so a cluster administrator would need to specify a path to the file below via the `--admission-control-config-file` flag on the API server.
In the following resource we are enforcing the baseline policy and warning and auditing the baseline policy. We are also making the kube-system namespace exempt from this policy.
It's not recommended to alter control plane / clusters after install, so let's build a new cluster with a default policy on all namespaces.
First, delete the current cluster.
```shell
kind delete cluster
```
Create a Pod Security configuration that `enforce` and `audit` baseline policies while using a restricted profile to `warn` the end user.
For additional options, check out the official [_standards admission controller_](/docs/tasks/configure-pod-container/enforce-standards-admission-controller/#configure-the-admission-controller) docs.
We now have a default baseline policy. Next pass it to the kind configuration to enable the `--admission-control-config-file` API server argument and pass the policy file. To pass a file to a kind cluster, use a configuration file to pass additional setup instructions. Kind uses `kubeadm` to provision the cluster and the configuration file has the ability to pass `kubeadmConfigPatches` for further customization. In our case, the local file is mounted into the control plane node as `/etc/kubernetes/policies/pod-security.yaml` which is then mounted into the `apiServer` container. We also pass the `--admission-control-config-file` argument pointing to the policy's location.
```shell
cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
# enable admission-control-config flag on the API server
Let's create a new namespace and see if the labels apply there.
```shell
kubectl create namespace test-defaults
kubectl describe namespace test-defaults
```
Same.
```
Name: test-defaults
Labels: kubernetes.io/metadata.name=test-defaults
Annotations: <none>
Status: Active
No resource quota.
No LimitRange resource.
```
Can a privileged workload be deployed?
```shell
cat <<EOF | kubectl -n test-defaults apply -f -
apiVersion: v1
kind: Pod
metadata:
name: busybox-privileged
spec:
containers:
- name: busybox
image: busybox
args:
- sleep
- "1000000"
securityContext:
allowPrivilegeEscalation: true
EOF
```
Hmm... yep. The default `warn` level is working at least.
```
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/busybox-privileged created
```
Let's delete the pod with `kubectl -n test-defaults delete pod/busybox-privileged`.
Is my config even working?
```shell
# if using docker, shell into the control plane
docker exec -it kind-control-plane bash
# cat out the file we mounted
cat /etc/kubernetes/policies/pod-security.yaml
# check the api server logs
cat /var/log/containers/kube-apiserver*.log
# check the api server config
cat /etc/kubernetes/manifests/kube-apiserver.yaml
```
**UPDATE:** The baseline policy permits `allowPrivilegeEscalation`. While I cannot see the Pod Security default levels of enforcement, they are there. Let's try to provide a manifest that violates the baseline by requesting hostNetwork access.
```shell
# delete the original pod
kubectl -n test-defaults delete pod busybox-privileged
cat <<EOF | kubectl -n test-defaults apply -f -
apiVersion: v1
kind: Pod
metadata:
name: busybox-privileged
spec:
containers:
- name: busybox
image: busybox
args:
- sleep
- "1000000"
hostNetwork: true
EOF
```
The output is similar to this:
```
Error from server (Forbidden): error when creating "STDIN": pods "busybox-privileged" is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true)
```
#### Yes!!! It worked! 🎉🎉🎉 {#it-worked}
I later found out, another way to check if things are operating as intended is to check the raw API server metrics endpoint.
Run the following command:
```
kubectl get --raw /metrics | grep pod_security_evaluations_total
```
The output is similar to this:
```
# HELP pod_security_evaluations_total [ALPHA] Number of policy evaluations that occurred, not counting ignored or exempt requests.
A monitoring tool could ingest these metrics too for reporting, assessments, or measuring trends.
## Clean up
When finished, delete the kind cluster.
```shell
kind delete cluster
```
## Auditing
Auditing is another way to track what policies are being enforced in your cluster. To set up auditing with kind, review the official docs for [enabling auditing](https://kind.sigs.k8s.io/docs/user/auditing/). As of [version 1.11](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.11.md#sig-auth), Kubernetes audit logs include two annotations that indicate whether or not a request was authorized (`authorization.k8s.io/decision`) and the reason for the decision (`authorization.k8s.io/reason`). Audit events can be streamed to a webhook for monitoring, tracking, or alerting.
The audit events look similar to the following:
```yaml
{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"","pod-security.kubernetes.io/audit":"allowPrivilegeEscalation != false (container \"busybox\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"busybox\" must set securityContext.capabilities.drop=[\"ALL\"]), runAsNonRoot != true (pod or container \"busybox\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"busybox\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"}}
```
Auditing is also a good first step in evaluating your cluster's current compliance with Pod Security. The Kubernetes Enhancement Proposal (KEP) hints at a future where `baseline` [could be the default for unlabeled namespaces](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/2579-psp-replacement/README.md#rollout-of-baseline-by-default-for-unlabeled-namespaces).
Example `audit-policy.yaml` configuration tuned for Pod Security events:
Once auditing is enabled, look at the configured local file if using `--audit-log-path` or the destination of a webhook if using `--audit-webhook-config-file`.
If using a file (`--audit-log-path`), run `cat /PATH/TO/API/AUDIT.log | grep "is forbidden:"` to see all rejected workloads audited.
## PSP migrations
If you're already using PSP, SIG Auth has created a guide and [published the steps to migrate off of PSP](/docs/tasks/configure-pod-container/migrate-from-psp/).
To summarize the process:
- Update all existing PSPs to be non-mutating
- Apply Pod Security policies in `warn` or `audit` mode
- Upgrade Pod Security policies to `enforce` mode
- Remove `PodSecurityPolicy` from `--enable-admission-plugins`
Listed as "optional future extensions" and currently out of scope, SIG Auth has kicked around the idea of providing a tool to assist with migrations. More [details in the KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/2579-psp-replacement/README.md#automated-psp-migration-tooling).
## Wrap up
Pod Security is a promising new feature that provides an out-of-the-box way to allow users to improve the security posture of their workloads. Like any new enhancement that has matured to beta, we ask that you try it out, provide feedback, or share your experience via either raising a Github issue or joining SIG Auth community meetings. It's our hope that Pod Security will be deployed on every cluster in our ongoing pursuit as a community to make Kubernetes security a priority.
For a step by step guide on how to enable "baseline" Pod Security Standards with Pod Security Admission feature please refer to these dedicated [tutorials](/docs/tutorials/security/) that cover the configuration needed at cluster level and namespace level.
## Additional resources
- [Official Pod Security Docs](/docs/concepts/security/pod-security-admission/)
- [Enforce Pod Security Standards with Namespace Labels](/docs/tasks/configure-pod-container/enforce-standards-namespace-labels/)
- [Enforce Pod Security Standards by Configuring the Built-in Admission Controller](/docs/tasks/configure-pod-container/enforce-standards-admission-controller/)
The Kubernetes in-tree storage plugin to [Container Storage Interface (CSI)](/blog/2019/01/15/container-storage-interface-ga/) migration infrastructure has already been [beta](/blog/2019/12/09/kubernetes-1-17-feature-csi-migration-beta/) since v1.17. CSI migration was introduced as alpha in Kubernetes v1.14.
Since then, SIG Storage and other Kubernetes special interest groups are working to ensure feature stability and compatibility in preparation for GA.
This article is intended to give a status update to the feature as well as changes between Kubernetes 1.17 and 1.23. In addition, I will also cover the future roadmap for the CSI migration feature GA for each storage plugin.
## Quick recap: What is CSI Migration, and why migrate?
The Container Storage Interface (CSI) was designed to help Kubernetes replace its existing, in-tree storage driver mechanisms - especially vendor specific plugins.
Kubernetes support for the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md#README) has been
[generally available](/blog/2019/01/15/container-storage-interface-ga/) since Kubernetes v1.13.
Support for using CSI drivers was introduced to make it easier to add and maintain new integrations between Kubernetes and storage backend technologies. Using CSI drivers allows for for better maintainability (driver authors can define their own release cycle and support lifecycle) and reduce the opportunity for vulnerabilities (with less in-tree code, the risks of a mistake are reduced, and cluster operators can select only the storage drivers that their cluster requires).
As more CSI Drivers were created and became production ready, SIG Storage group wanted all Kubernetes users to benefit from the CSI model. However, we cannot break API compatibility with the existing storage API types. The solution we came up with was CSI migration: a feature that translates in-tree APIs to equivalent CSI APIs and delegates operations to a replacement CSI driver.
The CSI migration effort enables the replacement of existing in-tree storage plugins such as `kubernetes.io/gce-pd` or `kubernetes.io/aws-ebs` with a corresponding [CSI driver](https://kubernetes-csi.github.io/docs/introduction.html) from the storage backend.
If CSI Migration is working properly, Kubernetes end users shouldn’t notice a difference. Existing `StorageClass`, `PersistentVolume` and `PersistentVolumeClaim` objects should continue to work.
When a Kubernetes cluster administrator updates a cluster to enable CSI migration, existing workloads that utilize PVCs which are backed by in-tree storage plugins will continue to function as they always have.
However, behind the scenes, Kubernetes hands control of all storage management operations (previously targeting in-tree drivers) to CSI drivers.
For example, suppose you are a `kubernetes.io/gce-pd` user, after CSI migration, you can still use `kubernetes.io/gce-pd` to provision new volumes, mount existing GCE-PD volumes or delete existing volumes. All existing API/Interface will still function correctly. However, the underlying function calls are all going through the [GCE PD CSI driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) instead of the in-tree Kubernetes function.
This enables a smooth transition for end users. Additionally as storage plugin developers, we can reduce the burden of maintaining the in-tree storage plugins and eventually remove them from the core Kubernetes binary.
## What has been changed, and what's new?
Building on the work done in Kubernetes v1.17 and earlier, the releases since then have
made a series of changes:
### New feature gates
The Kubernetes v1.21 release deprecated the `CSIMigration{provider}Complete` feature flags, and stopped honoring them. In their place came new feature flags named `InTreePlugin{vendor}Unregister`, that replace the old feature flag and retain all the functionality that `CSIMigration{provider}Complete` provided.
`CSIMigration{provider}Complete` was introduced before as a supplementary feature gate once CSI migration is enabled on all of the nodes. This flag unregisters the in-tree storage plugin you specify with the `{provider}` part of the flag name.
When you enable that feature gate, then instead of using the in-tree driver code, your cluster directly selects and uses the relevant CSI driver. This happens without any check for whether CSI migration is enabled on the node, or whether you have in fact deployed that CSI driver.
While this feature gate is a great helper, SIG Storage (and, I'm sure, lots of cluster operators) also wanted a feature gate that lets you disable an in-tree storage plugin, even without also enabling CSI migration. For example, you might want to disable the EBS storage plugin on a GCE cluster, because EBS volumes are specific to a different vendor's cloud (AWS).
To make this possible, Kubernetes v1.21 introduced a new feature flag set: `InTreePlugin{vendor}Unregister`.
`InTreePlugin{vendor}Unregister` is a standalone feature gate that can be enabled and disabled independently from CSI Migration. When enabled, the component will not register the specific in-tree storage plugin to the supported list. If the cluster operator only enables this flag, end users will get an error from PVC saying it cannot find the plugin when the plugin is used. The cluster operator may want to enable this regardless of CSI Migration if they do not want to support the legacy in-tree APIs and only support CSI moving forward.
### Observability
Kubernetes v1.21 introduced [metrics](https://github.com/kubernetes/kubernetes/issues/98279) for tracking CSI migration.
You can use these metrics to observe how your cluster is using storage services and whether access to that storage is using the legacy in-tree driver or its CSI-based replacement.
| Kube-Controller-Manager | storage_operation_duration_seconds | A new label `migrated` is added to the metric to indicate whether this storage operation is a CSI migration operation(string value `true` for enabled and `false` for not enabled). |
| Kubelet | csi_operations_seconds | The new metric exposes labels including `driver_name`, `method_name`, `grpc_status_code` and `migrated`. The meaning of these labels is identical to `csi_sidecar_operations_seconds`. |
| CSI Sidecars(provisioner, attacher, resizer) | csi_sidecar_operations_seconds | A new label `migrated` is added to the metric to indicate whether this storage operation is a CSI migration operation(string value `true` for enabled and `false` for not enabled). |
### Bug fixes and feature improvement
We have fixed numerous bugs like dangling attachment, garbage collection, incorrect topology label through the help of our beta testers.
SIG Storage has been working closely with SIG Cloud Provider and SIG Cluster Lifecycle on the rollout of CSI migration.
If you are a user of a managed Kubernetes service, check with your provider if anything needs to be done. In many cases, the provider will manage the migration and no additional work is required.
If you use a distribution of Kubernetes, check its official documentation for information about support for this feature. For the CSI Migration feature graduation to GA, SIG Storage and SIG Cluster Lifecycle are collaborating towards making the migration mechanisms available in tooling (such as kubeadm) as soon as they're available in Kubernetes itself.
## What is the timeline / status? {#timeline-and-status}
The current and targeted releases for each individual driver is shown in the table below:
The following storage drivers will not have CSI migration support. The ScaleIO driver was already removed; the others are deprecated and will be removed from core Kubernetes.
| Driver | Deprecated | Code Removal |
| --------- | ---------- | ------------- |
| ScaleIO | 1.16 | 1.22 |
| Flocker | 1.22 | 1.25 (Target) |
| Quobyte | 1.22 | 1.25 (Target) |
| StorageOS | 1.22 | 1.25 (Target) |
## What's next?
With more CSI drivers graduating to GA, we hope to soon mark the overall CSI Migration feature as GA. We are expecting cloud provider in-tree storage plugins code removal to happen by Kubernetes v1.26 and v1.27.
## What should I do as a user?
Note that all new features for the Kubernetes storage system (such as volume snapshotting) will only be added to the CSI interface. Therefore, if you are starting up a new cluster, creating stateful applications for the first time, or require these new features we recommend using CSI drivers natively (instead of the in-tree volume plugin API). Follow the [updated user guides for CSI drivers](https://kubernetes-csi.github.io/docs/drivers.html) and use the new CSI APIs.
However, if you choose to roll a cluster forward or continue using specifications with the legacy volume APIs, CSI Migration will ensure we continue to support those deployments with the new CSI drivers. However, if you want to leverage new features like snapshot, it will require a manual migration to re-import an existing intree PV as a CSI PV.
## How do I get involved?
The Kubernetes Slack channel [#csi-migration](https://kubernetes.slack.com/messages/csi-migration) along with any of the standard [SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and migration working group teams.
This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. We offer a huge thank you to the contributors who stepped up these last quarters to help move the project forward:
* Michelle Au (msau42)
* Jan Šafránek (jsafrane)
* Hemant Kumar (gnufied)
Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution to the CSI migration feature:
* Andy Zhang (andyzhangz)
* Divyen Patel (divyenpatel)
* Deep Debroy (ddebroy)
* Humble Devassy Chirammal (humblec)
* Jing Xu (jingxu97)
* Jordan Liggitt (liggitt)
* Matthew Cary (mattcary)
* Matthew Wong (wongma7)
* Neha Arora (nearora-msft)
* Oksana Naumov (trierra)
* Saad Ali (saad-ali)
* Tim Bannister (sftim)
* Xing Yang (xing-yang)
Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage). We’re rapidly growing and always welcome new contributors.
The PV object from the cluster also gets deleted. When attempting to retrieve the PV
it will be observed that the PV is no longer found:
```
kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
```
```
Error from server (NotFound): persistentvolumes "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" not found
```
Although the PV is deleted the underlying storage resource is not deleted, and
needs to be removed manually.
To sum it up, the reclaim policy associated with the Persistent Volume is currently
ignored under certain circumstance. For a `Bound` PV-PVC pair the ordering of PV-PVC
deletion determines whether the PV reclaim policy is honored. The reclaim policy
is honored if the PVC is deleted first, however, if the PV is deleted prior to
deleting the PVC then the reclaim policy is not exercised. As a result of this behavior,
the associated storage asset in the external infrastructure is not removed.
## PV reclaim policy with Kubernetes v1.23
The new behavior ensures that the underlying storage object is deleted from the backend when users attempt to delete a PV manually.
#### How to enable new behavior?
To make use of the new behavior, you must have upgraded your cluster to the v1.23 release of Kubernetes.
You need to make sure that you are running the CSI [`external-provisioner`](https://github.com/kubernetes-csi/external-provisioner) version `4.0.0`, or later.
You must also enable the `HonorPVReclaimPolicy` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the
`external-provisioner` and for the `kube-controller-manager`.
If you're not using a CSI driver to integrate with your storage backend, the fix isn't
available. The Kubernetes project doesn't have a current plan to fix the bug for in-tree
storage drivers: the future of those in-tree drivers is deprecation and migration to CSI.
#### How does it work?
The new behavior is achieved by adding a finalizer `external-provisioner.volume.kubernetes.io/finalizer` on new and existing PVs, the finalizer is only removed after the storage from backend is deleted.
An example of a PV with the finalizer, notice the new finalizer in the finalizers list
```
kubectl get pv pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53 -o yaml
The presence of the finalizer prevents the PV object from being removed from the
cluster. As stated previously, the finalizer is only removed from the PV object
after it is successfully deleted from the storage backend. To learn more about
finalizers, please refer to [Using Finalizers to Control Deletion](/blog/2021/05/14/using-finalizers-to-control-deletion/).
#### What about CSI migrated volumes?
The fix is applicable to CSI migrated volumes as well. However, when the feature
`HonorPVReclaimPolicy` is enabled on 1.23, and CSI Migration is disabled, the finalizer
is removed from the PV object if it exists.
### Some caveats
1. The fix is applicable only to CSI volumes and migrated volumes. In-tree volumes will exhibit older behavior.
2. The fix is introduced as an alpha feature in the [external-provisioner](https://github.com/kubernetes-csi/external-provisioner) under the feature gate `HonorPVReclaimPolicy`. The feature is disabled by default, and needs to be enabled explicitly.
The Kubernetes Slack channel [SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and migration working group teams.
Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution:
* Jan Šafránek (jsafrane)
* Xing Yang (xing-yang)
* Matthew Wong (wongma7)
Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage). We’re rapidly growing and always welcome new contributors.
release of the operator. We added new features, fine-tuned existing ones and
reworked our documentation in 290 commits over the past half year.
One of the highlights is that we're now able to record seccomp and SELinux
profiles using the operators [log enricher](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#log-enricher-based-recording).
This allows us to reduce the dependencies required for profile recording to have
[auditd](https://linux.die.net/man/8/auditd) or
[syslog](https://en.wikipedia.org/wiki/Syslog) (as fallback) running on the
nodes. All profile recordings in the operator work in the same way by using the
`ProfileRecording` CRD as well as their corresponding [label
selectors](/docs/concepts/overview/working-with-objects/labels). The log
enricher itself can be also used to gather meaningful insights about seccomp and
SELinux messages of a node. Checkout the [official
in our repository. From the user's point of view it works pretty much the same
as recording of seccomp profiles.
Under the hood, to know what the workload is doing SPO installs a special
permissive policy called [selinuxrecording](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/deploy/base/profiles/selinuxrecording.cil)
on startup which allows everything and logs all AVCs to `audit.log`.
These AVC messages are scraped by the log enricher component and when
the recorded workload exits, the policy is created.
#### `SELinuxProfile` CRD graduation
An `v1alpha2` version of the `SelinuxProfile` object has been introduced. This
removes the raw Common Intermediate Language (CIL) from the object itself and
instead adds a simple policy language to ease the writing and parsing
experience.
Alongside, a `RawSelinuxProfile` object was also introduced. This contains a
wrapped and raw representation of the policy. This was intended for folks to be
able to take their existing policies into use as soon as possible. However, on
validations are done here.
### AppArmor support
This version introduces the initial support for AppArmor, allowing users to load and
unload AppArmor profiles into cluster nodes by using the new [AppArmorProfile](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/deploy/base/crds/apparmorprofile.yaml) CRD.
To enable AppArmor support use the [enableAppArmor feature gate](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/examples/config.yaml#L10) switch of your SPO configuration.
Then use our [apparmor example](https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/examples/apparmorprofile.yaml) to deploy your first profile across your cluster.
### Metrics
The operator now exposes metrics, which are described in detail in
our new [metrics documentation](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#using-metrics).
We decided to secure the metrics retrieval process by using
[kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy), while we ship an
additional `spo-metrics-client` cluster role (and binding) to retrieve the
Beside all those new features, we decided to restructure parts of the Security
Profiles Operator internally to make it better to debug and more robust. For
example, we now maintain an internal [gRPC](https://grpc.io) API to communicate
within the operator across different features. We also improved the performance
of the log enricher, which now caches results for faster retrieval of the log
data. The operator can be put into a more [verbose log mode](https://github.com/kubernetes-sigs/security-profiles-operator/blob/71b3915/installation-usage.md#set-logging-verbosity)
by setting `verbosity` from `0` to `1`.
We also print the used `libseccomp` and `libbpf` versions on startup, as well as
expose CPU and memory profiling endpoints for each container via the
title: "Using Admission Controllers to Detect Container Drift at Runtime"
date: 2021-12-21
slug: admission-controllers-for-container-drift
---
**Author:** Saifuding Diliyaer (Box)
{{< figure src="intro-illustration.png" alt="Introductory illustration" attr="Illustration by Munire Aireti" >}}
At Box, we use Kubernetes (K8s) to manage hundreds of micro-services that enable Box to stream data at a petabyte scale. When it comes to the deployment process, we run [kube-applier](https://github.com/box/kube-applier) as part of the GitOps workflows with declarative configuration and automated deployment. Developers declare their K8s apps manifest into a Git repository that requires code reviews and automatic checks to pass, before any changes can get merged and applied inside our K8s clusters. With `kubectl exec` and other similar commands, however, developers are able to directly interact with running containers and alter them from their deployed state. This interaction could then subvert the change control and code review processes that are enforced in our CI/CD pipelines. Further, it allows such impacted containers to continue receiving traffic long-term in production.
To solve this problem, we developed our own K8s component called [kube-exec-controller](https://github.com/box/kube-exec-controller) along with its corresponding [kubectl plugin](https://github.com/box/kube-exec-controller#kubectl-pi). They function together in detecting and terminating potentially mutated containers (caused by interactive kubectl commands), as well as revealing the interaction events directly to the target Pods for better visibility.
## Admission control for interactive kubectl commands
Once a request is sent to K8s, it needs to be authenticated and authorized by the API server to proceed. Additionally, K8s has a separate layer of protection called [admission controllers](/docs/reference/access-authn-authz/admission-controllers/), which can intercept the request before an object is persisted in *etcd*. There are various predefined admission controls compiled into the API server binary (e.g. ResourceQuota to enforce hard resource usage limits per namespace). Besides, there are two dynamic admission controls named [MutatingAdmissionWebhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) and [ValidatingAdmissionWebhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook), used for mutating or validating K8s requests respectively. The latter is what we adopted to detect container drift at runtime caused by interactive kubectl commands. This whole process can be divided into three steps as explained in detail below.
### 1. Admit interactive kubectl command requests
First of all, we needed to enable a validating webhook that sends qualified requests to *kube-exec-controller*. To add the new validation mechanism applying to interactive kubectl commands specifically, we configured the webhook’s rules with resources as `[pods/exec, pods/attach]`, and operations as `CONNECT`. These rules tell the cluster's API server that all `exec` and `attach` requests should be subject to our admission control webhook. In the ValidatingAdmissionWebhook that we configured, we specified a `service` reference (could also be replaced with `url` that gives the location of the webhook) and `caBundle` to allow validating its X.509 certificate, both under the `clientConfig` stanza.
Here is a short example of what our ValidatingWebhookConfiguration object looks like:
```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: example-validating-webhook-config
webhooks:
- name: validate-pod-interaction.example.com
sideEffects: None
rules:
- apiGroups: ["*"]
apiVersions: ["*"]
operations: ["CONNECT"]
resources: ["pods/exec", "pods/attach"]
failurePolicy: Fail
clientConfig:
service:
# reference to kube-exec-controller service deployed inside the K8s cluster
name: example-service
namespace: kube-exec-controller
path: "/admit-pod-interaction"
caBundle: "{{VALUE}}" # PEM encoded CA bundle to validate kube-exec-controller's certificate
admissionReviewVersions: ["v1", "v1beta1"]
```
### 2. Label the target Pod with potentially mutated containers
Once a request of `kubectl exec` comes in, *kube-exec-controller* makes an internal note to label the associated Pod. The added labels mean that we can not only query all the affected Pods, but also enable the security mechanism to retrieve previously identified Pods, in case the controller service itself gets restarted.
The admission control process cannot directly modify the targeted in its admission response. This is because the `pods/exec` request is against a subresource of the Pod API, and the API kind for that subresource is `PodExecOptions`. As a result, there is a separate process in *kube-exec-controller* that patches the labels asynchronously. The admission control always permits the `exec` request, then acts as a client of the K8s API to label the target Pod and to log related events. Developers can check whether their Pods are affected or not using `kubectl` or similar tools. For example:
Warning PodInteraction 5s admission-controller-service Pod was interacted with 'kubectl exec' command by user 'username-1' initially at time 2021-09-24 16:00:00 -0800 PST
Warning PodInteraction 5s admission-controller-service Pod will be evicted at time 2021-09-24 17:00:00 -0800 PST (in about 1h0m0s).
```
### 3. Evict the target Pod after a predefined period
As you can see in the above event messages, the affected Pod is not evicted immediately. At times, developers might have to get into their running containers necessarily for debugging some live issues. Therefore, we define a time to live (TTL) of affected Pods based on the environment of clusters they are running. In particular, we allow a longer time in our dev clusters as it is more common to run `kubectl exec` or other interactive commands for active development.
For our production clusters, we specify a lower time limit so as to avoid the impacted Pods serving traffic abidingly. The *kube-exec-controller* internally sets and tracks a timer for each Pod that matches the associated TTL. Once the timer is up, the controller evicts that Pod using K8s API. The eviction (rather than deletion) is to ensure service availability, since the cluster respects any configured [PodDisruptionBudget](/docs/concepts/workloads/pods/disruptions/) (PDB). Let's say if a user has defined *x* number of Pods as critical in their PDB, the eviction (as requested by *kube-exec-controller*) does not continue when the target workload has fewer than *x* Pods running.
Here comes a sequence diagram of the entire workflow mentioned above:
<!-- Mermaid Live Editor link - https://mermaid-js.github.io/mermaid-live-editor/edit/#pako:eNp9kjFPAzEMhf-KlalIbWd0QpUQdGJB3JrFTUyJmjhHzncFof53nGtpqYTYEuu958-Wv4zLnkxjenofiB09BtwWTJbRSS6QCLCHu01ZPdJIMXdUYNZTGYOjRd4zlRvLHRYJLnTIArvbtozV83TbAnZhUcVUrkXo04OU2I6uKu99Cn0fMsNDZik5Rm3SHntYTrRYrabUBl4GBmt2w4acRKAPcrBcLq0Bl1NC9pYnoRouHZopX9RX9aotddJeADaf4DDGwFuQN4IRY_Ao9bunzVvOO13COeYCcR9j3k-OCQDP9KfgC8TJsFbZIHSxnGljzp1lgKs2v9HXugMBwe2WPHTZ94CvottB6Ap5eg2s9cBaUnrLVEP_Yp5ynrOf3fxPV2V1lBOhmZtEJWHweiFfldQa1SWyptGnAuAQxRrLB5UOna6P1j7o4ZhGykBzg4Pk9pPdz_-oOR3ZsXj4BjrP5rU-->
## A new kubectl plugin for better user experience
Our admission controller component works great for solving the container drift issue we had on the platform. It is also able to submit all related Events to the target Pod that has been affected. However, K8s clusters don't retain Events very long (the default retention period is one hour). We need to provide other ways for developers to get their Pod interaction activity. A [kubectl plugin](/docs/tasks/extend-kubectl/kubectl-plugins/) is a perfect choice for us to expose this information. We named our plugin `kubectl pi` (short for `pod-interaction`) and provide two subcommands: `get` and `extend`.
When the `get` subcommand is called, the plugin checks the metadata attached by our admission controller and transfers it to human-readable information. Here is an example output from running `kubectl pi get`:
The plugin can also be used to extend the TTL for a Pod that is marked for future eviction. This is useful in case developers need extra time to debug ongoing issues. To achieve this, a developer uses the `kubectl pi extend` subcommand, where the plugin patches the relevant *annotations* for the given Pod. These *annotations* include the duration and username who made the extension request for transparency (displayed in the table returned from the `kubectl pi get` command).
Correspondingly, there is another webhook defined in *kube-exec-controller* which admits valid annotation updates. Once admitted, those updates reset the eviction timer of the target Pod as requested. An example of requesting the extension from the developer side would be:
```
$ kubectl pi extend test-pod --duration=30m
Successfully extended the termination time of pod/test-pod with a duration=30m
Although our admission controller service works great in handling interactive requests to a Pod, it could as well evict the Pod while the actual commands are no-op in these requests. For instance, developers sometimes run `kubectl exec` merely to check their service logs stored on hosts. Nevertheless, the target Pods would still get bounced despite the state of their containers not changing at all. One of the improvements here could be adding the ability to distinguish the commands that are passed to the interactive requests, so that no-op commands should not always force a Pod eviction. However, this becomes challenging when developers get a shell to a running container and execute commands inside the shell, since they will no longer be visible to our admission controller service.
Another item worth pointing out here is the choice of using K8s *labels* and *annotations*. In our design, we decided to have all immutable metadata attached as *labels* for better enforcing the immutability in our admission control. Yet some of these metadata could fit better as *annotations*. For instance, we had a label with the key `box.com/podInitialInteractionTimestamp` used to list all affected Pods in *kube-exec-controller* code, although its value would be unlikely to query for. As a more ideal design in the K8s world, a single *label* could be preferable in our case for identification with other metadata applied as *annotations* instead.
## Summary
With the power of admission controllers, we are able to secure our K8s clusters by detecting potentially mutated containers at runtime, and evicting their Pods without affecting service availability. We also utilize kubectl plugins to provide flexibility of the eviction time and hence, bringing a better and more self-independent experience to service owners. We are proud to announce that we have open-sourced the whole project for the community to leverage in their own K8s clusters. Any contribution is more than welcomed and appreciated. You can find this project hosted on GitHub at https://github.com/box/kube-exec-controller
*Special thanks to Ayush Sobti and Ethan Goldblum for their technical guidance on this project.*
When you own two data centers, thousands of physical servers, virtual machines and hosting for hundreds of thousands sites, Kubernetes can actually simplify the management of all these things. As practice has shown, by using Kubernetes, you can declaratively describe and manage not only applications, but also the infrastructure itself. I work for the largest Czech hosting provider **WEDOS Internet a.s** and today I'll show you two of my projects — [Kubernetes-in-Kubernetes](https://github.com/kvaps/kubernetes-in-kubernetes) and [Kubefarm](https://github.com/kvaps/kubefarm).
With their help you can deploy a fully working Kubernetes cluster inside another Kubernetes using Helm in just a couple of commands. How and why?
Let me introduce you to how our infrastructure works. All our physical servers can be divided into two groups: **control-plane** and **compute** nodes. Control plane nodes are usually set up manually, have a stable OS installed, and designed to run all cluster services including Kubernetes control-plane. The main task of these nodes is to ensure the smooth operation of the cluster itself. Compute nodes do not have any operating system installed by default, instead they are booting the OS image over the network directly from the control plane nodes. Their work is to carry out the workload.
Once nodes have downloaded their image, they can continue to work without keeping connection to the PXE server. That is, a PXE server is just keeping rootfs image and does not hold any other complex logic. After our nodes have booted, we can safely restart the PXE server, nothing critical will happen to them.
{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/scheme02.svg" alt="Kubernetes cluster after bootstrapping" >}}
After booting, the first thing our nodes do is join to the existing Kubernetes cluster, namely, execute the **kubeadm join** command so that kube-scheduler could schedule some pods on them and launch various workloads afterwards. From the beginning we used the scheme when nodes were joined into the same cluster used for the control-plane nodes.
{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/scheme03.svg" alt="Kubernetes scheduling containers to the compute nodes" >}}
This scheme worked stably for over two years. However later we decided to add containerized Kubernetes to it. And now we can spawn new Kubernetes-clusters very easily right on our control-plane nodes which are now member special admin-clusters. Now, compute nodes can be joined directly to their own clusters - depending on the configuration.
{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/scheme04.svg" alt="Multiple clusters are running in single Kubernetes, compute nodes joined to them" >}}
## Kubefarm
This project came with the goal of enabling anyone to deploy such an infrastructure in just a couple of commands using Helm and get about the same in the end.
At this time, we moved away from the idea of a monocluster. Because it turned out to be not very convenient for managing work of several development teams in the same cluster. The fact is that Kubernetes was never designed as a multi-tenant solution and at the moment it does not provide sufficient means of isolation between projects. Therefore, running separate clusters for each team turned out to be a good idea. However, there should not be too many clusters, to let them be convenient to manage. Nor is it too small to have sufficient independence between development teams.
The scalability of our clusters became noticeably better after that change. The more clusters you have per number of nodes, the smaller the failure domain and the more stable they work. And as a bonus, we got a fully declaratively described infrastructure. Thus, now you can deploy a new Kubernetes cluster in the same way as deploying any other application in Kubernetes.
It uses [Kubernetes-in-Kubernetes](http://github.com/kvaps/kubernetes-in-kubernetes) as a basis, [LTSP](https://github.com/ltsp/ltsp/) as PXE-server from which the nodes are booted, and automates the DHCP server configuration using [dnsmasq-controller](https://github.com/kvaps/dnsmasq-controller):
Now let's see how it works. In general, if you look at Kubernetes as from an application perspective, you can note that it follows all the principles of [The Twelve-Factor App](https://12factor.net/), and is actually written very well. Thus, it means running Kubernetes as an app in a different Kubernetes shouldn't be a big deal.
### Running Kubernetes in Kubernetes
Now let's take a look at the [Kubernetes-in-Kubernetes](https://github.com/kvaps/kubernetes-in-kubernetes) project, which provides a ready-made Helm chart for running Kubernetes in Kubernetes.
Here is the parameters that you can pass to Helm in the values file:
<img alt="Kubernetes is just five binaries" style="float: right; max-height: 280px;" src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/5binaries.png">
Beside **persistence** (storage parameters for the cluster), the Kubernetes control-plane components are described here: namely: **etcd cluster**, **apiserver**, **controller-manager** and **scheduler**. These are pretty much standard Kubernetes components. There is a light-hearted saying that “Kubernetes is just five binaries”. So here is where the configuration for these binaries is located.
If you ever tried to bootstrap a cluster using kubeadm, then this config will remind you it's configuration. But in addition to Kubernetes entities, you also have an admin container. In fact, it is a container which holds two binaries inside: **kubectl** and **kubeadm**. They are used to generate kubeconfig for the above components and to perform the initial configuration for the cluster. Also, in an emergency, you can always exec into it to check and manage your cluster.
After the release [has been deployed](https://asciinema.org/a/407280), you can see a list of pods: **admin-container**, **apiserver** in two replicas, **controller-manager**, **etcd-cluster**, **scheduller** and the initial job that initializes the cluster. In the end you have a command, which allows you to get shell into the admin container, you can use it to see what is happening inside:
Also, let's take look at the certificates. If you've ever installed Kubernetes, then you know that it has a _scary_ directory `/etc/kubernetes/pki` with a bunch of some certificates. In case of Kubernetes-in-Kubernetes, you have fully automated management of them with cert-manager. Thus, it is enough to pass all certificates parameters to Helm during installation, and all the certificates will automatically be generated for your cluster.
Looking at one of the certificates, eg. apiserver, you can see that it has a list of DNS names and IP addresses. If you want to make this cluster accessible outside, then just describe the additional DNS names in the values file and update the release. This will update the certificate resource, and cert-manager will regenerate the certificate. You'll no longer need to think about this. If kubeadm certificates need to be renewed at least once a year, here the cert-manager will take care and automatically renew them.
Now let's log into the admin container and look at the cluster and nodes. Of course, there are no nodes, yet, because at the moment you have deployed just the blank control-plane for Kubernetes. But in kube-system namespace you can see some coredns pods waiting for scheduling and configmaps already appeared. That is, you can conclude that the cluster is working:
Here is the [diagram of the deployed cluster](https://kvaps.github.io/images/posts/Kubernetes-in-Kubernetes-and-PXE-bootable-servers-farm/Argo_CD_kink_network.html). You can see services for all Kubernetes components: **apiserver**, **controller-manager**, **etcd-cluster** and **scheduler**. And the pods on right side to which they forward traffic.
*By the way, the diagram, is drawn in [ArgoCD](https://argoproj.github.io/argo-cd/) — the GitOps tool we use to manage our clusters, and cool diagrams are one of its features.*
### Orchestrating physical servers
OK, now you can see the way how is our Kubernetes control-plane deployed, but what about worker nodes, how are we adding them? As I already said, all our servers are bare metal. We do not use virtualization to run Kubernetes, but we orchestrate all physical servers by ourselves.
Also, we do use Linux network boot feature very actively. Moreover, this is exactly the booting, not some kind of automation of the installation. When the nodes are booting, they just run a ready-made system image for them. That is, to update any node, we just need to reboot it - and it will download a new image. It is very easy, simple and convenient.
For this, the [Kubefarm](https://github.com/kvaps/kubefarm) project was created, which allows you to automate this. The most commonly used examples can be found in the [examples](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples) directory. The most standard of them named [generic](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/generic). Let's take a look at values.yaml:
Here you can specify the parameters which are passed into the upstream Kubernetes-in-Kubernetes chart. In order for you control-plane to be accessible from the outside, it is enough to specify the IP address here, but if you wish, you can specify some DNS name here.
In the PXE server configuration you can specify a timezone. You can also add an SSH key for logging in without a password (but you can also specify a password), as well as kernel modules and parameters that should be applied during booting the system.
Next comes the **nodePools** configuration, i.e. the nodes themselves. If you've ever used a terraform module for gke, then this logic will remind you of it. Here you statically describe all nodes with a set of parameters:
- **Name** (hostname);
- **MAC-addresses** — we have nodes with two network cards, and each one can boot from any of the MAC addresses specified here.
- **IP-address**, which the DHCP server should issue to this node.
In this example, you have two pools: the first has five nodes, the second has only one, the second pool has also two tags assigned. Tags are the way to describe configuration for specific nodes. For example, you can add specific DHCP options for some pools, options for the PXE server for booting (e.g. here is debug option enabled) and set of **kubernetesLabels** and **kubernetesTaints** options. What does that mean?
For example, in this configuration you have a second nodePool with one node. The pool has **debug** and **foo** tags assigned. Now see the options for **foo** tag in **kubernetesLabels**. This means that the m1c43 node will boot with these two labels and taint assigned. Everything seems to be simple. Now [let's try](https://asciinema.org/a/407282) this in practice.
### Demo
Go to [examples](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples) and update previously deployed chart to Kubefarm. Just use the [generic](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/generic) parameters and look at the pods. You can see that a PXE server and one more job were added. This job essentially goes to the deployed Kubernetes cluster and creates a new token. Now it will run repeatedly every 12 hours to generate a new token, so that the nodes can connect to your cluster.
In a [graphical representation](https://kvaps.github.io/images/posts/Kubernetes-in-Kubernetes-and-PXE-bootable-servers-farm/Argo_CD_Applications_kubefarm-network.html), it looks about the same, but now apiserver started to be exposed outside.
In the diagram, the IP is highlighted in green, the PXE server can be reached through it. At the moment, Kubernetes does not allow creating a single LoadBalancer service for TCP and UDP protocols by default, so you have to create two different services with the same IP address. One is for TFTP, and the second for HTTP, through which the system image is downloaded.
But this simple example is not always enough, sometimes you might need to modify the logic at boot. For example, here is a directory [advanced_network](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/advanced_network), inside which there is a [values file](https://github.com/kvaps/kubefarm/tree/v0.13.1/examples/advanced_network) with a simple shell script. Let's call it `network.sh`:
All this script does is take environment variables at boot time, and generates a network configuration based on them. It creates a directory and puts the netplan config inside. For example, a bonding interface is created here. Basically, this script can contain everything you need. It can hold the network configuration or generate the system services, add some hooks or describe any other logic. Anything that can be described in bash or shell languages will work here, and it will be executed at boot time.
Let's see how it can be [deployed](https://asciinema.org/a/407284). Let's pass the generic values file as the first parameter, and an additional values file as the second parameter. This is a standard Helm feature. This way you can also pass the secrets, but in this case, the configuration is just expanded by the second file:
Let's look at the configmap **foo-kubernetes-ltsp** for the netboot server and make sure that `network.sh` script is really there. These commands used to configure the network at boot time:
[Here](https://asciinema.org/a/407286) you can see how it works in principle. The chassis interface (we use HPE Moonshots 1500) have the nodes, you can enter `show node list` command to get a list of all the nodes. Now you can see the booting process.
You can also get their MAC addresses by `show node macaddr all` command. We have a clever operator that collects MAC-addresses from chassis automatically and passes them to the DHCP server. Actually, it's just creating custom configuration resources for dnsmasq-controller which is running in same admin Kubernetes cluster. Also, trough this interface you can control the nodes themselves, e.g. turn them on and off.
If you have no such opportunity to enter the chassis through iLO and collect a list of MAC addresses for your nodes, you can consider using [catchall cluster](https://asciinema.org/a/407287) pattern. Purely speaking, it is just a cluster with a dynamic DHCP pool. Thus, all nodes that are not described in the configuration to other clusters will automatically join to this cluster.
For example, you can see a special cluster with some nodes. They are joined to the cluster with an auto-generated name based on their MAC address. Starting from this point you can connect to them and see what happens there. Here you can somehow prepare them, for example, set up the file system and then rejoin them to another cluster.
Now let's try connecting to the node terminal and see how it is booting. After the BIOS, the network card is configured, here it sends a request to the DHCP server from a specific MAC address, which redirects it to a specific PXE server. Later the kernel and initrd image are downloaded from the server using the standard HTTP protocol:
After loading the kernel, the node downloads the rootfs image and transfers control to systemd. Then the booting proceeds as usual, and after that the node joins Kubernetes:
If you take a look at **fstab**, you can see only two entries there: **/var/lib/docker** and **/var/lib/kubelet**, they are mounted as **tmpfs** (in fact, from RAM). At the same time, the root partition is mounted as **overlayfs**, so all changes that you make here on the system will be lost on the next reboot.
Looking into the block devices on the node, you can see some nvme disk, but it has not yet been mounted anywhere. There is also a loop device - this is the exact rootfs image downloaded from the server. At the moment it is located in RAM, occupies 653 MB and mounted with the **loop** option.
If you look in **/etc/ltsp**, you find the `network.sh` file that was executed at boot. From containers, you can see running `kube-proxy` and `pause` container for it.
But where does the main image come from? There is a little trick here. The image for the nodes is built through the [Dockerfile](https://github.com/kvaps/kubefarm/tree/v0.13.1/build/ltsp) along with the server. The [Docker multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) feature allows you to easily add any packages and kernel modules exactly at the stage of the image build. It looks like this:
What's going on here? First, we take a regular Ubuntu 20.04 and install all the packages we need. First of all we install the **kernel**, **lvm**, **systemd**, **ssh**. In general, everything that you want to see on the final node should be described here. Here we also install `docker` with `kubelet` and `kubeadm`, which are used to join the node to the cluster.
And then we perform an additional configuration. In the last stage, we simply install `tftp` and `nginx` (which serves our image to clients), **grub** (bootloader). Then root of the previous stages copied into the final image and generate squashed image from it. That is, in fact, we get a docker image, which has both the server and the boot image for our nodes. At the same time, it can be easily updated by changing the Dockerfile.
### Webhooks and API aggregation layer
I want to pay special attention to the problem of webhooks and aggregation layer. In general, webhooks is a Kubernetes feature that allows you to respond to the creation or modification of any resources. Thus, you can add a handler so that when resources are applied, Kubernetes must send request to some pod and check if configuration of this resource is correct, or make additional changes to it.
But the point is, in order for the webhooks to work, the apiserver must have direct access to the cluster for which it is running. And if it is started in a separate cluster, like our case, or even separately from any cluster, then Konnectivity service can help us here. Konnectivity is one of the optional but officially supported Kubernetes components.
Let's take cluster of four nodes for example, each of them is running a `kubelet` and we have other Kubernetes components running outside: `kube-apiserver`, `kube-scheduler` and `kube-controller-manager`. By default, all these components interact with the apiserver directly - this is the most known part of the Kubernetes logic. But in fact, there is also a reverse connection. For example, when you want to view the logs or run a `kubectl exec command`, the API server establishes a connection to the specific kubelet independently:
But the problem is that if we have a webhook, then it usually runs as a standard pod with a service in our cluster. And when apiserver tries to reach it, it will fail because it will try to access an in-cluster service named **webhook.namespace.svc** being outside of the cluster where it is actually running:
And here Konnectivity comes to our rescue. Konnectivity is a tricky proxy server developed especially for Kubernetes. It can be deployed as a server next to the apiserver. And Konnectivity-agent is deployed in several replicas directly in the cluster you want to access. The agent establishes a connection to the server and sets up a stable channel to make apiserver able to access all webhooks and all kubelets in the cluster. Thus, now all communication with the cluster will take place through the Konnectivity-server:
{{< figure src="/images/blog/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm/konnectivity03.svg" alt="Kubernetes apiserver reaching webhook via konnectivity" >}}
## Our plans
Of course, we are not going to stop at this stage. People interested in the project often write to me. And if there will be a sufficient number of interested people, I hope to move Kubernetes-in-Kubernetes project under [Kubernetes SIGs](https://github.com/kubernetes-sigs), by representing it in form of the official Kubernetes Helm chart. Perhaps, by making this project independent we'll gather an even larger community.
I am also thinking of integrating it with the Machine Controller Manager, which would allow creating worker nodes, not only of physical servers, but also, for example, for creating virtual machines using kubevirt and running them in the same Kubernetes cluster. By the way, it also allows to spawn virtual machines in the clouds, and have a control-plane deployed locally.
I am also considering the option of integrating with the Cluster-API so that you can create physical Kubefarm clusters directly through the Kubernetes environment. But at the moment I'm not completely sure about this idea. If you have any thoughts on this matter, I'll be happy to listen to them.
title: "Kubernetes is Moving on From Dockershim: Commitments and Next Steps"
date: 2022-01-07
slug: kubernetes-is-moving-on-from-dockershim
---
**Authors:** Sergey Kanzhelev (Google), Jim Angel (Google), Davanum Srinivas (VMware), Shannon Kularathna (Google), Chris Short (AWS), Dawn Chen (Google)
Kubernetes is removing dockershim in the upcoming v1.24 release. We're excited
to reaffirm our community values by supporting open source container runtimes,
enabling a smaller kubelet, and increasing engineering velocity for teams using
Kubernetes. If you [use Docker Engine as a container runtime](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/)
for your Kubernetes cluster, get ready to migrate in 1.24! To check if you're
affected, refer to [Check whether dockershim deprecation affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/).
## Why we’re moving away from dockershim
Docker was the first container runtime used by Kubernetes. This is one of the
reasons why Docker is so familiar to many Kubernetes users and enthusiasts.
Docker support was hardcoded into Kubernetes – a component the project refers to
as dockershim.
As containerization became an industry standard, the Kubernetes project added support
for additional runtimes. This culminated in the implementation of the
container runtime interface (CRI), letting system components (like the kubelet)
talk to container runtimes in a standardized way. As a result, dockershim became
an anomaly in the Kubernetes project.
Dependencies on Docker and dockershim have crept into various tools
and projects in the CNCF ecosystem ecosystem, resulting in fragile code.
By removing the
dockershim CRI, we're embracing the first value of CNCF: "[Fast is better than
channel on [Kubernetes Slack](https://slack.kubernetes.io/)!
## Final thoughts
As a project, we've already seen cluster operators increasingly adopt other
container runtimes through 2021.
We believe there are no major blockers to migration. The steps we're taking to
improve the migration experience will light the path more clearly for you.
We understand that migration from dockershim is yet another action you may need to
do to keep your Kubernetes infrastructure up to date. For most of you, this step
will be straightforward and transparent. In some cases, you will encounter
hiccups or issues. The community has discussed at length whether postponing the
dockershim removal would be helpful. For example, we recently talked about it in
the [SIG Node discussion on November 11th](https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#bookmark=id.r77y11bgzid)
and in the [Kubernetes Steering committee meeting held on December 6th](https://docs.google.com/document/d/1qazwMIHGeF3iUh5xMJIJ6PDr-S3bNkT8tNLRkSiOkOU/edit#bookmark=id.m0ir406av7jx).
We already [postponed](https://github.com/kubernetes/enhancements/pull/2481/) it
once in 2021 because the adoption rate of other
runtimes was lower than we wanted, which also gave us more time to identify
potential blocking issues.
At this point, we believe that the value that you (and Kubernetes) gain from
dockershim removal makes up for the migration effort you'll have. Start planning
now to avoid surprises. We'll have more updates and guides before Kubernetes
Welcome to the first episode of the APAC edition of the "Meet Our Contributors" blog post series.
In this post, we'll introduce you to five amazing folks from the India region who have been actively contributing to the upstream Kubernetes projects in a variety of ways, as well as being the leaders or maintainers of numerous community initiatives.
💫 *Let's get started, so without further ado…*
## [Arsh Sharma](https://github.com/RinkiyaKeDad)
Arsh is currently employed with Okteto as a Developer Experience engineer. As a new contributor, he realised that 1:1 mentorship opportunities were quite beneficial in getting him started with the upstream project.
He is presently a CI Signal shadow on the Kubernetes 1.23 release team. He is also contributing to the SIG Testing and SIG Docs projects, as well as to the [cert-manager](https://github.com/cert-manager/infrastructure) tools development work that is being done under the aegis of SIG Architecture.
To the newcomers, Arsh helps plan their early contributions sustainably.
> _I would encourage folks to contribute in a way that's sustainable. What I mean by that
> is that it's easy to be very enthusiastic early on and take up more stuff than one can
> actually handle. This can often lead to burnout in later stages. It's much more sustainable
Kunal Kushwaha is a core member of the Kubernetes marketing council. He is also a CNCF ambassador and one of the founders of the [CNCF Students Program](https://community.cncf.io/cloud-native-students/).. He also served as a Communications role shadow during the 1.22 release cycle.
At the end of his first year, Kunal began contributing to the [fabric8io kubernetes-client](https://github.com/fabric8io/kubernetes-client) project. He was then selected to work on the same project as part of Google Summer of Code. Kunal mentored people on the same project, first through Google Summer of Code then through Google Code-in.
As an open-source enthusiast, he believes that diverse participation in the community is beneficial since it introduces new perspectives and opinions and respect for one's peers. He has worked on various open-source projects, and his participation in communities has considerably assisted his development as a developer.
> _I believe if you find yourself in a place where you do not know much about the
> project, that's a good thing because now you can learn while contributing and the
> community is there to help you. It has helped me a lot in gaining skills, meeting
> people from around the world and also helping them. You can learn on the go,
> you don't have to be an expert. Make sure to also check out no code contributions
> because being a beginner is a skill and you can bring new perspectives to the
Madhav Jivarajani works on the VMware Upstream Kubernetes stability team. He began contributing to the Kubernetes project in January 2021 and has since made significant contributions to several areas of work under SIG Architecture, SIG API Machinery, and SIG ContribEx (contributor experience).
Among several significant contributions are his recent efforts toward the Archival of [design proposals](https://github.com/kubernetes/community/issues/6055), refactoring the ["groups" codebase](https://github.com/kubernetes/k8s.io/pull/2713) under k8s-infra repository to make it mockable and testable, and improving the functionality of the [GitHub k8s bot](https://github.com/kubernetes/test-infra/issues/23129).
In addition to his technical efforts, Madhav oversees many projects aimed at assisting new contributors. He organises bi-weekly "KEP reading club" sessions to help newcomers understand the process of adding new features, deprecating old ones, and making other key changes to the upstream project. He has also worked on developing [Katacoda scenarios](https://github.com/kubernetes-sigs/contributor-katacoda) to assist new contributors to become acquainted with the process of contributing to k/k. In addition to his current efforts to meet with community members every week, he has organised several [new contributors workshops (NCW)](https://www.youtube.com/watch?v=FgsXbHBRYIc).
> _I initially did not know much about Kubernetes. I joined because the community was
> super friendly. But what made me stay was not just the people, but the project itself.
> My solution to not feeling overwhelmed in the community was to gain as much context
> and knowledge into the topics that I was interested in and were being discussed. And
> as a result I continued to dig deeper into Kubernetes and the design of it.
> I am a systems nut & thus Kubernetes was an absolute goldmine for me._
Rajas Kakodkar currently works at VMware as a Member of Technical Staff. He has been engaged in many aspects of the upstream Kubernetes project since 2019.
He is now a key contributor to the Testing special interest group. He is also active in the SIG Network community. Lately, Rajas has contributed significantly to the [NetworkPolicy++](https://docs.google.com/document/d/1AtWQy2fNa4qXRag9cCp5_HsefD7bxKe3ea2RPn8jnSs/) and [`kpng`](https://github.com/kubernetes-sigs/kpng) sub-projects.
One of the first challenges he ran across was that he was in a different time zone than the upstream project's regular meeting hours. However, async interactions on community forums progressively corrected that problem.
> _I enjoy contributing to Kubernetes not just because I get to work on
> cutting edge tech but more importantly because I get to work with
> awesome people and help in solving real world problems._
Rajula Vineet Reddy, a Junior Engineer at CERN, is a member of the Marketing Council team under SIG ContribEx . He also served as a release shadow for SIG Release during the 1.22 and 1.23 Kubernetes release cycles.
He started looking at the Kubernetes project as part of a university project with the help of one of his professors. Over time, he spent a significant amount of time reading the project's documentation, Slack discussions, GitHub issues, and blogs, which helped him better grasp the Kubernetes project and piqued his interest in contributing upstream. One of his key contributions was his assistance with automation in the SIG ContribEx Upstream Marketing subproject.
According to Rajula, attending project meetings and shadowing various project roles are vital for learning about the community.
> _I find the community very helpful and it's always_
> “you get back as much as you contribute”.
> _The more involved you are, the more you will understand, get to learn and
> contribute new things._
>
> _The first step to_ “come forward and start” _is hard. But it's all gonna be
> smooth after that. Just take that jump._
---
If you have any recommendations/suggestions for who we should interview next, please let us know in #sig-contribex. We're thrilled to have other folks assisting us in reaching out to even more wonderful individuals of the community. Your suggestions would be much appreciated.
We'll see you all in the next one. Everyone, till then, have a happy contributing! 👋
[Admission control](/docs/reference/access-authn-authz/admission-controllers/) is a key part of Kubernetes security, alongside authentication and authorization. Webhook admission controllers are extensively used to help improve the security of Kubernetes clusters in a variety of ways including restricting the privileges of workloads and ensuring that images deployed to the cluster meet organization’s security requirements.
However, as with any additional component added to a cluster, security risks can present themselves. A security risk example is if the deployment and management of the admission controller are not handled correctly. To help admission controller users and designers manage these risks appropriately, the [security documentation](https://github.com/kubernetes/community/tree/master/sig-security#security-docs) subgroup of SIG Security has spent some time developing a [threat model for admission controllers](https://github.com/kubernetes/sig-security/tree/main/sig-security-docs/papers/admission-control). This threat model looks at likely risks which may arise from the incorrect use of admission controllers, which could allow security policies to be bypassed, or even allow an attacker to get unauthorised access to the cluster.
From the threat model, we developed a set of security best practices that should be adopted to ensure that cluster operators can get the security benefits of admission controllers whilst avoiding any risks from using them.
## Admission controllers and good practices for security
From the threat model, a couple of themes emerged around how to ensure the security of admission controllers.
### Secure webhook configuration
It’s important to ensure that any security component in a cluster is well configured and admission controllers are no different here. There are a couple of security best practices to consider when using admission controllers
* **Correctly configured TLS for all webhook traffic**. Communications between the API server and the admission controller webhook should be authenticated and encrypted to ensure that attackers who may be in a network position to view or modify this traffic cannot do so. To achieve this access the API server and webhook must be using certificates from a trusted certificate authority so that they can validate their mutual identities
* **Only authenticated access allowed**. If an attacker can send an admission controller large numbers of requests, they may be able to overwhelm the service causing it to fail. Ensuring all access requires strong authentication should mitigate that risk.
* **Admission controller fails closed**. This is a security practice that has a tradeoff, so whether a cluster operator wants to configure it will depend on the cluster’s threat model. If an admission controller fails closed, when the API server can’t get a response from it, all deployments will fail. This stops attackers bypassing the admission controller by disabling it, but, can disrupt the cluster’s operation. As clusters can have multiple webhooks, one approach to hit a middle ground might be to have critical controls on a fail closed setups and less critical controls allowed to fail open.
* **Regular reviews of webhook configuration**. Configuration mistakes can lead to security issues, so it’s important that the admission controller webhook configuration is checked to make sure the settings are correct. This kind of review could be done automatically by an Infrastructure As Code scanner or manually by an administrator.
### Secure cluster configuration for admission control
In most cases, the admission controller webhook used by a cluster will be installed as a workload in the cluster. As a result, it’s important to ensure that Kubernetes' security features that could impact its operation are well configured.
* **Restrict [RBAC](/docs/reference/access-authn-authz/rbac/) rights**. Any user who has rights which would allow them to modify the configuration of the webhook objects or the workload that the admission controller uses could disrupt its operation. So it’s important to make sure that only cluster administrators have those rights.
* **Prevent privileged workloads**. One of the realities of container systems is that if a workload is given certain privileges, it will be possible to break out to the underlying cluster node and impact other containers on that node. Where admission controller services run in the cluster they’re protecting, it’s important to ensure that any requirement for privileged workloads is carefully reviewed and restricted as much as possible.
* **Strictly control external system access**. As a security service in a cluster admission controller systems will have access to sensitive information like credentials. To reduce the risk of this information being sent outside the cluster, [network policies](/docs/concepts/services-networking/network-policies/) should be used to restrict the admission controller services access to external networks.
* **Each cluster has a dedicated webhook**. Whilst it may be possible to have admission controller webhooks that serve multiple clusters, there is a risk when using that model that an attack on the webhook service would have a larger impact where it’s shared. Also where multiple clusters use an admission controller there will be increased complexity and access requirements, making it harder to secure.
### Admission controller rules
A key element of any admission controller used for Kubernetes security is the rulebase it uses. The rules need to be able to accurately meet their goals avoiding false positive and false negative results.
* **Regularly test and review rules**. Admission controller rules need to be tested to ensure their accuracy. They also need to be regularly reviewed as the Kubernetes API will change with each new version, and rules need to be assessed with each Kubernetes release to understand any changes that may be required to keep them up to date.
**Authors:** Dewan Ahmed (Aiven) and Chris Short (AWS)
## Introduction
[SIG Multicluster](https://github.com/kubernetes/community/tree/master/sig-multicluster) is the SIG focused on how Kubernetes concepts are expanded and used beyond the cluster boundary. Historically, Kubernetes resources only interacted within that boundary - KRU or Kubernetes Resource Universe (not an actual Kubernetes concept). Kubernetes clusters, even now, don't really know anything about themselves or, about other clusters. Absence of cluster identifiers is a case in point. With the growing adoption of multicloud and multicluster deployments, the work SIG Multicluster doing is gaining a lot of attention. In this blog, [Jeremy Olmsted-Thompson, Google](https://twitter.com/jeremyot) and [Chris Short, AWS](https://twitter.com/ChrisShort) discuss the interesting problems SIG Multicluster is solving and how you can get involved. Their initials **JOT** and **CS** will be used for brevity.
## A summary of their conversation
**CS**: How long has the SIG Multicluster existed and how was the SIG in its infancy? How long have you been with this SIG?
**JOT**: I've been around for almost two years in the SIG Multicluster. All I know about the infancy years is from the lore but even in the early days, it was always about solving this same problem. Early efforts have been things like [KubeFed](https://github.com/kubernetes-sigs/kubefed). I think there are still folks using KubeFed but it's a smaller slice. Back then, I think people out there deploying large numbers of Kubernetes clusters were really not at a point where we had a ton of real concrete use cases. Projects like KubeFed and [Cluster Registry](https://github.com/kubernetes-retired/cluster-registry) were developed around that time and the need back then can be associated to these projects. The motivation for these projects were how do we solve the problems that we think people are **going to have**, when they start expanding to multiple clusters. Honestly, in some ways, it was trying to do too much at that time.
**CS**: How does KubeFed differ from the current state of SIG Multicluster? How does the **lore** differ from the **now**?
**JOT**: Yeah, it was like trying to get ahead of potential problems instead of addressing specific problems. I think towards the end of 2019, there was a slow down in SIG multicluster work and we kind of picked it back up with one of the most active recent projects that is the [SIG Multicluster services (MCS)](https://github.com/kubernetes-sigs/mcs-api).
Now this is the shift to solving real specific problems. For example,
> I've got workloads that are spread across multiple clusters and I need them to talk to each other.
Okay, that's very straightforward and we know that we need to solve that. To get started, let's make sure that these projects can work together on a common API so you get the same kind of portability that you get with Kubernetes.
There's a few implementations of the MCS API out there and more are being developed. But, we didn't build an implementation because depending on how you're deploying things there could be hundreds of implementations. As long as you only need the basic Multicluster service functionality, it'll just work on whatever background you want, whether it's Submariner, GKE, or a service mesh.
My favorite example of "then vs. now" is cluster ID. A few years ago, there was an effort to define a cluster ID. A lot of really good thought went into this concept, for example, how do we make a cluster ID is unique across multiple clusters. How do we make this ID globally unique so it'll work in every contact? Let's say, there's an acquisition or merger of teams - does the cluster IDs still remain unique for those teams?
With Multicluster services, we found the need for an actual cluster ID, and it has a very specific need. To address this specific need, we're no longer considering every single Kubernetes cluster out there rather the ClusterSets - a grouping of clusters that work together in some kind of bounds. That's a much narrower scope than considering clusters everywhere in time and space. It also leaves flexibility for an implementer to define the boundary (a ClusterSet) beyond which this cluster ID will no longer be unique.
**CS**: How do you feel about the current state of SIG Multicluster versus where you're hoping to be in future?
**JOT**: There's a few projects that are kind of getting started, for example, Work API. In the future, I think that some common practices around how do we deploy things across clusters are going to develop.
> If I have clusters deployed in a bunch of different regions; what's the best way to actually do that?
The answer is, almost always, "it depends". Why are you doing this? Is it because there's some kind of compliance that makes you care about locality? Is it performance? Is it availability?
I think revisiting registry patterns will probably be a natural step after we have cluster IDs, that is, how do you actually associate these clusters together? Maybe you've got a distributed deployment that you run in your own data centers all over the world. I imagine that expanding the API in that space is going to be important as more multi cluster features develop. It really depends on what the community starts doing with these tools.
**CS**: In the early days of Kubernetes, we used to have a few large Kubernetes clusters and now we're dealing with many small Kubernetes clusters - even multiple clusters for our own dev environments. How has this shift from a few large clusters to many small clusters affected the SIG? Has it accelerated the work or make it challenging in any way?
**JOT**: I think that it has created a lot of ambiguity that needs solving. Originally, you'd have a dev cluster, a staging cluster, and a prod cluster. When the multi region thing came in, we started needing dev/staging/prod clusters, per region. And then, sometimes clusters really need more isolation due to compliance or some regulations issues. Thus, we're ending up with a lot of clusters. I think figuring out the right balance on how many clusters should you actually have is important. The power of Kubernetes is being able to deploy a lot of things managed by a single control plane. So, it's not like every single workload that gets deployed should be in its own cluster. But I think it's pretty clear that we can't put every single workload in a single cluster.
**CS**: What are some of your favorite things about this SIG?
**JOT**: The complexity of the problems, the people and the newness of the space. We don't have right answers and we have to figure this out. At the beginning, we couldn't even think about multi clusters because there was no way to connect services across clusters. Now there is and we're starting to go tackle those problems, I think that this is a really fun place to be in because I expect that the SIG is going to get a lot busier the next couple of years. It's a very collaborative group and we definitely would like more people to come join us, get involved, raise their problems and bring their ideas.
**CS**: What do you think keeps people in this group? How has the pandemic affected you?
**JOT**: I think it definitely got a little bit quieter during the pandemic. But for the most part; it's a very distributed group so whether you're calling in to our weekly meetings from a conference room or from your home, it doesn't make that huge of a difference. During the pandemic, a lot of people had time to focus on what's next for their scale and growth. I think that's what keeps people in the group - we have real problems that need to be solved which are very new in this space. And it's fun :)
## Wrap up
**CS**: That's all we have for today. Thanks Jeremy for your time.
**JOT**: Thanks Chris. Everybody is welcome at our [bi-weekly meetings](https://github.com/kubernetes/community/tree/master/sig-multicluster#meetings). We love as many people to come as possible and welcome all questions and all ideas. It's a new space and it'd be great to grow the community.
Ensuring the reliability of SIG Node upstream code is a continuous effort
that takes a lot of behind-the-scenes effort from many contributors.
There are frequent releases of Kubernetes, base operating systems,
container runtimes, and test infrastructure that result in a complex matrix that
requires attention and steady investment to "keep the lights on."
In May 2020, the Kubernetes node special interest group ("SIG Node") organized a new
subproject for continuous integration (CI) for node-related code and tests. Since its
inauguration, the SIG Node CI subproject has run a weekly meeting, and even the full hour
is often not enough to complete triage of all bugs, test-related PRs and issues, and discuss all
related ongoing work within the subgroup.
Over the past two years, we've fixed merge-blocking and release-blocking tests, reducing time to merge Kubernetes contributors' pull requests thanks to reduced test flakes. When we started, Node test jobs only passed 42% of the time, and through our efforts, we now ensure a consistent >90% job pass rate. We've closed 144 test failure issues and merged 176 pull requests just in kubernetes/kubernetes. And we've helped subproject participants ascend the Kubernetes contributor ladder, with 3 new org members, 6 new reviewers, and 2 new approvers.
The Node CI subproject is an approachable first stop to help new contributors
get started with SIG Node. There is a low barrier to entry for new contributors
to address high-impact bugs and test fixes, although there is a long
road before contributors can climb the entire contributor ladder:
it took over a year to establish two new approvers for the group.
The complexity of all the different components that power Kubernetes nodes
and its test infrastructure requires a sustained investment over a long period
for developers to deeply understand the entire system,
both at high and low levels of detail.
We have several regular contributors at our meetings, however; our reviewers
and approvers pool is still small. It is our goal to continue to grow
contributors to ensure a sustainable distribution of work
that does not just fall to a few key approvers.
It's not always obvious how subprojects within SIGs are formed, operate,
and work. Each is unique to its sponsoring SIG and tailored to the projects
that the group is intended to support. As a group that has welcomed many
first-time SIG Node contributors, we'd like to share some of the details and
accomplishments over the past two years,
helping to demystify our inner workings and celebrate the hard work
of all our dedicated contributors!
## Timeline
***May 2020.*** SIG Node CI group was formed on May 11, 2020, with more than
We give group updates at our maintainer track sessions, such as
[KubeCon + CloudNativeCon Europe 2021](https://kccnceu2021.sched.com/event/iE8E/kubernetes-sig-node-intro-and-deep-dive-elana-hashman-red-hat-sergey-kanzhelev-google) and
[KubeCon + CloudNative North America 2021](https://kccncna2021.sched.com/event/lV9D/kubenetes-sig-node-intro-and-deep-dive-elana-hashman-derek-carr-red-hat-sergey-kanzhelev-dawn-chen-google?iframe=no&w=100%&sidebar=yes&bg=no).
Join us in our mission to keep the kubelet and other SIG Node components reliable and ensure smooth and uneventful releases!
as a part of the Kubernetes v1.20 release. For more detail
on what that means, check out the blog post
[Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/).
Also, you can read [check whether dockershim removal affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/)
to determine how much impact the removal of dockershim would have for you
or for your organization.
As the Kubernetes 1.24 release has become imminent, we've been working hard to try to make this a smooth transition.
- We've written a blog post detailing our [commitment and next steps](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/).
- We believe there are no major blockers to migration to [other container runtimes](/docs/setup/production-environment/container-runtimes/#container-runtimes).
- There is also a [Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/) guide available.
- We've also created a page to list
[articles on dockershim removal and on using CRI-compatible runtimes](/docs/reference/node/topics-on-dockershim-and-cri-compatible-runtimes/).
That list includes some of the already mentioned docs, and also covers selected external sources
(including vendor guides).
### Why is the dockershim being removed from Kubernetes?
Early versions of Kubernetes only worked with a specific container runtime:
Docker Engine. Later, Kubernetes added support for working with other container runtimes.
The CRI standard was [created](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) to
enable interoperability between orchestrators (like Kubernetes) and many different container
runtimes.
Docker Engine doesn't implement that interface (CRI), so the Kubernetes project created
special code to help with the transition, and made that _dockershim_ code part of Kubernetes
itself.
The dockershim code was always intended to be a temporary solution (hence the name: shim).
You can read more about the community discussion and planning in the
### What should I look out for when changing CRI implementations?
While the underlying containerization code is the same between Docker and most
CRIs (including containerd), there are a few differences around the edges. Some
common things to consider when migrating are:
- Logging configuration
- Runtime resource limitations
- Node provisioning scripts that call docker or use docker via it's control socket
- Kubectl plugins that require docker CLI or the control socket
- Tools from the Kubernetes project that require direct access to Docker Engine
(for example: the deprecated `kube-imagepuller` tool)
- Configuration of functionality like `registry-mirrors` and insecure registries
- Other support scripts or daemons that expect Docker Engine to be available and are run
outside of Kubernetes (for example, monitoring or security agents)
- GPUs or special hardware and how they integrate with your runtime and Kubernetes
If you use Kubernetes resource requests/limits or file-based log collection
DaemonSets then they will continue to work the same, but if you’ve customized
your `dockerd` configuration, you’ll need to adapt that for your new container
runtime where possible.
Another thing to look out for is anything expecting to run for system maintenance
or nested inside a container when building images will no longer work. For the
former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
latter you can use newer container build options like [img], [buildah],
[kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker.
Welcome back to the second episode of the "Meet Our Contributors" blog post series for APAC.
This post will feature four outstanding contributors from the Australia and New Zealand regions, who have played diverse leadership and community roles in the Upstream Kubernetes project.
So, without further ado, let's get straight to the blog.
Caleb Woodbine is currently a member of the ii.nz organisation.
He began contributing to the Kubernetes project in 2018 as a member of the Kubernetes Conformance working group. His experience was positive, and he benefited from early guidance from [Hippie Hacker](https://github.com/hh), a fellow contributor from New Zealand.
He has made major contributions to Kubernetes project since then through `SIG k8s-infra` and `k8s-conformance` working group.
Caleb is also a co-organizer of the [CloudNative NZ](https://www.meetup.com/cloudnative-nz/) community events, which aim to expand the reach of Kubernetes project throughout New Zealand in order to encourage technical education and improved employment opportunities.
> _There need to be more outreach in APAC and the educators and universities must pick up Kubernetes, as they are very slow and about 8+ years out of date. NZ tends to rather pay overseas than educate locals on the latest cloud tech Locally._
## [Dylan Graham](https://github.com/DylanGraham)
Dylan Graham is a cloud engineer from Adeliade, Australia. He has been contributing to the upstream Kubernetes project since 2018.
He stated that being a part of such a large-scale project was initially overwhelming, but that the community's friendliness and openness assisted him in getting through it.
He began by contributing to the project documentation and is now mostly focused on the community support for the APAC region.
He believes that consistent attendance at community/project meetings, taking on project tasks, and seeking community guidance as needed can help new aspiring developers become effective contributors.
> _The feeling of being a part of a large community is really special. I've met some amazing people, even some before the pandemic in real life :)_
## [Hippie Hacker](https://github.com/hh)
Hippie has worked for the CNCF.io as a Strategic Initiatives contractor from New Zealand for almost 5+ years. He is an active contributor to k8s-infra, API conformance testing, Cloud provider conformance submissions, and apisnoop.cncf.io domains of the upstream Kubernetes & CNCF projects.
He recounts their early involvement with the Kubernetes project, which began roughly 5 years ago when their firm, ii.nz, demonstrated [network booting from a Raspberry Pi using PXE and running Gitlab in-cluster to install Kubernetes on servers](https://ii.nz/post/bringing-the-cloud-to-your-community/).
He describes their own contributing experience as someone who, at first, tried to do all of the hard lifting on their own, but eventually saw the benefit of group contributions which reduced burnout and task division which allowed folks to keep moving forward on their own momentum.
He recommends that new contributors use pair programming.
> _The cross pollination of approaches and two pairs of eyes on the same work can often yield a much more amplified effect than a PR comment / approval alone can afford._
## [Nick Young](https://github.com/youngnick)
Nick Young works at VMware as a technical lead for Contour, a CNCF ingress controller. He was active with the upstream Kubernetes project from the beginning, and eventually became the chair of the LTS working group, where he advocated user concerns. He is currently the SIG Network Gateway API subproject's maintainer.
His contribution path was notable in that he began working on major areas of the Kubernetes project early on, skewing his trajectory.
He asserts the best thing a new contributor can do is to "start contributing". Naturally, if it is relevant to their employment, that is excellent; however, investing non-work time in contributing can pay off in the long run in terms of work. He believes that new contributors, particularly those who are currently Kubernetes users, should be encouraged to participate in higher-level project discussions.
> _Just being active and contributing will get you a long way. Once you've been active for a while, you'll find that you're able to answer questions, which will mean you're asked questions, and before you know it you are an expert._
---
If you have any recommendations/suggestions for who we should interview next, please let us know in #sig-contribex. Your suggestions would be much appreciated. We're thrilled to have additional folks assisting us in reaching out to even more wonderful individuals of the community.
We'll see you all in the next one. Everyone, till then, have a happy contributing! 👋
Way back in December of 2020, Kubernetes announced the [deprecation of Dockershim](/blog/2020/12/02/dont-panic-kubernetes-and-docker/). In Kubernetes, dockershim is a software shim that allows you to use the entire Docker engine as your container runtime within Kubernetes. In the upcoming v1.24 release, we are removing Dockershim - the delay between deprecation and removal in line with the [project’s policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/) of supporting features for at least one year after deprecation. If you are a cluster operator, this guide includes the practical realities of what you need to know going into this release. Also, what do you need to do to ensure your cluster doesn’t fall over!
## First, does this even affect you?
If you are rolling your own cluster or are otherwise unsure whether or not this removal affects you, stay on the safe side and [check to see if you have any dependencies on Docker Engine](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/). Please note that using Docker Desktop to build your application containers is not a Docker dependency for your cluster. Container images created by Docker are compliant with the [Open Container Initiative (OCI)](https://opencontainers.org/), a Linux Foundation governance structure that defines industry standards around container formats and runtimes. They will work just fine on any container runtime supported by Kubernetes.
If you are using a managed Kubernetes service from a cloud provider, and you haven’t explicitly changed the container runtime, there may be nothing else for you to do. Amazon EKS, Azure AKS, and Google GKE all default to containerd now, though you should make sure they do not need updating if you have any node customizations. To check the runtime of your nodes, follow [Find Out What Container Runtime is Used on a Node](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/).
Regardless of whether you are rolling your own cluster or using a managed Kubernetes service from a cloud provider, you may need to [migrate telemetry or security agents that rely on Docker Engine](/docs/tasks/administer-cluster/migrating-from-dockershim/migrating-telemetry-and-security-agents/).
## I have a Docker dependency. What now?
If your Kubernetes cluster depends on Docker Engine and you intend to upgrade to Kubernetes v1.24 (which you should eventually do for security and similar reasons), you will need to change your container runtime from Docker Engine to something else or use [cri-dockerd](https://github.com/Mirantis/cri-dockerd). Since [containerd](https://containerd.io/) is a graduated CNCF project and the runtime within Docker itself, it’s a safe bet as an alternative container runtime. Fortunately, the Kubernetes project has already documented the process of [changing a node’s container runtime](/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/), using containerd as an example. Instructions are similar for switching to one of the other supported runtimes.
## I want to upgrade Kubernetes, and I need to maintain compatibility with Docker as a runtime. What are my options?
Fear not, you aren’t being left out in the cold and you don’t have to take the security risk of staying on an old version of Kubernetes. Mirantis and Docker have jointly released, and are maintaining, a replacement for dockershim. That replacement is called [cri-dockerd](https://github.com/Mirantis/cri-dockerd). If you do need to maintain compatibility with Docker as a runtime, install cri-dockerd following the instructions in the project’s documentation.
## Is that it?
Yes. As long as you go into this release aware of the changes being made and the details of your own clusters, and you make sure to communicate clearly with your development teams, it will be minimally dramatic. You may have some changes to make to your cluster, application code, or scripts, but all of these requirements are documented. Switching from using Docker Engine as your runtime to using [one of the other supported container runtimes](/docs/setup/production-environment/container-runtimes/) effectively means removing the middleman, since the purpose of dockershim is to access the container runtime used by Docker itself. From a practical perspective, this removal is better both for you and for Kubernetes maintainers in the long-run.
If you still have questions, please first check the [Dockershim Removal FAQ](/blog/2022/02/17/dockershim-faq/).
It's safe to say that the removal receiving the most attention with the release of Kubernetes 1.24
is Dockershim. Dockershim was deprecated in v1.20. As noted in the [Kubernetes 1.20 changelog](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#deprecation):
"Docker support in the kubelet is now deprecated and will be removed in a future release. The kubelet
uses a module called "dockershim" which implements CRI support for Docker and it has seen maintenance
issues in the Kubernetes community." With the upcoming release of Kubernetes 1.24, the Dockershim will
finally be removed.
In the article [Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/),
the authors succinctly captured the change's impact and encouraged users to remain calm:
> Docker as an underlying runtime is being deprecated in favor of runtimes that use the
> Container Runtime Interface (CRI) created for Kubernetes. Docker-produced images
> will continue to work in your cluster with all runtimes, as they always have.
Several guides have been created with helpful information about migrating from dockershim
to container runtimes that are directly compatible with Kubernetes. You can find them on the
[Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/)
page in the Kubernetes documentation.
For more information about why Kubernetes is moving away from dockershim, check out the aptly
named: [Kubernetes is Moving on From Dockershim](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/)
and the [updated dockershim removal FAQ](/blog/2022/02/17/dockershim-faq/).
Take a look at the [Is Your Cluster Ready for v1.24?](/blog/2022/03/31/ready-for-dockershim-removal/) post to learn about how to ensure your cluster continues to work after upgrading from v1.23 to v1.24.
## The Kubernetes API removal and deprecation process
Kubernetes contains a large number of components that evolve over time. In some cases, this
evolution results in APIs, flags, or entire features, being removed. To prevent users from facing
breaking changes, Kubernetes contributors adopted a feature [deprecation policy](/docs/reference/using-api/deprecation-policy/).
This policy ensures that stable APIs may only be deprecated when a newer stable version of that
same API is available and that APIs have a minimum lifetime as indicated by the following stability levels:
* Generally available (GA) or stable API versions may be marked as deprecated but must not be removed within a major version of Kubernetes.
* Beta or pre-release API versions must be supported for 3 releases after deprecation.
* Alpha or experimental API versions may be removed in any release without prior deprecation notice.
Removals follow the same deprecation policy regardless of whether an API is removed due to a beta feature
graduating to stable or because that API was not proven to be successful. Kubernetes will continue to make
sure migration options are documented whenever APIs are removed.
**Deprecated** APIs are those that have been marked for removal in a future Kubernetes release. **Removed**
APIs are those that are no longer available for use in current, supported Kubernetes versions after having
been deprecated. These removals have been superseded by newer, stable/generally available (GA) APIs.
## API removals, deprecations, and other changes for Kubernetes 1.24
* [Dynamic kubelet configuration](https://github.com/kubernetes/enhancements/issues/281): `DynamicKubeletConfig` is used to enable the dynamic configuration of the kubelet. The `DynamicKubeletConfig` flag was deprecated in Kubernetes 1.22. In v1.24, this feature gate will be removed from the kubelet. See [Reconfigure kubelet](/docs/tasks/administer-cluster/reconfigure-kubelet/). Refer to the ["Dynamic kubelet config is removed" KEP](https://github.com/kubernetes/enhancements/issues/281) for more information.
* [Dynamic log sanitization](https://github.com/kubernetes/kubernetes/pull/107207): The experimental dynamic log sanitization feature is deprecated and will be removed in v1.24. This feature introduced a logging filter that could be applied to all Kubernetes system components logs to prevent various types of sensitive information from leaking via logs. Refer to [KEP-1753: Kubernetes system components logs sanitization](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#deprecation) for more information and an [alternative approach](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#alternatives=).
* In-tree provisioner to CSI driver migration: This applies to a number of in-tree plugins, including [Portworx](https://github.com/kubernetes/enhancements/issues/2589). Refer to the [In-tree Storage Plugin to CSI Migration Design Doc](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/csi-migration.md#background-and-motivations) for more information.
* [Removing Dockershim from kubelet](https://github.com/kubernetes/enhancements/issues/2221): the Container Runtime Interface (CRI) for Docker (i.e. Dockershim) is currently a built-in container runtime in the kubelet code base. It was deprecated in v1.20. As of v1.24, the kubelet will no longer have dockershim. Check out this blog on [what you need to do be ready for v1.24](/blog/2022/03/31/ready-for-dockershim-removal/).
* [Storage capacity tracking for pod scheduling](https://github.com/kubernetes/enhancements/issues/1472): The CSIStorageCapacity API supports exposing currently available storage capacity via CSIStorageCapacity objects and enhances scheduling of pods that use CSI volumes with late binding. In v1.24, the CSIStorageCapacity API will be stable. The API graduating to stable initates the deprecation of the v1beta1 CSIStorageCapacity API. Refer to the [Storage Capacity Constraints for Pod Scheduling KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1472-storage-capacity-tracking) for more information.
* [The `master` label is no longer present on kubeadm control plane nodes](https://github.com/kubernetes/kubernetes/pull/107533). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to [KEP-2067: Rename the kubeadm "master" label and taint](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint).
* [VolumeSnapshot v1beta1 CRD will be removed](https://github.com/kubernetes/enhancements/issues/177). Volume snapshot and restore functionality for Kubernetes and the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, entered beta in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.21 and is now unsupported. Refer to [KEP-177: CSI Snapshot](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot#kep-177-csi-snapshot) and [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v4.1.0) for more information.
## What to do
### Dockershim removal
As stated earlier, there are several guides about
[Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/).
You can start with [Finding what container runtime are on your nodes](/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/).
If your nodes are using dockershim, there are other possible Docker Engine dependencies such as
Pods or third-party tools executing Docker commands or private registries in the Docker configuration file. You can follow the
[Check whether Dockershim deprecation affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/) guide to review possible
Docker Engine dependencies. Before upgrading to v1.24, you decide to either remain using Docker Engine and
[Migrate Docker Engine nodes from dockershim to cri-dockerd](/docs/tasks/administer-cluster/migrating-from-dockershim/migrate-dockershim-dockerd/) or migrate to a CRI-compatible runtime. Here's a guide to
[change the container runtime on a node from Docker Engine to containerd](/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/).
### `kubectl convert`
The [`kubectl convert`](/docs/tasks/tools/included/kubectl-convert-overview/) plugin for `kubectl`
can be helpful to address migrating off deprecated APIs. The plugin facilitates the conversion of
manifests between different API versions, for example, from a deprecated to a non-deprecated API
version. More general information about the API migration process can be found in the [Deprecated API Migration Guide](/docs/reference/using-api/deprecation-guide/).
Follow the [install `kubectl convert` plugin](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-kubectl-convert-plugin)
documentation to download and install the `kubectl-convert` binary.
### Looking ahead
The Kubernetes 1.25 and 1.26 releases planned for later this year will stop serving beta versions
of several currently stable Kubernetes APIs. The v1.25 release will also remove PodSecurityPolicy,
which was deprecated with Kubernetes 1.21 and will not graduate to stable. See [PodSecurityPolicy
Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) for more information.
The official [list of API removals planned for Kubernetes 1.25](/docs/reference/using-api/deprecation-guide/#v1-25) is:
* The beta CronJob API (batch/v1beta1)
* The beta EndpointSlice API (discovery.k8s.io/v1beta1)
* The beta Event API (events.k8s.io/v1beta1)
* The beta HorizontalPodAutoscaler API (autoscaling/v2beta1)
* The beta PodDisruptionBudget API (policy/v1beta1)
* The beta PodSecurityPolicy API (policy/v1beta1)
* The beta RuntimeClass API (node.k8s.io/v1beta1)
The official [list of API removals planned for Kubernetes 1.26](/docs/reference/using-api/deprecation-guide/#v1-26) is:
* The beta FlowSchema and PriorityLevelConfiguration APIs (flowcontrol.apiserver.k8s.io/v1beta1)
* The beta HorizontalPodAutoscaler API (autoscaling/v2beta2)
### Want to know more?
Deprecations are announced in the Kubernetes release notes. You can see the announcements of pending deprecations in the release notes for:
* We will formally announce the deprecations that come with [Kubernetes 1.24](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#deprecation) as part of the CHANGELOG for that release.
For information on the process of deprecation and removal, check out the official Kubernetes [deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api) document.
title: 'Increasing the security bar in Ingress-NGINX v1.2.0'
date: 2022-04-28
slug: ingress-nginx-1-2-0
---
**Authors:** Ricardo Katz (VMware), James Strong (Chainguard)
The [Ingress](/docs/concepts/services-networking/ingress/) may be one of the most targeted components
of Kubernetes. An Ingress typically defines an HTTP reverse proxy, exposed to the Internet, containing
multiple websites, and with some privileged access to Kubernetes API (such as to read Secrets relating to
TLS certificates and their private keys).
While it is a risky component in your architecture, it is still the most popular way to properly expose your services.
Ingress-NGINX has been part of security assessments that figured out we have a big problem: we don't
do all proper sanitization before turning the configuration into an `nginx.conf` file, which may lead to information
disclosure risks.
While we understand this risk and the real need to fix this, it's not an easy process to do, so we took another approach to reduce (but not remove!) this risk in the current (v1.2.0) release.
## Meet Ingress NGINX v1.2.0 and the chrooted NGINX process
One of the main challenges is that Ingress-NGINX runs the web proxy server (NGINX) alongside the Ingress
controller (the component that has access to Kubernetes API that and that creates the `nginx.conf` file).
So, NGINX does have the same access to the filesystem of the controller (and Kubernetes service account token, and other configurations from the container). While splitting those components is our end goal, the project needed a fast response; that lead us to the idea of using `chroot()`.
Let's take a look into what an Ingress-NGINX container looked like before this change:
As we can see, the same container (not the Pod, the container!) that provides HTTP Proxy is the one that watches Ingress objects and writes the Container Volume
Now, meet the new architecture:
What does all of this mean? A basic summary is: that we are isolating the NGINX service as a container inside the
controller container.
While this is not strictly true, to understand what was done here, it's good to understand how
Linux containers (and underlying mechanisms such as kernel namespaces) work.
You can read about cgroups in the Kubernetes glossary: [`cgroup`](https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-cgroup) and learn more about cgroups interact with namespaces in the NGINX project article
[What Are Namespaces and cgroups, and How Do They Work?](https://www.nginx.com/blog/what-are-namespaces-cgroups-how-do-they-work/).
(As you read that, bear in mind that Linux kernel namespaces are a different thing from
## Skip the talk, what do I need to use this new approach?
While this increases the security, we made this feature an opt-in in this release so you can have
time to make the right adjustments in your environment(s). This new feature is only available from
release v1.2.0 of the Ingress-NGINX controller.
There are two required changes in your deployments to use this feature:
* Append the suffix "-chroot" to the container image name. For example: `gcr.io/k8s-staging-ingress-nginx/controller-chroot:v1.2.0`
* In your Pod template for the Ingress controller, find where you add the capability `NET_BIND_SERVICE` and add the capability `SYS_CHROOT`. After you edit the manifest, you'll see a snippet like:
```yaml
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
- SYS_CHROOT
```
If you deploy the controller using the official Helm chart then change the following setting in
`values.yaml`:
```yaml
controller:
image:
chroot: true
```
Ingress controllers are normally set up cluster-wide (the IngressClass API is cluster scoped). If you manage the
Ingress-NGINX controller but you're not the overall cluster operator, then check with your cluster admin about
whether you can use the `SYS_CHROOT` capability, **before** you enable it in your deployment.
## OK, but how does this increase the security of my Ingress controller?
Take the following configuration snippet and imagine, for some reason it was added to your `nginx.conf`:
```
location /randomthing/ {
alias /;
autoindex on;
}
```
If you deploy this configuration, someone can call `http://website.example/randomthing` and get some listing (and access) to the whole filesystem of the Ingress controller.
Now, can you spot the difference between chrooted and non chrooted Nginx on the listings below?
| Without extra `chroot()` | With extra `chroot()` |
|----------------------------|--------|
| `bin` | `bin` |
| `dev` | `dev` |
| `etc` | `etc` |
| `home` | |
| `lib` | `lib` |
| `media` | |
| `mnt` | |
| `opt` | `opt` |
| `proc` | `proc` |
| `root` | |
| `run` | `run` |
| `sbin` | |
| `srv` | |
| `sys` | |
| `tmp` | `tmp` |
| `usr` | `usr` |
| `var` | `var` |
| `dbg` | |
| `nginx-ingress-controller` | |
| `wait-shutdown` | |
The one in left side is not chrooted. So NGINX has full access to the filesystem. The one in right side is chrooted, so a new filesystem with only the required files to make NGINX work is created.
## What about other security improvements in this release?
We know that the new `chroot()` mechanism helps address some portion of the risk, but still, someone
can try to inject commands to read, for example, the `nginx.conf` file and extract sensitive information.
So, another change in this release (this is opt-out!) is the _deep inspector_.
We know that some directives or regular expressions may be dangerous to NGINX, so the deep inspector
checks all fields from an Ingress object (during its reconciliation, and also with a
to verify if any fields contains these dangerous directives.
The ingress controller already does this for annotations, and our goal is to move this existing validation to happen inside
deep inspection as part of a future release.
You can take a look into the existing rules in [https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go](https://github.com/kubernetes/ingress-nginx/blob/main/internal/ingress/inspector/rules.go).
Due to the nature of inspecting and matching all strings within relevant Ingress objects, this new feature may consume a bit more CPU. You can disable it by running the ingress controller with the command line argument `--deep-inspect=false`.
## What's next?
This is not our final goal. Our final goal is to split the control plane and the data plane processes.
In fact, doing so will help us also achieve a [Gateway](https://gateway-api.sigs.k8s.io/) API implementation,
as we may have a different controller as soon as it "knows" what to provide to the data plane
(we need some help here!!)
Some other projects in Kubernetes already take this approach
(like [KPNG](https://github.com/kubernetes-sigs/kpng), the proposed replacement for `kube-proxy`),
and we plan to align with them and get the same experience for Ingress-NGINX.
## Further reading
If you want to take a look into how chrooting was done in Ingress NGINX, take a look
into [https://github.com/kubernetes/ingress-nginx/pull/8337](https://github.com/kubernetes/ingress-nginx/pull/8337)
The release v1.2.0 containing all the changes can be found at
title: "Frontiers, fsGroups and frogs: the Kubernetes 1.23 release interview"
date: 2022-04-29
---
**Author**: Craig Box (Google)
One of the highlights of hosting the weekly [Kubernetes Podcast from Google](https://kubernetespodcast.com/) is talking to the release managers for each new Kubernetes version. The release team is constantly refreshing. Many working their way from small documentation fixes, step up to shadow roles, and then eventually lead a release.
As we prepare for the 1.24 release next week, [in accordance with long-standing tradition](https://www.google.com/search?q=%22release+interview%22+site%3Akubernetes.io%2Fblog), I'm pleased to bring you a look back at the story of 1.23. The release was led by [Rey Lejano](https://twitter.com/reylejano), a Field Engineer at SUSE. [I spoke to Rey](https://kubernetespodcast.com/episode/167-kubernetes-1.23/) in December, as he was awaiting the birth of his first child.
Make sure you [subscribe, wherever you get your podcasts](https://kubernetespodcast.com/subscribe/), so you hear all our stories from the Cloud Native community, including the story of 1.24 next week.
*This transcript has been lightly edited and condensed for clarity.*
---
**CRAIG BOX: I'd like to start with what is, of course, on top of everyone's mind at the moment. Let's talk African clawed frogs!**
REY LEJANO: [CHUCKLES] Oh, you mean [Xenopus lavis](https://en.wikipedia.org/wiki/African_clawed_frog), the scientific name for the African clawed frog?
**CRAIG BOX: Of course.**
REY LEJANO: Not many people know, but my background and my degree is actually in microbiology, from the University of California Davis. I did some research for about four years in biochemistry, in a biochemistry lab, and I [do have a research paper published](https://www.sciencedirect.com/science/article/pii/). It's actually on glycoproteins, particularly something called "cortical granule lectin". We used frogs, because they generate lots and lots of eggs, from which we can extract the protein. That protein prevents polyspermy. When the sperm goes into the egg, the egg releases a glycoprotein, cortical granule lectin, to the membrane, and prevents any other sperm from going inside the egg.
**CRAIG BOX: Were you able to take anything from the testing that we did on frogs and generalize that to higher-order mammals, perhaps?**
REY LEJANO: Yes. Since mammals also have cortical granule lectin, we were able to analyze both the convergence and the evolutionary pattern, not just from multiple species of frogs, but also into mammals as well.
**CRAIG BOX: Now, there's a couple of different threads to unravel here. When you were young, what led you into the fields of biology, and perhaps more the technical side of it?**
REY LEJANO: I think it was mostly from family, since I do have a family history in the medical field that goes back generations. So I kind of felt like that was the natural path going into college.
**CRAIG BOX: Now, of course, you're working in a more abstract tech field. What led you out of microbiology?**
REY LEJANO: [CHUCKLES] Well, I've always been interested in tech. Taught myself a little programming when I was younger, before high school, did some web dev stuff. Just kind of got burnt out being in a lab. I was literally in the basement. I had a great opportunity to join a consultancy that specialized in [ITIL](https://www.axelos.com/certifications/itil-service-management/what-is-itil). I actually started off with application performance management, went into monitoring, went into operation management and also ITIL, which is aligning your IT asset management and service managements with business services. Did that for a good number of years, actually.
**CRAIG BOX: It's very interesting, as people describe the things that they went through and perhaps the technologies that they worked on, you can pretty much pinpoint how old they might be. There's a lot of people who come into tech these days that have never heard of ITIL. They have no idea what it is. It's basically just SRE with more process.**
REY LEJANO: Yes, absolutely. It's not very cloud native. [CHUCKLES]
**CRAIG BOX: Not at all.**
REY LEJANO: You don't really hear about it in the cloud native landscape. Definitely, you can tell someone's been in the field for a little bit, if they specialize or have worked with ITIL before.
**CRAIG BOX: You mentioned that you wanted to get out of the basement. That is quite often where people put the programmers. Did they just give you a bit of light in the new basement?**
REY LEJANO: [LAUGHS] They did give us much better lighting. Able to get some vitamin D sometimes, as well.
**CRAIG BOX: To wrap up the discussion about your previous career — over the course of the last year, with all of the things that have happened in the world, I could imagine that microbiology skills may be more in demand than perhaps they were when you studied them?**
REY LEJANO: Oh, absolutely. I could definitely see a big increase of numbers of people going into the field. Also, reading what's going on with the world currently kind of brings back all the education I've learned in the past, as well.
**CRAIG BOX: Do you keep in touch with people you went through school with?**
REY LEJANO: Just some close friends, but not in the microbiology field.
**CRAIG BOX: One thing that I think will probably happen as a result of the pandemic is a renewed interest in some of these STEM fields. It will be interesting to see what impact that has on society at large.**
REY LEJANO: Yeah. I think that'll be great.
**CRAIG BOX: You mentioned working at a consultancy doing IT management, application performance monitoring, and so on. When did Kubernetes come into your professional life?**
REY LEJANO: One of my good friends at the company I worked at, left in mid-2015. He went on to a company that was pretty heavily into Docker. He taught me a little bit. I did my first "docker run" around 2015, maybe 2016. Then, one of the applications we were using for the ITIL framework was containerized around 2018 or so, also in Kubernetes. At that time, it was pretty buggy. That was my initial introduction to Kubernetes and containerised applications.
Then I left that company, and I actually joined my friend over at [RX-M](https://rx-m.com/), which is a cloud native consultancy and training firm. They specialize in Docker and Kubernetes. I was able to get my feet wet. I got my CKD, got my CKA as well. And they were really, really great at encouraging us to learn more about Kubernetes and also to be involved in the community.
**CRAIG BOX: You will have seen, then, the life cycle of people adopting Kubernetes and containerization at large, through your own initial journey and then through helping customers. How would you characterize how that journey has changed from the early days to perhaps today?**
REY LEJANO: I think the early days, there was a lot of questions of, why do I have to containerize? Why can't I just stay with virtual machines?
**CRAIG BOX: It's a line item on your CV.**
REY LEJANO: [CHUCKLES] It is. And nowadays, I think people know the value of using containers, of orchestrating containers with Kubernetes. I don't want to say "jumping on the bandwagon", but it's become the de-facto standard to orchestrate containers.
**CRAIG BOX: It's not something that a consultancy needs to go out and pitch to customers that they should be doing. They're just taking it as, that will happen, and starting a bit further down the path, perhaps.**
REY LEJANO: Absolutely.
**CRAIG BOX: Working at a consultancy like that, how much time do you get to work on improving process, perhaps for multiple customers, and then looking at how you can upstream that work, versus paid work that you do for just an individual customer at a time?**
REY LEJANO: Back then, it would vary. They helped me introduce myself, and I learned a lot about the cloud native landscape and Kubernetes itself. They helped educate me as to how the cloud native landscape, and the tools around it, can be used together. My boss at that company, Randy, he actually encouraged us to start contributing upstream, and encouraged me to join the release team. He just said, this is a great opportunity. Definitely helped me with starting with the contributions early on.
**CRAIG BOX: Was the release team the way that you got involved with upstream Kubernetes contribution?**
REY LEJANO: Actually, no. My first contribution was with SIG Docs. I met Taylor Dolezal — he was the release team lead for 1.19, but he is involved with SIG Docs as well. I met him at KubeCon 2019, I sat at his table during a luncheon. I remember Paris Pittman was hosting this luncheon at the Marriott. Taylor says he was involved with SIG Docs. He encouraged me to join. I started joining into meetings, started doing a few drive-by PRs. That's what we call them — drive-by — little typo fixes. Then did a little bit more, started to send better or higher quality pull requests, and also reviewing PRs.
**CRAIG BOX: When did you first formally take your release team role?**
REY LEJANO: That was in [1.18](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.18/release_team.md), in December. My boss at the time encouraged me to apply. I did, was lucky enough to get accepted for the release notes shadow. Then from there, stayed in with release notes for a few cycles, then went into Docs, naturally then led Docs, then went to Enhancements, and now I'm the release lead for 1.23.
**CRAIG BOX: I don't know that a lot of people think about what goes into a good release note. What would you say does?**
REY LEJANO: [CHUCKLES] You have to tell the end user what has changed or what effect that they might see in the release notes. It doesn't have to be highly technical. It could just be a few lines, and just saying what has changed, what they have to do if they have to do anything as well.
**CRAIG BOX: As you moved through the process of shadowing, how did you learn from the people who were leading those roles?**
REY LEJANO: I said this a few times when I was the release lead for this cycle. You get out of the release team as much as you put in, or it directly aligns to how much you put in. I learned a lot. I went into the release team having that mindset of learning from the role leads, learning from the other shadows, as well. That's actually a saying that my first role lead told me. I still carry it to heart, and that was back in 1.18. That was Eddie, in the very first meeting we had, and I still carry it to heart.
**CRAIG BOX: You, of course, were [the release lead for 1.23](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.23). First of all, congratulations on the release.**
REY LEJANO: Thank you very much.
**CRAIG BOX: The theme for this release is [The Next Frontier](https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/). Tell me the story of how we came to the theme and then the logo.**
REY LEJANO: The Next Frontier represents a few things. It not only represents the next enhancements in this release, but Kubernetes itself also has a history of Star Trek references. The original codename for Kubernetes was Project Seven, a reference to Seven of Nine, originally from Star Trek Voyager. Also the seven spokes in the helm in the logo of Kubernetes as well. And, of course, Borg, the predecessor to Kubernetes.
The Next Frontier continues that Star Trek reference. It's a fusion of two titles in the Star Trek universe. One is [Star Trek V, the Final Frontier](https://en.wikipedia.org/wiki/Star_Trek_V:_The_Final_Frontier), and the Star Trek: The Next Generation.
**CRAIG BOX: Do you have any opinion on the fact that Star Trek V was an odd-numbered movie, and they are [canonically referred to as being lesser than the even-numbered ones](https://screenrant.com/star-trek-movies-odd-number-curse-explained/)?**
REY LEJANO: I can't say, because I am such a sci-fi nerd that I love all of them even though they're bad. Even the post-Next Generation movies, after the series, I still liked all of them, even though I know some weren't that great.
**CRAIG BOX: Am I right in remembering that Star Trek V was the one directed by William Shatner?**
REY LEJANO: Yes, that is correct.
**CRAIG BOX: I think that says it all.**
REY LEJANO: [CHUCKLES] Yes.
**CRAIG BOX: Now, I understand that the theme comes from a part of the [SIG Release charter](https://github.com/kubernetes/community/blob/master/sig-release/charter.md)?**
REY LEJANO: Yes. There's a line in the SIG Release charter, "ensure there is a consistent group of community members in place to support the release process across time." With the release team, we have new shadows that join every single release cycle. With this, we're growing with this community. We're growing the release team members. We're growing SIG Release. We're growing the Kubernetes community itself. For a lot of people, this is their first time contributing to open source, so that's why I say it's their new open source frontier.
**CRAIG BOX: And the logo is obviously very Star Trek-inspired. It sort of surprised me that it took that long for someone to go this route.**
REY LEJANO: I was very surprised as well. I had to relearn Adobe Illustrator to create the logo.
**CRAIG BOX: This your own work, is it?**
REY LEJANO: This is my own work.
**CRAIG BOX: It's very nice.**
REY LEJANO: Thank you very much. Funny, the galaxy actually took me the longest time versus the ship. Took me a few days to get that correct. I'm always fine-tuning it, so there might be a final change when this is actually released.
**CRAIG BOX: No frontier is ever truly final.**
REY LEJANO: True, very true.
**CRAIG BOX: Moving now from the theme of the release to the substance, perhaps, what is new in 1.23?**
REY LEJANO: We have 47 enhancements. I'm going to run through most of the stable ones, if not all of them, some of the key Beta ones, and a few of the Alpha enhancements for 1.23.
One of the key enhancements is [dual-stack IPv4/IPv6](https://github.com/kubernetes/enhancements/issues/563), which went GA in 1.23.
Some background info: dual-stack was introduced as Alpha in 1.15. You probably saw a keynote at KubeCon 2019. Back then, the way dual-stack worked was that you needed two services — you needed a service per IP family. You would need a service for IPv4 and a service for IPv6. It was refactored in 1.20. In 1.21, it was in Beta; clusters were enabled to be dual-stack by default.
And then in 1.23 we did remove the IPv6 dual-stack feature flag. It's not mandatory to use dual-stack. It's actually not "default" still. The pods, the services still default to single-stack. There are some requirements to be able to use dual-stack. The nodes have to be routable on IPv4 and IPv6 network interfaces. You need a CNI plugin that supports dual-stack. The pods themselves have to be configured to be dual-stack. And the services need the ipFamilyPolicy field to specify prefer dual-stack, or require dual-stack.
**CRAIG BOX: This sounds like there's an implication in this that v4 is still required. Do you see a world where we can actually move to v6-only clusters?**
REY LEJANO: I think we'll be talking about IPv4 and IPv6 for many, many years to come. I remember a long time ago, they kept saying "it's going to be all IPv6", and that was decades ago.
**CRAIG BOX: I think I may have mentioned on the show before, but there was [a meeting in London that Vint Cerf attended](https://www.youtube.com/watch?v=AEaJtZVimqs), and he gave a public presentation at the time to say, now is the time of v6. And that was 10 years ago at least. It's still not the time of v6, and my desktop still doesn't have Linux on it. One day.**
REY LEJANO: [LAUGHS] In my opinion, that's one of the big key features that went stable for 1.23.
One of the other highlights of 1.23 is [pod security admission going to Beta](/blog/2021/12/09/pod-security-admission-beta/). I know this feature is going to Beta, but I highlight this because as some people might know, PodSecurityPolicy, which was deprecated in 1.21, is targeted to be removed in 1.25. Pod security admission replaces pod security policy. It's an admission controller. It evaluates the pods against a predefined set of pod security standards to either admit or deny the pod for running.
There's three levels of pod security standards. Privileged, that's totally open. Baseline, known privileges escalations are minimized. Or Restricted, which is hardened. And you could set pod security standards either to run in three modes, which is enforce: reject any pods that are in violation; to audit: pods are allowed to be created, but the violations are recorded; or warn: it will send a warning message to the user, and the pod is allowed.
**CRAIG BOX: You mentioned there that PodSecurityPolicy is due to be deprecated in two releases' time. Are we lining up these features so that pod security admission will be GA at that time?**
REY LEJANO: Yes. Absolutely. I'll talk about that for another feature in a little bit as well. There's also another feature that went to GA. It was an API that went to GA, and therefore the Beta API is now deprecated. I'll talk about that a little bit.
**CRAIG BOX: All right. Let's talk about what's next on the list.**
REY LEJANO: Let's move on to more stable enhancements. One is the [TTL controller](https://github.com/kubernetes/enhancements/issues/592). This cleans up jobs and pods after the jobs are finished. There is a TTL timer that starts when the job or pod is finished. This TTL controller watches all the jobs, and ttlSecondsAfterFinished needs to be set. The controller will see if the ttlSecondsAfterFinished, combined with the last transition time, if it's greater than now. If it is, then it will delete the job and the pods of that job.
**CRAIG BOX: Loosely, it could be called a garbage collector?**
REY LEJANO: Yes. Garbage collector for pods and jobs, or jobs and pods.
**CRAIG BOX: If Kubernetes is truly becoming a programming language, it of course has to have a garbage collector implemented.**
REY LEJANO: Yeah. There's another one, too, coming in Alpha. [CHUCKLES]
**CRAIG BOX: Tell me about that.**
REY LEJANO: That one is coming in in Alpha. It's actually one of my favorite features, because there's only a few that I'm going to highlight today. [PVCs for StafeulSet will be cleaned up](https://github.com/kubernetes/enhancements/issues/1847). It will auto-delete PVCs created by StatefulSets, when you delete that StatefulSet.
**CRAIG BOX: What's next on our tour of stable features?**
REY LEJANO: Next one is, [skip volume ownership change goes to stable](https://github.com/kubernetes/enhancements/issues/695). This is from SIG Storage. There are times when you're running a stateful application, like many databases, they're sensitive to permission bits changing underneath. Currently, when a volume is bind mounted inside the container, the permissions of that volume will change recursively. It might take a really long time.
Now, there's a field, the fsGroupChangePolicy, which allows you, as a user, to tell Kubernetes how you want the permission and ownership change for that volume to happen. You can set it to always, to always change permissions, or just on mismatch, to only do it when the permission ownership changes at the top level is different from what is expected.
**CRAIG BOX: It does feel like a lot of these enhancements came from a very particular use case where someone said, "hey, this didn't work for me and I've plumbed in a feature that works with exactly the thing I need to have".**
REY LEJANO: Absolutely. People create issues for these, then create Kubernetes enhancement proposals, and then get targeted for releases.
**CRAIG BOX: Another GA feature in this release — ephemeral volumes.**
REY LEJANO: We've always been able to use empty dir for ephemeral volumes, but now we could actually have [ephemeral inline volumes](https://github.com/kubernetes/enhancements/issues/1698), meaning that you could take your standard CSI driver and be able to use ephemeral volumes with it.
**CRAIG BOX: And, a long time coming, [CronJobs](https://github.com/kubernetes/enhancements/issues/19).**
REY LEJANO: CronJobs is a funny one, because it was stable before 1.23. For 1.23, it was still tracked,but it was just cleaning up some of the old controller. With CronJobs, there's a v2 controller. What was cleaned up in 1.23 is just the old v1 controller.
**CRAIG BOX: Were there any other duplications or major cleanups of note in this release?**
REY LEJANO: Yeah. There were a few you might see in the major themes. One's a little tricky, around FlexVolumes. This is one of the efforts from SIG Storage. They have an effort to migrate in-tree plugins to CSI drivers. This is a little tricky, because FlexVolumes were actually deprecated in November 2020. We're [formally announcing it in 1.23](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#kubernetes-volume-plugin-faq-for-storage-vendors).
**CRAIG BOX: FlexVolumes, in my mind, predate CSI as a concept. So it's about time to get rid of them.**
REY LEJANO: Yes, it is. There's another deprecation, just some [klog specific flags](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog), but other than that, there are no other big deprecations in 1.23.
**CRAIG BOX: The buzzword of the last KubeCon, and in some ways the theme of the last 12 months, has been secure software supply chain. What work is Kubernetes doing to improve in this area?**
REY LEJANO: For 1.23, Kubernetes is now SLSA compliant at Level 1, which means that provenance attestation files that describe the staging and release phases of the release process are satisfactory for the SLSA framework.
**CRAIG BOX: What needs to happen to step up to further levels?**
REY LEJANO: Level 1 means a few things — that the build is scripted; that the provenance is available, meaning that the artifacts are verified and they're handed over from one phase to the next; and describes how the artifact is produced. Level 2 means that the source is version-controlled, which it is, provenance is authenticated, provenance is service-generated, and there is a build service. There are four levels of SLSA compliance.
**CRAIG BOX: It does seem like the levels were largely influenced by what it takes to build a big, secure project like this. It doesn't seem like it will take a lot of extra work to move up to verifiable provenance, for example. There's probably just a few lines of script required to meet many of those requirements.**
REY LEJANO: Absolutely. I feel like we're almost there; we'll see what will come out of 1.24. And I do want to give a big shout-out to SIG Release and Release Engineering, primarily to Adolfo García Veytia, who is aka Puerco on GitHub and on Slack. He's been driving this forward.
**CRAIG BOX: You've mentioned some APIs that are being graduated in time to replace their deprecated version. Tell me about the new HPA API.**
REY LEJANO: The [horizontal pod autoscaler v2 API](https://github.com/kubernetes/enhancements/issues/2702), is now stable, which means that the v2beta2 API is deprecated. Just for everyone's knowledge, the v1 API is not being deprecated. The difference is that v2 adds support for multiple and custom metrics to be used for HPA.
**CRAIG BOX: There's also now a facility to validate my CRDs with an expression language.**
REY LEJANO: Yeah. You can use the [Common Expression Language, or CEL](https://github.com/google/cel-spec), to validate your CRDs, so you no longer need to use webhooks. This also makes the CRDs more self-contained and declarative, because the rules are now kept within the CRD object definition.
**CRAIG BOX: What new features, perhaps coming in Alpha or Beta, have taken your interest?**
REY LEJANO: Aside from pod security policies, I really love [ephemeral containers](https://github.com/kubernetes/enhancements/issues/277) supporting kubectl debug. It launches an ephemeral container and a running pod, shares those pod namespaces, and you can do all your troubleshooting with just running kubectl debug.
**CRAIG BOX: There's also been some interesting changes in the way that events are handled with kubectl.**
REY LEJANO: Yeah. kubectl events has always had some issues, like how things weren't sorted. [kubectl events improved](https://github.com/kubernetes/enhancements/issues/1440) that so now you can do `--watch`, and it will also sort with the `--watch` option as well. That is something new. You can actually combine fields and custom columns. And also, you can list events in the timeline with doing the last N number of minutes. And you can also sort events using other criteria as well.
**CRAIG BOX: You are a field engineer at SUSE. Are there any things that are coming in that your individual customers that you deal with are looking out for?**
REY LEJANO: More of what I look out for to help the customers.
**CRAIG BOX: Right.**
REY LEJANO: I really love kubectl events. Really love the PVCs being cleaned up with StatefulSets. Most of it's for selfish reasons that it will improve troubleshooting efforts. [CHUCKLES]
**CRAIG BOX: I have always hoped that a release team lead would say to me, "yes, I have selfish reasons. And I finally got something I wanted in."**
REY LEJANO: [LAUGHS]
**CRAIG BOX: Perhaps I should run to be release team lead, just so I can finally get init containers fixed once and for all.**
REY LEJANO: Oh, init containers, I've been looking for that for a while. I've actually created animated GIFs on how init containers will be run with that Kubernetes enhancement proposal, but it's halted currently.
**CRAIG BOX: One day.**
REY LEJANO: One day. Maybe I shouldn't stay halted.
**CRAIG BOX: You mentioned there are obviously the things you look out for. Are there any things that are coming down the line, perhaps Alpha features or maybe even just proposals you've seen lately, that you're personally really looking forward to seeing which way they go?**
REY LEJANO: Yeah. Oone is a very interesting one, it affects the whole community, so it's not just for personal reasons. As you may have known, Dockershim is deprecated. And we did release a blog that it will be removed in 1.24.
**CRAIG BOX: Scared a bunch of people.**
REY LEJANO: Scared a bunch of people. From a survey, we saw that a lot of people are still using Docker and Dockershim. One of the enhancements for 1.23 is, [kubelet CRI goes to Beta](https://github.com/kubernetes/enhancements/issues/2040). This promotes the CRI API, which is required. This had to be in Beta for Dockershim to be removed in 1.24.
**CRAIG BOX: Now, in the last release team lead interview, [we spoke with Savitha Raghunathan](https://kubernetespodcast.com/episode/157-kubernetes-1.22/), and she talked about what she would advise you as her successor. It was to look out for the mental health of the team members. How were you able to take that advice on board?**
REY LEJANO: That was great advice from Savitha. A few things I've made note of with each release team meeting. After each release team meeting, I stop the recording, because we do record all the meetings and post them on YouTube. And I open up the floor to anyone who wants to say anything that's not recorded, that's not going to be on the agenda. Also, I tell people not to work on weekends. I broke this rule once, but other than that, I told people it could wait. Just be mindful of your mental health.
**CRAIG BOX: It's just been announced that [James Laverack from Jetstack](https://twitter.com/JamesLaverack/status/1466834312993644551) will be the release team lead for 1.24. James and I shared an interesting Mexican dinner at the last KubeCon in San Diego.**
REY LEJANO: Oh, nice. I didn't know you knew James.
**CRAIG BOX: The British tech scene. We're a very small world. What will your advice to James be?**
REY LEJANO: What I would tell James for 1.24 is use teachable moments in the release team meetings. When you're a shadow for the first time, it's very daunting. It's very difficult, because you don't know the repos. You don't know the release process. Everyone around you seems like they know the release process, and very familiar with what the release process is. But as a first-time shadow, you don't know all the vernacular for the community. I just advise to use teachable moments. Take a few minutes in the release team meetings to make it a little easier for new shadows to ramp up and to be familiar with the release process.
**CRAIG BOX: Has there been major evolution in the process in the time that you've been involved? Or do you think that it's effectively doing what it needs to do?**
REY LEJANO: It's always evolving. I remember my first time in release notes, 1.18, we said that our goal was to automate and program our way out so that we don't have a release notes team anymore. That's changed [CHUCKLES] quite a bit. Although there's been significant advancements in the release notes process by Adolfo and also James, they've created a subcommand in krel to generate release notes.
But nowadays, all their release notes are richer. Still not there at the automation process yet. Every release cycle, there is something a little bit different. For this release cycle, we had a production readiness review deadline. It was a soft deadline. A production readiness review is a review by several people in the community. It's actually been required since 1.21, and it ensures that the enhancements are observable, scalable, supportable, and it's safe to operate in production, and could also be disabled or rolled back. In 1.23, we had a deadline to have the production readiness review completed by a specific date.
**CRAIG BOX: How have you found the change of schedule to three releases per year rather than four?**
REY LEJANO: Moving to three releases a year from four, in my opinion, has been an improvement, because we support the last three releases, and now we can actually support the last releases in a calendar year instead of having 9 months out of 12 months of the year.
**CRAIG BOX: The next event on the calendar is a [Kubernetes contributor celebration](https://www.kubernetes.dev/events/kcc2021/) starting next Monday. What can we expect from that event?**
REY LEJANO: This is our second time running this virtual event. It's a virtual celebration to recognize the whole community and all of our accomplishments of the year, and also contributors. There's a number of events during this week of celebration. It starts the week of December 13.
There's events like the Kubernetes Contributor Awards, where SIGs honor and recognize the hard work of the community and contributors. There's also a DevOps party game as well. There is a cloud native bake-off. I do highly suggest people to go to [kubernetes.dev/celebration](https://www.kubernetes.dev/events/past-events/2021/kcc2021/) to learn more.
**CRAIG BOX: How exactly does one judge a virtual bake-off?**
REY LEJANO: That I don't know. [CHUCKLES]
**CRAIG BOX: I tasted my scones. I think they're the best. I rate them 10 out of 10.**
REY LEJANO: Yeah. That is very difficult to do virtually. I would have to say, probably what the dish is, how closely it is tied with Kubernetes or open source or to CNCF. There's a few judges. I know Josh Berkus and Rin Oliver are a few of the judges running the bake-off.
**CRAIG BOX: Yes. We spoke with Josh about his love of the kitchen, and so he seems like a perfect fit for that role.**
REY LEJANO: He is.
**CRAIG BOX: Finally, your wife and yourself are expecting your first child in January. Have you had a production readiness review for that?**
REY LEJANO: I think we failed that review. [CHUCKLES]
**CRAIG BOX: There's still time.**
REY LEJANO: We are working on refactoring. We're going to refactor a little bit in December, and `--apply` again.
---
_[Rey Lejano](https://twitter.com/reylejano) is a field engineer at SUSE, by way of Rancher Labs, and was the release team lead for Kubernetes 1.23. He is now also a co-chair for SIG Docs. His son Liam is now 3 and a half months old._
_You can find the [Kubernetes Podcast from Google](http://www.kubernetespodcast.com/) at [@KubernetesPod](https://twitter.com/KubernetesPod) on Twitter, and you can [subscribe](https://kubernetespodcast.com/subscribe/) so you never miss an episode._
Dockershim has been removed as of Kubernetes v1.24, and this is a positive move for the project. However, context is important for fully understanding something, be it socially or in software development, and this deserves a more in-depth review. Alongside the dockershim removal in Kubernetes v1.24, we’ve seen some confusion (sometimes at a panic level) and dissatisfaction with this decision in the community, largely due to a lack of context around this removal. The decision to deprecate and eventually remove dockershim from Kubernetes was not made quickly or lightly. Still, it’s been in the works for so long that many of today’s users are newer than that decision, and certainly newer than the choices that led to the dockershim being necessary in the first place.
So what is the dockershim, and why is it going away?
In the early days of Kubernetes, we only supported one container runtime. That runtime was Docker Engine. Back then, there weren’t really a lot of other options out there and Docker was the dominant tool for working with containers, so this was not a controversial choice. Eventually, we started adding more container runtimes, like rkt and hypernetes, and it became clear that Kubernetes users want a choice of runtimes working best for them. So Kubernetes needed a way to allow cluster operators the flexibility to use whatever runtime they choose.
The [Container Runtime Interface](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) (CRI) was released to allow that flexibility. The introduction of CRI was great for the project and users alike, but it did introduce a problem: Docker Engine’s use as a container runtime predates CRI, and Docker Engine is not CRI-compatible. To solve this issue, a small software shim (dockershim) was introduced as part of the kubelet component specifically to fill in the gaps between Docker Engine and CRI, allowing cluster operators to continue using Docker Engine as their container runtime largely uninterrupted.
However, this little software shim was never intended to be a permanent solution. Over the course of years, its existence has introduced a lot of unnecessary complexity to the kubelet itself. Some integrations are inconsistently implemented for Docker because of this shim, resulting in an increased burden on maintainers, and maintaining vendor-specific code is not in line with our open source philosophy. To reduce this maintenance burden and move towards a more collaborative community in support of open standards, [KEP-2221 was introduced](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim), proposing the removal of the dockershim. With the release of Kubernetes v1.20, the deprecation was official.
We didn’t do a great job communicating this, and unfortunately, the deprecation announcement led to some panic within the community. Confusion around what this meant for Docker as a company, if container images built by Docker would still run, and what Docker Engine actually is led to a conflagration on social media. This was our fault; we should have more clearly communicated what was happening and why at the time. To combat this, we released [a blog](/blog/2020/12/02/dont-panic-kubernetes-and-docker/) and [accompanying FAQ](/blog/2020/12/02/dockershim-faq/) to allay the community’s fears and correct some misconceptions about what Docker is and how containers work within Kubernetes. As a result of the community’s concerns, Docker and Mirantis jointly agreed to continue supporting the dockershim code in the form of [cri-dockerd](https://www.mirantis.com/blog/the-future-of-dockershim-is-cri-dockerd/), allowing you to continue using Docker Engine as your container runtime if need be. For the interest of users who want to try other runtimes, like containerd or cri-o, [migration documentation was written](/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/).
We later [surveyed the community](https://kubernetes.io/blog/2021/11/12/are-you-ready-for-dockershim-removal/) and [discovered that there are still many users with questions and concerns](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim). In response, Kubernetes maintainers and the CNCF committed to addressing these concerns by extending documentation and other programs. In fact, this blog post is a part of this program. With so many end users successfully migrated to other runtimes, and improved documentation, we believe that everyone has a paved way to migration now.
Docker is not going away, either as a tool or as a company. It’s an important part of the cloud native community and the history of the Kubernetes project. We wouldn’t be where we are without them. That said, removing dockershim from kubelet is ultimately good for the community, the ecosystem, the project, and open source at large. This is an opportunity for all of us to come together to support open standards, and we’re glad to be doing so with the help of Docker and the community.
and to only supply a small controller wrapper around the library, and a pod image
capable of attaching to volumes and writing the appropriate data to the volume.
Individual populators can be extremely generic such that they work with every type
of PVC, or they can do vendor specific things to rapidly fill a volume with data
if the volume was provisioned by a specific CSI driver from the same vendor, for
example, by communicating directly with the storage for that volume.
## How can I learn more?
The enhancement proposal,
[Volume Populators](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators), includes lots of detail about the history and technical implementation
of this feature.
[Volume populators and data sources](/docs/concepts/storage/persistent-volumes/#volume-populators-and-data-sources), within the documentation topic about persistent volumes,
explains how to use this feature in your cluster.
Please get involved by joining the Kubernetes storage SIG to help us enhance this
feature. There are a lot of good ideas already and we'd be thrilled to have more!
<p>In 2016, the company began moving their code from Heroku to<ahref="https://www.docker.com/">Docker</a> containers running on <ahref="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <ahref="http://kubernetes.io/">Kubernetes</a> and monitored with <ahref="https://prometheus.io/">Prometheus</a>.</p>
<p>In 2016, the company began moving their code from Heroku to containers running on <ahref="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <ahref="http://kubernetes.io/">Kubernetes</a> and monitored with <ahref="https://prometheus.io/">Prometheus</a>.</p>
<h2>Impact</h2>
@@ -42,7 +42,7 @@ With the speed befitting a startup, Pear Deck delivered its first prototype to c
<p>On top of that, many of Pear Deck's customers are behind government firewalls and connect through Firebase, not Pear Deck's servers, making troubleshooting even more difficult.</p>
<p>The team began looking around for another solution, and finally decided in early 2016 to start moving the app from Heroku to<ahref="https://www.docker.com/">Docker</a> containers running on <ahref="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <ahref="http://kubernetes.io/">Kubernetes</a> and monitored with <ahref="https://prometheus.io/">Prometheus</a>.</p>
<p>The team began looking around for another solution, and finally decided in early 2016 to start moving the app from Heroku to containers running on <ahref="https://cloud.google.com/kubernetes-engine/">Google Kubernetes Engine</a>, orchestrated by <ahref="http://kubernetes.io/">Kubernetes</a> and monitored with <ahref="https://prometheus.io/">Prometheus</a>.</p>
"When it became clear that Google Kubernetes Engine was going to have a lot of support from Google and be a fully-managed Kubernetes platform, it seemed very obvious to us that was the way to go," says Eynon-Lynch.
@@ -70,7 +70,7 @@ If you haven't set foot in a school in awhile, you might be surprised by what yo
<p>Recently, the team launched a new single sign-on solution for use in an internal application. "Due to the resource based architecture of the Kubernetes platform, we were able to bring that application into an entirely new production environment in less than a day, most of that time used for testing after applying the already well-known resource definitions from staging to the new environment," says van den Bosch. "On a traditional VM this would have likely cost a day or two, and then probably a few weeks to iron out the kinks in our provisioning scripts as we apply updates."</p>
<p>Legacy applications are also being moved to Kubernetes. Not long ago, the team needed to set up a Java-based application for compiling and running a frontend. "On a traditional VM, it would have taken quite a bit of time to set it up and keep it up to date, not to mention maintenance for that setup down the line," says van den Bosch. Instead, it took less than half a day to Dockerize it and get it running on Kubernetes. "It was much easier, and we were able to save costs too because we didn't have to spin up new VMs specially for it."</p>
<p>Legacy applications are also being moved to Kubernetes. Not long ago, the team needed to set up a Java-based application for compiling and running a frontend. "On a traditional VM, it would have taken quite a bit of time to set it up and keep it up to date, not to mention maintenance for that setup down the line," says van den Bosch. Instead, it took less than half a day to containerize it and get it running on Kubernetes. "It was much easier, and we were able to save costs too because we didn't have to spin up new VMs specially for it."</p>
{{<case-studies/quoteauthor="VICTOR VAN DEN BOSCH, SENIOR DEVOPS ENGINEER, PROWISE">}}
"We're really trying to deliver integrated solutions with our hardware and software and making it as easy as possible for users to use and collaborate from different places," says van den Bosch. And, says Haalstra, "We cannot do it without Kubernetes."
@@ -46,7 +46,7 @@ Since it was started in a dorm room in 2003, Squarespace has made it simple for
After experimenting with another container orchestration platform and "breaking it in very painful ways," Lynch says, the team began experimenting with Kubernetes in mid-2016 and found that it "answered all the questions that we had."
{{< /case-studies/quote >}}
<p>Within a couple months, they had a stable cluster for their internal use, and began rolling out Kubernetes for production. They also added Zipkin and CNCF projects <ahref="https://prometheus.io/">Prometheus</a> and <ahref="https://www.fluentd.org/">fluentd</a> to their cloud native stack. "We switched to Kubernetes, a new world, and we revamped all our other tooling as well," says Lynch. "It allowed us to streamline our process, so we can now easily create an entire microservice project from templates, generate the code and deployment pipeline for that, generate the Dockerfile, and then immediately just ship a workable, deployable project to Kubernetes." Deployments across Dev/QA/Stage/Prod were also "simplified drastically," Lynch adds. "Now there is little configuration variation."</p>
<p>Within a couple months, they had a stable cluster for their internal use, and began rolling out Kubernetes for production. They also added Zipkin and CNCF projects <ahref="https://prometheus.io/">Prometheus</a> and <ahref="https://www.fluentd.org/">fluentd</a> to their cloud native stack. "We switched to Kubernetes, a new world, and we revamped all our other tooling as well," says Lynch. "It allowed us to streamline our process, so we can now easily create an entire microservice project from templates, generate the code and deployment pipeline for that, generate the Dockerfile, and then immediately just ship a workable, deployable project to Kubernetes." Deployments across Dev/QA/Stage/Prod were also "simplified drastically," Lynch adds. "Now there is little configuration variation."</p>
<p>And the whole process takes only five minutes, an almost 85% reduction in time compared to their VM deployment. "From end to end that probably took half an hour, and that's not accounting for the fact that an infrastructure engineer would be responsible for doing that, so there's some business delay in there as well."</p>
@@ -58,9 +58,9 @@ How many people does it take to turn on a light bulb?
<p>In addition, Wink had other requirements: horizontal scalability, the ability to encrypt everything quickly, connections that could be easily brought back up if something went wrong. "Looking at this whole structure we started, we decided to make a secure socket-based service," says Klein. "We've always used, I would say, some sort of clustering technology to deploy our services and so the decision we came to was, this thing is going to be containerized, running on Docker."</p>
<p>At the time – just over two years ago – Docker wasn't yet widely used, but as Klein points out, "it was certainly understood by the people who were on the frontier of technology. We started looking at potential technologies that existed. One of the limiting factors was that we needed to deploy multi-port non-http/https services. It wasn't really appropriate for some of the early cluster technology. We liked the project a lot and we ended up using it on other stuff for a while, but initially it was too targeted toward http workloads."</p>
<p>In 2015, Docker wasn't yet widely used, but as Klein points out, "it was certainly understood by the people who were on the frontier of technology. We started looking at potential technologies that existed. One of the limiting factors was that we needed to deploy multi-port non-http/https services. It wasn't really appropriate for some of the early cluster technology. We liked the project a lot and we ended up using it on other stuff for a while, but initially it was too targeted toward http workloads."</p>
<p>Once Wink's backend engineering team decided on a Dockerized workload, they had to make decisions about the OS and the container orchestration platform. "Obviously you can't just start the containers and hope everything goes well," Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure."</p>
<p>Once Wink's backend engineering team decided on a containerized workload, they had to make decisions about the OS and the container orchestration platform. "Obviously you can't just start the containers and hope everything goes well," Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure."</p>
"Obviously you can't just start the containers and hope everything goes well," Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure."
@@ -68,7 +68,7 @@ How many people does it take to turn on a light bulb?
<p>Wink considered building directly on a general purpose Linux distro like Ubuntu (which would have required installing tools to run a containerized workload) and cluster management systems like Mesos (which was targeted toward enterprises with larger teams/workloads), but ultimately set their sights on CoreOS Container Linux. "A container-optimized Linux distribution system was exactly what we needed," he says. "We didn't have to futz around with trying to take something like a Linux distro and install everything. It's got a built-in container orchestration system, which is Fleet, and an easy-to-use API. It's not as feature-rich as some of the heavier solutions, but we realized that, at that moment, it was exactly what we needed."</p>
<p>Wink's hub (along with a revamped app) was introduced in July 2014 with a short-term deployment, and within the first month, they had moved the service to the Dockerized CoreOS deployment. Since then, they've moved almost every other piece of their infrastructure – from third-party cloud-to-cloud integrations to their customer service and payment portals – onto CoreOS Container Linux clusters.</p>
<p>Wink's hub (along with a revamped app) was introduced in July 2014 with a short-term deployment, and within the first month, they had moved the service to the containerized CoreOS deployment. Since then, they've moved almost every other piece of their infrastructure – from third-party cloud-to-cloud integrations to their customer service and payment portals – onto CoreOS Container Linux clusters.</p>
<p>Using this setup did require some customization. "Fleet is really nice as a basic container orchestration system, but it doesn't take care of routing, sharing configurations, secrets, et cetera, among instances of a service," Klein says. "All of those layers of functionality can be implemented, of course, but if you don't want to spend a lot of time writing unit files manually – which of course nobody does – you need to create a tool to automate some of that, which we did."</p>
<p>The Kubernetes community -- users, contributors, and the culture we've built together -- is one of the biggest reasons for the meteoric rise of this open source project. Our culture and values continue to grow and change as the project itself grows and changes. We all work together toward constant improvement of the project and the ways we work on it.
<br><br>We are the people who file issues and pull requests, attend SIG meetings, Kubernetes meetups, and KubeCon, advocate for its adoption and innovation, run <code>kubectl get pods</code>, and contribute in a thousand other vital ways. Read on to learn how you can get involved and become part of this amazing community.</p>
The Kubernetes Community values are the keystone to the ongoing success of the project.<br>
These principles guide every aspect of the Kubernetes project.
<br>
<ahref="/community/values/">
<brclass="mobile"><brclass="mobile">
<spanclass="fullbutton">
READ MORE
</span>
</a>
</div><aname="conduct"></a>
</div>
</div>
<divclass="conduct">
<divclass="conducttext">
<brclass="mobile"><brclass="mobile">
<brclass="tablet"><brclass="tablet">
<divclass="conducttextnobutton"style="margin-bottom:2%"><h1>Code of Conduct</h1>
The Kubernetes community values respect and inclusiveness, and enforces a Code of Conduct in all interactions. If you notice a violation of the Code of Conduct at an event or meeting, in Slack, or in another communication mechanism, reach out to the Kubernetes Code of Conduct Committee at <ahref="mailto:conduct@kubernetes.io"style="color:#0662EE;font-weight:300">conduct@kubernetes.io</a>. All reports are kept confidential. You can read about the committee <ahref="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct"style="color:#0662EE;font-weight:300">here</a>.
<divstyle="margin-bottom:4%;font-weight:300;text-align:center;padding-left:10%;padding-right:10%">We're on YouTube, a lot. Subscribe for a wide range of topics.</div>
<iframewidth="100%"height="250"src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ"title="Weekly community meetings"frameborder="0"allow="autoplay; encrypted-media"allowfullscreen></iframe>
<iframewidth="100%"height="250"src="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F"title="Talk from a community member"frameborder="0"allow="autoplay; encrypted-media"allowfullscreen></iframe>
<divclass="resourceboxtext"style="font-size:12px;text-transform:none !important;font-weight:300;line-height:1.4em;color:#333333;margin-top:4%">Real-time announcements of blog posts, events, news, ideas
<aclass="twitter-timeline"data-tweet-limit="1"href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">Tweets by kubernetesio</a><scriptasyncsrc="https://platform.twitter.com/widgets.js"charset="utf-8"></script>
<p>The Kubernetes community values respect and inclusiveness, and enforces a Code of Conduct in all interactions.</p>
<p>If you notice a violation of the Code of Conduct at an event or meeting, in <ahref="#slack">Slack</a>, or in another communication mechanism, reach out to the Kubernetes Code of Conduct Committee at <ahref="mailto:conduct@kubernetes.io">conduct@kubernetes.io</a>. All reports are kept confidential. You can read <ahref="https://github.com/kubernetes/community/tree/master/committee-code-of-conduct">about the committee</a> in the Kubernetes community repository on GitHub.</p>
<spanclass="videocta">Watch weekly community meetings ▶</span>
</a>
</div>
<divclass="video youtube"id="discuss">
<iframesrc="https://www.youtube.com/embed/videoseries?list=PL69nYSiGNLP3QpQrhZq_sLYo77BVKv09F"title="Talk from a community member"allow="camera 'none'; microphone 'none'; geolocation 'none'; fullscreen https://www.youtube.com/"></iframe>
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Kubernetes Prow Robot
