As an alpha feature, rolling updates for StatefulSets can now have a
.spec.minReadySeconds field. Improve the documentation for StatefulSets
to better explain how this is useful.
The HTML5 named character reference such as '>', '<' should be terminated
with semicolon character. This commit fixes the missing termination character
with the kubernetes-basics tutorials.
Compared with the other distro guide about installation kubectl, the macOS installation guide has a typo about validation of kubectl-convert.
This commits fixes the typo on "Install kubectl on macOS" documentation.
This change adds a deprecation notice for the in-tree quobyte storage class and offers a link to the out-of-tree example.
Signed-off-by: Victor Palade <victor@cloudflavor.io>
* Rebasing HostProcess security changes.
* Incorporated initial round of feedback
* Minor wording updates
* Finished up remaining todo items
* Apply suggestions from code review
Co-authored-by: Jordan Liggitt <jordan@liggitt.net>
Co-authored-by: Mark Rossetti <marosset@microsoft.com>
* Moved HostProcess security documentation into PSS and create-host-process-pod docs
* Updated with for James' review
* Apply suggestions from code review
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: James Sturtevant <jsturtevant@gmail.com>
* Minor edits
* Modifications for additional feedback
Co-authored-by: Jordan Liggitt <jordan@liggitt.net>
Co-authored-by: Mark Rossetti <marosset@microsoft.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: James Sturtevant <jsturtevant@gmail.com>
* Clarifying gcr vs mcr pause iamge usage for Windows
Signed-off-by: Mark Rossetti <marosset@microsoft.com>
* updating more pause image references
* Link to pause image section instead of duplicating container image multiple places
* Update content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md
Co-authored-by: James Sturtevant <jsturtevant@gmail.com>
* update link to wincat source
* Removing here link
* end of file blank line
* specify pause image v3.5 is for v1.22
* Update content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* removing v1.15 reference from docs
* Update content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md
Co-authored-by: James Sturtevant <jsturtevant@gmail.com>
* Update content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md
Co-authored-by: James Sturtevant <jsturtevant@gmail.com>
Co-authored-by: James Sturtevant <jsturtevant@gmail.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Here is a second batch for feature gate updates in 1.22.
- CPUManagerPolicyOptions kubernetes/kubernetes#101432
- ControllerManagerLeaderMigration kubernetes/kubernetes#103533
- DynamicKubeletConfig kubernetes/kubernetes#102966
- EndpointSliceProxying kubernetes/kubernetes#103451
- EndpointSliceTerminatingCondition kubernetes/kubernetes#103596
- HugePageStorageMediumSize kubernetes/kubernetes#99144
- JobTrackingWithFinalizers kubernetes/kubernetes#98817
(also tracked in #28841, can rebase).
- ServiceInternalTrafficPolicy kubernetes/kubernetes#103462
- WindowsEndpointSliceProxying kubernetes/kubernetes#103451
Some of these needs more detailed documentation.
Supports multiple architectures for the "Without a Package Manager" step such as arm64. Useful in Arch + ARM's scenario as kubeadm pacman repo is currently not maintained for ARM.
The term "control plane" is written as title case (e.g. Control Plane) only in this document. When it is referenced, it creates inconsistency in other documents. This commit will fix it by lower-casing the second word.
the english version is :"no pods without the toleration will schedule on them"，the Chinese translation in kubernetes documents is ：“没有对应容忍度的 Pod会被调度到这些节点”，Here the translation omits the keyword "no"，it should be translated into Chinese：“没有对应容忍度的 Pod不会被调度到这些节点”。
The `v1beta3` version is a 1.22 thing, not 1.21 (main). This PR replaces the `v1beta3` version by `v1beta2`. A follow up PR will add the `v1beta3` to the `dev-1.22` branch.
Following the "if you do not have root access" installation docs for kubectl results in a "permission denied" error, so this change adds the appropriate chmod command to minimise confusion.
Since 1.22 it will be possible to skip phases using the configuration
file. Add note about that in the relevant sections of the kubeadm
init and join reference pages.
- Re-purpose the page to include more general details about
customizing components.
- Add details about using patches via the config API (v1.22 feature).
This adds a documentation section about the `SeccompDefault` feature
gate for the kubelet. Beside that, the feature gate has been added to
the list of features, too.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
The capture shortcode is no longer used: the switch was made on
2020-06-15, in commit 0625ced466.
Remove it. After this change, it won't be possible to use this legacy
shortcode (builds would fail).
We now add the information about the regression introduced in the latest
patches. The final 1.18 release is now part of the documentation, too.
Beside that, we keep the cherry-pick deadline column and add a new one
"NOTE" to add required information.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
This PR fixes the links in the concepts section. Most of the links are
not dead links in English version because there are redirects for them.
However, these links will be broken when localized to any languages
other English.
Where appropriate. this PR also wraps some long lines which are
unfriendly to any localization team that try to catch the differences
between revisions.
* Remove Google SDK instructions.
Ref: https://github.com/kubernetes/website/issues/20232
* Remove Google SDK instructions for MacOS
* Remove Google SDK instructions for Windows
* Delete install-kubectl-gcloud.md
* Adjust yaml indentation on allowedHostPaths
allowedHostPaths is an attribute inside spec: of PSP, so the sample needs to be shifted to match the YAML.
* Adjusted the PSP example allowedHostPaths coments
Adjusted the whole example (including the #coments)
In 1.22 kubeadm is adding a new API version - v1beta3.
Adapt links and examples to use v1beta3 instead of v1beta2.
v1beta2 is not deprecated yet, but v1beta3 is preferred
at this point.
* Fix few grammar issues and typos in the blog page
> see the API documentation more information
Should be corrected to "documentation for more information" as the sentence is incorrect.
> may only compare two resource version for equality
Everywhere else in the paragraph "resource versions" are used, I believe this is just a typo.
> The get, list and watch
Missing comma after list to separate the elements
* Fix typos in blog post
> On rare occurences, a
"occurrences" is misspelled
> whole list, map or struct
3 elements are missing a comma seperator after map
> there are no way
are is a plural form while "no way" is a singular subject
> `MergePatch`, `StrategicMergePatch`, `JSONPatch` or `Update`
Missing comma after JSONPatch
break long lines
simplify wording (suggested by neolit123)
phrasing error/typo
don't mention creating a new kuebadm config file, not really relevant here
remove unnecessary stop from kubelet cert troubleshoot
minor phrasing improvement: *the* kublet
make the steps for fixing kubelet certs separate from other troubleshooting suggestions
move kubelet cert troubleshooting to a better place
improve note wording and add link
add missing dot
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
add missing dot
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
remove steps not related to the issue
Make instructions work on worker nodes too
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
Hi There, When I follow these instructions, I could not install correctly kubectl. After kubectl file downloading, That file needs to be given execution permission.
I want to contribute for that. Best regards.
Improvement in Configure Service Accounts for Pods Task
Signed-off-by: Shubham Kuchhal <shubham.kuchhal@india.nec.com>
Remove annotations field from metadata.
* Add missing git submodules init to german README.
* Add links and comments to docsy theme installation
* Adjust the JavaScript dependency managment from the english Readme
The original documentation was using `Profile` instead of the `KubeSchedulerConfiguration`. Also clarified how you pass this configuration to the scheduler.
We have some problems processing "<" and ">" in the code comment. The
library we used are treating them as unsafe HTML tags. This PR fixes the
problem by regenerating the references using the updated generator.
This commit adds a missing word to a sentence to correct the grammar and adds a `the` before `Gateway API` for consistency.
Signed-off-by: Lucas Servén Marín <lserven@gmail.com>
Fix typo for the *volumeMount* field
Running the previous snippets with `kubectl apply -k .` would generate this error:
```
error validating ".": error validating data: ValidationError(Deployment.spec.template.spec.containers[0]): unknown field "volumeMount" in io.k8s.api.core.v1.Container;
```
According to [this page](https://kubernetes.io/docs/concepts/storage/volumes/), the correct field is "volumeMounts".
When reading these sentences, I thought that each pod must have the same values as each other pod. In other words, pod1's memory limit must equal pod2's memory limit.
It looks like I misunderstood; "must be the same" means that the limit and request values on each individual pod must match.
Clarify what "must be the same".
This removes the description of the ephemeral containers API from the
concepts section of the website. Most (all?) concepts don't have a
detailed description of their API in concepts. We added it for the
initial release of ephemeral containers because using the raw API was
the only way to add an ephemeral container.
This description of the API is no longer correct. Since it's now easy to
add an ephemeral container using `kubectl debug`, and this is too much
detail for a concepts section, we should remove the API description
entirely.
- Created glossary entries for manual and kubelet eviction
- Created node-pressure-eviction.md and api-eviction.md
- Merged content from eviction-policy.md and out-of-resource.md
- Added note to kube-scheduler about node conditions
- Deleted eviction-policy and out-of-resource.md
- Redirected old topics
- Change topic weights
The current markdown text for the Windows introduction page is very
difficult for tracking changes. There are many paragraphs shown as a
single line. We need line breaks to ease the change tracking, especially
for localization teams. There are other nits related to indentation and
numbered lists. This PR is an attempt to reformat the page so that
future revisions can use it as an example.
Before
```sh
kubectl get po -l
run=nginx-app
No resources found in default namespace.
```
Because:
```sh
kubectl get po --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-app-58cfd4c468-jx6qp 1/1 Running 0 17m app=nginx-app,pod-template-hash=58cfd4c468
```
After:
```sh
kubectl get po -l app=nginx-app
NAME READY STATUS RESTARTS AGE
nginx-app-58cfd4c468-jx6qp 1/1 Running 0 18m
```
The surname Vixie should be in title case as it is a proper noun, even
as part of the name of Vixie cron.
"vixie cron" (with a space in it) is not the name of any scheduling
tool; there are tools named vixie-cron but that is different from the
existing text.
According to https://github.com/kubernetes/kubernetes/pull/98678 and the release notes for 1.21 the feature gate `TTLAfterFinished` is in beta now and enable by default. This change is not yet reflected in the Feature Gates documentation.
Review comments resolved
Small edits
Edits from reylejano comments
Edits from a-mcarthy and kbhawkey comments
More edits from sftim comments
Moved index.md to _index.md to pick up directory links
A volume with `volumeMode: Filesystem` is *mounted* into Pods into a directory. If the volume
is backed by a block device and the device is empty, Kubernetes creates a filesystem
on the device before mounting it for the first time.
This PR updates the component reference generated for 1.21. A bug was
found where the upstream source code is now applying a normalization
function to all flags.
This PR also removes the dangling kubelet-config API file.
Delete a reference to the "node.kubernetes.io/out-of-disk" taint, which
Kubernetes 1.14 deleted (commit 8f87e71e0cc3a2b714291ecd7210dfd8f4de33d6).
Document that the DaemonSet controller adds a toleration for the
"node.kubernetes.io/pid-pressure" taint since Kubernetes 1.14 (commit
ad6e3617d3e92f47fa0874ca258ffe31d1d2ff4c).
* content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md:
Delete references to "node.kubernetes.io/out-of-disk" and document that the
DaemonSet adds a toleration for "node.kubernetes.io/pid-pressure".
Issues fixed by this PR:
- Link in PodDeletionCost is wrong.
- Default value for APIResponseCompression since 1.16 should be true
- Default value for KubeletPodResources in 1.13, 1.14 should be false
- CSIVolumeHealth is an Alpha feature introduced in 1.21, not graduated or deprecated
- EndpointSlice is still Beta in 1.20
- EndpointSliceNodeName is only GA’ed in 1.21
- IndexedJob is an Alpha starting 1.21, not graduated or deprecated
- KubeletPodResources has been GA’ed in 1.20, thus should be removed from the first table
- RequestManagement was deprecated in 1.17 and replaced by APIPriorityAndFaireness
- Several field names are incorrect (e.g. `appProtocol`, `internalTrafficPolicy`, `loadBalancerClass`)
- SCTPSupport should be removed from the first table because it was GA’ed in 1.20.
- ServcieAppProtocol is Beta until 1.20
- StreamingProxyRedirects is GA’ed in 1.19, not deprecated
- No description for ValidateProxyRedirects is provided
- VolumeCapacityPriority is an Alpha feature introduced in 1.21, not graduated
- The order of VolumePVCDataSource is wrong in the table
- No description provided for Initializers
- The deprecated status should not be repeated in the description since it is already indicated by the table in which a feature is listed.
- No description is provided for RequestManagement
* Move portuguese content to pt-br to correct shortcode problems and add a redirection
* Correct small shortcode
* Move portuguese content to pt-br to correct shortcode problems and add a redirection
* Add missing print page shortcodes
* add metrics stability blog post
* Update content/en/blog/_posts/2021-03-30-metrics-stability-hits-ga.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2021-03-30-metrics-stability-hits-ga.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* add diagram for metric lifecycle
* add link to stable metrics list
* Update content/en/blog/_posts/2021-03-30-metrics-stability-ga/2021-03-30-metrics-stability-hits-ga.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/en/blog/_posts/2021-03-30-metrics-stability-ga/2021-03-30-metrics-stability-hits-ga.md
Co-authored-by: Bob Killen <killen.bob@gmail.com>
* fix dates on blog
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: Bob Killen <killen.bob@gmail.com>
1st review reflected
some parts below prerequistes modified
2nd review reflected
'in a Cluster' part modified
gitignore file restored
final review reflected
Final review reflected and error modified
Jsonpath modified
Adding comments to ensure default namespace, container and that the command is supported.
Add note to show you can substitute `env` for any supported command.
kops releases started including a `v` prefix in their tags and GitHub releases.
This updates to a more recent kops version and includes that prefix in the URL.
Signed-off-by: Peter Rifel <peter.rifel@datadoghq.com>
I noticed a typo in a recent blog post, and a similar typo appears in
the comments for some examples. This PR corrects these typos.
Signed-off-by: Bridget Kromhout <bridget@kromhout.org>
* multi-tenancy usage models blog post
Signed-off-by: Jim Bugwadia <jim@nirmata.com>
* update slug, date, filename
Signed-off-by: Jim Bugwadia <jim@nirmata.com>
* rename file
Signed-off-by: Jim Bugwadia <jim@nirmata.com>
* make links relative
Signed-off-by: Jim Bugwadia <jim@nirmata.com>
* update publish date to 4/15
Signed-off-by: Jim Bugwadia <jim@nirmata.com>
* add file
Signed-off-by: Jim Bugwadia <jim@nirmata.com>
Before:
```sh
kubectl get po -l run=nginx-app
No resources found in default namespace.
```
Because:
```sh
kubectl get po --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-app-58cfd4c468-jx6qp 1/1 Running 0 17m app=nginx-app,pod-template-hash=58cfd4c468
```
After:
```sh
kubectl get po -l app=nginx-app
NAME READY STATUS RESTARTS AGE
nginx-app-58cfd4c468-jx6qp 1/1 Running 0 18m
```
* Rename `docs/concepts/scheduling` to `docs/concepts/scheduling-eviction`
* Retitle concept header to "Penjadwalan dan Pengusiran"
* Update redirects
* Update internal links (id only)
Part of proposal #19081
Before:
```sh
kubectl get po -l run=nginx-app
No resources found in default namespace.
```
Because:
```sh
kubectl get po --show-labels
NAME READY STATUS RESTARTS AGE LABELS
nginx-app-58cfd4c468-jx6qp 1/1 Running 0 17m app=nginx-app,pod-template-hash=58cfd4c468
```
After:
```sh
kubectl get po -l app=nginx-app
NAME READY STATUS RESTARTS AGE
nginx-app-58cfd4c468-jx6qp 1/1 Running 0 18m
```
This PR fixes two problems. One was introduced by f265aed794
which folded the extend-cluster.md content into section index without
removing the original content. Another problem is that the PNG images
stored on google docs are not accessible by some people from the far end
of the planet.
This is a reference for WebhookAdmission config generated from kubernetes-sigs/reference-docs/genref tool.
More specifically, it is generated using the following command:
```shell
./genref -include apiserver-webhookadmission
```
Include a new section of the kubeadm certificate management page
to talk about kubelet serving certificates and how to make
them signed. Also include a note about using secure connection
with the metrics-server.
Before:
Example runs pod without creating deployment
```sh
kubectl run --image=nginx nginx-app --port=80
```
But later in document mentions:
```
deployment "nginx-app" created
```
After:
Pod is created with deployment. (Aligned with other language examples)
* Add initial portuguese translation to runtime class
* Apply suggestions from code review
Co-authored-by: Diego W. Antunes <devlware@gmail.com>
* Typo correct
Co-authored-by: Diego W. Antunes <devlware@gmail.com>
This is a reference for kube-scheduler policy config generated from
kubernetes-sigs/reference-docs/genref tool.
More specifically, it is generated using the following command:
```
./genref --include kube-scheduler-policy-config
```
This is a reference for audit.k8s.io/v1 API group generated from kubernetes-sigs/reference-docs/genref tool.
More specifically, it is generated using the following command:
```shell
./genref -include apiserver-audit
```
This is a reference for client authentication API generated from kubernetes-sigs/reference-docs/genref tool.
More specifically, it is generated using the following command:
```shell
./genref -include client-authentication
```
* #26765 - Ordered lists with wrong indexes
* The wrong output of "kubectl get service frontend" on LoadBalancer Service Type example.
* Fixing format
Signed-off-by: Jailton Lopes <jailton@gmail.com>
This is a reference for kube-proxy config generated from kubernetes-sigs/reference-docs/genref tool.
More specifically, it is generated using the following command:
./genref -include kube-proxy
* Actuallize podresources description
This commit updates description according to
https://github.com/kubernetes/enhancements/pull/1884
Update content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md
Signed-off-by: Alexey Perevalov <alexey.perevalov@huawei.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* podresources: document the new feature gate
Signed-off-by: Francesco Romani <fromani@redhat.com>
* device plugins: add clarifications after review
- fix the AllocatableResourcesResponse comment
- describe the NUMA ID and explain the meaning of the field.
Signed-off-by: Francesco Romani <fromani@redhat.com>
Co-authored-by: Alexey Perevalov <alexey.perevalov@huawei.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* add deprecation policy for metrics
* Update content/en/docs/reference/using-api/deprecation-policy.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* amend stability language given review comments
Co-authored-by: Tim Bannister <tim@scalefactory.com>
/docs/reference/setup-tools/kubeadm/kubeadm/ is redirected to /docs/reference/setup-tools/kubeadm/
This replaces the redirect links of kubeadm with the direct links.
NOTE: The pull request for `en` language has been already merged as https://github.com/kubernetes/website/pull/26919
removed [ and ] in the command ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=<trusted-ca-file> --cert=<cert-file> --key=<key-file> \
snapshot save <backup-file-location>
* fix a word translation
* Update content/zh/docs/concepts/workloads/_index.md
Co-authored-by: Qiming Teng <tengqm@outlook.com>
Co-authored-by: Qiming Teng <tengqm@outlook.com>
* Update kubelet-config-file.md
add note and link to kubeadm/kubelet-integration
* Update content/en/docs/tasks/administer-cluster/kubelet-config-file.md
Co-authored-by: Qiming Teng <tengqm@outlook.com>
Co-authored-by: Qiming Teng <tengqm@outlook.com>
(1) the help option for etcdctl is -h and not --h. Corrected the same
(2) in the snapshot command, the --endpoints option's value has been corrected to reflect the right endpoint.
* improve "look and feel" of tabs to make it clearer where they start/end
* add border-radius to tab-pane
* remove the .td-content .hightlight margin when inside the tab-content
* Deprecate `endpointslices.discovery.k8s.io/v1beta1`
* Removes `content/en/docs/tasks/administer-cluster/enabling-endpointslices.md`
and redirects requests to that page to
`content/en/docs/concepts/services-networking/endpoint-slices.md`
/docs/reference/setup-tools/kubeadm/kubeadm/ is redirected to
/docs/reference/setup-tools/kubeadm/
This replaces the redirect links of kubeadm with the direct links.
NOTE: The pull request for `en` language has been already merged as https://github.com/kubernetes/website/pull/26919
* manispulador -> manipulador
* dentro do cgroup e namespace do container -> dentro dos cgroups e namespaces
* tcpSocker -> tcpSocket
* conêiner-> contêiner
* Os logs para um manipulador de _hook_ não expostos em eventos de Pod -> Os logs para um manipulador de _hook_ não são expostos em eventos de Pod.
* failedPreStopHook -> FailedPreStopHook
* remove usage of the "certificates" API for cert renewal
"--use-api" option is removed from kubeadm alpha certs renew command since k8s 1.19
* Update kubeadm-certs.md
* Update content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
Co-authored-by: Lubomir I. Ivanov <neolit123@gmail.com>
* docs(configure-redis-using-configmap): update for clarity
Signed-off-by: Jai Govindani <jai@honestbank.com>
* fix(configure-redis-using-configmap): incorrect volumeMount index
Signed-off-by: Jai Govindani <jai@honestbank.com>
* fix(configure-redis-using-configmap): show Pod status as Running, separate commands from output
Signed-off-by: Jai Govindani <jai@honestbank.com>
* fix(configure-redis-using-configmap): typo
Signed-off-by: Jai Govindani <jai@honestbank.com>
* fix(configure-redis-using-configmap): configmap name
Signed-off-by: Jai Govindani <jai@honestbank.com>
There is a warning deprecation message for kubernetes v1.19:
```
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
```
Switched apiversion in the example to `networking.k8s.io/v1`
* Fixed a misconception that I had and I forgot to fix before the merge
* Update content/es/docs/reference/glossary/limitrange.md
Added suggested glossary tooltip : line23.
Thanks
Co-authored-by: Rael Garcia <rael@rael.io>
Co-authored-by: Rael Garcia <rael@rael.io>
* Sync with english version in 'Update Ubuntu/Debian installation instructions to use Signed-By option (#26952)'
Signed-off-by: ydFu <ader.ydfu@gmail.com>
/docs/tasks/tools/install-kubectl/ is redirected to
/docs/tasks/tools/
This commit replace the redirect links for installing kubectl
with direct links.
* Fix: title is put in quotes. Update: Added description, probably deleted by mistake reviewing change suggestions
* Apply suggestions from code review
Realmente no estoy entendiendo por qué el "Commit suggestions" no está realizando el commit al fork. Lo voy a intentar nuevamente, pero esta vez haciendo un review ...
Éste mensaje se corresponde con el Commit suggestion al utilizar el "Add suggestion to batch" no creo que funcione tampoco en cuyo caso voy a intentar hacer (como mencioné en el párrafo anterior) un review para que los cambios persistan.
Co-authored-by: Rael Garcia <rael@rael.io>
Co-authored-by: Rael Garcia <rael@rael.io>
/docs/reference/setup-tools/kubeadm/kubeadm/ is redirected to /docs/reference/setup-tools/kubeadm/
This replaces the redirect links of kubeadm with the direct links.
NOTE: The pull request for `en` language has been already merged as https://github.com/kubernetes/website/pull/26919
* Move pod-overhead.md to schedulling-eviction
* Add pod-overhead.md translation
* Update content/pt/docs/concepts/scheduling-eviction/pod-overhead.md
* Fix name: test-Pod to name: test-pod
* Small fixes
* remove reviewers
* change 'fonte' to 'código fonte'
* Add the suggestion by code review
* Update the references with the original doc
* Update the _index.md
/docs/reference/setup-tools/kubeadm/kubeadm/ is redirected to
/docs/reference/setup-tools/kubeadm/
This replaces the redirect links of kubeadm with the direct links.
* README.md Added description/suggestion to install git submodules.
Created _index.md for es/docs/concept/policy.
Created limit-range.md, content-type:concept for es/docs/concepts/policy
* Changes to the README-es.md file discarded to apply a separate commit for it
* Fast fix, missed word
* Update content/es/docs/concepts/policy/limit-range.md
Suggestion, Remove "en" (fix). Personal note: I'm not sure if the suggested change makes clear that the system administrator can force the users to follow specifications with new politics.. Not always the user have control on the cluster
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
lgtm
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
Nice
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
look at that trained eye! Good job thnaks
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
lgtm
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
Not sure about the translation, it sound better but it should be clear with terminology. A pod or container request resources it's not a requirement. But the same time we have [this definition](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/))
"Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value."
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
Lgtm
Co-authored-by: Rael Garcia <rael@rael.io>
* Suggested change discarded, reason: Contenedores is not defined
* Sorry my bad, Contenedores definition was there but as a singular noun not plural. Fiexed (line 51).. as well as pods (line 53
* Apply suggestions from code review
Wrong button last time. Added suggestions (as single commit)
Co-authored-by: Rael Garcia <rael@rael.io>
Co-authored-by: Victor Morales <chipahuac@hotmail.com>
* · Added: description/overview, file: content/es/docs/concepts/policy/_index.md
· Added: description, file: content/es/docs/concepts/policy/limit-range.md
· Midified: structure and dreafting, file: content/es/docs/concepts/policy/limit-range.md
· Added: term to glossary (es), file: content/es/docs/reference/glossary/limitrange.md
* Added Container word I deleted earlier today, lines 25,26,27.
Unnecessary repeated tooltip were removed allowing only once possible new concept per paragraph. Too many tooltip
* Fixed weight property commited by mistake previously.
* Added Suggested change manually
* Fixed Last suggested change
* All right I found the suggested change. Done
* Update content/es/docs/concepts/policy/limit-range.md
Co-authored-by: Rael Garcia <rael@rael.io>
* Update content/es/docs/concepts/policy/limit-range.md
Co-authored-by: Rael Garcia <rael@rael.io>
* Replaced occurrences of "Peticiones" by "Solicitudes" as it was suggested previously for line 29
Co-authored-by: Rael Garcia <rael@rael.io>
Co-authored-by: Victor Morales <chipahuac@hotmail.com>
This commit fixes the packages not found error during
Docker installation. The packages containerd.io 1.2.13-2,
docker-ce 19.03.11, and docker-cs-cli 19.03.11 are currently not in
Ubuntu 20.10. This commit instead points users to validated
versions of Docker.
fixes https://github.com/kubernetes/kubernetes/issues/99831
Signed-off-by: Enyinna Ochulor <eochulor@vmware.com>
Port 6379 was errantly identified as the mongo service port however mongo-service.yaml defines port 27017. I've updated the service command output to reflect that.
when use `kubectl run source-ip-app --image=k8s.gcr.io/echoserver:1.4` . it will be create a pod, the output is: `pod/source-ip-app created`, but the svc will find `deployment` when use `kubectl expose deployment source-ip-app --name=clusterip --port=80 --target-port=8080`
There is an extra space before the field 'type', so to create a secret failed. The message is "error: error parsing nginxsecret.yaml: error converting YAML to JSON: yaml: line 5: did not find expected key".
This simplifies the containerd installation instructions. All Linux distros will
now download from Docker repos, including Ubuntu 18.04 which previously
installed from Ubuntu repos.
The Docker runtime instructions have also been simplified. The RHEL/CentOS
specific option overlay2.override_kernel_check=true option has been removed.
It seems Docker will now autodetect overlay2 support on older Linux kernels
<4.0.0,>=3.10.0-514, which have back-ported overlay2 support on RHEL/CentOS 7.4+
See moby/moby#34368
Instead of putting "stringSlice" or "mapStringString" as data type for
flag data type, we can show more user friendly data type description
that is not GoLang specific.
* Add Concepts/Overview for Kubernetes and Components, and glossary itens in pt
Modify Glossary/Control-Plane definition in pt
Changes to be committed:
new file: content/pt/docs/concepts/overview/_index.md
new file: content/pt/docs/concepts/overview/components.md
new file: content/pt/docs/concepts/overview/what-is-kubernetes.md
new file: content/pt/docs/reference/glossary/cloud-controller-manager.md
new file: content/pt/docs/reference/glossary/cncf.md
new file: content/pt/docs/reference/glossary/container-runtime.md
modified: content/pt/docs/reference/glossary/control-plane.md
new file: content/pt/docs/reference/glossary/etcd.md
new file: content/pt/docs/reference/glossary/kube-apiserver.md
new file: content/pt/docs/reference/glossary/kube-controller-manager.md
new file: content/pt/docs/reference/glossary/kube-proxy.md
new file: content/pt/docs/reference/glossary/kube-scheduler.md
modified: package-lock.json
* Changes to be committed:
modified: package-lock.json
Untracked files:
content/pt/docs/concepts/overview/kubernetes-api.md
* Changes to be committed:
modified: package-lock.json
Untracked files:
content/pt/docs/concepts/overview/kubernetes-api.md
* ---
reviewers:
- lavalamp
- rikatz
title: Componentes do Kubernetes
content_type: concept
description: >
Um cluster Kubernetes consiste de componentes que representam a camada de gerenciamento, e um conjunto de máquinas chamadas nós.
weight: 20
card:
name: concepts
weight: 20
---
<!-- overview -->
Ao implantar o Kubernetes, você obtém um cluster.
{{< glossary_definition term_id="cluster" length="all" prepend="Um cluster Kubernetes consiste em">}}
Este documento descreve os vários componentes que você precisa ter para implantar um cluster Kubernetes completo e funcional.
Esse é o diagrama de um cluster Kubernetes com todos os componentes interligados.
<!-- body -->
## Componentes da camada de gerenciamento
Os componentes da camada de gerenciamento tomam decisões globais sobre o cluster (por exemplo, agendamento de _pods_), bem como detectam e respondem aos eventos do cluster (por exemplo, iniciando um novo _{{< glossary_tooltip text="pod" term_id="pod" >}}_ quando o campo `replicas` de um _Deployment_ não está atendido).
Os componentes da camada de gerenciamento podem ser executados em qualquer máquina do cluster. Contudo, para simplificar, os _scripts_ de configuração normalmente iniciam todos os componentes da camada de gerenciamento na mesma máquina, e não executa contêineres de usuário nesta máquina. Veja [Construindo clusters de alta disponibilidade](/docs/admin/high-availability/) para um exemplo de configuração de múltiplas VMs para camada de gerenciamento (_multi-main-VM_).
### kube-apiserver
{{< glossary_definition term_id="kube-apiserver" length="all" >}}
### etcd
{{< glossary_definition term_id="etcd" length="all" >}}
### kube-scheduler
{{< glossary_definition term_id="kube-scheduler" length="all" >}}
### kube-controller-manager
{{< glossary_definition term_id="kube-controller-manager" length="all" >}}
Alguns tipos desses controladores são:
* Controlador de nó: responsável por perceber e responder quando os nós caem.
* Controlador de _Job_: Observa os objetos _Job_ que representam tarefas únicas e, em seguida, cria _pods_ para executar essas tarefas até a conclusão.
* Controlador de _endpoints_: preenche o objeto _Endpoints_ (ou seja, junta os Serviços e os _pods_).
* Controladores de conta de serviço e de _token_: crie contas padrão e _tokens_ de acesso de API para novos _namespaces_.
### cloud-controller-manager
{{< glossary_definition term_id="cloud-controller-manager" length="short" >}}
O cloud-controller-manager executa apenas controladores que são específicos para seu provedor de nuvem.
Se você estiver executando o Kubernetes em suas próprias instalações ou em um ambiente de aprendizagem dentro de seu
próprio PC, o cluster não possui um gerenciador de controlador de nuvem.
Tal como acontece com o kube-controller-manager, o cloud-controller-manager combina vários ciclos de controle logicamente independentes em um binário único que você executa como um processo único. Você pode escalar horizontalmente (exectuar mais de uma cópia) para melhorar o desempenho ou para auxiliar na tolerância a falhas.
Os seguintes controladores podem ter dependências de provedor de nuvem:
* Controlador de nó: para verificar junto ao provedor de nuvem para determinar se um nó foi excluído da nuvem após parar de responder.
* Controlador de rota: para configurar rotas na infraestrutura de nuvem subjacente.
* Controlador de serviço: Para criar, atualizar e excluir balanceadores de carga do provedor de nuvem.
## Node Components
Os componentes de nó são executados em todos os nós, mantendo os _pods_ em execução e fornecendo o ambiente de execução do Kubernetes.
### kubelet
{{< glossary_definition term_id="kubelet" length="all" >}}
### kube-proxy
{{< glossary_definition term_id="kube-proxy" length="all" >}}
### Container runtime
{{< glossary_definition term_id="container-runtime" length="all" >}}
## Addons
Complementos (_addons_) usam recursos do Kubernetes ({{< glossary_tooltip term_id="daemonset" >}}, {{< glossary_tooltip term_id="deployment" >}}, etc) para implementar funcionalidades do cluster. Como fornecem funcionalidades em nível do cluster, recursos de _addons_ que necessitem ser criados dentro de um _namespace_ pertencem ao _namespace_ `kube-system`.
Alguns _addons_ selecionados são descritos abaixo; para uma lista estendida dos _addons_ disponíveis, por favor consulte [Addons](/docs/concepts/cluster-administration/addons/).
### DNS
Embora os outros complementos não sejam estritamente necessários, todos os clusters do Kubernetes devem ter um [DNS do cluster](/docs/concepts/services-networking/dns-pod-service/), já que muitos exemplos dependem disso.
O DNS do cluster é um servidor DNS, além de outros servidores DNS em seu ambiente, que fornece registros DNS para serviços do Kubernetes.
Os contêineres iniciados pelo Kubernetes incluem automaticamente esse servidor DNS em suas pesquisas DNS.
### Web UI (Dashboard)
[Dashboard](/docs/tasks/access-application-cluster/web-ui-dashboard/) é uma interface de usuário Web, de uso geral, para clusters do Kubernetes. Ele permite que os usuários gerenciem e solucionem problemas de aplicações em execução no cluster, bem como o próprio cluster.
### Monitoramento de recursos do contêiner
[Monitoramento de recursos do contêiner](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) registra métricas de série temporal genéricas sobre os contêineres em um banco de dados central e fornece uma interface de usuário para navegar por esses dados.
### Logging a nivel do cluster
Um mecanismo de [_logging_ a nível do cluster](/docs/concepts/cluster-administration/logging/) é responsável por guardar os _logs_ dos contêineres em um armazenamento central de _logs_ com um interface para navegação/pesquisa.
## {{% heading "whatsnext" %}}
* Aprenda sobre [Nós](/docs/concepts/architecture/nodes/).
* Aprenda sobre [Controladores](/docs/concepts/architecture/controller/).
* Aprenda sobre [kube-scheduler](/docs/concepts/scheduling-eviction/kube-scheduler/).
* Leia a [documentação](https://etcd.io/docs/) oficial do **etcd**.
* ---
reviewers:
title: Componentes do Kubernetes
content_type: concept
description: >
Um cluster Kubernetes consiste de componentes que representam a camada de gerenciamento, e um conjunto de máquinas chamadas nós.
weight: 20
card:
name: concepts
weight: 20
---
<!-- overview -->
Ao implantar o Kubernetes, você obtém um cluster.
{{< glossary_definition term_id="cluster" length="all" prepend="Um cluster Kubernetes consiste em">}}
Este documento descreve os vários componentes que você precisa ter para implantar um cluster Kubernetes completo e funcional.
Esse é o diagrama de um cluster Kubernetes com todos os componentes interligados.
<!-- body -->
## Componentes da camada de gerenciamento
Os componentes da camada de gerenciamento tomam decisões globais sobre o cluster (por exemplo, agendamento de _pods_), bem como detectam e respondem aos eventos do cluster (por exemplo, iniciando um novo _{{< glossary_tooltip text="pod" term_id="pod" >}}_ quando o campo `replicas` de um _Deployment_ não está atendido).
Os componentes da camada de gerenciamento podem ser executados em qualquer máquina do cluster. Contudo, para simplificar, os _scripts_ de configuração normalmente iniciam todos os componentes da camada de gerenciamento na mesma máquina, e não executa contêineres de usuário nesta máquina. Veja [Construindo clusters de alta disponibilidade](/docs/admin/high-availability/) para um exemplo de configuração de múltiplas VMs para camada de gerenciamento (_multi-main-VM_).
### kube-apiserver
{{< glossary_definition term_id="kube-apiserver" length="all" >}}
### etcd
{{< glossary_definition term_id="etcd" length="all" >}}
### kube-scheduler
{{< glossary_definition term_id="kube-scheduler" length="all" >}}
### kube-controller-manager
{{< glossary_definition term_id="kube-controller-manager" length="all" >}}
Alguns tipos desses controladores são:
* Controlador de nó: responsável por perceber e responder quando os nós caem.
* Controlador de _Job_: Observa os objetos _Job_ que representam tarefas únicas e, em seguida, cria _pods_ para executar essas tarefas até a conclusão.
* Controlador de _endpoints_: preenche o objeto _Endpoints_ (ou seja, junta os Serviços e os _pods_).
* Controladores de conta de serviço e de _token_: crie contas padrão e _tokens_ de acesso de API para novos _namespaces_.
### cloud-controller-manager
{{< glossary_definition term_id="cloud-controller-manager" length="short" >}}
O cloud-controller-manager executa apenas controladores que são específicos para seu provedor de nuvem.
Se você estiver executando o Kubernetes em suas próprias instalações ou em um ambiente de aprendizagem dentro de seu
próprio PC, o cluster não possui um gerenciador de controlador de nuvem.
Tal como acontece com o kube-controller-manager, o cloud-controller-manager combina vários ciclos de controle logicamente independentes em um binário único que você executa como um processo único. Você pode escalar horizontalmente (exectuar mais de uma cópia) para melhorar o desempenho ou para auxiliar na tolerância a falhas.
Os seguintes controladores podem ter dependências de provedor de nuvem:
* Controlador de nó: para verificar junto ao provedor de nuvem para determinar se um nó foi excluído da nuvem após parar de responder.
* Controlador de rota: para configurar rotas na infraestrutura de nuvem subjacente.
* Controlador de serviço: Para criar, atualizar e excluir balanceadores de carga do provedor de nuvem.
## Node Components
Os componentes de nó são executados em todos os nós, mantendo os _pods_ em execução e fornecendo o ambiente de execução do Kubernetes.
### kubelet
{{< glossary_definition term_id="kubelet" length="all" >}}
### kube-proxy
{{< glossary_definition term_id="kube-proxy" length="all" >}}
### Container runtime
{{< glossary_definition term_id="container-runtime" length="all" >}}
## Addons
Complementos (_addons_) usam recursos do Kubernetes ({{< glossary_tooltip term_id="daemonset" >}}, {{< glossary_tooltip term_id="deployment" >}}, etc) para implementar funcionalidades do cluster. Como fornecem funcionalidades em nível do cluster, recursos de _addons_ que necessitem ser criados dentro de um _namespace_ pertencem ao _namespace_ `kube-system`.
Alguns _addons_ selecionados são descritos abaixo; para uma lista estendida dos _addons_ disponíveis, por favor consulte [Addons](/docs/concepts/cluster-administration/addons/).
### DNS
Embora os outros complementos não sejam estritamente necessários, todos os clusters do Kubernetes devem ter um [DNS do cluster](/docs/concepts/services-networking/dns-pod-service/), já que muitos exemplos dependem disso.
O DNS do cluster é um servidor DNS, além de outros servidores DNS em seu ambiente, que fornece registros DNS para serviços do Kubernetes.
Os contêineres iniciados pelo Kubernetes incluem automaticamente esse servidor DNS em suas pesquisas DNS.
### Web UI (Dashboard)
[Dashboard](/docs/tasks/access-application-cluster/web-ui-dashboard/) é uma interface de usuário Web, de uso geral, para clusters do Kubernetes. Ele permite que os usuários gerenciem e solucionem problemas de aplicações em execução no cluster, bem como o próprio cluster.
### Monitoramento de recursos do contêiner
[Monitoramento de recursos do contêiner](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) registra métricas de série temporal genéricas sobre os contêineres em um banco de dados central e fornece uma interface de usuário para navegar por esses dados.
### Logging a nivel do cluster
Um mecanismo de [_logging_ a nível do cluster](/docs/concepts/cluster-administration/logging/) é responsável por guardar os _logs_ dos contêineres em um armazenamento central de _logs_ com um interface para navegação/pesquisa.
## {{% heading "whatsnext" %}}
* Aprenda sobre [Nós](/docs/concepts/architecture/nodes/).
* Aprenda sobre [Controladores](/docs/concepts/architecture/controller/).
* Aprenda sobre [kube-scheduler](/docs/concepts/scheduling-eviction/kube-scheduler/).
* Leia a [documentação](https://etcd.io/docs/) oficial do **etcd**.
* ---
reviewers:
title: O que é Kubernetes?
description: >
Kubernetes é um plataforma de código aberto, portável e extensiva para o gerenciamento de cargas de trabalho e serviços distribuídos em contêineres, que facilita tanto a configuração declarativa quanto a automação. Ele possui um ecossistema grande, e de rápido crescimento. Serviços, suporte, e ferramentas para Kubernetes estão amplamente disponíveis.
content_type: concept
weight: 10
card:
name: concepts
weight: 10
sitemap:
priority: 0.9
---
<!-- overview -->
Essa página é uma visão geral do Kubernetes.
<!-- body -->
Kubernetes é um plataforma de código aberto, portável e extensiva para o gerenciamento de cargas de trabalho e serviços distribuídos em contêineres, que facilita tanto a configuração declarativa quanto a automação. Ele possui um ecossistema grande, e de rápido crescimento. Serviços, suporte, e ferramentas para Kubernetes estão amplamente disponíveis.
O Google tornou Kubernetes um projeto de código-aberto em 2014. O Kubernetes combina [mais de 15 anos de experiência do Google](/blog/2015/04/borg-predecessor-to-kubernetes/) executando cargas de trabalho produtivas em escala, com as melhores idéias e práticas da comunidade.
O nome **Kubernetes** tem origem no Grego, significando _timoneiro_ ou _piloto_. **K8s** é a abreviação derivada pela troca das oito letras "ubernete" por "8", se tornado _K"8"s_.
## Voltando no tempo
Vamos dar uma olhada no porque o Kubernetes é tão útil, voltando no tempo.
**Era da implantação tradicional:** No início, as organizações executavam aplicações em servidores físicos. Não havia como definir limites de recursos para aplicações em um mesmo servidor físico, e isso causava problemas de alocação de recursos. Por exemplo, se várias aplicações fossem executadas em um mesmo servidor físico, poderia haver situações em que uma aplicação ocupasse a maior parte dos recursos e, como resultado, o desempenho das outras aplicações seria inferior. Uma solução para isso seria executar cada aplicação em um servidor físico diferente. Mas isso não escalava, pois os recursos eram subutilizados, e se tornava custoso para as organizações manter muitos servidores físicos.
**Era da implantação virtualizada:** Como solução, a virtualização foi introduzida. Esse modelo permite que você execute várias máquinas virtuais (VMs) em uma única CPU de um servidor físico. A virtualização permite que as aplicações sejam isoladas entre as VMs, e ainda fornece um nível de segurança, pois as informações de uma aplicação não podem ser acessadas livremente por outras aplicações.
A virtualização permite melhor utilização de recursos em um servidor físico, e permite melhor escalabilidade porque uma aplicação pode ser adicionada ou atualizada facilmente, reduz os custos de hardware e muito mais. Com a virtualização, você pode apresentar um conjunto de recursos físicos como um cluster de máquinas virtuais descartáveis.
Cada VM é uma máquina completa que executa todos os componentes, incluindo seu próprio sistema operacional, além do hardware virtualizado.
**Era da implantação em contêineres:** Contêineres são semelhantes às VMs, mas têm propriedades de isolamento flexibilizados para compartilhar o sistema operacional (SO) entre as aplicações. Portanto, os contêineres são considerados leves. Semelhante a uma VM, um contêiner tem seu próprio sistema de arquivos, compartilhamento de CPU, memória, espaço de processo e muito mais. Como eles estão separados da infraestrutura subjacente, eles são portáveis entre nuvens e distribuições de sistema operacional.
Contêineres se tornaram populares porque eles fornecem benefícios extra, tais como:
* Criação e implantação ágil de aplicações: aumento da facilidade e eficiência na criação de imagem de contêiner comparado ao uso de imagem de VM.
* Desenvolvimento, integração e implantação contínuos: fornece capacidade de criação e de implantação de imagens de contêiner de forma confiável e frequente, com a funcionalidade de efetuar reversões rápidas e eficientes (devido à imutabilidade da imagem).
* Separação de interesses entre Desenvolvimento e Operações: crie imagens de contêineres de aplicações no momento de construção/liberação em vez de no momento de implantação, desacoplando as aplicações da infraestrutura.
* A capacidade de observação (Observabilidade) não apenas apresenta informações e métricas no nível do sistema operacional, mas também a integridade da aplicação e outros sinais.
* Consistência ambiental entre desenvolvimento, teste e produção: funciona da mesma forma em um laptop e na nuvem.
* Portabilidade de distribuição de nuvem e sistema operacional: executa no Ubuntu, RHEL, CoreOS, localmente, nas principais nuvens públicas e em qualquer outro lugar.
* Gerenciamento centrado em aplicações: eleva o nível de abstração da execução em um sistema operacional em hardware virtualizado à execução de uma aplicação em um sistema operacional usando recursos lógicos.
* Microserviços fracamente acoplados, distribuídos, elásticos e livres: as aplicações são divididas em partes menores e independentes e podem ser implantados e gerenciados dinamicamente - não uma pilha monolítica em execução em uma grande máquina de propósito único.
* Isolamento de recursos: desempenho previsível de aplicações.
* Utilização de recursos: alta eficiência e densidade.
## Por que você precisa do Kubernetes e o que ele pode fazer{#why-you-need-kubernetes-and-what-can-it-do}
Os contêineres são uma boa maneira de agrupar e executar suas aplicações. Em um ambiente de produção, você precisa gerenciar os contêineres que executam as aplicações e garantir que não haja tempo de inatividade. Por exemplo, se um contêiner cair, outro contêiner precisa ser iniciado. Não seria mais fácil se esse comportamento fosse controlado por um sistema?
É assim que o Kubernetes vem ao resgate! O Kubernetes oferece uma estrutura para executar sistemas distribuídos de forma resiliente. Ele cuida do escalonamento e do recuperação à falha de sua aplicação, fornece padrões de implantação e muito mais. Por exemplo, o Kubernetes pode gerenciar facilmente uma implantação no método canário para seu sistema.
O Kubernetes oferece a você:
* **Descoberta de serviço e balanceamento de carga**
O Kubernetes pode expor um contêiner usando o nome DNS ou seu próprio endereço IP. Se o tráfego para um contêiner for alto, o Kubernetes pode balancear a carga e distribuir o tráfego de rede para que a implantação seja estável.
* **Orquestração de armazenamento**
O Kubernetes permite que você monte automaticamente um sistema de armazenamento de sua escolha, como armazenamentos locais, provedores de nuvem pública e muito mais.
* **Lançamentos e reversões automatizadas**
Você pode descrever o estado desejado para seus contêineres implantados usando o Kubernetes, e ele pode alterar o estado real para o estado desejado em um ritmo controlada. Por exemplo, você pode automatizar o Kubernetes para criar novos contêineres para sua implantação, remover os contêineres existentes e adotar todos os seus recursos para o novo contêiner.
* **Empacotamento binário automático**
Você fornece ao Kubernetes um cluster de nós que pode ser usado para executar tarefas nos contêineres. Você informa ao Kubernetes de quanta CPU e memória (RAM) cada contêiner precisa. O Kubernetes pode encaixar contêineres em seus nós para fazer o melhor uso de seus recursos.
* **Autocorreção**
O Kubernetes reinicia os contêineres que falham, substitui os contêineres, elimina os contêineres que não respondem à verificação de integridade definida pelo usuário e não os anuncia aos clientes até que estejam prontos para servir.
* **Gerenciamento de configuração e de segredos**
O Kubernetes permite armazenar e gerenciar informações confidenciais, como senhas, tokens OAuth e chaves SSH. Você pode implantar e atualizar segredos e configuração de aplicações sem reconstruir suas imagens de contêiner e sem expor segredos em sua pilha de configuração.
## O que o Kubernetes não é
O Kubernetes não é um sistema PaaS (plataforma como serviço) tradicional e completo. Como o Kubernetes opera no nível do contêiner, e não no nível do hardware, ele fornece alguns recursos geralmente aplicáveis comuns às ofertas de PaaS, como implantação, escalonamento, balanceamento de carga, e permite que os usuários integrem suas soluções de _logging_, monitoramento e alerta. No entanto, o Kubernetes não é monolítico, e essas soluções padrão são opcionais e conectáveis. O Kubernetes fornece os blocos de construção para a construção de plataformas de desenvolvimento, mas preserva a escolha e flexibilidade do usuário onde é importante.
Kubernetes:
* Não limita os tipos de aplicações suportadas. O Kubernetes visa oferecer suporte a uma variedade extremamente diversa de cargas de trabalho, incluindo cargas de trabalho sem estado, com estado e de processamento de dados. Se uma aplicação puder ser executada em um contêiner, ele deve ser executado perfeitamente no Kubernetes.
* Não implanta código-fonte e não constrói sua aplicação. Os fluxos de trabalho de integração contínua, entrega e implantação (CI/CD) são determinados pelas culturas e preferências da organização, bem como pelos requisitos técnicos.
* Não fornece serviços em nível de aplicação, tais como middleware (por exemplo, barramentos de mensagem), estruturas de processamento de dados (por exemplo, Spark), bancos de dados (por exemplo, MySQL), caches, nem sistemas de armazenamento em cluster (por exemplo, Ceph), como serviços integrados. Esses componentes podem ser executados no Kubernetes e/ou podem ser acessados por aplicações executadas no Kubernetes por meio de mecanismos portáteis, como o [Open Service Broker](https://openservicebrokerapi.org/).
* Não dita soluções de _logging_, monitoramento ou alerta. Ele fornece algumas integrações como prova de conceito e mecanismos para coletar e exportar métricas.
* Não fornece nem exige um sistema/idioma de configuração (por exemplo, Jsonnet). Ele fornece uma API declarativa que pode ser direcionada por formas arbitrárias de especificações declarativas.
* Não fornece nem adota sistemas abrangentes de configuração de máquinas, manutenção, gerenciamento ou autocorreção.
* Adicionalmente, o Kubernetes não é um mero sistema de orquestração. Na verdade, ele elimina a necessidade de orquestração. A definição técnica de orquestração é a execução de um fluxo de trabalho definido: primeiro faça A, depois B e depois C. Em contraste, o Kubernetes compreende um conjunto de processos de controle independentes e combináveis que conduzem continuamente o estado atual em direção ao estado desejado fornecido. Não importa como você vai de A para C. O controle centralizado também não é necessário. Isso resulta em um sistema que é mais fácil de usar e mais poderoso, robusto, resiliente e extensível.
## {{% heading "whatsnext" %}}
* Dê uma olhada em [Componentes do Kubernetes](/docs/concepts/overview/components/).
* Pronto para [Iniciar](/docs/setup/)?
The use of `apt-key` to install has also been removed as it is now deprecated
and will be last available in Debian 11 and Ubuntu 22.04.
Also updates the Docker repository setup instructions in container-runtimes.md,
to now refer to the respective instructions at https://docs.docker.com/engine/install/
which has already made the move to use the signed-by option.
* Add Dont Panic blog post in pt language
Add Dont Panic blog post in pt language.
Partial fix for kubernetes #13939
* Update content/pt/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update content/pt/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Update 2020-12-02-dont-panic-kubernetes-and-docker.md
typo `:%s/\<kubernetes\>/Kubernetes/gc` and `:%s/é/e`
* Update 2020-12-02-dont-panic-kubernetes-and-docker.md
removendo depreciação e arrumando typos
* remove enterprise name
* Update content/pt/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Update content/pt/blog/_posts/2020-12-02-dont-panic-kubernetes-and-docker.md
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Update 2020-12-02-dont-panic-kubernetes-and-docker.md
typo `:%s/\<kubernetes\>/Kubernetes/gc` and `:%s/é/e`
Update 2020-12-02-dont-panic-kubernetes-and-docker.md
removendo depreciação e arrumando typos
remove enterprise name
Co-authored-by: Tim Bannister <tim@scalefactory.com>
/docs/reference/setup-tools/kubeadm/kubeadm/ is redirected to
/docs/reference/setup-tools/kubeadm/
This replaces the redirect links of kubeadm with the direct links.
- Add the new page "Configuring a cgroup driver"
The page includes details on how to configure the cgroup driver
using kubeadm. It also includes a migration guide for users to
move to the "systemd" driver.
- Link to the new page from install-kubeadm.md and container-runtimes.md
* Move accessing API from within pod to tasks
* Remove reviewers, version check; Add whatsnext
* Move to run applications
* Fix what's next section link
The `imagePullPolicy` field is set automatically based on the image tag
if it's initially omitted, but it is not updated if the image tag later
changes. This can lead to [confusing
behaviour](https://itnext.io/defaults-are-hard-kubernetes-deployment-edition-3b11095792f2).
This change attempts to warn users of this potential pitfall.
Fix `Source files` section in localization document, and remove {{< release-branch >}} variable
due to `release-1.20` branch does not exist.
- improve sentence
- Fix latest branch for latest version
- Add description if latest branch does not exist
- Describe about master branch
- Apply suggestions from code review
- Unify `development branch` to` localization branch`
- Remove description for switching upstream branch
- Add description for switching upstream
- Also, add description for merging to master and new release branch.
Co-authored-by: Qiming Teng <tengqim@cn.ibm.com>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
Co-authored-by: Seokho Son <shsongist@gmail.com>
* Add EndpointSlice blog post in pt language
* Apply corrections into endpointslice blog post translation
Co-authored-by: Jhon Mike <jhon.msdev@gmail.com>
Co-authored-by: Jhon Mike <jhon.msdev@gmail.com>
This page listed `/var/run/docker.sock` as the UNIX domain socket path for the Docker container runtime, but it's actually `/var/run/dockershim.sock`, as the kubelet documentation indicates as the default value for the `--container-runtime-endpoint` argument:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
The socket file specified (`/var/run/docker.sock`) is where the Docker daemon listens for requests for the Docker API, not the CRI interface.
1. Print a message to tell if modules need to be initialized when running `make serve` or other similar targets, and stop current make procedure.
2. Provide `module-init` target to initialize all dependencies
Co-authored-by: Tim Bannister <tim@scalefactory.com>
* Add (stub) localization guide for Spanish
Co-authored-by: Victor Morales <chipahuac@hotmail.com>
* docs: Add brief page introduction
* docs: use archivo as discussed with the team
Co-authored-by: Victor Morales <chipahuac@hotmail.com>
Co-authored-by: Rael Garcia <rael@rael.io>
There is a warning deprecation message for kubernetes v1.19:
```
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
```
Switched apiversion in the example to `networking.k8s.io/v1`
* Translate Explore You App in Portuguese
* Update Explore Your App link in Learn Kubernetes Basics _index.html
* Update link to Explore Interactive page translation in Portuguese
* Fix typo and change the translation of the word *worker*.
Signed-off-by: Jailton Lopes <jailton@gmail.com>
* Fix typo
Signed-off-by: Jailton Lopes <jailton@gmail.com>
* Resync with english version in 'Dual-stack docs correction #26386'
* Update in 'administer-cluster\safely-drain-node.md' #25996
* Fix git rebase in services-networking\dual-stack.md
Signed-off-by: ydFu <ader.ydfu@gmail.com>
- Ko: add l10n contributor name in a blog (#26285)
- Translate reference/glossary/secret.md in Korean (#26391)
- Update outdated files in the dev-1.20-ko.4 branch (2) (#26342)
- Translate reference/glossary/quantity.md in Korean (#26390)
- Update outdated files in the dev-1.20-ko.4 branch (1) (#26427)
- Translate reference/glossary/storage-class.md in Korean (#26419)
- Update outdated files in dev-1.20-ko.4 branch (3) (#26599)
Co-authored-by: seokho-son <shsongist@gmail.com>
Co-authored-by: Jerry Park <jaehwa@gmail.com>
Co-authored-by: santachopa <santachopa@naver.com>
Co-authored-by: jmyung <jesang.myung@gmail.com>
Operator Pattern link was pointing to https://coreos.com/operator. This link was redirecting to the openshift.com homepage.
Provided new link on openshift.com pointing to Operator Pattern documentation.
The docs don't mention when the kubelet will attempt to renew a cert,
which causes concern when one notices that certain certificates are being
renewed and others are not. Adding the time frame adds certainty, so that
if an user notices a kubelet cert expiring in less than 30d, they know
something is misconfigured and should be looked at.
Without first mentioning where the number 40 is coming from, it was a bit unclear why 40 would be the threshold value for policy selection in this case.
I hope this patch clarifies it.
update based on English version of statefulset.md
1. added a block inside #stable-network-id
2. sync tables and blocks
3. added a sentence in #parallel-pod-management
the `!` in `!=` should use halfwidth form instead of fullwidth, otherwise it will cause trouble in K8s.
Signed-off-by: Hollow Man <hollowman@hollowman.ml>
The "Automatic mounting of manually created Secrets" section of the
Secrets documentation previously suggesting using PodPresets. PodPresets
have been removed, there is no alternate facility described, and it's
unclear if auto-mounting secrets based on associations with
ServiceAccounts was ever supported. Accordingly, the section should be
removed.
This is part of work in umbrella issue:
[zh] Umbrella issue: pages out of sync in tasks section #26178
Topology Manageer (L):
content/zh/docs/tasks/administer-cluster/topology-manager.md
This is part of the work in the following umbrella issue:
[zh] Umbrella issue: pages out of sync in tasks section
CRICTL (L)
content/zh/docs/tasks/debug-application-cluster/crictl.md
This is part of work in umbrella issue:
Umbrella issue: pages out of sync in tasks section #26178
HA (L):
content/zh/docs/tasks/administer-cluster/highly-available-master.md
The current layout partial is using scripts from a site not accessible
from behind the dam great firewall. This is a fix to cache the scripts
as we do before so that tabs, top menu works for everyone.
The original link has moved due to docs restructuring in the
cloud-provider-openstack repo. Also correcting the capitalization.
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This commit adds a cosmetic change by adding backticks to show the env
var in monospace font when rendered.
Mention Flatcar Linux in kubeadm troubleshooting doc, alongwith Fedora
CoreOS.
Signed-off-by: Suraj Deshmukh <surajd.service@gmail.com>
The old link points to the code base of 1.11. Since then there have been
several changes to the configuration, either new flags added or removed.
Signed-off-by: Suraj Deshmukh <surajd.service@gmail.com>
1. change keyword replicationController to ReplicaSet
2. change word 连续 to 持续
3. other minor changes (e.g: add connection word for readability, add 'shell' for command line input)
This is for part of work in the umbrella issue:
[zh] Umbrella issue: pages out of sync in tasks section #26178
Portforward (L)
content/zh/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md
- Update outdated files in the dev-1.20-ko.3 (1) (#26131)
- Update outdated files in the dev-1.20-ko.3 branch (2) (#26122)
Co-authored-by: seokho-son <shsongist@gmail.com>
Co-authored-by: Jerry Park <jaehwa@gmail.com>
Unless that first part of the paragraph is meant to be more like a subheading (in which case it could use some reformatting), I would like to suggest a minor rephrase so that it becomes a proper sentence and has a predicate.
Thanks for considering this!
This is for part of the items in the following umbrella issue:
[zh] Umbrella issue: pages out of sync in tasks section #26178
Service Account (L)
content/zh/docs/tasks/configure-pod-container/configure-service-account.md
Sync web page for part of the items in umbrella issue:
[zh] Umbrella issue: pages out of sync in tasks section #26178
content/zh/docs/tasks/administer-cluster/nodelocaldns.md
Sync web page for part of the items in umbrella issue:
Misc Batch 2 (S)
content/zh/docs/tasks/debug-application-cluster/debug-pod-replication-controller.md
content/zh/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch.md
content/zh/docs/tasks/access-application-cluster/configure-access-multiple-clusters.md
Just a small clarification on the description of the responsibilities of a Job. Kubernetes doesn't ensure that the Pods are successful, but the scheduler's responsibility is to continue retrying until success. Updating the docs with a slight wording change to more accurately reflect that a Job won't ensure success, but it will continue retrying until success.
Sync web page for part of the items in umbrella issue:
[zh] Umbrella issue: pages out of sync in concepts section #26177
Garbage Collection
content/zh/docs/concepts/workloads/controllers/garbage-collection.md
* [zh] Umbrella issue: pages out of sync in concepts section(Misc Batch 5)
```
[x] content/zh/docs/concepts/policy/resource-quotas.md
[x] content/zh/docs/concepts/cluster-administration/system-metrics.md
[x] content/zh/docs/concepts/cluster-administration/flow-control.md
```
* sync with english version in policy/pod-security-policy.md
Signed-off-by: ydFu ader.ydfu@gmail.com
* [zh] Umbrella issue: pages out of sync in concepts section #26177(Misc Batch 4)
```
[x] content/zh/docs/concepts/architecture/controller.md
[x] content/zh/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md
[x] content/zh/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md
[x] content/zh/docs/concepts/extend-kubernetes/operator.md
```
* sync with english version in home/_index.md
Signed-off-by: ydFu <ader.ydfu@gmail.com>
A few links to the flexVolume documentation do not resolve correctly due
to case sensitivity in the page anchor. This updates those links to
resolve to the correct section of the volumes doc.
Signed-off-by: hasheddan <georgedanielmangum@gmail.com>
* # This is a combination of 14 commits.
# The first commit's message is:
Update patch release manager to patch release team in version skew po… (#17008)
* Update patch release manager to patch release team in skew support policy.
* Adding link for additional patch-release information to version skew policy.
# This is the 2nd commit message:
translate the first part of secret.md document
# This is the 3rd commit message:
translate secrets montados se actualizan automáticamente
# This is the 4th commit message:
translation until Montaje Automatico de Secrets Creados Manualmente
# This is the 5th commit message:
translate restrictions
# This is the 6th commit message:
translate Interacción del Secret y Pod de por Vida
# This is the 7th commit message:
finish first translation version secret.md
# This is the 8th commit message:
fix spaces
# This is the 9th commit message:
checking first part
# This is the 10th commit message:
review translation ultil line 232
# This is the 11th commit message:
fix some issues line 509
# This is the 12th commit message:
finish first revision secret documentation
# This is the 13th commit message:
remove pod-overview
# This is the 14th commit message:
added secret content in spanish content/es/docs/concepts/configuration/secret.md
* “Secret” es un objet Kubernetes
* capital letter in Secret Kubernetes Object
* Remove ` from k8s native objects
Co-Authored-By: Tim Bannister <tim@scalefactory.com>
* Remove ` from k8s native objects
Co-Authored-By: Tim Bannister <tim@scalefactory.com>
* Capitalize Secret
Co-Authored-By: Tim Bannister <tim@scalefactory.com>
* Update reviewer
* Apply suggestions from code review
Co-authored-by: Mitesh Jain <47820816+miteshskj@users.noreply.github.com>
Co-authored-by: Rael Garcia <rael@rael.io>
Co-authored-by: Tim Bannister <tim@scalefactory.com>
`service2` is mapped to `second.bar.com` in the given example
```
- host: second.bar.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: service2
port:
number: 80
```
The stateless application tutorial is littered with unsuitable terminology. This update switches the database technology to a database that has already updated their own terminology as well as removes the terminology from the tutorial itself.
The advanced logging tutorial followup is set to Draft as it will take a larger effort to convert that, and I'm not convinced its even in a working state right now.
A followup PR to the examples repo to be referenced here will be made.
Addresses #22918
Signed-off-by: Paul Czarkowski <username.taken@gmail.com>
address comments in PR
Signed-off-by: Paul Czarkowski <username.taken@gmail.com>
remove confusing comment about dns, we can assume k8s has kube-dns
Signed-off-by: Paul Czarkowski <username.taken@gmail.com>
This PR updates the go.mod file so that tests of the example manifests
are run against the 1.20 branch. The missing test cases for newly added
examples are also added. To perform tests on your local machine, run the
following command on the root of your local clone:
```
go test k8s.io/website/content/en/examples
```
This patch updates the CRI-O documentation to the latest available
version. It also adds a dedicated cgroup driver section to mention that
the configuration has to be in sync.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
- Update outdated files in the dev-1.20-ko.2(p1) (#25915)
- Update outdated files in the dev-1.20-ko.2(p2) (#25916)
- Fix issue with links to already translated ko documents (#25991)
- Translate reference/glossary/dynamic-volume-provisioning.md in Korean (#26047)
Co-authored-by: seokho-son <shsongist@gmail.com>
Co-authored-by: Jerry Park <jaehwa@gmail.com>
Co-authored-by: santachopa <santachopa@naver.com>
* Enhancements in clarity to docs/tasks/access-application-cluster/connecting-frontend-backend.md
* Code review comments
* Reverting name of service due to backed value in Dockerimage
As suggested, removed the language related to common vernacular. I think the documentation is well written in the common labels section, and can possibly be enhanced as more and more of these labels are implemented. So, just a link in the best practice section is sufficient as suggested by you.
as the CKA requires taking these code snippets and using them quickly, spaces can be an issue
add two spaces to begin of line to make pod spec copying even faster (if this is the case)
* Add the code blocks in the Markdown spec to make it easy to read.
* Add description that distinguish between **command** and **output** make it easy to read.
* Adjust description in Kubernetes components for smoother reading.
Signed-off-by: ydFu <ader.ydfu@gmail.com>
I added a <br> after the end of the third bullet and backed out all of the other changes I suggested in the original pull request. I think this better matches the author's original intent. The only difference now between what's currently published and this edit is the line break coded after the third bullet.
The current explanation is misleading (for example: for consistently decreasing resource usage, the HPA stabilization does not impact the frequency of its actuation). This has confused users recently ( https://github.com/kubernetes/kubernetes/issues/96671 )
This change allows announcements to have an expiry date and / or a
“do not show until” date.
Separating this out also leaves room for future changes to enforce
a set of approvers.
Co-Authored-By: Karen Bradshaw <kbhawkey@gmail.com>
* Add Code blocks in the Markdown spec to make it easy to read.
* Uniform case and adjustment description for smoother writing.
Signed-off-by: ydFu <ader.ydfu@gmail.com>
The `/var/lib/kubelet/pod-resources/kubelet.sock` is required by device monitoring agent but not device plugin.
This word `plugin` is ambiguous.
plugin -> device monitoring agent
- Translate reference/glossary/persistent-volume.md in Korean (#25474)
- Update outdated files in the dev-1.20-ko.1 branch (1) (#25576)
- Translate blog/dont-panic-kubernetes-and-docker into Korean (#25596)
- Update outdated files in the dev-1.20-ko.1 branch(3) (#25728)
- Ko: enhance tutorials//cluster-intro translation (#25754)
- Fix Outdated files in the dev-1.20-ko.1 branch (2) (#25765)
Co-authored-by: seokho-son <shsongist@gmail.com>
Co-authored-by: jmyung <jesang.myung@gmail.com>
Co-authored-by: Jerry Park <jaehwa@gmail.com>
Co-authored-by: santachopa <santachopa@naver.com>
Co-authored-by: SEUNGHYUN KO <kosehy@gmail.com>
Co-authored-by: June Yi <gochist@gmail.com>
Add containerd installation on Debian
write docker keyring in a seprate file
Co-authored-by: Tim Bannister <tim@scalefactory.com>
edit comment
Co-authored-by: Tim Bannister <tim@scalefactory.com>
remove sudo
Co-authored-by: Tim Bannister <tim@scalefactory.com>
The language "For all service accounts in the "qa" namespace" in the example is confusing namespaces and groups. Language fixed to disambiguate between group and namespace. An additional example provided which uses both the group ("dev") AND the namespace ("development") to further illustrate this point
There is no need to restart the kubelet as part of configuring kubeadm to set the cgroup driver. At this point in the setup instructions, the kubernetes cluster isn't even up and running yet, as kubeadm init hasn't been run. The previous step even says the kubelet is in a crashloop waiting for kubeadm.
The improvements include:
- timezone-aware date formatting
- if getJSON() errors during a build, we can now ignore it
- You can set Hugo configuration via environment variables even if
snake_case.
sync with english version
Update debug-running-pod.md
apply suggestions
Update debug-running-pod.md
sync the left part
Update debug-running-pod.md
apply suggestions
Adding instructions for how to validate kubectl binaries against checksum files (Linux, MacOS, Windows)
Updating links to download from https://dl.k8s.io/
Updating Linux-specific install instructions to use install command, and
macOS-specific instructions to chown root the install to provide a trusted
kubectl.
Adding note annotation around optional download instructions
Markdown updates
* Updating numbered lists to use markdown syntax ("1." for each entry), should make it easier to add and remove list items in future
* Adding some syntax highlighting to the command snippets
Correcting "PowerShell" spelling
fixes: https://github.com/kubernetes/website/issues/25040
Signed-off-by: Nate W <4453979+nate-double-u@users.noreply.github.com>
resond to PR comments
change publish date and respond to some comments
respond to some more comments
fix formatting
fix image links and respond to comment
respond to formatting comments
remove gks reference
fix metrics names
remove gcp
pr feedback
* All base64 commands need `-w0` argument or else the base64_encoded_ca bash variable will contain space chars (" ") where newlines were
* All sed command are missing final "/" at the end of the expression. Command fails with the following error
```bash
/bin/sed: -e expression #1, char 95: unterminated `s' command
```
The code example uses `preferredDuringScheduling...`, as opposed to
`requiredDuringScheduling...`. So the anti-affinity rule is a soft one. It
says that the pod _should_ not be scheduled on [...] rather than that the pod
_cannot_ be scheduled on [...].
command: ["start", "--host", "\$(MY_SERVICE_NAME)"]
The back slash causes an error. So removed it.
The error detail below
```
Error: rawResources failed to read Resources: YAML file [deployment.yaml] encounters a format error.
error converting YAML to JSON: yaml: line 18: found unknown escape character
```
```
kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
```
sync with english version
Apply suggestions from code review
Co-authored-by: Qiming Teng <tengqim@cn.ibm.com>
Co-authored-by: Qiming Teng <tengqim@cn.ibm.com>
adjust the format to rearrange the numbered list which is not continue
Update expose-external-ip-address.md
Update expose-external-ip-address.md
Update expose-external-ip-address.md
indent by 3 spaces from line 51 to 57
The "Set up the Docker daemon" step fails because the initial files have not been created. I copied the step of creating of the /etc/docker/ folder from the CentOS/RHEL instructions to the Ubuntu Instructions
A missing newline after the last list item caused an important note about plugins being hidden to be append to an unrelated note about debugging nodes.
The buildkit-cli-for-kubectl project aims to provide a drop-in replacement for
`docker build` with compatible UX. Some users looking to migrate off
dockershim may find this CLI plugin useful.
Signed-off-by: Daniel Hiltgen <hiltgend@vmware.com>
The description talks about the server one way and the overview text talks about it a bit differently.
This change aligns them and make them easier to understand in my opinion.
Adds a `caution` note that SSH key pairs do not establish trust between
clients and servers. A secondary method is required to establish trust
between an SSH client and host server, such as fixed `known_hosts` file.
Clients which do not establish adequate trust are vulnerable to "man in
the middle" impersonation attacks.
Signed-off-by: Adam Kaplan <adam.kaplan@redhat.com>
For someone following the guide to setup a cluster, sudo is being used in the initial commands, I think it would great to carry this on till the last commands.
If the EBS volume is partitioned, you have to specify which partition to
mount or the mounting process will fail.
```
mount: /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-<volumeid>: wrong fs type, bad option, bad superblock on /dev/nvme3n1, missing codepage or helper program, or other error.
```
This PR adds a paragraph explaining the insecure by default nature of k8s secrets, and points users at the documentation to turn on encryption at rest and RBAC.
I think a second page needs to be created showing the correct combination of RBAC rules for various cases, which should eventually replace the link to the RBAC documentation.
In 1.20 an optional metrics endpoint is added to the scheduler that
allows administrators to see the requested pod resources and the
imposed pod limits as metrics to better understand the capacity and
usage of the system under the pod model. Describe how these metrics
can be seen under cluster-administration.
fixing content in tabs on adding-windows-nodes.md
reworking tabbed content on adding-windows-nodes.md
intro-windows-in-kubernetes.md
Updating Windows pause image details
PR feedback for adding-windows-nodes.md
Updating comments got GMSA on containerD
Update the kubeadm upgrade documentation:
- Re-order the steps to drain / upgrade kubelet/kubectl / uncordon
- Remove the detailed output from apply and plan.
- Use {{ skew ... }} markers so that we can avoid more verbose changes
to the document.
- Apply minor / general cleanup (whitespace, missing ':')
The command resided under "kubeadm alpha certs".
It will be promoted to GA in 1.20 as "kubeadm certs".
The existing command "kubeadm alpha" will
remain present for one more release, but it will be hidden
from documentation as it is deprecated.
Default for periodSeconds field of startupProbe resource is 10 seconds.
$ kubectl explain pod.spec.containers.startupProbe.periodSeconds
KIND: Pod
VERSION: v1
FIELD: periodSeconds <integer>
DESCRIPTION:
How often (in seconds) to perform the probe. Default to 10 seconds. Minimum
value is 1.
As suggested by BenTheElder, we need to add a cloudbuild.yaml spec that
will be used by the prow job to automate the container build and push
the tags.
Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
@git submodule status --recursive | awk '/^[+-]/ {printf "\033[31mWARNING\033[0m Submodule not initialized: \033[34m%s\033[0m\n",$$2}' 1>&2
@git submodule status --recursive | awk '/^[+-]/ {err = 1; printf "\033[31mWARNING\033[0m Submodule not initialized: \033[34m%s\033[0m\n",$$2} END { if (err != 0) print "You need to run \033[32mmake module-init\033[0m to initialize missing modules first"; exit err }' 1>&2
@@ -9,14 +9,14 @@ Herzlich willkommen! Dieses Repository enthält alle Assets, die zur Erstellung
Sie können auf die Schaltfläche **Fork** im oberen rechten Bereich des Bildschirms klicken, um eine Kopie dieses Repositorys in Ihrem GitHub-Konto zu erstellen. Diese Kopie wird als *Fork* bezeichnet. Nehmen Sie die gewünschten Änderungen an Ihrem Fork vor. Wenn Sie bereit sind, diese Änderungen an uns zu senden, gehen Sie zu Ihrem Fork und erstellen Sie eine neue Pull-Anforderung, um uns darüber zu informieren.
Sobald Ihre Pull-Anfrage erstellt wurde, übernimmt ein Rezensent von Kubernetes die Verantwortung für klares, umsetzbares Feedback. Als Eigentümer des Pull-Request **liegt es in Ihrer Verantwortung Ihren Pull-Reqest enstsprechend des Feedbacks, dass Sie vom Kubernetes-Reviewer erhalten haben abzuändern.** Beachten Sie auch, dass Sie am Ende mehr als einen Rezensenten von Kubernetes erhalten, der Ihnen Feedback gibt, oder dass Sie Rückmeldungen von einem Rezensenten von Kubernetes erhalten, der sich von demjenigen unterscheidet, der ursprünglich für das Feedback zugewiesen wurde. In einigen Fällen kann es vorkommen, dass einer Ihrer Prüfer bei Bedarf eine technische Überprüfung von einem [Kubernetes Tech-Reviewer](https://github.com/kubernetes/website/wiki/tech-reviewers) anfordert. Reviewer geben ihr Bestes, um zeitnah Feedback zu geben, die Antwortzeiten können jedoch je nach den Umständen variieren.
Sobald Ihre Pull-Anfrage erstellt wurde, übernimmt ein Rezensent von Kubernetes die Verantwortung für klares, umsetzbares Feedback. Als Eigentümer des Pull-Request **liegt es in Ihrer Verantwortung Ihren Pull-Reqest entsprechend des Feedbacks, dass Sie vom Kubernetes-Reviewer erhalten haben abzuändern.** Beachten Sie auch, dass Sie am Ende mehr als einen Rezensenten von Kubernetes erhalten, der Ihnen Feedback gibt, oder dass Sie Rückmeldungen von einem Rezensenten von Kubernetes erhalten, der sich von demjenigen unterscheidet, der ursprünglich für das Feedback zugewiesen wurde. In einigen Fällen kann es vorkommen, dass einer Ihrer Prüfer bei Bedarf eine technische Überprüfung von einem [Kubernetes Tech-Reviewer](https://github.com/kubernetes/website/wiki/tech-reviewers) anfordert. Reviewer geben ihr Bestes, um zeitnah Feedback zu geben, die Antwortzeiten können jedoch je nach den Umständen variieren.
Weitere Informationen zum Beitrag zur Kubernetes-Dokumentation finden Sie unter:
@@ -37,6 +37,13 @@ Um die Kubernetes-Website lokal laufen zu lassen, empfiehlt es sich, ein speziel
> Wenn Sie die Website lieber lokal ohne Docker ausführen möchten, finden Sie weitere Informationen unter [Website lokal mit Hugo ausführen](#Die-Site-lokal-mit-Hugo-ausführen).
Das benötigte [Docsy Hugo theme](https://github.com/google/docsy#readme) muss als git submodule installiert werden:
```
#Füge das Docsy submodule hinzu
git submodule update --init --recursive --depth 1
```
Wenn Sie Docker [installiert](https://www.docker.com/get-started) haben, erstellen Sie das Docker-Image `kubernetes-hugo` lokal:
```bash
@@ -55,9 +62,18 @@ make container-serve
Hugo-Installationsanweisungen finden Sie in der [offiziellen Hugo-Dokumentation](https://gohugo.io/getting-started/installing/). Stellen Sie sicher, dass Sie die Hugo-Version installieren, die in der Umgebungsvariablen `HUGO_VERSION` in der Datei [`netlify.toml`](netlify.toml#L9) angegeben ist.
Das benötigte [Docsy Hugo theme](https://github.com/google/docsy#readme) muss als git submodule installiert werden:
```
#Füge das Docsy submodule hinzu
git submodule update --init --recursive --depth 1
```
So führen Sie die Site lokal aus, wenn Sie Hugo installiert haben:
```bash
# Installieren der JavaScript Abhängigkeiten
npm ci
make serve
```
@@ -65,7 +81,7 @@ Dadurch wird der lokale Hugo-Server an Port 1313 gestartet. Öffnen Sie Ihren Br
## Community, Diskussion, Beteiligung und Unterstützung
Erfahren Sie auf der [Community-Seite](http://kubernetes.io/community/) wie Sie mit der Kubernetes-Community interagieren können.
Erfahren Sie auf der [Community-Seite](https://kubernetes.io/community/) wie Sie mit der Kubernetes-Community interagieren können.
Sie können die Betreuer dieses Projekts unter folgender Adresse erreichen:
@@ -17,9 +17,9 @@ Los revisores harán todo lo posible para proporcionar toda la información nece
Para obtener más información sobre cómo contribuir a la documentación de Kubernetes, puede consultar:
* [Empezando a contribuir](https://kubernetes.io/docs/contribute/start/)
* [Visualizando sus cambios en su entorno local](http://kubernetes.io/docs/contribute/intermediate#view-your-changes-locally)
* [Utilizando las plantillas de las páginas](http://kubernetes.io/docs/contribute/style/page-content-types/)
* [Guía de estilo de la documentación](http://kubernetes.io/docs/contribute/style/style-guide/)
* [Visualizando sus cambios en su entorno local](https://kubernetes.io/docs/contribute/intermediate#view-your-changes-locally)
* [Utilizando las plantillas de las páginas](https://kubernetes.io/docs/contribute/style/page-content-types/)
* [Guía de estilo de la documentación](https://kubernetes.io/docs/contribute/style/style-guide/)
* [Traduciendo la documentación de Kubernetes](https://kubernetes.io/docs/contribute/localization/)
## Levantando el sitio web kubernetes.io en su entorno local con Docker
@@ -30,6 +30,17 @@ El método recomendado para levantar una copia local del sitio web kubernetes.io
> Si prefiere levantar el sitio web sin utilizar **Docker**, puede seguir las instrucciones disponibles en la sección [Levantando kubernetes.io en local con Hugo](#levantando-kubernetesio-en-local-con-hugo).
**`Nota`: Para el procedimiento de construir una imagen de Docker e iniciar el servidor.**
El sitio web de Kubernetes utiliza Docsy Hugo theme. Se sugiere que se instale si aún no se ha hecho, los **submódulos** y otras dependencias de herramientas de desarrollo ejecutando el siguiente comando de `git`:
```bash
# pull de los submódulos del repositorio
git submodule update --init --recursive --depth 1
```
Si identifica que `git` reconoce una cantidad innumerable de cambios nuevos en el proyecto, la forma más simple de solucionarlo es cerrando y volviendo a abrir el proyecto en el editor. Los submódulos son automáticamente detectados por `git`, pero los plugins usados por los editores pueden tener dificultades para ser cargados.
Una vez tenga Docker [configurado en su máquina](https://www.docker.com/get-started), puede construir la imagen de Docker `kubernetes-hugo` localmente ejecutando el siguiente comando en la raíz del repositorio:
```bash
@@ -73,4 +84,4 @@ La participación en la comunidad de Kubernetes está regulada por el [Código d
Kubernetes es posible gracias a la participación de la comunidad y la documentación es vital para facilitar el acceso al proyecto.
Agradecemos muchísimo sus contribuciones a nuestro sitio web y nuestra documentación.
Agradecemos muchísimo sus contribuciones a nuestro sitio web y nuestra documentación.
W tym repozytorium znajdziesz wszystko, czego potrzebujesz do zbudowania [strony internetowej Kubernetesa wraz z dokumentacją](https://kubernetes.io/). Bardzo nam miło, że chcesz wziąć udział w jej współtworzeniu!
## Twój wkład w dokumentację
+ [Twój wkład w dokumentację](#twój-wkład-w-dokumentację)
+ [Informacje o wersjach językowych](#informacje-o-wersjach-językowych)
Możesz kliknąć w przycisk **Fork** w prawym górnym rogu ekranu, aby stworzyć kopię tego repozytorium na swoim koncie GitHub. Taki rodzaj kopii (odgałęzienia) nazywa się *fork*. Zmieniaj w nim, co chcesz, a kiedy będziesz już gotowy/a przesłać te zmiany do nas, przejdź do swojej kopii i stwórz nowy *pull request*, abyśmy zostali o tym poinformowani.
# Jak używać tego repozytorium
Po stworzeniu *pull request*, jeden z recenzentów projektu Kubernetes podejmie się przekazania jasnych wskazówek pozwalających podjąć następne działania. Na Tobie, jako właścicielu *pull requesta*, **spoczywa odpowiedzialność za wprowadzenie poprawek zgodnie z uwagami recenzenta.** Może też się zdarzyć, że swoje uwagi zgłosi więcej niż jeden recenzent, lub że recenzję będzie robił ktoś inny, niż ten, kto został przydzielony na początku. W niektórych przypadkach, jeśli zajdzie taka potrzeba, recenzent może poprosić dodatkowo o recenzję jednego z [recenzentów technicznych](https://github.com/kubernetes/website/wiki/Tech-reviewers). Recenzenci zrobią wszystko, aby odpowiedzieć sprawnie, ale konkretny czas odpowiedzi zależy od wielu czynników.
Możesz uruchomić serwis lokalnie poprzez Hugo (Extended version) lub ze środowiska kontenerowego. Zdecydowanie zalecamy korzystanie z kontenerów, bo dzięki temu lokalna wersja będzie spójna z tym, co jest na oficjalnej stronie.
Więcej informacji na temat współpracy przy tworzeniu dokumentacji znajdziesz na stronach:
| [README po angielsku](README.md) | [README po francusku](README-fr.md) |
| [README po koreańsku](README-ko.md) | [README po niemiecku](README-de.md) |
| [README po portugalsku](README-pt.md) | [README w hindi](README-hi.md) |
| [README po hiszpańsku](README-es.md) | [README po indonezyjsku](README-id.md) |
| [README po chińsku](README-zh.md) | [README po japońsku](README-ja.md) |
| [README po wietnamsku](README-vi.md) | [README po rosyjsku](README-ru.md) |
| [README po włosku](README-it.md) | [README po ukraińsku](README-uk.md) |
| | |
Przed rozpoczęciem zainstaluj niezbędne zależności. Sklonuj repozytorium i przejdź do odpowiedniego katalogu:
## Jak uruchomić lokalną kopię strony przy pomocy Dockera?
Zalecaną metodą uruchomienia serwisu internetowego Kubernetesa lokalnie jest użycie specjalnego obrazu [Dockera](https://docker.com), który zawiera generator stron statycznych [Hugo](https://gohugo.io).
> Użytkownicy Windows będą potrzebowali dodatkowych narzędzi, które mogą zainstalować przy pomocy [Chocolatey](https://chocolatey.org).
> Jeśli wolisz uruchomić serwis lokalnie bez Dockera, przeczytaj [jak uruchomić serwis lokalnie przy pomocy Hugo](#jak-uruchomić-lokalną-kopię-strony-przy-pomocy-hugo) poniżej.
Strona Kubernetesa używa [Docsy Hugo theme](https://github.com/google/docsy#readme). Nawet jeśli planujesz uruchomić serwis w środowisku kontenerowym, zalecamy pobranie podmodułów i innych zależności za pomocą polecenia:
Jeśli [zainstalowałeś i uruchomiłeś](https://www.docker.com/get-started) już Dockera, zbuduj obraz `kubernetes-hugo` lokalnie:
```
# pull in the Docsy submodule
git submodule update --init --recursive --depth 1
```
```bash
## Uruchomienie serwisu w kontenerze
Aby zbudować i uruchomić serwis wewnątrz środowiska kontenerowego, wykonaj następujące polecenia:
```
make container-image
```
Po zbudowaniu obrazu, możesz uruchomić serwis lokalnie:
```bash
make container-serve
```
@@ -62,29 +47,106 @@ Aby obejrzeć zawartość serwisu otwórz w przeglądarce adres http://localhost
## Jak uruchomić lokalną kopię strony przy pomocy Hugo?
Zajrzyj do [oficjalnej dokumentacji Hugo](https://gohugo.io/getting-started/installing/) po instrukcję instalacji. Upewnij się, że instalujesz rozszerzoną wersję Hugo, określoną przez zmienną środowiskową `HUGO_VERSION` w pliku [`netlify.toml`](netlify.toml#L9).
Upewnij się, że zainstalowałeś odpowiednią wersję Hugo "extended", określoną przez zmienną środowiskową `HUGO_VERSION` w pliku [`netlify.toml`](netlify.toml#L10).
Aby uruchomić serwis lokalnie po instalacji Hugo, napisz:
Aby uruchomić i przetestować serwis lokalnie, wykonaj:
```bash
# install dependencies
npm ci
make serve
```
Zostanie uruchomiony lokalny serwer Hugo na porcie 1313. Otwórz w przeglądarce adres http://localhost:1313, aby obejrzeć zawartość serwisu. Po każdej zmianie plików źródłowych, Hugo automatycznie aktualizuje stronę i odświeża jej widok w przeglądarce.
## Społeczność, listy dyskusyjne, uczestnictwo i wsparcie
## Budowanie dokumentacji źródłowej API
Zajrzyj na stronę [społeczności](http://kubernetes.io/community/), aby dowiedzieć się, jak możesz zaangażować się w jej działania.
Budowanie dokumentacji źródłowej API zostało opisane w [angielskiej wersji pliku README.md](README.md#building-the-api-reference-pages).
## Rozwiązywanie problemów
### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version
Z przyczyn technicznych, Hugo jest rozprowadzany w dwóch wersjach. Aktualny serwis używa tylko wersji **Hugo Extended**. Na stronie z [wydaniami](https://github.com/gohugoio/hugo/releases) poszukaj archiwum z `extended` w nazwie. Dla potwierdzenia, uruchom `hugo version` i poszukaj słowa `extended`.
### Błąd w środowisku macOS: "too many open files"
Jeśli po uruchomieniu `make serve` na macOS widzisz następujący błąd:
```
ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files
make: *** [serve] Error 1
```
sprawdź aktualny limit otwartych plików:
`launchctl limit maxfiles`
Uruchom następujące polecenia: (na podstawie https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c):
```shell
#!/bin/sh
# These are the original gist links, linking to my gists now.
Udział w działaniach społeczności Kubernetes jest regulowany przez [Kodeks postępowania](code-of-conduct.md).
Możesz kliknąć w przycisk **Fork** w prawym górnym rogu ekranu, aby stworzyć kopię tego repozytorium na swoim koncie GitHub. Taki rodzaj kopii (odgałęzienia) nazywa się *fork*. Zmieniaj w nim, co chcesz, a kiedy będziesz już gotowy/a przesłać te zmiany do nas, przejdź do swojej kopii i stwórz nowy *pull request*, abyśmy zostali o tym poinformowani.
## Dziękujemy!
Po stworzeniu *pull request*, jeden z recenzentów projektu Kubernetes podejmie się przekazania jasnych wskazówek pozwalających podjąć następne działania. Na Tobie, jako właścicielu *pull requesta*, **spoczywa odpowiedzialność za wprowadzenie poprawek zgodnie z uwagami recenzenta.**
Może też się zdarzyć, że swoje uwagi zgłosi więcej niż jeden recenzent, lub że recenzję będzie robił ktoś inny, niż ten, kto został przydzielony na początku.
W niektórych przypadkach, jeśli zajdzie taka potrzeba, recenzent może poprosić dodatkowo o recenzję jednego z [recenzentów technicznych](https://github.com/kubernetes/website/wiki/Tech-reviewers). Recenzenci zrobią wszystko, aby odpowiedzieć sprawnie, ale konkretny czas odpowiedzi zależy od wielu czynników.
Więcej informacji na temat współpracy przy tworzeniu dokumentacji znajdziesz na stronach:
* [Udział w rozwijaniu dokumentacji](https://kubernetes.io/docs/contribute/)
Udział w działaniach społeczności Kubernetesa jest regulowany przez [Kodeks postępowania CNCF](https://github.com/cncf/foundation/blob/master/code-of-conduct-languages/pl.md).
# Dziękujemy!
Kubernetes rozkwita dzięki zaangażowaniu społeczności — doceniamy twój wkład w tworzenie naszego serwisu i dokumentacji!
Bemvindos! Este repositório abriga todos os recursos necessários para criar o [site e documentação do Kubernetes](https://kubernetes.io/). Estamos muito satisfeitos por você querer contribuir!
Bem-vindos! Este repositório contém todos os recursos necessários para criar o [website e documentação do Kubernetes](https://kubernetes.io/). Estamos muito satisfeitos por você querer contribuir!
## Contribuindo com os documentos
# Utilizando este repositório
Você pode clicar no botão **Fork** na área superior direita da tela para criar uma cópia desse repositório na sua conta do GitHub. Esta cópia é chamada de *fork*. Faça as alterações desejadas no seu fork e, quando estiver pronto para enviar as alterações para nós, vá até o fork e crie uma nova solicitação de pull para nos informar sobre isso.
Você pode executar o website localmente utilizando o Hugo (versão Extended), ou você pode executa-ló em um container runtime. É altamente recomendável utilizar um container runtime, pois garante a consistência na implantação do website real.
Depois que seu **pull request** for criado, um revisor do Kubernetes assumirá a responsabilidade de fornecer um feedback claro e objetivo. Como proprietário do pull request, **é sua responsabilidade modificar seu pull request para abordar o feedback que foi fornecido a você pelo revisor do Kubernetes.** Observe também que você pode acabar tendo mais de um revisor do Kubernetes para fornecer seu feedback ou você pode acabar obtendo feedback de um revisor do Kubernetes que é diferente daquele originalmente designado para lhe fornecer feedback. Além disso, em alguns casos, um de seus revisores pode solicitar uma revisão técnica de um [revisor de tecnologia Kubernetes](https://github.com/kubernetes/website/wiki/Tech-reviewers) quando necessário. Os revisores farão o melhor para fornecer feedback em tempo hábil, mas o tempo de resposta pode variar de acordo com as circunstâncias.
## Pré-requisitos
Para usar este repositório, você precisa instalar:
- [npm](https://www.npmjs.com/)
- [Go](https://golang.org/)
- [Hugo (versão Extended)](https://gohugo.io/)
- Um container runtime, por exemplo [Docker](https://www.docker.com/).
Antes de você iniciar, instale as dependências, clone o repositório e navegue até o diretório:
O website do Kubernetes utiliza o [tema Docsy Hugo](https://github.com/google/docsy#readme). Mesmo se você planeje executar o website em um container, é altamente recomendado baixar os submódulos e outras dependências executando o seguinte comando:
```
# Baixar o submódulo Docsy
git submodule update --init --recursive --depth 1
```
## Executando o website usando um container
Para executar o build do website em um container, execute o comando abaixo para criar a imagem do container e executa-lá:
```
make container-image
make container-serve
```
Abra seu navegador em http://localhost:1313 para visualizar o website. Conforme você faz alterações nos arquivos fontes, o Hugo atualiza o website e força a atualização do navegador.
## Executando o website localmente utilizando o Hugo
Consulte a [documentação oficial do Hugo](https://gohugo.io/getting-started/installing/) para instruções de instalação do Hugo. Certifique-se de instalar a versão do Hugo especificada pela variável de ambiente `HUGO_VERSION` no arquivo [`netlify.toml`](netlify.toml#L9).
Para executar o build e testar o website localmente, execute:
```bash
# instalar dependências
npm ci
make serve
```
Isso iniciará localmente o Hugo na porta 1313. Abra o seu navegador em http://localhost:1313 para visualizar o website. Conforme você faz alterações nos arquivos fontes, o Hugo atualiza o website e força uma atualização no navegador.
## Construindo a página de referência da API
A página de referência da API localizada em `content/en/docs/reference/kubernetes-api` é construída a partir da especificação do Swagger utilizando https://github.com/kubernetes-sigs/reference-docs/tree/master/gen-resourcesdocs.
Siga os passos abaixo para atualizar a página de referência para uma nova versão do Kubernetes:
OBS: modifique o "v1.20" no exemplo a seguir pela versão a ser atualizada
1. Obter o submódulo `kubernetes-resources-reference`:
```
git submodule update --init --recursive --depth 1
```
2. Criar a nova versão da API no submódulo e adicionar à especificação do Swagger:
4. Ajustar os arquivos `toc.yaml` e `fields.yaml` para refletir as mudanças entre as duas versões.
5. Em seguida, gerar as páginas:
```
make api-reference
```
Você pode validar o resultado localmente gerando e disponibilizando o site a partir da imagem do container:
```
make container-image
make container-serve
```
Abra o seu navegador em http://localhost:1313/docs/reference/kubernetes-api/ para visualizar a página de referência da API.
6. Quando todas as mudanças forem refletidas nos arquivos de configuração `toc.yaml` e `fields.yaml`, crie um pull request com a nova página de referência de API.
## Troubleshooting
### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version
Por motivos técnicos, o Hugo é disponibilizado em dois conjuntos de binários. O website atual funciona apenas na versão **Hugo Extended**. Na [página de releases](https://github.com/gohugoio/hugo/releases) procure por arquivos com `extended` no nome. Para confirmar, execute `hugo version` e procure pela palavra `extended`.
### Troubleshooting macOS for too many open files
Se você executar o comando `make serve` no macOS e retornar o seguinte erro:
```
ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files
make: *** [serve] Error 1
```
Verifique o limite atual para arquivos abertos:
`launchctl limit maxfiles`
Em seguida, execute os seguintes comandos (adaptado de https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c):
```shell
#!/bin/sh
# Esse são os links do gist original, vinculados ao meu gists agora.
Esta solução funciona tanto para o MacOS Catalina quanto para o MacOS Mojave.
### Erro de "Out of Memory"
Se você executar o comando `make container-serve` e retornar o seguinte erro:
```
make: *** [container-serve] Error 137
```
Verifique a quantidade de memória disponível para o agente de execução de contêiner. No caso do Docker Desktop para macOS, abra o menu "Preferences..." -> "Resources..." e tente disponibilizar mais memória.
# Comunidade, discussão, contribuição e apoio
Saiba mais sobre a comunidade Kubernetes SIG Docs e reuniões na [página da comunidade](http://kubernetes.io/community/).
Você também pode entrar em contato com os mantenedores deste projeto em:
- [Slack](https://kubernetes.slack.com/messages/sig-docs) ([Obter o convide para o este slack](https://slack.k8s.io/))
Você pode clicar no botão **Fork** na área superior direita da tela para criar uma cópia desse repositório na sua conta do GitHub. Esta cópia é chamada de *fork*. Faça as alterações desejadas no seu fork e, quando estiver pronto para enviar as alterações para nós, vá até o fork e crie um novo **pull request** para nos informar sobre isso.
Depois que seu **pull request** for criado, um revisor do Kubernetes assumirá a responsabilidade de fornecer um feedback claro e objetivo. Como proprietário do pull request, **é sua responsabilidade modificar seu pull request para atender ao feedback que foi fornecido a você pelo revisor do Kubernetes.**
Observe também que você pode acabar tendo mais de um revisor do Kubernetes para fornecer seu feedback ou você pode acabar obtendo feedback de um outro revisor do Kubernetes diferente daquele originalmente designado para lhe fornecer o feedback.
Além disso, em alguns casos, um de seus revisores pode solicitar uma revisão técnica de um [revisor técnico do Kubernetes](https://github.com/kubernetes/website/wiki/Tech-reviewers) quando necessário. Os revisores farão o melhor para fornecer feedbacks em tempo hábil, mas o tempo de resposta pode variar de acordo com as circunstâncias.
Para mais informações sobre como contribuir com a documentação do Kubernetes, consulte:
* [Comece a contribuir](https://kubernetes.io/docs/contribute/start/)
* [Preparando suas alterações na documentação](http://kubernetes.io/docs/contribute/intermediate#view-your-changes-locally)
* [Usando Modelos de Página](http://kubernetes.io/docs/contribute/style/page-templates/)
* [Contribua com a documentação do Kubernetes](https://kubernetes.io/docs/contribute/)
* [Tipos de conteúdo de página](https://kubernetes.io/docs/contribute/style/page-content-types/)
* [Guia de Estilo da Documentação](http://kubernetes.io/docs/contribute/style/style-guide/)
* [Localizando documentação do Kubernetes](https://kubernetes.io/docs/contribute/localization/)
Você pode contactar os mantenedores da localização em Português em:
Você pode contatar os mantenedores da localização em Português em:
* Felipe ([GitHub - @femrtnz](https://github.com/femrtnz))
A maneira recomendada de executar o site do Kubernetes localmente é executar uma imagem especializada do [Docker](https://docker.com) que inclui o gerador de site estático [Hugo](https://gohugo.io).
> Se você está rodando no Windows, você precisará de mais algumas ferramentas que você pode instalar com o [Chocolatey](https://chocolatey.org). `choco install make`
> Se você preferir executar o site localmente sem o Docker, consulte [Executando o site localmente usando o Hugo](#executando-o-site-localmente-usando-o-hugo) abaixo.
Se você tiver o Docker [em funcionamento](https://www.docker.com/get-started), crie a imagem do Docker do `kubernetes-hugo` localmente:
```bash
make container-image
```
Depois que a imagem foi criada, você pode executar o site localmente:
```bash
make container-serve
```
Abra seu navegador para http://localhost:1313 para visualizar o site. Conforme você faz alterações nos arquivos de origem, Hugo atualiza o site e força a atualização do navegador.
## Executando o site localmente usando o Hugo
Veja a [documentação oficial do Hugo](https://gohugo.io/getting-started/installing/) para instruções de instalação do Hugo. Certifique-se de instalar a versão do Hugo especificada pela variável de ambiente `HUGO_VERSION` no arquivo [`netlify.toml`](netlify.toml#L9).
Para executar o site localmente quando você tiver o Hugo instalado:
```bash
make serve
```
Isso iniciará o servidor Hugo local na porta 1313. Abra o navegador para http://localhost:1313 para visualizar o site. Conforme você faz alterações nos arquivos de origem, Hugo atualiza o site e força a atualização do navegador.
## Comunidade, discussão, contribuição e apoio
Aprenda a se envolver com a comunidade do Kubernetes na [página da comunidade](http://kubernetes.io/community/).
Você pode falar com os mantenedores deste projeto:
Добро пожаловать! Данный репозиторий содержит все необходимые файлы для сборки [сайта Kubernetes и документации](https://kubernetes.io/). Мы благодарим вас за старания!
Данный репозиторий содержит все необходимые файлы для сборки [сайта Kubernetes и документации](https://kubernetes.io/). Мы благодарим вас за желание внести свой вклад!
## Запуск сайта с помощью Hugo
# Использование этого репозитория
Обратитесь к [официальной документации Hugo](https://gohugo.io/getting-started/installing/), чтобы установить Hugo. Убедитесь, что вы установили правильную версию Hugo, которая устанавливается в переменной окружения `HUGO_VERSION` в файле [`netlify.toml`](netlify.toml#L10).
Запустить сайт локально можно с помощью Hugo (Extended version) или же в исполняемой среде для контейнеров. Мы настоятельно рекомендуем воспользоваться контейнерной средой, поскольку она обеспечивает консистивность развёртывания с оригинальным сайтом.
После установки Hugo, чтобы запустить сайт, выполните в консоли:
## Предварительные требования
```bash
Чтобы работать с этим репозиторием, понадобятся следующие компоненты, установленные локально:
- [npm](https://www.npmjs.com/)
- [Go](https://golang.org/)
- [Hugo (Extended version)](https://gohugo.io/)
- Исполняемая среда для контейнеров вроде [Docker](https://www.docker.com/)
Перед тем, как начать, установите зависимости. Склонируйте репозиторий и перейдите в его директорию:
Эта команда запустит сервер Hugo на порту 1313. Откройте браузер и перейдите по ссылке http://localhost:1313, чтобы открыть сайт. Если вы отредактируете исходные файлы сайта, Hugo автоматически применит изменения и обновит страницу в браузере.
Сайт Kubernetes использует [тему для Hugo под названием Docsy](https://github.com/google/docsy). Даже если вы планируете запускать сайт в контейнере, мы настоятельно рекомендуем загрузить соответствующий подмодуль и другие зависимости для разработки, выполнив следующую команду:
## Сообщество, обсуждение, вклад и поддержка
```
# загружаем Git-подмодуль Docsy
git submodule update --init --recursive --depth 1
```
Узнайте, как поучаствовать в жизни сообщества Kubernetes на [странице сообщества](http://kubernetes.io/community/).
## Запуск сайта в контейнере
Вы можете связаться с сопровождающими этого проекта по следующим ссылкам:
Чтобы собрать сайт в контейнере, выполните следующие команды — они собирают образ контейнера и запускают его:
- [Канал в Slack](https://kubernetes.slack.com/messages/sig-docs)
Откройте браузер и перейдите по ссылке http://localhost:1313, чтобы увидеть сайт. Если вы отредактируете исходные файлы сайта, Hugo автоматически обновит сам сайт и выполнит обновление страницы в браузере.
## Запуск сайта с помощью Hugo
Нажмите на кнопку **Fork** в правом верхнем углу, чтобы создать копию этого репозитория в ваш GitHub-аккаунт. Эта копия называется *форк-репозиторием*. Делайте любые изменения в вашем форк-репозитории, и когда вы будете готовы опубликовать изменения, откройте форк-репозиторий и создайте новый пулреквест, чтобы уведомить нас.
Убедитесь, что вы установили расширенную версию Hugo (extended version): она определена в переменной окружения `HUGO_VERSION` в файле [`netlify.toml`](netlify.toml#L10).
После того, как вы отправите пулреквест, ревьювер Kubernetes даст по нему обратную связь. Вы, как автор пулреквеста, **должны обновить свой пулреквест после его рассмотрения ревьювером Kubernetes.**
Чтобы собрать и протестировать сайт локально, выполните:
Вполне возможно, что более одного ревьювера Kubernetes оставят свои комментарии или даже может быть так, что новый комментарий ревьювера Kubernetes будет отличаться от первоначального назначенного ревьювера. Кроме того, в некоторых случаях один из ревьюверов может запросить технический обзор у [технического ревьювера Kubernetes](https://github.com/kubernetes/website/wiki/Tech-reviewers), если это будет необходимо. Ревьюверы сделают все возможное, чтобы как можно оперативно оставить свои предложения и пожелания, но время ответа может варьироваться в зависимости от обстоятельств.
```bash
# install dependencies
npm ci
make serve
```
Эти команды запустят локальный сервер Hugo на порту 1313. Откройте браузер и перейдите по ссылке http://localhost:1313, чтобы увидеть сайт. Если вы отредактируете исходные файлы сайта, Hugo автоматически обновит сам сайт и выполнит обновление страницы в браузере.
## Решение проблем
### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version
По техническим причинам Hugo поставляется с двумя наборами бинарников. Текущий сайт Kubernetes работает только в версии **Hugo Extended**. На [странице релизов](https://github.com/gohugoio/hugo/releases) ищите архивы со словом `extended` в названии. Чтобы убедиться в корректности, запустите команду `hugo version` и найдите в выводе слово `extended`.
### Решение проблемы на macOS с "too many open files"
Если вы запускаете `make serve` на macOS и получаете следующую ошибку:
```
ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files
make: *** [serve] Error 1
```
Попробуйте проверить текущий лимит для открытых файлов:
`launchctl limit maxfiles`
Затем выполните следующие команды (они взяты и адаптированы из https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c):
```shell
#!/bin/sh
# Ссылки на оригинальные gist-файлы закомментированы в пользу моих адаптированных.
Данное решение работает для macOS Catalina и Mojave.
# Участие в SIG Docs
Узнайте о Kubernetes-сообществе SIG Docs и его встречах на [странице сообщества](https://github.com/kubernetes/community/tree/master/sig-docs#meetings).
Вы можете связаться с сопровождающими этот проект по следующим ссылкам:
- [Канал в Slack](https://kubernetes.slack.com/messages/sig-docs) ([получите приглашение в этот Slack](https://slack.k8s.io/))
Нажмите на кнопку **Fork** в правом верхнем углу, чтобы создать копию этого репозитория для вашего GitHub-аккаунта. Эта копия называется *форк-репозиторием*. Делайте любые изменения в своем форк-репозитории и, когда будете готовы опубликовать изменения, зайдите в свой форк-репозиторий и создайте новый pull-запрос (PR), чтобы уведомить нас.
После того, как вы отправите pull-запрос, ревьювер из проекта Kubernetes даст по нему обратную связь. Вы, как автор pull-запроса, **должны обновить свой PR после его рассмотрения ревьювером Kubernetes.**
Вполне возможно, что более одного ревьювера Kubernetes оставят свои комментарии. Может быть даже так, что вы будете получать обратную связь уже не от того ревьювера, что был первоначально вам назначен. Кроме того, в некоторых случаях один из ревьюверов может запросить техническую рецензию от [технического ревьювера Kubernetes](https://github.com/kubernetes/website/wiki/Tech-reviewers), если это потребуется. Ревьюверы сделают все возможное, чтобы как можно оперативнее оставить свои предложения и пожелания, но время ответа может варьироваться в зависимости от обстоятельств.
Узнать подробнее о том, как поучаствовать в документации Kubernetes, вы можете по ссылкам ниже:
@@ -42,21 +121,22 @@ hugo server --buildFuture
* [Руководство по оформлению документации](https://kubernetes.io/docs/contribute/style/style-guide/)
* [Руководство по локализации Kubernetes](https://kubernetes.io/docs/contribute/localization/)
Участие в сообществе Kubernetes регулируется [кодексом поведения CNCF](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
Участие в сообществе Kubernetes регулируется [кодексом поведения CNCF](https://github.com/cncf/foundation/blob/master/code-of-conduct-languages/ru.md).
## Спасибо!
# Спасибо!
Kubernetes процветает благодаря сообществу и мы ценим ваш вклад в сайт и документацию!
<!-- This repository contains the assets required to build the [Kubernetes website and documentation](https://kubernetes.io/). We're glad that you want to contribute! -->
Вітаємо! В цьому репозиторії міститься все необхідне для роботи над [сайтом і документацією Kubernetes](https://kubernetes.io/). Ми щасливі, що ви хочете зробити свій внесок!
<!-- This will start the local Hugo server on port 1313. Open up your browser to http://localhost:1313 to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh. -->
@@ -82,4 +83,4 @@ hugo server --buildFuture
## Дякуємо!
<!-- Kubernetes thrives on community participation, and we appreciate your contributions to our website and our documentation! -->
Долучення до спільноти - запорука успішного розвитку Kubernetes. Ми цінуємо ваш внесок у наш сайт і документацію!
Долучення до спільноти - запорука успішного розвитку Kubernetes. Ми цінуємо ваш внесок у наш сайт і документацію!
This repository contains the assets required to build the [Kubernetes website and documentation](https://kubernetes.io/). We're glad that you want to contribute!
@@ -174,7 +174,7 @@ Learn more about SIG Docs Kubernetes community and meetings on the [community pa
You can also reach the maintainers of this project at:
This repository contains the assets required to build the [Kubernetes website and documentation](https://kubernetes.io/). We're glad that you want to contribute!
# Using this repository
- [Contributing to the docs](#contributing-to-the-docs)
- [Localization ReadMes](#localization-readmemds)
## Using this repository
You can run the website locally using Hugo (Extended version), or you can run it in a container runtime. We strongly recommend using the container runtime, as it gives deployment consistency with the live website.
@@ -19,14 +22,14 @@ To use this repository, you need the following installed locally:
Before you start, install the dependencies. Clone the repository and navigate to the directory:
The Kubernetes website uses the [Docsy Hugo theme](https://github.com/google/docsy#readme). Even if you plan to run the website in a container, we strongly recommend pulling in the submodule and other development dependencies by running the following:
To build the site in a container, run the following to build the container image and run it:
```
```bash
make container-image
make container-serve
```
Open up your browser to http://localhost:1313 to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh.
If you see errors, it probably means that the hugo container did not have enough computing resources available. To solve it, increase the amount of allowed CPU and memory usage for Docker on your machine ([MacOSX](https://docs.docker.com/docker-for-mac/#resources) and [Windows](https://docs.docker.com/docker-for-windows/#resources)).
Open up your browser to <http://localhost:1313> to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh.
## Running the website locally using Hugo
@@ -54,9 +59,47 @@ npm ci
make serve
```
This will start the local Hugo server on port 1313. Open up your browser to http://localhost:1313 to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh.
This will start the local Hugo server on port 1313. Open up your browser to <http://localhost:1313> to view the website. As you make changes to the source files, Hugo updates the website and forces a browser refresh.
## Building the API reference pages
The API reference pages located in `content/en/docs/reference/kubernetes-api` are built from the Swagger specification, using <https://github.com/kubernetes-sigs/reference-docs/tree/master/gen-resourcesdocs>.
To update the reference pages for a new Kubernetes release (replace v1.20 in the following examples with the release to update to):
1. Pull the `kubernetes-resources-reference` submodule:
3. In `api-ref-assets/config/`, adapt the files `toc.yaml` and `fields.yaml` to reflect the changes of the new release.
4. Next, build the pages:
```bash
make api-reference
```
You can test the results locally by making and serving the site from a container image:
```bash
make container-image
make container-serve
```
In a web browser, go to <http://localhost:1313/docs/reference/kubernetes-api/> to view the API reference.
5. When all changes of the new contract are reflected into the configuration files `toc.yaml` and `fields.yaml`, create a Pull Request with the newly generated API reference pages.
## Troubleshooting
### error: failed to transform resource: TOCSS: failed to transform "scss/main.scss" (text/x-scss): this feature is not available in your current Hugo version
Hugo is shipped in two set of binaries for technical reasons. The current website runs based on the **Hugo Extended** version only. In the [release page](https://github.com/gohugoio/hugo/releases) look for archives with `extended` in the name. To confirm, run `hugo version` and look for the word `extended`.
@@ -65,7 +108,7 @@ Hugo is shipped in two set of binaries for technical reasons. The current websit
If you run `make serve` on macOS and receive the following error:
```
```bash
ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files
make: *** [serve] Error 1
```
@@ -74,9 +117,9 @@ Try checking the current limit for open files:
`launchctl limit maxfiles`
Then run the following commands (adapted from https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c):
Then run the following commands (adapted from <https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c>):
```
```shell
#!/bin/sh
# These are the original gist links, linking to my gists now.
Learn more about SIG Docs Kubernetes community and meetings on the [community page](https://github.com/kubernetes/community/tree/master/sig-docs#meetings).
@@ -107,39 +149,39 @@ You can also reach the maintainers of this project at:
- [Slack](https://kubernetes.slack.com/messages/sig-docs) [Get an invite for this Slack](https://slack.k8s.io/)
You can click the **Fork** button in the upper-right area of the screen to create a copy of this repository in your GitHub account. This copy is called a *fork*. Make any changes you want in your fork, and when you are ready to send those changes to us, go to your fork and create a new pull request to let us know about it.
You can click the **Fork** button in the upper-right area of the screen to create a copy of this repository in your GitHub account. This copy is called a _fork_. Make any changes you want in your fork, and when you are ready to send those changes to us, go to your fork and create a new pull request to let us know about it.
Once your pull request is created, a Kubernetes reviewer will take responsibility for providing clear, actionable feedback. As the owner of the pull request, **it is your responsibility to modify your pull request to address the feedback that has been provided to you by the Kubernetes reviewer.**
Once your pull request is created, a Kubernetes reviewer will take responsibility for providing clear, actionable feedback. As the owner of the pull request, **it is your responsibility to modify your pull request to address the feedback that has been provided to you by the Kubernetes reviewer.**
Also, note that you may end up having more than one Kubernetes reviewer provide you feedback or you may end up getting feedback from a Kubernetes reviewer that is different than the one initially assigned to provide you feedback.
Furthermore, in some cases, one of your reviewers might ask for a technical review from a Kubernetes tech reviewer when needed. Reviewers will do their best to provide feedback in a timely fashion but response time can vary based on circumstances.
Furthermore, in some cases, one of your reviewers might ask for a technical review from a Kubernetes tech reviewer when needed. Reviewers will do their best to provide feedback in a timely fashion but response time can vary based on circumstances.
For more information about contributing to the Kubernetes documentation, see:
* [Contribute to Kubernetes docs](https://kubernetes.io/docs/contribute/)
Participation in the Kubernetes community is governed by the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
# Thank you!
## Thank you
Kubernetes thrives on community participation, and we appreciate your contributions to our website and our documentation!
- Replace these instructions and the following text with your content.
- Replace `<angle bracket placeholders>` with actual values. For example, you would update `date: <yyyy>-<mm>-<dd>` to look something like `date: 2021-10-21`.
- For convenience, use third-party tools to author and collaborate on your content.
- To save time and effort in reviews, check your content's spelling, grammar, and style before contributing.
- Feel free to ask for assistance in the Kubernetes Slack channel, [#sig-docs-blog](https://kubernetes.slack.com/archives/CJDHVD54J).
-->
Replace this first line of your content with one to three sentences that summarize the blog post.
## This is a section heading
To help the reader, organize your content into sections that contain about three to six paragraphs.
If you're documenting commands, separate the commands from the outputs, like this:
1. Verify that the Secret exists by running the following command:
```shell
kubectl get secrets
```
The response should be like this:
```shell
NAME TYPE DATA AGE
mysql-pass-c57bb4t7mf Opaque 1 9s
```
You're free to create any sections you like. Below are a few common patterns we see at the end of blog posts.
## What’s next?
This optional section describes the future of the thing you've just described in the post.
## How can I learn more?
This optional section provides links to more information. Please avoid promoting and over-represent your organization.
## How do I get involved?
An optional section that links to resources for readers to get involved, and acknowledgments of individual contributors, such as:
* [The name of a channel on Slack, #a-channel](https://<a-workspace>.slack.com/messages/<a-channel>)
* [A link to a "contribute" page with more information](<https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact>).
* Acknowledgements and thanks to the contributors. <person's name> ([<github id>](https://github.com/<github id>)) who did X, Y, and Z.
* Those interested in getting involved with the design and development of <project>, join the [<name of the SIG>](https://github.com/project/community/tree/master/<sig-group>). We’re rapidly growing and always welcome new contributors.
<h3>Die Gewissheit, dass Kubernetes überall und für alle gut funktioniert.</h3>
<p>Verbinden Sie sich mit der Kubernetes-Community in unserem <ahref="http://slack.k8s.io/">Slack Kanal</a>, <ahref="https://discuss.kubernetes.io/">Diskussionsforum</a>, oder beteiligen Sie sich an der <ahref="https://groups.google.com/forum/#!forum/kubernetes-dev"> Kubernetes-dev-Google-Gruppe</a>. Eine wöchentlichesCommunity-Meeting findet per Videokonferenz statt, um den Stand der Dinge zu diskutieren, folgen Sie
<p>Verbinden Sie sich mit der Kubernetes-Community in unserem <ahref="http://slack.k8s.io/">Slack Kanal</a>, <ahref="https://discuss.kubernetes.io/">Diskussionsforum</a>, oder beteiligen Sie sich an der <ahref="https://groups.google.com/g/kubernetes-dev"> Kubernetes-dev-Google-Gruppe</a>. Eine wöchentlichesCommunity-Meeting findet per Videokonferenz statt, um den Stand der Dinge zu diskutieren, folgen Sie
<ahref="https://github.com/kubernetes/community/blob/master/events/community-meeting.md">diesen Anweisungen</a> für Informationen wie Sie teilnehmen können.</p>
<p>Sie können Kubernetes auch auf der ganzen Welt über unsere
<ahref="https://www.meetup.com/topics/kubernetes/">Kubernetes Meetup Community</a> und der
@@ -23,7 +23,7 @@ Dieser Verhaltenskodex gilt sowohl innerhalb von Projekträumen als auch in öff
Fälle von missbräuchlichem, belästigendem oder anderweitig unzumutbarem Verhalten in Kubernetes können gemeldet werden, indem Sie sich an das [Kubernetes Komitee für Verhaltenskodex](https://git.k8s.io/community/committee-code-of-conduct) wenden unter <conduct@kubernetes.io>. Für andere Projekte wenden Sie sich bitte an einen CNCF-Projektbetreuer oder an unseren Mediator, Mishi Choudhary <mishi@linux.com>.
Dieser Verhaltenskodex wurde aus dem Contributor Covenant übernommen (http://contributor-covenant.org), Version 1.2.0, verfügbar unter http://contributor-covenant.org/version/1/2/0/
Dieser Verhaltenskodex wurde aus dem Contributor Covenant übernommen (https://contributor-covenant.org), Version 1.2.0, verfügbar unter https://contributor-covenant.org/version/1/2/0/
@@ -147,7 +147,8 @@ Die zweite ist, die interne Node-Liste des Node Controllers mit der Liste der ve
Wenn ein Node in einer Cloud-Umgebung ausgeführt wird und sich in einem schlechten Zustand befindet, fragt der Node Controller den Cloud-Anbieter, ob die virtuelle Maschine für diesen Node noch verfügbar ist. Wenn nicht, löscht der Node Controller den Node aus seiner Node-Liste.
Der dritte ist die Überwachung des Zustands der Nodes. Der Node Controller ist dafür verantwortlich,
die NodeReady-Bedingung von NodeStatus auf ConditionUnknown zu aktualisieren, wenn ein wenn ein Node unerreichbar wird (der Node Controller empfängt aus irgendeinem Grund keine Herzschläge mehr, z.B. weil der Node heruntergefahren ist) und später alle Pods aus dem Node zu entfernen (und diese ordnungsgemäss zu beenden), wenn der Node weiterhin unzugänglich ist. (Die Standard-Timeouts sind 40s, um ConditionUnknown zu melden und 5 Minuten, um mit der Evakuierung der Pods zu beginnen).
die NodeReady-Bedingung von NodeStatus auf ConditionUnknown zu aktualisieren, wenn ein Node unerreichbar wird (der Node Controller empfängt aus irgendeinem Grund keine Herzschläge mehr, z.B. weil der Node heruntergefahren ist) und später alle Pods aus dem Node zu entfernen (und diese ordnungsgemäss zu beenden), wenn der Node weiterhin unzugänglich ist. (Die Standard-Timeouts sind 40s, um ConditionUnknown zu melden und 5 Minuten, um mit der Evakuierung der Pods zu beginnen).
Der Node Controller überprüft den Zustand jedes Nodes alle `--node-monitor-period` Sekunden.
@@ -26,7 +26,7 @@ Die Add-Ons in den einzelnen Kategorien sind alphabetisch sortiert - Die Reihenf
* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) ermöglicht das nahtlose Verbinden von Kubernetes mit einer Reihe an CNI-Plugins wie z.B. Calico, Canal, Flannel, Romana, oder Weave.
* [Contiv](http://contiv.github.io) bietet konfigurierbares Networking (Native L3 auf BGP, Overlay mit vxlan, Klassisches L2, Cisco-SDN/ACI) für verschiedene Anwendungszwecke und auch umfangreiches Policy-Framework. Das Contiv-Projekt ist vollständig [Open Source](http://github.com/contiv). Der [installer](http://github.com/contiv/install) bietet sowohl kubeadm als auch nicht-kubeadm basierte Installationen.
* [Contrail](http://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), basierend auf [Tungsten Fabric](https://tungsten.io), ist eine Open Source, multi-Cloud Netzwerkvirtualisierungs- und Policy-Management Plattform. Contrail und Tungsten Fabric sind mit Orechstratoren wie z.B. Kubernetes, OpenShift, OpenStack und Mesos integriert und bieten Isolationsmodi für Virtuelle Maschinen, Container (bzw. Pods) und Bare Metal workloads.
* [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kubernetes.md) ist ein Overlay-Network-Provider der mit Kubernetes genutzt werden kann.
* [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) ist ein Overlay-Network-Provider der mit Kubernetes genutzt werden kann.
* [Knitter](https://github.com/ZTE/Knitter/) ist eine Network-Lösung die Mehrfach-Network in Kubernetes ermöglicht.
* [Multus](https://github.com/Intel-Corp/multus-cni) ist ein Multi-Plugin für Mehrfachnetzwerk-Unterstützung um alle CNI-Plugins (z.B. Calico, Cilium, Contiv, Flannel), zusätzlich zu SRIOV-, DPDK-, OVS-DPDK- und VPP-Basierten Workloads in Kubernetes zu unterstützen.
* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) bietet eine Integration zwischen VMware NSX-T und einem Orchestator wie z.B. Kubernetes. Außerdem bietet es eine Integration zwischen NSX-T und Containerbasierten CaaS/PaaS-Plattformen wie z.B. Pivotal Container Service (PKS) und OpenShift.
Diese Sektion umfasst verschiedene Optionen zum Einrichten und Betrieb von Kubernetes.
Verschiedene Kubernetes Lösungen haben verschiedene Anforderungen: Einfache Wartung, Sicherheit, Kontrolle, verfügbare Resourcen und erforderliches Fachwissen zum Betrieb und zur Verwaltung dess folgende Diagramm zeigt die möglichen Abstraktionen eines Kubernetes-Clusters und ob eine Abstraktion selbst verwaltet oder von einem Anbieter verwaltet wird.
Verschiedene Kubernetes Lösungen haben verschiedene Anforderungen: Einfache Wartung, Sicherheit, Kontrolle, verfügbare Resourcen und erforderliches Fachwissen zum Betrieb und zur Verwaltung. Das folgende Diagramm zeigt die möglichen Abstraktionen eines Kubernetes-Clusters und ob eine Abstraktion selbst verwaltet oder von einem Anbieter verwaltet wird.
Sie können einen Kubernetes-Cluster auf einer lokalen Maschine, Cloud, On-Prem Datacenter bereitstellen; oder wählen Sie einen verwalteten Kubernetes-Cluster. Sie können auch eine individuelle Lösung über eine grosse Auswahl an Cloud Anbietern oder Bare-Metal-Umgebungen nutzen.
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna20"buttonid="desktopKCButton">Attend KubeCon NA virtually on November 17-20, 2020</a>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccncna21"buttonid="desktopKCButton">Attend KubeCon North America on October 11-15, 2021</a>
<br>
<br>
<br>
<br>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu21"buttonid="desktopKCButton">Attend KubeCon EU virtually on May 4 – 7, 2021</a>
<ahref="https://events.linuxfoundation.org/kubecon-cloudnativecon-europe-2022/?utm_source=kubernetes.io&utm_medium=nav&utm_campaign=kccnceu22"buttonid="desktopKCButton">Attend KubeCon Europe on May 17-20, 2022</a>
@@ -26,7 +26,7 @@ On the other hand, CNI is more philosophically aligned with Kubernetes. It's far
Additionally, it's trivial to wrap a CNI plugin and produce a more customized CNI plugin — it can be done with a simple shell script. CNM is much more complex in this regard. This makes CNI an attractive option for rapid development and iteration. Early prototypes have proven that it's possible to eject almost 100% of the currently hard-coded network logic in kubelet into a plugin.
We investigated [writing a "bridge" CNM driver](https://groups.google.com/forum/#!topic/kubernetes-sig-network/5MWRPxsURUw) for Docker that ran CNI drivers. This turned out to be very complicated. First, the CNM and CNI models are very different, so none of the "methods" lined up. We still have the global vs. local and key-value issues discussed above. Assuming this driver would declare itself local, we have to get info about logical networks from Kubernetes.
We investigated [writing a "bridge" CNM driver](https://groups.google.com/g/kubernetes-sig-network/c/5MWRPxsURUw) for Docker that ran CNI drivers. This turned out to be very complicated. First, the CNM and CNI models are very different, so none of the "methods" lined up. We still have the global vs. local and key-value issues discussed above. Assuming this driver would declare itself local, we have to get info about logical networks from Kubernetes.
Unfortunately, Docker drivers are hard to map to other control planes like Kubernetes. Specifically, drivers are not told the name of the network to which a container is being attached — just an ID that Docker allocates internally. This makes it hard for a driver to map back to any concept of network that exists in another system.
@@ -34,6 +34,6 @@ This and other issues have been brought up to Docker developers by network vendo
For all of these reasons we have chosen to invest in CNI as the Kubernetes plugin model. There will be some unfortunate side-effects of this. Most of them are relatively minor (for example, `docker inspect` will not show an IP address), but some are significant. In particular, containers started by `docker run` might not be able to communicate with containers started by Kubernetes, and network integrators will have to provide CNI drivers if they want to fully integrate with Kubernetes. On the other hand, Kubernetes will get simpler and more flexible, and a lot of the ugliness of early bootstrapping (such as configuring Docker to use our bridge) will go away.
As we proceed down this path, we’ll certainly keep our eyes and ears open for better ways to integrate and simplify. If you have thoughts on how we can do that, we really would like to hear them — find us on [slack](http://slack.k8s.io/) or on our [network SIG mailing-list](https://groups.google.com/forum/#!forum/kubernetes-sig-network).
As we proceed down this path, we’ll certainly keep our eyes and ears open for better ways to integrate and simplify. If you have thoughts on how we can do that, we really would like to hear them — find us on [slack](http://slack.k8s.io/) or on our [network SIG mailing-list](https://groups.google.com/g/kubernetes-sig-network).
@@ -20,21 +20,14 @@ For example, if we want to require scheduling on a node that is in the us-centra
```
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "failure-domain.beta.kubernetes.io/zone"
operator: In
values: ["us-central1-a"]
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "failure-domain.beta.kubernetes.io/zone"
operator: In
values: ["us-central1-a"]
```
@@ -44,21 +37,14 @@ Preferred rules mean that if nodes match the rules, they will be chosen first, a
```
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "failure-domain.beta.kubernetes.io/zone"
operator: In
values: ["us-central1-a"]
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "failure-domain.beta.kubernetes.io/zone"
operator: In
values: ["us-central1-a"]
```
@@ -67,21 +53,14 @@ Node anti-affinity can be achieved by using negative operators. So for instance
```
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "failure-domain.beta.kubernetes.io/zone"
operator: NotIn
values: ["us-central1-a"]
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "failure-domain.beta.kubernetes.io/zone"
operator: NotIn
values: ["us-central1-a"]
```
@@ -99,7 +78,7 @@ The kubectl command allows you to set taints on nodes, for example:
```
kubectl taint nodes node1 key=value:NoSchedule
```
```
creates a taint that marks the node as unschedulable by any pods that do not have a toleration for taint with key key, value value, and effect NoSchedule. (The other taint effects are PreferNoSchedule, which is the preferred version of NoSchedule, and NoExecute, which means any pods that are running on the node when the taint is applied will be evicted unless they tolerate the taint.) The toleration you would add to a PodSpec to have the corresponding pod tolerate this taint would look like this
@@ -107,15 +86,11 @@ creates a taint that marks the node as unschedulable by any pods that do not hav
```
tolerations:
- key: "key"
operator: "Equal"
value: "value"
effect: "NoSchedule"
tolerations:
- key: "key"
operator: "Equal"
value: "value"
effect: "NoSchedule"
```
@@ -138,21 +113,13 @@ Let’s look at an example. Say you have front-ends in service S1, and they comm
@@ -56,13 +56,13 @@ Cri-containerd uses containerd to manage the full container lifecycle and all co
Let’s use an example to demonstrate how cri-containerd works for the case when Kubelet creates a single-container pod:
1.1.Kubelet calls cri-containerd, via the CRI runtime service API, to create a pod;
2.2.cri-containerd uses containerd to create and start a special [pause container](https://www.ianlewis.org/en/almighty-pause-container) (the _sandbox container_) and put that container inside the pod’s cgroups and namespace (steps omitted for brevity);
3.3.cri-containerd configures the pod’s network namespace using CNI;
4.4.Kubelet subsequently calls cri-containerd, via the CRI image service API, to pull the application container image;
5.5.cri-containerd further uses containerd to pull the image if the image is not present on the node;
6.6.Kubelet then calls cri-containerd, via the CRI runtime service API, to create and start the application container inside the pod using the pulled container image;
7.7.cri-containerd finally calls containerd to create the application container, put it inside the pod’s cgroups and namespace, then to start the pod’s new application container.
1. Kubelet calls cri-containerd, via the CRI runtime service API, to create a pod;
2. cri-containerd uses containerd to create and start a special [pause container](https://www.ianlewis.org/en/almighty-pause-container) (the _sandbox container_) and put that container inside the pod’s cgroups and namespace (steps omitted for brevity);
3. cri-containerd configures the pod’s network namespace using CNI;
4. Kubelet subsequently calls cri-containerd, via the CRI image service API, to pull the application container image;
5. cri-containerd further uses containerd to pull the image if the image is not present on the node;
6. Kubelet then calls cri-containerd, via the CRI runtime service API, to create and start the application container inside the pod using the pulled container image;
7. cri-containerd finally calls containerd to create the application container, put it inside the pod’s cgroups and namespace, then to start the pod’s new application container.
After these steps, a pod and its corresponding application container is created and running.
@@ -140,7 +140,7 @@ The local persistent volume beta feature is not complete by far. Some notable en
## Complementary features
[Pod priority and preemption](/docs/concepts/configuration/pod-priority-preemption/) is another Kubernetes feature that is complementary to local persistent volumes. When your application uses local storage, it must be scheduled to the specific node where the local volume resides. You can give your local storage workload high priority so if that node ran out of room to run your workload, Kubernetes can preempt lower priority workloads to make room for it.
[Pod priority and preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/) is another Kubernetes feature that is complementary to local persistent volumes. When your application uses local storage, it must be scheduled to the specific node where the local volume resides. You can give your local storage workload high priority so if that node ran out of room to run your workload, Kubernetes can preempt lower priority workloads to make room for it.
[Pod disruption budget](/docs/concepts/workloads/pods/disruptions/) is also very important for those workloads that must maintain quorum. Setting a disruption budget for your workload ensures that it does not drop below quorum due to voluntary disruption events, such as node drains during upgrade.
@@ -94,7 +94,7 @@ JOSH BERKUS: That goes into release notes. I mean, keep in mind that one of the
However, stuff happens, and we do occasionally have to do those. And so far, our main way to identify that to people actually is in the release notes. If you look at [the current release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#no-really-you-must-do-this-before-you-upgrade), there are actually two things in there right now that are sort of breaking changes.
One of them is the bit with [priority and preemption](/docs/concepts/configuration/pod-priority-preemption/) in that preemption being on by default now allows badly behaved users of the system to cause trouble in new ways. I'd actually have to look at the release notes to see what the second one was...
One of them is the bit with [priority and preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/) in that preemption being on by default now allows badly behaved users of the system to cause trouble in new ways. I'd actually have to look at the release notes to see what the second one was...
TIM PEPPER: The [JSON capitalization case sensitivity](https://github.com/kubernetes/kubernetes/issues/64612).
@@ -104,7 +104,7 @@ Master and Worker nodes should be protected from overload and resource exhaustio
Resource consumption by the control plane will correlate with the number of pods and the pod churn rate. Very large and very small clusters will benefit from non-default [settings](/docs/reference/command-line-tools-reference/kube-apiserver/) of kube-apiserver request throttling and memory. Having these too high can lead to request limit exceeded and out of memory errors.
On worker nodes, [Node Allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/) should be configured based on a reasonable supportable workload density at each node. Namespaces can be created to subdivide the worker node cluster into multiple virtual clusters with resource CPU and memory [quotas](/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/). Kubelet handling of [out of resource](/docs/tasks/administer-cluster/out-of-resource/) conditions can be configured.
On worker nodes, [Node Allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/) should be configured based on a reasonable supportable workload density at each node. Namespaces can be created to subdivide the worker node cluster into multiple virtual clusters with resource CPU and memory [quotas](/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/). Kubelet handling of [out of resource](/docs/concepts/scheduling-eviction/node-pressure-eviction/) conditions can be configured.
## Security
@@ -166,7 +166,7 @@ Some critical state is held outside etcd. Certificates, container images, and ot
* Cloud provider specific account and configuration data
## Considerations for your production workloads
Anti-affinity specifications can be used to split clustered services across backing hosts, but at this time the settings are used only when the pod is scheduled. This means that Kubernetes can restart a failed node of your clustered application, but does not have a native mechanism to rebalance after a fail back. This is a topic worthy of a separate blog, but supplemental logic might be useful to achieve optimal workload placements after host or worker node recoveries or expansions. The [Pod Priority and Preemption feature](/docs/concepts/configuration/pod-priority-preemption/) can be used to specify a preferred triage in the event of resource shortages caused by failures or bursting workloads.
Anti-affinity specifications can be used to split clustered services across backing hosts, but at this time the settings are used only when the pod is scheduled. This means that Kubernetes can restart a failed node of your clustered application, but does not have a native mechanism to rebalance after a fail back. This is a topic worthy of a separate blog, but supplemental logic might be useful to achieve optimal workload placements after host or worker node recoveries or expansions. The [Pod Priority and Preemption feature](/docs/concepts/scheduling-eviction/pod-priority-preemption/) can be used to specify a preferred triage in the event of resource shortages caused by failures or bursting workloads.
For stateful services, external attached volume mounts are the standard Kubernetes recommendation for a non-clustered service (e.g., a typical SQL database). At this time Kubernetes managed snapshots of these external volumes is in the category of a [roadmap feature request](https://docs.google.com/presentation/d/1dgxfnroRAu0aF67s-_bmeWpkM1h2LCxe6lB1l1oS0EQ/edit#slide=id.g3ca07c98c2_0_47), likely to align with the Container Storage Interface (CSI) integration. Thus performing backups of such a service would involve application specific, in-pod activity that is beyond the scope of this document. While awaiting better Kubernetes support for a snapshot and backup workflow, running your database service in a VM rather than a container, and exposing it to your Kubernetes workload may be worth considering.
@@ -176,7 +176,7 @@ Cluster-distributed stateful services (e.g., Cassandra) can benefit from splitti
[Logs](/docs/concepts/cluster-administration/logging/) and [metrics](/docs/tasks/debug-application-cluster/resource-usage-monitoring/) (if collected and persistently retained) are valuable to diagnose outages, but given the variety of technologies available it will not be addressed in this blog. If Internet connectivity is available, it may be desirable to retain logs and metrics externally at a central location.
Your production deployment should utilize an automated installation, configuration and update tool (e.g., [Ansible](https://github.com/kubernetes-incubator/kubespray), [BOSH](https://github.com/cloudfoundry-incubator/kubo-deployment), [Chef](https://github.com/chef-cookbooks/kubernetes), [Juju](/docs/getting-started-guides/ubuntu/installation/), [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/), [Puppet](https://forge.puppet.com/puppetlabs/kubernetes), etc.). A manual process will have repeatability issues, be labor intensive, error prone, and difficult to scale. [Certified distributions](https://www.cncf.io/certification/software-conformance/#logos) are likely to include a facility for retaining configuration settings across updates, but if you implement your own install and config toolchain, then retention, backup and recovery of the configuration artifacts is essential. Consider keeping your deployment components and settings under a version control system such as Git.
Your production deployment should utilize an automated installation, configuration and update tool (e.g., [Ansible](https://github.com/kubernetes-incubator/kubespray), [BOSH](https://github.com/cloudfoundry-incubator/kubo-deployment), [Chef](https://github.com/chef-cookbooks/kubernetes), [Juju](/docs/getting-started-guides/ubuntu/installation/), [kubeadm](/docs/reference/setup-tools/kubeadm/), [Puppet](https://forge.puppet.com/puppetlabs/kubernetes), etc.). A manual process will have repeatability issues, be labor intensive, error prone, and difficult to scale. [Certified distributions](https://www.cncf.io/certification/software-conformance/#logos) are likely to include a facility for retaining configuration settings across updates, but if you implement your own install and config toolchain, then retention, backup and recovery of the configuration artifacts is essential. Consider keeping your deployment components and settings under a version control system such as Git.
@@ -17,7 +17,7 @@ Let’s dive into the key features of this release:
## Simplified Kubernetes Cluster Management with kubeadm in GA
Most people who have gotten hands-on with Kubernetes have at some point been hands-on with kubeadm. It's an essential tool for managing the cluster lifecycle, from creation to configuration to upgrade; and now kubeadm is officially GA. [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/) handles the bootstrapping of production clusters on existing hardware and configuring the core Kubernetes components in a best-practice-manner to providing a secure yet easy joining flow for new nodes and supporting easy upgrades. What’s notable about this GA release are the now graduated advanced features, specifically around pluggability and configurability. The scope of kubeadm is to be a toolbox for both admins and automated, higher-level system and this release is a significant step in that direction.
Most people who have gotten hands-on with Kubernetes have at some point been hands-on with kubeadm. It's an essential tool for managing the cluster lifecycle, from creation to configuration to upgrade; and now kubeadm is officially GA. [kubeadm](/docs/reference/setup-tools/kubeadm/) handles the bootstrapping of production clusters on existing hardware and configuring the core Kubernetes components in a best-practice-manner to providing a secure yet easy joining flow for new nodes and supporting easy upgrades. What’s notable about this GA release are the now graduated advanced features, specifically around pluggability and configurability. The scope of kubeadm is to be a toolbox for both admins and automated, higher-level system and this release is a significant step in that direction.
Kubernetes is well-known for running scalable workloads. It scales your workloads based on their resource usage. When a workload is scaled up, more instances of the application get created. When the application is critical for your product, you want to make sure that these new instances are scheduled even when your cluster is under resource pressure. One obvious solution to this problem is to over-provision your cluster resources to have some amount of slack resources available for scale-up situations. This approach often works, but costs more as you would have to pay for the resources that are idle most of the time.
[Pod priority and preemption](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/) is a scheduler feature made generally available in Kubernetes 1.14 that allows you to achieve high levels of scheduling confidence for your critical workloads without overprovisioning your clusters. It also provides a way to improve resource utilization in your clusters without sacrificing the reliability of your essential workloads.
[Pod priority and preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/) is a scheduler feature made generally available in Kubernetes 1.14 that allows you to achieve high levels of scheduling confidence for your critical workloads without overprovisioning your clusters. It also provides a way to improve resource utilization in your clusters without sacrificing the reliability of your essential workloads.
@@ -58,7 +58,7 @@ Take maven project as example, adding the following dependencies into your depen
Then we can make use of the provided builder libraries to write your own controller.
For example, the following one is a simple controller prints out node information
on watch notification, see complete example [here](https://github.com/kubernetes-client/java/blob/master/examples/src/main/java/io/kubernetes/client/examples/ControllerExample.java):
on watch notification, see complete example [here](https://github.com/kubernetes-client/java/blob/master/examples/examples-release-13/src/main/java/io/kubernetes/client/examples/ControllerExample.java):
@@ -31,9 +31,9 @@ Standard labels are used by Kubernetes components to support some features. For
The labels are reaching general availability in this release. Kubernetes components have been updated to populate the GA and beta labels and to react to both. However, if you are using the beta labels in your pod specs for features such as node affinity, or in your custom controllers, we recommend that you start migrating them to the new GA labels. You can find the documentation for the new labels here:
@@ -55,7 +55,7 @@ The team has made progress in the last few months that is well worth celebrating
- The K8s-Infrastructure Working Group released an automated billing report that they start every meeting off by reviewing as a group.
- DNS for k8s.io and kubernetes.io are also fully [community-owned](https://groups.google.com/g/kubernetes-dev/c/LZTYJorGh7c/m/u-ydk-yNEgAJ), with community members able to [file issues](https://github.com/kubernetes/k8s.io/issues/new?assignees=&labels=wg%2Fk8s-infra&template=dns-request.md&title=DNS+REQUEST%3A+%3Cyour-dns-record%3E) to manage records.
- The container registry [k8s.gcr.io](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io) is also fully community-owned and available for all Kubernetes subprojects to use.
- The container registry [k8s.gcr.io](https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io) is also fully community-owned and available for all Kubernetes subprojects to use.
- The Kubernetes [publishing-bot](https://github.com/kubernetes/publishing-bot) responsible for keeping k8s.io/kubernetes/staging repositories published to their own top-level repos (For example: [kubernetes/api](https://github.com/kubernetes/api)) runs on a community-owned cluster.
- The gcsweb.k8s.io service used to provide anonymous access to GCS buckets for kubernetes artifacts runs on a community-owned cluster.
- There is also an automated process of promoting all our container images. This includes a fully documented infrastructure, managed by the Kubernetes community, with automated processes for provisioning permissions.
@@ -325,7 +325,7 @@ Now that we have a way to communicate helpful information to users in context,
we're already considering other ways we can use this to improve people's experience with Kubernetes.
A couple areas we're looking at next are warning about [known problematic values](http://issue.k8s.io/64841#issuecomment-395141013)
we cannot reject outright for compatibility reasons, and warning about use of deprecated fields or field values
(like selectors using beta os/arch node labels, [deprecated in v1.14](/docs/reference/kubernetes-api/labels-annotations-taints/#beta-kubernetes-io-arch-deprecated)).
(like selectors using beta os/arch node labels, [deprecated in v1.14](/docs/reference/labels-annotations-taints/#beta-kubernetes-io-arch-deprecated)).
I'm excited to see progress in this area, continuing to make it easier to use Kubernetes.
title: "A Custom Kubernetes Scheduler to Orchestrate Highly Available Applications"
date: 2020-12-21
slug: writing-crl-scheduler
---
**Author**: Chris Seto (Cockroach Labs)
As long as you're willing to follow the rules, deploying on Kubernetes and air travel can be quite pleasant. More often than not, things will "just work". However, if one is interested in travelling with an alligator that must remain alive or scaling a database that must remain available, the situation is likely to become a bit more complicated. It may even be easier to build one's own plane or database for that matter. Travelling with reptiles aside, scaling a highly available stateful system is no trivial task.
Scaling any system has two main components:
1. Adding or removing infrastructure that the system will run on, and
2. Ensuring that the system knows how to handle additional instances of itself being added and removed.
Most stateless systems, web servers for example, are created without the need to be aware of peers. Stateful systems, which includes databases like CockroachDB, have to coordinate with their peer instances and shuffle around data. As luck would have it, CockroachDB handles data redistribution and replication. The tricky part is being able to tolerate failures during these operations by ensuring that data and instances are distributed across many failure domains (availability zones).
One of Kubernetes' responsibilities is to place "resources" (e.g, a disk or container) into the cluster and satisfy the constraints they request. For example: "I must be in availability zone _A_" (see [Running in multiple zones](/docs/setup/best-practices/multiple-zones/#nodes-are-labeled)), or "I can't be placed onto the same node as this other Pod" (see [Affinity and anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)).
As an addition to those constraints, Kubernetes offers [Statefulsets](/docs/concepts/workloads/controllers/statefulset/) that provide identity to Pods as well as persistent storage that "follows" these identified pods. Identity in a StatefulSet is handled by an increasing integer at the end of a pod's name. It's important to note that this integer must always be contiguous: in a StatefulSet, if pods 1 and 3 exist then pod 2 must also exist.
Under the hood, CockroachCloud deploys each region of CockroachDB as a StatefulSet in its own Kubernetes cluster - see [Orchestrate CockroachDB in a Single Kubernetes Cluster](https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html).
In this article, I'll be looking at an individual region, one StatefulSet and one Kubernetes cluster which is distributed across at least three availability zones.
A three-node CockroachCloud cluster would look something like this:
When adding additional resources to the cluster we also distribute them across zones. For the speediest user experience, we add all Kubernetes nodes at the same time and then scale up the StatefulSet.
Note that anti-affinities are satisfied no matter the order in which pods are assigned to Kubernetes nodes. In the example, pods 0, 1 and 2 were assigned to zones A, B, and C respectively, but pods 3 and 4 were assigned in a different order, to zones B and A respectively. The anti-affinity is still satisfied because the pods are still placed in different zones.
To remove resources from a cluster, we perform these operations in reverse order.
We first scale down the StatefulSet and then remove from the cluster any nodes lacking a CockroachDB pod.
Now, remember that pods in a StatefulSet of size _n_ must have ids in the range `[0,n)`. When scaling down a StatefulSet by _m_, Kubernetes removes _m_ pods, starting from the highest ordinals and moving towards the lowest, [the reverse in which they were added](/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees).
Consider the cluster topology below:
As ordinals 5 through 3 are removed from this cluster, the statefulset continues to have a presence across all 3 availability zones.
However, Kubernetes' scheduler doesn't _guarantee_ the placement above as we expected at first.
Our combined knowledge of the following is what lead to this misconception.
* Kubernetes' ability to [automatically spread Pods across zone](/docs/setup/best-practices/multiple-zones/#pods-are-spread-across-zones)
* The behavior that a StatefulSet with _n_ replicas, when Pods are being deployed, they are created sequentially, in order from `{0..n-1}`. See [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees) for more details.
Consider the following topology:
These pods were created in order and they are spread across all availability zones in the cluster. When ordinals 5 through 3 are terminated, this cluster will lose its presence in zone C!
Worse yet, our automation, at the time, would remove Nodes A-2, B-2, and C-2. Leaving CRDB-1 in an unscheduled state as persistent volumes are only available in the zone they are initially created in.
To correct the latter issue, we now employ a "hunt and peck" approach to removing machines from a cluster. Rather than blindly removing Kubernetes nodes from the cluster, only nodes without a CockroachDB pod would be removed. The much more daunting task was to wrangle the Kubernetes scheduler.
## A session of brainstorming left us with 3 options:
### 1. Upgrade to kubernetes 1.18 and make use of Pod Topology Spread Constraints
While this seems like it could have been the perfect solution, at the time of writing Kubernetes 1.18 was unavailable on the two most common managed Kubernetes services in public cloud, EKS and GKE.
Furthermore, [pod topology spread constraints](/docs/concepts/workloads/pods/pod-topology-spread-constraints/) were still a [beta feature in 1.18](https://v1-18.docs.kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) which meant that it [wasn't guaranteed to be available in managed clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#kubernetes_feature_choices) even when v1.18 became available.
The entire endeavour was concerningly reminiscent of checking [caniuse.com](https://caniuse.com/) when Internet Explorer 8 was still around.
### 2. Deploy a statefulset _per zone_.
Rather than having one StatefulSet distributed across all availability zones, a single StatefulSet with node affinities per zone would allow manual control over our zonal topology.
Our team had considered this as an option in the past which made it particularly appealing.
Ultimately, we decided to forego this option as it would have required a massive overhaul to our codebase and performing the migration on existing customer clusters would have been an equally large undertaking.
### 3. Write a custom Kubernetes scheduler.
Thanks to an example from [Kelsey Hightower](https://github.com/kelseyhightower/scheduler) and a blog post from [Banzai Cloud](https://banzaicloud.com/blog/k8s-custom-scheduler/), we decided to dive in head first and write our own [custom Kubernetes scheduler](/docs/tasks/extend-kubernetes/configure-multiple-schedulers/).
Once our proof-of-concept was deployed and running, we quickly discovered that the Kubernetes' scheduler is also responsible for mapping persistent volumes to the Pods that it schedules.
The output of [`kubectl get events`](/docs/tasks/extend-kubernetes/configure-multiple-schedulers/#verifying-that-the-pods-were-scheduled-using-the-desired-schedulers) had led us to believe there was another system at play.
In our journey to find the component responsible for storage claim mapping, we discovered the [kube-scheduler plugin system](/docs/concepts/scheduling-eviction/scheduling-framework/). Our next POC was a `Filter` plugin that determined the appropriate availability zone by pod ordinal, and it worked flawlessly!
Our [custom scheduler plugin](https://github.com/cockroachlabs/crl-scheduler) is open source and runs in all of our CockroachCloud clusters.
Having control over how our StatefulSet pods are being scheduled has let us scale out with confidence.
We may look into retiring our plugin once pod topology spread constraints are available in GKE and EKS, but the maintenance overhead has been surprisingly low.
Better still: the plugin's implementation is orthogonal to our business logic. Deploying it, or retiring it for that matter, is as simple as changing the `schedulerName` field in our StatefulSet definitions.
---
_[Chris Seto](https://twitter.com/_ostriches) is a software engineer at Cockroach Labs and works on their Kubernetes automation for [CockroachCloud](https://cockroachlabs.cloud), CockroachDB._
### What should I look out for when changing CRI implementations?
@@ -129,7 +138,7 @@ common things to consider when migrating are:
- Kubectl plugins that require docker CLI or the control socket
- Kubernetes tools that require direct access to Docker (e.g. kube-imagepuller)
- Configuration of functionality like `registry-mirrors` and insecure registries
- Other support scripts or daemons that expect docker to be available and are run
- Other support scripts or daemons that expect Docker to be available and are run
outside of Kubernetes (e.g. monitoring or security agents)
- GPUs or special hardware and how they integrate with your runtime and Kubernetes
@@ -140,14 +149,15 @@ runtime where possible.
Another thing to look out for is anything expecting to run for system maintenance
or nested inside a container when building images will no longer work. For the
former, you can use the [`crictl`][cr] tool as a drop-in replacement and for the
latter you can use newer container build options like [img], [buildah], or
[kaniko] that don’t require Docker.
former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
latter you can use newer container build options like [img], [buildah],
[kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker.
We’re pleased to announce the release of Kubernetes 1.20, our third and final release of 2020! This release consists of 42 enhancements: 11 enhancements have graduated to stable, 15 enhancements are moving to beta, and 16 enhancements are entering alpha.
The 1.20 release cycle returned to its normal cadence of 11 weeks following the previous extended release cycle. This is one of the most feature dense releases in a while: the Kubernetes innovation cycle is still trending upward. This release has more alpha than stable enhancements, showing that there is still much to explore in the cloud native ecosystem.
## Major Themes
### Volume Snapshot Operations Goes Stable
This feature provides a standard way to trigger volume snapshot operations and allows users to incorporate snapshot operations in a portable manner on any Kubernetes environment and supported storage providers.
Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced, enterprise-grade, storage administration features for Kubernetes, including application or cluster level backup solutions.
Note that snapshot support requires Kubernetes distributors to bundle the Snapshot controller, Snapshot CRDs, and validation webhook. A CSI driver supporting the snapshot functionality must also be deployed on the cluster.
### Kubectl Debug Graduates to Beta
The `kubectl alpha debug` features graduates to beta in 1.20, becoming `kubectl debug`. The feature provides support for common debugging workflows directly from kubectl. Troubleshooting scenarios supported in this release of kubectl include:
* Troubleshoot workloads that crash on startup by creating a copy of the pod that uses a different container image or command.
* Troubleshoot distroless containers by adding a new container with debugging tools, either in a new copy of the pod or using an ephemeral container. (Ephemeral containers are an alpha feature that are not enabled by default.)
* Troubleshoot on a node by creating a container running in the host namespaces and with access to the host’s filesystem.
Note that as a new built-in command, `kubectl debug` takes priority over any kubectl plugin named “debug”. You must rename the affected plugin.
Invocations using `kubectl alpha debug` are now deprecated and will be removed in a subsequent release. Update your scripts to use `kubectl debug`. For more information about `kubectl debug`, see [Debugging Running Pods](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/).
### Beta: API Priority and Fairness
Introduced in 1.18, Kubernetes 1.20 now enables API Priority and Fairness (APF) by default. This allows `kube-apiserver` to categorize incoming requests by priority levels.
### Alpha with updates: IPV4/IPV6
The IPv4/IPv6 dual stack has been reimplemented to support dual stack services based on user and community feedback. This allows both IPv4 and IPv6 service cluster IP addresses to be assigned to a single service, and also enables a service to be transitioned from single to dual IP stack and vice versa.
### GA: Process PID Limiting for Stability
Process IDs (pids) are a fundamental resource on Linux hosts. It is trivial to hit the task limit without hitting any other resource limits and cause instability to a host machine.
Administrators require mechanisms to ensure that user pods cannot induce pid exhaustion that prevents host daemons (runtime, kubelet, etc) from running. In addition, it is important to ensure that pids are limited among pods in order to ensure they have limited impact to other workloads on the node.
After being enabled-by-default for a year, SIG Node graduates PID Limits to GA on both `SupportNodePidsLimit` (node-to-pod PID isolation) and `SupportPodPidsLimit` (ability to limit PIDs per pod).
### Alpha: Graceful node shutdown
Users and cluster administrators expect that pods will adhere to expected pod lifecycle including pod termination. Currently, when a node shuts down, pods do not follow the expected pod termination lifecycle and are not terminated gracefully which can cause issues for some workloads.
The `GracefulNodeShutdown` feature is now in Alpha. `GracefulNodeShutdown` makes the kubelet aware of node system shutdowns, enabling graceful termination of pods during a system shutdown.
## Major Changes
### Dockershim Deprecation
Dockershim, the container runtime interface (CRI) shim for Docker is being deprecated. Support for Docker is deprecated and will be removed in a future release. Docker-produced images will continue to work in your cluster with all CRI compliant runtimes as Docker images follow the Open Container Initiative (OCI) image specification.
The Kubernetes community has written a [detailed blog post about deprecation](https://blog.k8s.io/2020/12/02/dont-panic-kubernetes-and-docker/) with [a dedicated FAQ page for it](https://blog.k8s.io/2020/12/02/dockershim-faq/).
### Exec Probe Timeout Handling
A longstanding bug regarding exec probe timeouts that may impact existing pod definitions has been fixed. Prior to this fix, the field `timeoutSeconds` was not respected for exec probes. Instead, probes would run indefinitely, even past their configured deadline, until a result was returned. With this change, the default value of `1 second` will be applied if a value is not specified and existing pod definitions may no longer be sufficient if a probe takes longer than one second. A feature gate, called `ExecProbeTimeout`, has been added with this fix that enables cluster operators to revert to the previous behavior, but this will be locked and removed in subsequent releases. In order to revert to the previous behavior, cluster operators should set this feature gate to `false`.
Please review the updated documentation regarding [configuring probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes) for more details.
You can check out the full details of the 1.20 release in the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md).
# Availability of release
Kubernetes 1.20 is available for [download on GitHub](https://github.com/kubernetes/kubernetes/releases/tag/v1.20.0). There are some great resources out there for getting started with Kubernetes. You can check out some [interactive tutorials](https://kubernetes.io/docs/tutorials/) on the main Kubernetes site, or run a local cluster on your machine using Docker containers with [kind](https://kind.sigs.k8s.io). If you’d like to try building a cluster from scratch, check out the [Kubernetes the Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way) tutorial by Kelsey Hightower.
# Release Team
This release was made possible by a very dedicated group of individuals, who came together as a team in the midst of a lot of things happening out in the world. A huge thank you to the release lead Jeremy Rickard, and to everyone else on the release team for supporting each other, and working so hard to deliver the 1.20 release for the community.
> The Kubernetes 1.20 Release has been the raddest release yet.
2020 has been a challenging year for many of us, but Kubernetes contributors have delivered a record-breaking number of enhancements in this release. That is a great accomplishment, so the release lead wanted to end the year with a little bit of levity and pay homage to [Kubernetes 1.14 - Caturnetes](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.14) with a "rad" cat named Humphrey.
Humphrey is the release lead's cat and has a permanent [`blep`](https://www.inverse.com/article/42316-why-do-cats-blep-science-explains). *Rad* was pretty common slang in the 1990s in the United States, and so were laser backgrounds. Humphrey in a 1990s style school picture felt like a fun way to end the year. Hopefully, Humphrey and his *blep* bring you a little joy at the end of 2020!
The release logo was created by [Henry Hsu - @robotdancebattle](https://www.instagram.com/robotdancebattle/).
# User Highlights
- Apple is operating multi-thousand node Kubernetes clusters in data centers all over the world. Watch [Alena Prokharchyk's KubeCon NA Keynote](https://youtu.be/Tx8qXC-U3KM) to learn more about their cloud native journey.
# Project Velocity
The [CNCF K8s DevStats project](https://k8s.devstats.cncf.io/) aggregates a number of interesting data points related to the velocity of Kubernetes and various sub-projects. This includes everything from individual contributions to the number of companies that are contributing, and is a neat illustration of the depth and breadth of effort that goes into evolving this ecosystem.
In the v1.20 release cycle, which ran for 11 weeks (September 25 to December 9), we saw contributions from [967 companies](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=v1.19.0%20-%20now&var-metric=contributions) and [1335 individuals](https://k8s.devstats.cncf.io/d/66/developer-activity-counts-by-companies?orgId=1&var-period_name=v1.19.0%20-%20now&var-metric=contributions&var-repogroup_name=Kubernetes&var-country_name=All&var-companies=All) ([44 of whom](https://k8s.devstats.cncf.io/d/52/new-contributors?orgId=1&from=1601006400000&to=1607576399000&var-repogroup_name=Kubernetes) made their first Kubernetes contribution) from [26 countries](https://k8s.devstats.cncf.io/d/50/countries-stats?orgId=1&from=1601006400000&to=1607576399000&var-period_name=Quarter&var-countries=All&var-repogroup_name=Kubernetes&var-metric=rcommitters&var-cum=countries).
# Ecosystem Updates
- KubeCon North America just wrapped up three weeks ago, the second such event to be virtual! All talks are [now available to all on-demand](https://www.youtube.com/playlist?list=PLj6h78yzYM2Pn8RxfLh2qrXBDftr6Qjut) for anyone still needing to catch up!
- In June, the Kubernetes community formed a new working group as a direct response to the Black Lives Matter protests occurring across America. WG Naming's goal is to remove harmful and unclear language in the Kubernetes project as completely as possible and to do so in a way that is portable to other CNCF projects. A great introductory talk on this important work and how it is conducted was given [at KubeCon 2020 North America](https://sched.co/eukp), and the initial impact of this labor [can actually be seen in the v1.20 release](https://github.com/kubernetes/enhancements/issues/2067).
- Previously announced this summer, [The Certified Kubernetes Security Specialist (CKS) Certification](https://www.cncf.io/announcements/2020/11/17/kubernetes-security-specialist-certification-now-available/) was released during Kubecon NA for immediate scheduling! Following the model of CKA and CKAD, the CKS is a performance-based exam, focused on security-themed competencies and domains. This exam is targeted at current CKA holders, particularly those who want to round out their baseline knowledge in securing cloud workloads (which is all of us, right?).
# Event Updates
KubeCon + CloudNativeCon Europe 2021 will take place May 4 - 7, 2021! Registration will open on January 11. You can find more information about the conference [here](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/). Remember that [the CFP](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/program/cfp/) closes on Sunday, December 13, 11:59pm PST!
# Upcoming release webinar
Stay tuned for the upcoming release webinar happening this January.
# Get Involved
If you’re interested in contributing to the Kubernetes community, Special Interest Groups (SIGs) are a great starting point. Many of them may align with your interests! If there are things you’d like to share with the community, you can join the weekly community meeting, or use any of the following channels:
* Find out more about contributing to Kubernetes at the new [Kubernetes Contributor website](https://www.kubernetes.dev/)
* Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for latest updates
* Join the community discussion on [Discuss](https://discuss.kubernetes.io/)
* Join the community on [Slack](http://slack.k8s.io/)
* Share your Kubernetes [story](https://docs.google.com/a/linuxfoundation.org/forms/d/e/1FAIpQLScuI7Ye3VQHQTwBASrgkjQDSS5TP0g3AXfFhwSM9YpHgxRKFA/viewform)
* Read more about what’s happening with Kubernetes on the [blog](https://kubernetes.io/blog/)
* Learn more about the [Kubernetes Release Team](https://github.com/kubernetes/sig-release/tree/master/release-team)
title: 'Kubernetes 1.20: Kubernetes Volume Snapshot Moves to GA'
date: 2020-12-10
slug: kubernetes-1.20-volume-snapshot-moves-to-ga
---
**Authors**: Xing Yang, VMware & Xiangqian Yu, Google
The Kubernetes Volume Snapshot feature is now GA in Kubernetes v1.20. It was introduced as [alpha](https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/) in Kubernetes v1.12, followed by a [second alpha](https://kubernetes.io/blog/2019/01/17/update-on-volume-snapshot-alpha-for-kubernetes/) with breaking changes in Kubernetes v1.13, and promotion to [beta](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-cis-volume-snapshot-beta/) in Kubernetes 1.17. This blog post summarizes the changes releasing the feature from beta to GA.
## What is a volume snapshot?
Many storage systems (like Google Cloud Persistent Disks, Amazon Elastic Block Storage, and many on-premise storage systems) provide the ability to create a “snapshot” of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to rehydrate a new volume (pre-populated with the snapshot data) or to restore an existing volume to a previous state (represented by the snapshot).
## Why add volume snapshots to Kubernetes?
Kubernetes aims to create an abstraction layer between distributed applications and underlying clusters so that applications can be agnostic to the specifics of the cluster they run on and application deployment requires no “cluster-specific” knowledge.
The Kubernetes Storage SIG identified snapshot operations as critical functionality for many stateful workloads. For example, a database administrator may want to snapshot a database’s volumes before starting a database operation.
By providing a standard way to trigger volume snapshot operations in Kubernetes, this feature allows Kubernetes users to incorporate snapshot operations in a portable manner on any Kubernetes environment regardless of the underlying storage.
Additionally, these Kubernetes snapshot primitives act as basic building blocks that unlock the ability to develop advanced enterprise-grade storage administration features for Kubernetes, including application or cluster level backup solutions.
## What’s new since beta?
With the promotion of Volume Snapshot to GA, the feature is enabled by default on standard Kubernetes deployments and cannot be turned off.
Many enhancements have been made to improve the quality of this feature and to make it production-grade.
- The Volume Snapshot APIs and client library were moved to a separate Go module.
- A snapshot validation webhook has been added to perform necessary validation on volume snapshot objects. More details can be found in the [Volume Snapshot Validation Webhook Kubernetes Enhancement Proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1900-volume-snapshot-validation-webhook).
- Along with the validation webhook, the volume snapshot controller will start labeling invalid snapshot objects that already existed. This allows users to identify, remove any invalid objects, and correct their workflows. Once the API is switched to the v1 type, those invalid objects will not be deletable from the system.
- To provide better insights into how the snapshot feature is performing, an initial set of operation metrics has been added to the volume snapshot controller.
- There are more end-to-end tests, running on GCP, that validate the feature in a real Kubernetes cluster. Stress tests (based on Google Persistent Disk and `hostPath` CSI Drivers) have been introduced to test the robustness of the system.
Other than introducing tightening validation, there is no difference between the v1beta1 and v1 Kubernetes volume snapshot API. In this release (with Kubernetes 1.20), both v1 and v1beta1 are served while the stored API version is still v1beta1. Future releases will switch the stored version to v1 and gradually remove v1beta1 support.
## Which CSI drivers support volume snapshots?
Snapshots are only supported for CSI drivers, not for in-tree or FlexVolume drivers. Ensure the deployed CSI driver on your cluster has implemented the snapshot interfaces. For more information, see [Container Storage Interface (CSI) for Kubernetes GA](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/).
Currently more than [50 CSI drivers](https://kubernetes-csi.github.io/docs/drivers.html) support the Volume Snapshot feature. The [GCE Persistent Disk CSI Driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) has gone through the tests for upgrading from volume snapshots beta to GA. GA level support for other CSI drivers should be available soon.
## Who builds products using volume snapshots?
As of the publishing of this blog, the following participants from the [Kubernetes Data Protection Working Group](https://github.com/kubernetes/community/tree/master/wg-data-protection) are building products or have already built products using Kubernetes volume snapshots.
- CSI Driver along with [CSI Snapshotter sidecar](https://github.com/kubernetes-csi/external-snapshotter/tree/master/pkg/sidecar-controller)
It is strongly recommended that Kubernetes distributors bundle and deploy the volume snapshot controller, CRDs, and validation webhook as part of their Kubernetes cluster management process (independent of any CSI Driver).
{{< warning >}}
The snapshot validation webhook serves as a critical component to transition smoothly from using v1beta1 to v1 API. Not installing the snapshot validation webhook makes prevention of invalid volume snapshot objects from creation/updating impossible, which in turn will block deletion of invalid volume snapshot objects in coming upgrades.
{{< /warning >}}
If your cluster does not come pre-installed with the correct components, you may manually install them. See the [CSI Snapshotter](https://github.com/kubernetes-csi/external-snapshotter#readme) README for details.
## How to use volume snapshots?
Assuming all the required components (including CSI driver) have been already deployed and running on your cluster, you can create volume snapshots using the `VolumeSnapshot` API object, or use an existing `VolumeSnapshot` to restore a PVC by specifying the VolumeSnapshot data source on it. For more details, see the [volume snapshot documentation](/docs/concepts/storage/volume-snapshots/).
{{< note >}} The Kubernetes Snapshot API does not provide any application consistency guarantees. You have to prepare your application (pause application, freeze filesystem etc.) before taking the snapshot for data consistency either manually or using higher level APIs/controllers. {{< /note >}}
### Dynamically provision a volume snapshot
To dynamically provision a volume snapshot, create a `VolumeSnapshotClass` API object first.
Then create a `VolumeSnapshot` API object from a PVC by specifying the volume snapshot class.
```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: test-snapshot
namespace: ns1
spec:
volumeSnapshotClassName: test-snapclass
source:
persistentVolumeClaimName: test-pvc
```
### Importing an existing volume snapshot with Kubernetes
To import a pre-existing volume snapshot into Kubernetes, manually create a `VolumeSnapshotContent` object first.
```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
name: test-content
spec:
deletionPolicy: Delete
driver: testdriver.csi.k8s.io
source:
snapshotHandle: 7bdd0de3-xxx
volumeSnapshotRef:
name: test-snapshot
namespace: default
```
Then create a `VolumeSnapshot` object pointing to the `VolumeSnapshotContent` object.
```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: test-snapshot
spec:
source:
volumeSnapshotContentName: test-content
```
### Rehydrate volume from snapshot
A bound and ready `VolumeSnapshot` object can be used to rehydrate a new volume with data pre-populated from snapshotted data as shown here:
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-restore
namespace: demo-namespace
spec:
storageClassName: test-storageclass
dataSource:
name: test-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
## How to add support for snapshots in a CSI driver?
See the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md) and the [Kubernetes-CSI Driver Developer Guide](https://kubernetes-csi.github.io/docs/snapshot-restore-feature.html) for more details on how to implement the snapshot feature in a CSI driver.
## What are the limitations?
The GA implementation of volume snapshots for Kubernetes has the following limitations:
- Does not support reverting an existing PVC to an earlier state represented by a snapshot (only supports provisioning a new volume from a snapshot).
### How to learn more?
The code repository for snapshot APIs and controller is here: https://github.com/kubernetes-csi/external-snapshotter
Check out additional documentation on the snapshot feature here: http://k8s.io/docs/concepts/storage/volume-snapshots and https://kubernetes-csi.github.io/docs/
## How to get involved?
This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together.
We offer a huge thank you to the contributors who stepped up these last few quarters to help the project reach GA. We want to thank Saad Ali, Michelle Au, Tim Hockin, and Jordan Liggitt for their insightful reviews and thorough consideration with the design, thank Andi Li for his work on adding the support of the snapshot validation webhook, thank Grant Griffiths on implementing metrics support in the snapshot controller and handling password rotation in the validation webhook, thank Chris Henzie, Raunak Shah, and Manohar Reddy for writing critical e2e tests to meet the scalability and stability requirements for graduation, thank Kartik Sharma for moving snapshot APIs and client lib to a separate go module, and thank Raunak Shah and Prafull Ladha for their help with upgrade testing from beta to GA.
There are many more people who have helped to move the snapshot feature from beta to GA. We want to thank everyone who has contributed to this effort:
For those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG). We’re rapidly growing and always welcome new contributors.
We also hold regular [Data Protection Working Group meetings](https://docs.google.com/document/d/15tLCV3csvjHbKb16DVk-mfUmFry_Rlwo-2uG6KNGsfw/edit#). New attendees are welcome to join in discussions.
Typically when a [CSI](https://github.com/container-storage-interface/spec/blob/baa71a34651e5ee6cb983b39c03097d7aa384278/spec.md) driver mounts credentials such as secrets and certificates, it has to authenticate against storage providers to access the credentials. However, the access to those credentials are controlled on the basis of the pods' identities rather than the CSI driver's identity. CSI drivers, therefore, need some way to retrieve pod's service account token.
Currently there are two suboptimal approaches to achieve this, either by granting CSI drivers the permission to use TokenRequest API or by reading tokens directly from the host filesystem.
Both of them exhibit the following drawbacks:
- Violating the principle of least privilege
- Every CSI driver needs to re-implement the logic of getting the pod’s service account token
The second approach is more problematic due to:
- The audience of the token defaults to the kube-apiserver
- The token is not guaranteed to be available (e.g. `AutomountServiceAccountToken=false`)
- The approach does not work for CSI drivers that run as a different (non-root) user from the pods. See [file permission section for service account token](https://github.com/kubernetes/enhancements/blob/f40c24a5da09390bd521be535b38a4dbab09380c/keps/sig-storage/20180515-svcacct-token-volumes.md#file-permission)
- The token might be legacy Kubernetes service account token which doesn’t expire if `BoundServiceAccountTokenVolume=false`
Kubernetes 1.20 introduces an alpha feature, `CSIServiceAccountToken`, to improve the security posture. The new feature allows CSI drivers to receive pods' [bound service account tokens](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md).
This feature also provides a knob to re-publish volumes so that short-lived volumes can be refreshed.
## Pod Impersonation
### Using GCP APIs
Using [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity), a Kubernetes service account can authenticate as a Google service account when accessing Google Cloud APIs. If a CSI driver needs to access GCP APIs on behalf of the pods that it is mounting volumes for, it can use the pod's service account token to [exchange for GCP tokens](https://cloud.google.com/iam/docs/reference/sts/rest). The pod's service account token is plumbed through the volume context in `NodePublishVolume` RPC calls when the feature `CSIServiceAccountToken` is enabled. For example: accessing [Google Secret Manager](https://cloud.google.com/secret-manager/) via a [secret store CSI driver](https://github.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp).
### Using Vault
If users configure [Kubernetes as an auth method](https://www.vaultproject.io/docs/auth/kubernetes), Vault uses the `TokenReview` API to validate the Kubernetes service account token. For CSI drivers using Vault as resources provider, they need to present the pod's service account to Vault. For example, [secrets store CSI driver](https://github.com/hashicorp/secrets-store-csi-driver-provider-vault) and [cert manager CSI driver](https://github.com/jetstack/cert-manager-csi).
## Short-lived Volumes
To keep short-lived volumes such as certificates effective, CSI drivers can specify `RequiresRepublish=true` in their`CSIDriver` object to have the kubelet periodically call `NodePublishVolume` on mounted volumes. These republishes allow CSI drivers to ensure that the volume content is up-to-date.
## Next steps
This feature is alpha and projected to move to beta in 1.21. See more in the following KEP and CSI documentation:
- [KEP-1855: Service Account Token for CSI Driver](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1855-csi-driver-service-account-token/README.md)
- SIG-Auth [meets regularly](https://github.com/kubernetes/community/tree/master/sig-auth#meetings) and can be reached via [Slack and the mailing list](https://github.com/kubernetes/community/tree/master/sig-auth#contact)
- SIG-Storage [meets regularly](https://github.com/kubernetes/community/tree/master/sig-storage#meetings) and can be reached via [Slack and the mailing list](https://github.com/kubernetes/community/tree/master/sig-storage#contact).
**Authors**: Hemant Kumar, Red Hat & Christian Huffman, Red Hat
Kubernetes 1.20 brings two important beta features, allowing Kubernetes admins and users alike to have more adequate control over how volume permissions are applied when a volume is mounted inside a Pod.
### Allow users to skip recursive permission changes on mount
Traditionally if your pod is running as a non-root user ([which you should](https://twitter.com/thockin/status/1333892204490735617)), you must specify a `fsGroup` inside the pod’s security context so that the volume can be readable and writable by the Pod. This requirement is covered in more detail in [here](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/).
But one side-effect of setting `fsGroup` is that, each time a volume is mounted, Kubernetes must recursively `chown()` and `chmod()` all the files and directories inside the volume - with a few exceptions noted below. This happens even if group ownership of the volume already matches the requested `fsGroup`, and can be pretty expensive for larger volumes with lots of small files, which causes pod startup to take a long time. This scenario has been a [known problem](https://github.com/kubernetes/kubernetes/issues/69699) for a while, and in Kubernetes 1.20 we are providing knobs to opt-out of recursive permission changes if the volume already has the correct permissions.
When configuring a pod’s security context, set `fsGroupChangePolicy` to "OnRootMismatch" so if the root of the volume already has the correct permissions, the recursive permission change can be skipped. Kubernetes ensures that permissions of the top-level directory are changed last the first time it applies permissions.
```yaml
securityContext:
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
fsGroupChangePolicy: "OnRootMismatch"
```
You can learn more about this in [Configure volume permission and ownership change policy for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods).
### Allow CSI Drivers to declare support for fsGroup based permissions
Although the previous section implied that Kubernetes _always_ recursively changes permissions of a volume if a Pod has a `fsGroup`, this is not strictly true. For certain multi-writer volume types, such as NFS or Gluster, the cluster doesn’t perform recursive permission changes even if the pod has a `fsGroup`. Other volume types may not even support `chown()`/`chmod()`, which rely on Unix-style permission control primitives.
So how do we know when to apply recursive permission changes and when we shouldn't? For in-tree storage drivers, this was relatively simple. For [CSI](https://kubernetes-csi.github.io/docs/introduction.html#introduction) drivers that could span a multitude of platforms and storage types, this problem can be a bigger challenge.
Previously, whenever a CSI volume was mounted to a Pod, Kubernetes would attempt to automatically determine if the permissions and ownership should be modified. These methods were imprecise and could cause issues as we already mentioned, depending on the storage type.
The CSIDriver custom resource now has a `.spec.fsGroupPolicy` field, allowing storage drivers to explicitly opt in or out of these recursive modifications. By having the CSI driver specify a policy for the backing volumes, Kubernetes can avoid needless modification attempts. This optimization helps to reduce volume mount time and also cuts own reporting errors about modifications that would never succeed.
#### CSIDriver FSGroupPolicy API
Three FSGroupPolicy values are available as of Kubernetes 1.20, with more planned for future releases.
- **ReadWriteOnceWithFSType** - This is the default policy, applied if no `fsGroupPolicy` is defined; this preserves the behavior from previous Kubernetes releases. Each volume is examined at mount time to determine if permissions should be recursively applied.
- **File** - Always attempt to apply permission modifications, regardless of the filesystem type or PersistentVolumeClaim’s access mode.
- **None** - Never apply permission modifications.
#### How do I use it?
The only configuration needed is defining `fsGroupPolicy` inside of the `.spec` for a CSIDriver. Once that element is defined, any subsequently mounted volumes will automatically use the defined policy. There’s no additional deployment required!
#### What’s next?
Depending on feedback and adoption, the Kubernetes team plans to push these implementations to GA in either 1.21 or 1.22.
### How can I learn more?
This feature is explained in more detail in Kubernetes project documentation: [CSI Driver fsGroup Support](https://kubernetes-csi.github.io/docs/support-fsgroup.html) and [Configure volume permission and ownership change policy for Pods ](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods).
### How do I get involved?
The [Kubernetes Slack channel #csi](https://kubernetes.slack.com/messages/csi) and any of the [standard SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and the CSI team.
Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage). We’re rapidly growing and always welcome new contributors.
**Authors:** Renaud Gaubert (NVIDIA), David Ashpole (Google), and Pramod Ramarao (NVIDIA)
With Kubernetes 1.20, infrastructure teams who manage large scale Kubernetes clusters, are seeing the graduation of two exciting and long awaited features:
* The Pod Resources API (introduced in 1.13) is finally graduating to GA. This allows Kubernetes plugins to obtain information about the node’s resource usage and assignment; for example: which pod/container consumes which device.
* The `DisableAcceleratorMetrics` feature (introduced in 1.19) is graduating to beta and will be enabled by default. This removes device metrics reported by the kubelet in favor of the new plugin architecture.
Many of the features related to fundamental device support (device discovery, plugin, and monitoring) are reaching a strong level of stability.
Kubernetes users should see these features as stepping stones to enable more complex use cases (networking, scheduling, storage, etc.)!
One such example is Non Uniform Memory Access (NUMA) placement where, when selecting a device, an application typically wants to ensure that data transfer between CPU Memory and Device Memory is as fast as possible. In some cases, incorrect NUMA placement can nullify the benefit of offloading compute to an external device.
If these are topics of interest to you, consider joining the [Kubernetes Node Special Insterest Group](https://github.com/kubernetes/community/tree/master/sig-node) (SIG) for all topics related to the Kubernetes node, the COD (container orchestrated device) workgroup for topics related to runtimes, or the resource management forum for topics related to resource management!
## The Pod Resources API - Why does it need to exist?
Kubernetes is a vendor neutral platform. If we want it to support device monitoring, adding vendor-specific code in the Kubernetes code base is not an ideal solution. Ultimately, devices are a domain where deep expertise is needed and the best people to add and maintain code in that area are the device vendors themselves.
The Pod Resources API was built as a solution to this issue. Each vendor can build and maintain their own out-of-tree monitoring plugin. This monitoring plugin, often deployed as a separate pod within a cluster, can then associate the metrics a device emits with the associated pod that's using it.
For example, use the NVIDIA GPU dcgm-exporter to scrape metrics in Prometheus format:
```
$ curl -sL http://127.0.01:8080/metrics
# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).
# TYPE DCGM_FI_DEV_MEM_CLOCK gauge
# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).
Each agent is expected to adhere to the node monitoring guidelines. In other words, plugins are expected to generate metrics in Prometheus format, and new metrics should not have any dependency on the Kubernetes base directly.
This allows consumers of the metrics to use a compatible monitoring pipeline to collect and analyze metrics from a variety of agents, even if they are maintained by different vendors.
## Disabling the NVIDIA GPU metrics - Warning {#nvidia-gpu-metrics-deprecated}
With the graduation of the plugin monitoring system, Kubernetes is deprecating the NVIDIA GPU metrics that are being reported by the kubelet.
With the [DisableAcceleratorMetrics](/docs/concepts/cluster-administration/system-metrics/#disable-accelerator-metrics) feature being enabled by default in Kubernetes 1.20, NVIDIA GPUs are no longer special citizens in Kubernetes. This is a good thing in the spirit of being vendor-neutral, and enables the most suited people to maintain their plugin on their own release schedule!
Users will now need to either install the [NVIDIA GDGM exporter](https://github.com/NVIDIA/gpu-monitoring-tools) or use [bindings](https://github.com/nvidia/go-nvml) to gather more accurate and complete metrics about NVIDIA GPUs. This deprecation means that you can no longer rely on metrics that were reported by kubelet, such as `container_accelerator_duty_cycle` or `container_accelerator_memory_used_bytes` which were used to gather NVIDIA GPU memory utilization.
This means that users who used to rely on the NVIDIA GPU metrics reported by the kubelet, will need to update their reference and deploy the NVIDIA plugin. Namely the different metrics reported by Kubernetes map to the following metrics:
You might also be interested in other metrics such as `DCGM_FI_DEV_GPU_TEMP` (the GPU temperature) or DCGM_FI_DEV_POWER_USAGE (the power usage). The [default set](https://github.com/NVIDIA/gpu-monitoring-tools/blob/d5c9bb55b4d1529ca07068b7f81e690921ce2b59/etc/dcgm-exporter/default-counters.csv) is available in Nvidia's [Data Center GPU Manager documentation](https://docs.nvidia.com/datacenter/dcgm/latest/dcgm-api/group__dcgmFieldIdentifiers.html).
Note that for this release you can still set the `DisableAcceleratorMetrics` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to _false_, effectively re-enabling the ability for the kubelet to report NVIDIA GPU metrics.
Paired with the graduation of the Pod Resources API, these tools can be used to generate GPU telemetry [that can be used in visualization dashboards](https://grafana.com/grafana/dashboards/12239), below is an example:
## The Pod Resources API - What can I go on to do with this?
As soon as this interface was introduced, many vendors started using it for widely different use cases! To list a few examples:
The [kuryr-kubernetes](https://github.com/openstack/kuryr-kubernetes) CNI plugin in tandem with [intel-sriov-device-plugin](https://github.com/intel/sriov-network-device-plugin). This allowed the CNI plugin to know which allocation of SR-IOV Virtual Functions (VFs) the kubelet made and use that information to correctly setup the container network namespace and use a device with the appropriate NUMA node. We also expect this interface to be used to track the allocated and available resources with information about the NUMA topology of the worker node.
Another use-case is GPU telemetry, where GPU metrics can be associated with the containers and pods that the GPU is assigned to. One such example is the NVIDIA `dcgm-exporter`, but others can be easily built in the same paradigm.
The Pod Resources API is a simple gRPC service which informs clients of the pods the kubelet knows. The information concerns the devices assignment the kubelet made and the assignment of CPUs. This information is obtained from the internal state of the kubelet's Device Manager and CPU Manager respectively.
You can see below a sample example of the API and how a go client could use that information in a few lines:
Finally, note that you can watch the number of requests made to the Pod Resources endpoint by watching the new kubelet metric called `pod_resources_endpoint_requests_total` on the kubelet's `/metrics` endpoint.
## Is device monitoring suitable for production? Can I extend it? Can I contribute?
Yes! This feature released in 1.13, almost 2 years ago, has seen broad adoption, is already used by different cloud managed services, and with its graduation to G.A in Kubernetes 1.20 is production ready!
If you are a device vendor, you can start using it today! If you just want to monitor the devices in your cluster, go get the latest version of your monitoring plugin!
If you feel passionate about that area, join the kubernetes community, help improve the API or contribute the device monitoring plugins!
## Acknowledgements
We thank the members of the community who have contributed to this feature or given feedback including members of WG-Resource-Management, SIG-Node and the Resource management forum!
Authors: Marcin Maciaszczyk, Kubermatic & Sebastian Florek, Kubermatic
In October 2020, the Kubernetes Dashboard officially turned five. As main project maintainers, we can barely believe that so much time has passed since our very first commits to the project. However, looking back with a bit of nostalgia, we realize that quite a lot has happened since then. Now it’s due time to celebrate “our baby” with a short recap.
## How It All Began
The initial idea behind the Kubernetes Dashboard project was to provide a web interface for Kubernetes. We wanted to reflect the kubectl functionality through an intuitive web UI. The main benefit from using the UI is to be able to quickly see things that do not work as expected (monitoring and troubleshooting). Also, the Kubernetes Dashboard is a great starting point for users that are new to the Kubernetes ecosystem.
The very [first commit](https://github.com/kubernetes/dashboard/commit/5861187fa807ac1cc2d9b2ac786afeced065076c) to the Kubernetes Dashboard was made by Filip Grządkowski from Google on 16th October 2015 – just a few months from the initial commit to the Kubernetes repository. Our initial commits go back to November 2015 ([Sebastian committed on 16 November 2015](https://github.com/kubernetes/dashboard/commit/09e65b6bb08c49b926253de3621a73da05e400fd); [Marcin committed on 23 November 2015](https://github.com/kubernetes/dashboard/commit/1da4b1c25ef040818072c734f71333f9b4733f55)). Since that time, we’ve become regular contributors to the project. For the next two years, we worked closely with the Googlers, eventually becoming main project maintainers ourselves.
{{< figure src="first-ui.png" caption="The First Version of the User Interface" >}}
{{< figure src="along-the-way-ui.png" caption="Prototype of the New User Interface" >}}
{{< figure src="current-ui.png" caption="The Current User Interface" >}}
As you can see, the initial look and feel of the project were completely different from the current one. We have changed the design multiple times. The same has happened with the code itself.
## Growing Up - The Big Migration
At [the beginning of 2018](https://github.com/kubernetes/dashboard/pull/2727), we reached a point where AngularJS was getting closer to the end of its life, while the new Angular versions were published quite often. A lot of the libraries and the modules that we were using were following the trend. That forced us to spend a lot of the time rewriting the frontend part of the project to make it work with newer technologies.
The migration came with many benefits like being able to refactor a lot of the code, introduce design patterns, reduce code complexity, and benefit from the new modules. However, you can imagine that the scale of the migration was huge. Luckily, there were a number of contributions from the community helping us with the resource support, new Kubernetes version support, i18n, and much more. After many long days and nights, we finally released the [first beta version](https://github.com/kubernetes/dashboard/releases/tag/v2.0.0-beta1) in July 2019, followed by the [2.0 release](https://github.com/kubernetes/dashboard/releases/tag/v2.0.0) in April 2020 — our baby had grown up.
## Where Are We Standing in 2021?
Due to limited resources, unfortunately, we were not able to offer extensive support for many different Kubernetes versions. So, we’ve decided to always try and support the latest Kubernetes version available at the time of the Kubernetes Dashboard release. The latest release, [Dashboard v2.2.0](https://github.com/kubernetes/dashboard/releases/tag/v2.2.0) provides support for Kubernetes v1.20.
On top of that, we put in a great deal of effort into [improving resource support](https://github.com/kubernetes/dashboard/issues/5232). Meanwhile, we do offer support for most of the Kubernetes resources. Also, the Kubernetes Dashboard supports multiple languages: English, German, French, Japanese, Korean, Chinese (Traditional, Simplified, Traditional Hong Kong). Persian and Russian localizations are currently in progress. Moreover, we are working on the support for 3rd party themes and the design of the app in general. As you can see, quite a lot of things are going on.
Luckily, we do have regular contributors with domain knowledge who are taking care of the project, updating the Helm charts, translations, Go modules, and more. But as always, there could be many more hands on deck. So if you are thinking about contributing to Kubernetes, keep us in mind ;)
## What’s Next
The Kubernetes Dashboard has been growing and prospering for more than 5 years now. It provides the community with an intuitive Web UI, thereby decreasing the complexity of Kubernetes and increasing its accessibility to new community members. We are proud of what the project has achieved so far, but this is by far not the end. These are our priorities for the future:
* Keep providing support for the new Kubernetes versions
* Keep improving the support for the existing resources
* Keep working on auth system improvements
* [Rewrite the API to use gRPC and shared informers](https://github.com/kubernetes/dashboard/pull/5449): This will allow us to improve the performance of the application but, most importantly, to support live updates coming from the Kubernetes project. It is one of the most requested features from the community.
* Split the application into two containers, one with the UI and the second with the API running inside.
## The Kubernetes Dashboard in Numbers
* Initial commit made on October 16, 2015
* Over 100 million pulls from Dockerhub since the v2 release
* 8 supported languages and the next 2 in progress
* Over 3360 closed PRs
* Over 2260 closed issues
* 100% coverage of the supported core Kubernetes resources
* Over 9000 stars on GitHub
* Over 237 000 lines of code
## Join Us
As mentioned earlier, we are currently looking for more people to help us further develop and grow the project. We are open to contributions in multiple areas, i.e., [issues with help wanted label](https://github.com/kubernetes/dashboard/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22). Please feel free to reach out via GitHub or the #sig-ui channel in the [Kubernetes Slack](https://slack.k8s.io/).
**Author:** Tabitha Sable (Kubernetes SIG Security)
PodSecurityPolicy (PSP) is being deprecated in Kubernetes 1.21, to be released later this week. This starts the countdown to its removal, but doesn’t change anything else. PodSecurityPolicy will continue to be fully functional for several more releases before being removed completely. In the meantime, we are developing a replacement for PSP that covers key use cases more easily and sustainably.
What are Pod Security Policies? Why did we need them? Why are they going away, and what’s next? How does this affect you? These key questions come to mind as we prepare to say goodbye to PSP, so let’s walk through them together. We’ll start with an overview of how features get removed from Kubernetes.
## What does deprecation mean in Kubernetes?
Whenever a Kubernetes feature is set to go away, our [deprecation policy](/docs/reference/using-api/deprecation-policy/) is our guide. First the feature is marked as deprecated, then after enough time has passed, it can finally be removed.
Kubernetes 1.21 starts the deprecation process for PodSecurityPolicy. As with all feature deprecations, PodSecurityPolicy will continue to be fully functional for several more releases. The current plan is to remove PSP from Kubernetes in the 1.25 release.
Until then, PSP is still PSP. There will be at least a year during which the newest Kubernetes releases will still support PSP, and nearly two years until PSP will pass fully out of all supported Kubernetes versions.
## What is PodSecurityPolicy?
[PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) is a built-in [admission controller](/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/) that allows a cluster administrator to control security-sensitive aspects of the Pod specification.
First, one or more PodSecurityPolicy resources are created in a cluster to define the requirements Pods must meet. Then, RBAC rules are created to control which PodSecurityPolicy applies to a given pod. If a pod meets the requirements of its PSP, it will be admitted to the cluster as usual. In some cases, PSP can also modify Pod fields, effectively creating new defaults for those fields. If a Pod does not meet the PSP requirements, it is rejected, and cannot run.
One more important thing to know about PodSecurityPolicy: it’s not the same as [PodSecurityContext](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context).
A part of the Pod specification, PodSecurityContext (and its per-container counterpart `SecurityContext`) is the collection of fields that specify many of the security-relevant settings for a Pod. The security context dictates to the kubelet and container runtime how the Pod should actually be run. In contrast, the PodSecurityPolicy only constrains (or defaults) the values that may be set on the security context.
The deprecation of PSP does not affect PodSecurityContext in any way.
## Why did we need PodSecurityPolicy?
In Kubernetes, we define resources such as Deployments, StatefulSets, and Services that represent the building blocks of software applications. The various controllers inside a Kubernetes cluster react to these resources, creating further Kubernetes resources or configuring some software or hardware to accomplish our goals.
In most Kubernetes clusters, RBAC (Role-Based Access Control) [rules](/docs/reference/access-authn-authz/rbac/#role-and-clusterrole) control access to these resources. `list`, `get`, `create`, `edit`, and `delete` are the sorts of API operations that RBAC cares about, but _RBAC does not consider what settings are being put into the resources it controls_. For example, a Pod can be almost anything from a simple webserver to a privileged command prompt offering full access to the underlying server node and all the data. It’s all the same to RBAC: a Pod is a Pod is a Pod.
To control what sorts of settings are allowed in the resources defined in your cluster, you need Admission Control in addition to RBAC. Since Kubernetes 1.3, PodSecurityPolicy has been the built-in way to do that for security-related Pod fields. Using PodSecurityPolicy, you can prevent “create Pod” from automatically meaning “root on every cluster node,” without needing to deploy additional external admission controllers.
## Why is PodSecurityPolicy going away?
In the years since PodSecurityPolicy was first introduced, we have realized that PSP has some serious usability problems that can’t be addressed without making breaking changes.
The way PSPs are applied to Pods has proven confusing to nearly everyone that has attempted to use them. It is easy to accidentally grant broader permissions than intended, and difficult to inspect which PSP(s) apply in a given situation. The “changing Pod defaults” feature can be handy, but is only supported for certain Pod settings and it’s not obvious when they will or will not apply to your Pod. Without a “dry run” or audit mode, it’s impractical to retrofit PSP to existing clusters safely, and it’s impossible for PSP to ever be enabled by default.
For more information about these and other PSP difficulties, check out SIG Auth’s KubeCon NA 2019 Maintainer Track session video: {{< youtube "SFtHRmPuhEw?start=953" youtube-quote-sm >}}
Today, you’re not limited only to deploying PSP or writing your own custom admission controller. Several external admission controllers are available that incorporate lessons learned from PSP to provide a better user experience. [K-Rail](https://github.com/cruise-automation/k-rail), [Kyverno](https://github.com/kyverno/kyverno/), and [OPA/Gatekeeper](https://github.com/open-policy-agent/gatekeeper/) are all well-known, and each has its fans.
Although there are other good options available now, we believe there is still value in having a built-in admission controller available as a choice for users. With this in mind, we turn toward building what’s next, inspired by the lessons learned from PSP.
## What’s next?
Kubernetes SIG Security, SIG Auth, and a diverse collection of other community members have been working together for months to ensure that what’s coming next is going to be awesome. We have developed a Kubernetes Enhancement Proposal ([KEP 2579](https://github.com/kubernetes/enhancements/issues/2579)) and a prototype for a new feature, currently being called by the temporary name "PSP Replacement Policy." We are targeting an Alpha release in Kubernetes 1.22.
PSP Replacement Policy starts with the realization that since there is a robust ecosystem of external admission controllers already available, PSP’s replacement doesn’t need to be all things to all people. Simplicity of deployment and adoption is the key advantage a built-in admission controller has compared to an external webhook, so we have focused on how to best utilize that advantage.
PSP Replacement Policy is designed to be as simple as practically possible while providing enough flexibility to really be useful in production at scale. It has soft rollout features to enable retrofitting it to existing clusters, and is configurable enough that it can eventually be active by default. It can be deactivated partially or entirely, to coexist with external admission controllers for advanced use cases.
## What does this mean for you?
What this all means for you depends on your current PSP situation. If you’re already using PSP, there’s plenty of time to plan your next move. Please review the PSP Replacement Policy KEP and think about how well it will suit your use case.
If you’re making extensive use of the flexibility of PSP with numerous PSPs and complex binding rules, you will likely find the simplicity of PSP Replacement Policy too limiting. Use the next year to evaluate the other admission controller choices in the ecosystem. There are resources available to ease this transition, such as the [Gatekeeper Policy Library](https://github.com/open-policy-agent/gatekeeper-library).
If your use of PSP is relatively simple, with a few policies and straightforward binding to service accounts in each namespace, you will likely find PSP Replacement Policy to be a good match for your needs. Evaluate your PSPs compared to the Kubernetes [Pod Security Standards](/docs/concepts/security/pod-security-standards/) to get a feel for where you’ll be able to use the Restricted, Baseline, and Privileged policies. Please follow along with or contribute to the KEP and subsequent development, and try out the Alpha release of PSP Replacement Policy when it becomes available.
If you’re just beginning your PSP journey, you will save time and effort by keeping it simple. You can approximate the functionality of PSP Replacement Policy today by using the Pod Security Standards’ PSPs. If you set the cluster default by binding a Baseline or Restricted policy to the `system:serviceaccounts` group, and then make a more-permissive policy available as needed in certain Namespaces [using ServiceAccount bindings](/docs/concepts/policy/pod-security-policy/#run-another-pod), you will avoid many of the PSP pitfalls and have an easy migration to PSP Replacement Policy. If your needs are much more complex than this, your effort is probably better spent adopting one of the more fully-featured external admission controllers mentioned above.
We’re dedicated to making Kubernetes the best container orchestration tool we can, and sometimes that means we need to remove longstanding features to make space for better things to come. When that happens, the Kubernetes deprecation policy ensures you have plenty of time to plan your next move. In the case of PodSecurityPolicy, several options are available to suit a range of needs and use cases. Start planning ahead now for PSP’s eventual removal, and please consider contributing to its replacement! Happy securing!
**Acknowledgment:** It takes a wonderful group to make wonderful software. Thanks are due to everyone who has contributed to the PSP replacement effort, especially (in alphabetical order) Tim Allclair, Ian Coldwater, and Jordan Liggitt. It’s been a joy to work with y’all on this.
We’re pleased to announce the release of Kubernetes 1.21, our first release of 2021! This release consists of 51 enhancements: 13 enhancements have graduated to stable, 16 enhancements are moving to beta, 20 enhancements are entering alpha, and 2 features have been deprecated.
This release cycle, we saw a major shift in ownership of processes around the release team. We moved from a synchronous mode of communication, where we periodically asked the community for inputs, to a mode where the community opts-in to contribute features and/or blogs to the release. These changes have resulted in an increase in collaboration and teamwork across the community. The result of all that is reflected in Kubernetes 1.21 having the most number of features in the recent times.
## Major Themes
### CronJobs Graduate to Stable!
[CronJobs](/docs/concepts/workloads/controllers/cron-jobs/) (previously ScheduledJobs) has been a beta feature since Kubernetes 1.8! With 1.21 we get to finally see this widely used API graduate to stable.
CronJobs are meant for performing regular scheduled actions such as backups, report generation, and so on. Each of those tasks should be configured to recur indefinitely (for example: once a day / week / month); you can define the point in time within that interval when the job should start.
### Immutable Secrets and ConfigMaps
[Immutable Secrets](/docs/concepts/configuration/secret/#secret-immutable) and [ConfigMaps](/docs/concepts/configuration/configmap/#configmap-immutable) add a new field to those resource types that will reject changes to those objects if set. Secrets and ConfigMaps by default are mutable which is beneficial for pods that are able to consume changes. Mutating Secrets and ConfigMaps can also cause problems if a bad configuration is pushed for pods that use them.
By marking Secrets and ConfigMaps as immutable you can be sure your application configuration won't change. If you want to make changes you'll need to create a new, uniquly named Secret or ConfigMap and deploy a new pod to consume that resource. Immutable resources also have scaling benefits because controllers do not need to poll the API server to watch for changes.
This feature has graduated to stable in Kubernetes 1.21.
### IPv4/IPv6 dual-stack support
IP addresses are a consumable resource that cluster operators and administrators need to make sure are not exhausted. In particular, public IPv4 addresses are now scarce. Having dual-stack support enables native IPv6 routing to pods and services, whilst still allowing your cluster to talk IPv4 where needed. Dual-stack cluster networking also improves a possible scaling limitation for workloads.
Dual-stack support in Kubernetes means that pods, services, and nodes can get IPv4 addresses and IPv6 addresses. In Kubernetes 1.21 [dual-stack networking](/docs/concepts/services-networking/dual-stack/) has graduated from alpha to beta, and is now enabled by default.
### Graceful Node Shutdown
[Graceful Node shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown) also graduated to beta with this release (and will now be available to a much larger group of users)! This is a hugely beneficial feature that allows the kubelet to be aware of node shutdown, and gracefully terminate pods that are scheduled to that node.
Currently, when a node shuts down, pods do not follow the expected termination lifecycle and are not shut down gracefully. This can introduce problems with a lot of different workloads. Going forward, the kubelet will be able to detect imminent system shutdown through systemd, then inform running pods so they can terminate as gracefully as possible.
### PersistentVolume Health Monitor
Persistent Volumes (PV) are commonly used in applications to get local, file-based storage. They can be used in many different ways and help users migrate applications without needing to re-write storage backends.
Kubernetes 1.21 has a new alpha feature which allows PVs to be monitored for health of the volume and marked accordingly if the volume becomes unhealthy. Workloads will be able to react to the health state to protect data from being written or read from a volume that is unhealthy.
### Reducing Kubernetes Build Maintenance
Previously Kubernetes has maintained multiple build systems. This has often been a source of friction and complexity for new and current contributors.
Over the last release cycle, a lot of work has been put in to simplify the build process, and standardize on the native Golang build tools. This should empower broader community maintenance, and lower the barrier to entry for new contributors.
## Major Changes
### PodSecurityPolicy Deprecation
In Kubernetes 1.21, PodSecurityPolicy is deprecated. As with all Kubernetes feature deprecations, PodSecurityPolicy will continue to be available and fully-functional for several more releases. PodSecurityPolicy, previously in the beta stage, is planned for removal in Kubernetes 1.25.
What's next? We're developing a new built-in mechanism to help limit Pod privileges, with a working title of “PSP Replacement Policy.” Our plan is for this new mechanism to cover the key PodSecurityPolicy use cases, with greatly improved ergonomics and maintainability. To learn more, read [PodSecurityPolicy Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future).
### TopologyKeys Deprecation
The Service field `topologyKeys` is now deprecated; all the component features that used this field were previously alpha, and are now also deprecated.
We've replaced `topologyKeys` with a way to implement topology-aware routing, called topology-aware hints. Topology-aware hints are an alpha feature in Kubernetes 1.21. You can read more details about the replacement feature in [Topology Aware Hints](/docs/concepts/services-networking/service-topology/); the related [KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/2433-topology-aware-hints/README.md) explains the context for why we switched.
* [External client-go credential providers](https://github.com/kubernetes/enhancements/issues/541) - beta in 1.21
* [Structured logging](https://github.com/kubernetes/enhancements/issues/1602) - graduating to beta in 1.22
* [TTL after finish cleanup for Jobs and Pods](https://github.com/kubernetes/enhancements/issues/592) - graduated to beta
# Release notes
You can check out the full details of the 1.21 release in the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md).
# Availability of release
Kubernetes 1.21 is available for [download on GitHub](https://github.com/kubernetes/kubernetes/releases/tag/v1.21.0). There are some great resources out there for getting started with Kubernetes. You can check out some [interactive tutorials](https://kubernetes.io/docs/tutorials/) on the main Kubernetes site, or run a local cluster on your machine using Docker containers with [kind](https://kind.sigs.k8s.io). If you’d like to try building a cluster from scratch, check out the [Kubernetes the Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way) tutorial by Kelsey Hightower.
# Release Team
This release was made possible by a very dedicated group of individuals, who came together as a team in the midst of a lot of things happening out in the world. A huge thank you to the release lead Nabarun Pal, and to everyone else on the release team for supporting each other, and working so hard to deliver the 1.21 release for the community.
The Kubernetes 1.21 Release Logo portrays the global nature of the Release Team, with release team members residing in timezones from UTC+8 all the way to UTC-8. The diversity of the release team brought in a lot of challenges, but the team tackled them all by adopting more asynchronous communication practices. The heptagonal globe in the release logo signifies the sheer determination of the community to overcome the challenges as they come. It celebrates the amazing teamwork of the release team over the last 3 months to bring in a fun packed Kubernetes release!
The logo is designed by [Aravind Sekar](https://www.behance.net/noblebatman), an independent designer based out of India. Aravind helps open source communities like PyCon India in their design efforts.
# User Highlights
- CNCF welcomes 47 new organizations across the globe as members to advance Cloud Native technology further at the start of 2021! These [new members](https://www.cncf.io/announcements/2021/02/24/cloud-native-computing-foundation-welcomes-47-new-members-at-the-start-of-2021/) will join CNCF at the upcoming 2021 KubeCon + CloudNativeCon events, including [KubeCon + CloudNativeCom EU – Virtual](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/) from May 4 – 7, 2021, and [KubeCon + CloudNativeCon NA in Los Angeles](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/) from October 12 – 15, 2021.
# Project Velocity
The [CNCF K8s DevStats project](https://k8s.devstats.cncf.io/) aggregates a number of interesting data points related to the velocity of Kubernetes and various sub-projects. This includes everything from individual contributions to the number of companies that are contributing, and is a neat illustration of the depth and breadth of effort that goes into evolving this ecosystem.
In the v1.21 release cycle, which ran for 12 weeks (January 11 to April 8), we saw contributions from [999 companies](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=v1.20.0%20-%20now&var-metric=contributions) and [1279 individuals](https://k8s.devstats.cncf.io/d/66/developer-activity-counts-by-companies?orgId=1&var-period_name=v1.20.0%20-%20now&var-metric=contributions&var-repogroup_name=Kubernetes&var-country_name=All&var-companies=All).
# Ecosystem Updates
- In the wake of rising racism & attacks on global Asian communities, read the statement from CNCF General Priyanka Sharma on the [CNCF blog](https://www.cncf.io/blog/2021/03/18/statement-from-cncf-general-manager-priyanka-sharma-on-the-unacceptable-attacks-against-aapi-and-asian-communities/) reinstating the community's commitment towards inclusive values & diversity-powered resilience.
- We now have a process in place for migration of the default branch from master → main. Learn more about the guidelines [here](k8s.dev/rename)
- CNCF and the Linux Foundation have announced the availability of their new training course, [LFS260 – Kubernetes Security Essentials](https://training.linuxfoundation.org/training/kubernetes-security-essentials-lfs260/). In addition to providing skills and knowledge on a broad range of best practices for securing container-based applications and Kubernetes platforms, the course is also a great way to prepare for the recently launched [Certified Kubernetes Security Specialist](https://training.linuxfoundation.org/certification/certified-kubernetes-security-specialist/) certification exam.
# Event Updates
- KubeCon + CloudNativeCon Europe 2021 will take place May 4 - 7, 2021! You can find more information about the conference [here](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/).
- [Kubernetes Community Days](https://kubernetescommunitydays.org/) are being relaunched! Q2 2021 will start with Africa and Bengaluru.
# Upcoming release webinar
Join the members of the Kubernetes 1.21 release team on May 13th, 2021 to learn about the major features in this release including IPv4/IPv6 dual-stack support, PersistentVolume Health Monitor, Immutable Secrets and ConfigMaps, and many more. Register here: [https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-live-webinar-kubernetes-121-release/](https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-live-webinar-kubernetes-121-release/)
# Get Involved
If you’re interested in contributing to the Kubernetes community, Special Interest Groups (SIGs) are a great starting point. Many of them may align with your interests! If there are things you’d like to share with the community, you can join the weekly community meeting, or use any of the following channels:
* Find out more about contributing to Kubernetes at the [Kubernetes Contributor website](https://www.kubernetes.dev/)
* Follow us on Twitter [@Kubernetesio](https://twitter.com/kubernetesio) for latest updates
* Join the community discussion on [Discuss](https://discuss.kubernetes.io/)
* Join the community on [Slack](http://slack.k8s.io/)
* Share your Kubernetes [story](https://github.com/cncf/foundation/blob/master/case-study-guidelines.md)
* Read more about what’s happening with Kubernetes on the [blog](https://kubernetes.io/blog/)
* Learn more about the [Kubernetes Release Team](https://github.com/kubernetes/sig-release/tree/master/release-team)
solve use-cases that require Pods to run forever, Jobs are useful when Pods need
to run to completion. Commonly used in parallel batch processing, Jobs can be
used in a variety of applications ranging from video rendering and database
maintenance to sending bulk emails and scientific computing.
While the amount of parallelism and the conditions for Job completion are
configurable, the Kubernetes API lacked the ability to suspend and resume Jobs.
This is often desired when cluster resources are limited and a higher priority
Job needs to execute in the place of another Job. Deleting the lower priority
Job is a poor workaround as Pod completion history and other metrics associated
with the Job will be lost.
With the recent Kubernetes 1.21 release, you will be able to suspend a Job by
updating its spec. The feature is currently in **alpha** and requires you to
enable the `SuspendJob` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/)
and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/)
in order to use it.
## API changes
We introduced a new boolean field `suspend` into the `.spec` of Jobs. Let's say
I create the following Job:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: my-job
spec:
suspend: true
parallelism: 2
completions: 10
template:
spec:
containers:
- name: my-container
image: busybox
command: ["sleep", "5"]
restartPolicy: Never
```
Jobs are not suspended by default, so I'm explicitly setting the `suspend` field
to _true_ in the `.spec` of the above Job manifest. In the above example, the
Job controller will refrain from creating Pods until I'm ready to start the Job,
which I can do by updating `suspend` to false.
As another example, consider a Job that was created with the `suspend` field
omitted. The Job controller will happily create Pods to work towards Job
completion. However, before the Job completes, if I explicitly set the field to
true with a Job update, the Job controller will terminate all active Pods that
are running and will wait indefinitely for the flag to be flipped back to false.
Typically, Pod termination is done by sending a SIGTERM signal to all container
processes in the Pod; the [graceful termination period](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)
defined in the Pod spec will be honoured. Pods terminated this way will not be
counted as failures by the Job controller.
It is important to understand that succeeded and failed Pods from the past will
continue to exist after you suspend a Job. That is, that they will count towards
Job completion once you resume it. You can verify this by looking at Job's
status before and after suspension.
Read the [documentation](/docs/concepts/workloads/controllers/job#suspending-a-job)
for a full overview of this new feature.
## Where is this useful?
Let's say I'm the operator of a large cluster. I have many users submitting Jobs
to the cluster, but not all Jobs are created equal — some Jobs are more
important than others. Cluster resources aren't infinite either, so all users
must share resources. If all Jobs were created in the suspended state and placed
in a pending queue, I can achieve priority-based Job scheduling by resuming Jobs
in the right order.
As another motivational use-case, consider a cloud provider where compute
resources are cheaper at night than in the morning. If I have a long-running Job
that takes multiple days to complete, being able to suspend the Job in the
morning and then resume it in the evening every day can reduce costs.
Since this field is a part of the Job spec, [CronJobs](/docs/concepts/workloads/controllers/cron-jobs/)
automatically get this feature for free too.
## References and next steps
If you're interested in a deeper dive into the rationale behind this feature and
the decisions we have taken, consider reading the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2232-suspend-jobs).
There's more detail on suspending and resuming jobs in the documentation for [Job](/docs/concepts/workloads/controllers/job#suspending-a-job).
As previously mentioned, this feature is currently in alpha and is available
only if you explicitly opt-in through the `SuspendJob` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
If this is a feature you're interested in, please consider testing suspended
Jobs in your cluster and providing feedback. You can discuss this enhancement [on GitHub](https://github.com/kubernetes/enhancements/issues/2232).
The SIG Apps community also [meets regularly](https://github.com/kubernetes/community/tree/master/sig-apps#meetings)
and can be reached through [Slack or the mailing list](https://github.com/kubernetes/community/tree/master/sig-apps#contact).
Barring any unexpected changes to the API, we intend to graduate the feature to
beta in Kubernetes 1.22, so that the feature becomes available by default.
**Authors:** Lili Cosic (Red Hat), Frederic Branczyk (Polar Signals), Manuel Rüger (Sony Interactive Entertainment), Tariq Ibrahim (Salesforce)
## What?
[kube-state-metrics](https://github.com/kubernetes/kube-state-metrics), a project under the Kubernetes organization, generates Prometheus format metrics based on the current state of the Kubernetes native resources. It does this by listening to the Kubernetes API and gathering information about resources and objects, e.g. Deployments, Pods, Services, and StatefulSets. A full list of resources is available in the [documentation](https://github.com/kubernetes/kube-state-metrics/tree/master/docs) of kube-state-metrics.
## Why?
There are numerous useful metrics and insights provided by `kube-state-metrics` right out of the box! These metrics can be used to serve as an insight into your cluster: Either through metrics alone, in the form of dashboards, or through an alerting pipeline. To provide a few examples:
* `kube_pod_container_status_restarts_total` can be used to alert on a crashing pod.
* `kube_deployment_status_replicas` which together with `kube_deployment_status_replicas_available` can be used to alert on whether a deployment is rolled out successfully or stuck.
* `kube_pod_container_resource_requests` and `kube_pod_container_resource_limits` can be used in capacity planning dashboards.
And there are many more metrics available! To learn more about the other metrics and their details, please check out the [documentation](https://github.com/kubernetes/kube-state-metrics/tree/master/docs#readme).
## What is new in v2.0?
So now that we know what kube-state-metrics is, we are excited to announce the next release: kube-state-metrics v2.0! This release was long-awaited and started with an alpha release in September 2020. To ease maintenance we removed tech debt and also adjusted some confusing wording around user-facing flags and APIs. We also removed some metrics that caused unnecessarily high cardinality in Prometheus! For the 2.0 release, we took the time to set up scale and performance testing. This allows us to better understand if we hit any issues in large clusters and also to document resource request recommendations for your clusters. In this release (and v1.9.8) container builds providing support for multiple architectures were introduced allowing you to run kube-state-metrics on ARM, ARM64, PPC64 and S390x as well!
So without further ado, here is the list of more noteworthy user-facing breaking changes. A full list of changes, features and bug fixes is available in the changelog at the end of this post.
* Flag `--namespace` was renamed to `--namespaces`. If you are using the former, please make sure to update the flag before deploying the latest release.
* Flag `--collectors` was renamed to `--resources`.
* Flags `--metric-blacklist` and `--metric-whitelist` were renamed to `--metric-denylist` and `--metric-allowlist`.
* Flag `--metric-labels-allowlist` allows you to specify a list of Kubernetes labels that get turned into the dimensions of the `kube_<resource-name>_labels` metrics. By default, the metric contains only name and namespace labels.
* All metrics with a prefix of `kube_hpa_*` were renamed to `kube_horizontalpodautoscaler_*`.
* Metric labels that relate to Kubernetes were converted to snake_case.
* If you are importing kube-state-metrics as a library, we have updated our go module path to `k8s.io/kube-state-metrics/v2`
* All deprecated stable metrics were removed as per the [notice in the v1.9 release](https://github.com/kubernetes/kube-state-metrics/tree/release-1.9/docs#metrics-deprecation).
* `quay.io/coreos/kube-state-metrics` images will no longer be updated. `k8s.gcr.io/kube-state-metrics/kube-state-metrics` is the new canonical location.
* The helm chart that is part of the kubernetes/kube-state-metrics repository is deprecated. https://github.com/prometheus-community/helm-charts will be its new location.
For the full list of v2.0 release changes includes features, bug fixes and other breaking changes see the full [CHANGELOG](https://github.com/kubernetes/kube-state-metrics/blob/master/CHANGELOG.md).
## Found a problem?
Thanks to all our users for testing so far and thank you to all our contributors for your issue reports as well as code and documentation changes! If you find any problems, we the [maintainers](https://github.com/kubernetes/kube-state-metrics/blob/master/OWNERS) are more than happy to look into them, so please report them by opening a [GitHub issue](https://github.com/kubernetes/kube-state-metrics/issues/new/choose).
**Authors:** Ryan Bezdicek (Medtronic), Jim Bugwadia (Nirmata), Tasha Drew (VMware), Fei Guo (Alibaba), Adrian Ludwin (Google)
Kubernetes clusters are typically used by several teams in an organization. In other cases, Kubernetes may be used to deliver applications to end users requiring segmentation and isolation of resources across users from different organizations. Secure sharing of Kubernetes control plane and worker node resources allows maximizing productivity and saving costs in both cases.
The Kubernetes Multi-Tenancy Working Group is chartered with defining tenancy models for Kubernetes and making it easier to operationalize tenancy related use cases. This blog post, from the working group members, describes three common tenancy models and introduces related working group projects.
We will also be presenting on this content and discussing different use cases at our Kubecon EU 2021 panel session, [Multi-tenancy vs. Multi-cluster: When Should you Use What?](https://sched.co/iE66).
## Namespaces as a Service
With the *namespaces-as-a-service* model, tenants share a cluster and tenant workloads are restricted to a set of Namespaces assigned to the tenant. The cluster control plane resources like the API server and scheduler, and worker node resources like CPU, memory, etc. are available for use across all tenants.
To isolate tenant workloads, each namespace must also contain:
* **[role bindings](/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding):** for controlling access to the namespace
* **[network policies](/docs/concepts/services-networking/network-policies/):** to prevent network traffic across tenants
* **[resource quotas](/docs/concepts/policy/resource-quotas/):** to limit usage and ensure fairness across tenants
With this model, tenants share cluster-wide resources like ClusterRoles and CustomResourceDefinitions (CRDs) and hence cannot create or update these cluster-wide resources.
The [Hierarchical Namespace Controller (HNC)](/blog/2020/08/14/introducing-hierarchical-namespaces/) project makes it easier to manage namespace based tenancy by allowing users to create additional namespaces under a namespace, and propagating resources within the namespace hierarchy. This allows self-service namespaces for tenants, without requiring cluster-wide permissions.
The [Multi-Tenancy Benchmarks (MTB)](https://github.com/kubernetes-sigs/multi-tenancy/tree/master/benchmarks) project provides benchmarks and a command-line tool that performs several configuration and runtime checks to report if tenant namespaces are properly isolated and the necessary security controls are implemented.
## Clusters as a Service
With the *clusters-as-a-service* usage model, each tenant gets their own cluster. This model allows tenants to have different versions of cluster-wide resources such as CRDs, and provides full isolation of the Kubernetes control plane.
The tenant clusters may be provisioned using projects like [Cluster API (CAPI)](https://cluster-api.sigs.k8s.io/) where a management cluster is used to provision multiple workload clusters. A workload cluster is assigned to a tenant and tenants have full control over cluster resources. Note that in most enterprises a central platform team may be responsible for managing required add-on services such as security and monitoring services, and for providing cluster lifecycle management services such as patching and upgrades. A tenant administrator may be restricted from modifying the centrally managed services and other critical cluster information.
## Control planes as a Service
In a variation of the *clusters-as-a-service* model, the tenant cluster may be a **virtual cluster** where each tenant gets their own dedicated Kubernetes control plane but share worker node resources. As with other forms of virtualization, users of a virtual cluster see no significant differences between a virtual cluster and other Kubernetes clusters. This is sometimes referred to as `Control Planes as a Service` (CPaaS).
A virtual cluster of this type shares worker node resources and workload state independent control plane components, like the scheduler. Other workload aware control-plane components, like the API server, are created on a per-tenant basis to allow overlaps, and additional components are used to synchronize and manage state across the per-tenant control plane and the underlying shared cluster resources. With this model users can manage their own cluster-wide resources.
The [Virtual Cluster](https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster) project implements this model, where a `supercluster` is shared by multiple `virtual clusters`. The [Cluster API Nested](https://github.com/kubernetes-sigs/cluster-api-provider-nested) project is extending this work to conform to the CAPI model, allowing use of familiar API resources to provision and manage virtual clusters.
## Security considerations
Cloud native security involves different system layers and lifecycle phases as described in the [Cloud Native Security Whitepaper](/blog/2020/11/18/cloud-native-security-for-your-clusters) from CNCF SIG Security. Without proper security measures implemented across all layers and phases, Kubernetes tenant isolation can be compromised and a security breach with one tenant can threaten other tenants.
It is important for any new user to Kubernetes to realize that the default installation of a new upstream Kubernetes cluster is not secure, and you are going to need to invest in hardening it in order to avoid security issues.
At a minimum, the following security measures are required:
* image scanning: container image vulnerabilities can be exploited to execute commands and access additional resources.
* [RBAC](/docs/reference/access-authn-authz/rbac/): for *namespaces-as-a-service* user roles and permissions must be properly configured at a per-namespace level; for other models tenants may need to be restricted from accessing centrally managed add-on services and other cluster-wide resources.
* [network policies](/docs/concepts/services-networking/network-policies/): for *namespaces-as-a-service* default network policies that deny all ingress and egress traffic are recommended to prevent cross-tenant network traffic and may also be used as a best practice for other tenancy models.
* [Kubernetes Pod Security Standards](/docs/concepts/security/pod-security-standards/): to enforce Pod hardening best practices the `Restricted` policy is recommended as the default for tenant workloads with exclusions configured only as needed.
* [CIS Benchmarks for Kubernetes](https://www.cisecurity.org/benchmark/kubernetes/): the CIS Benchmarks for Kubernetes guidelines should be used to properly configure Kubernetes control-plane and worker node components.
Additional recommendations include using:
* policy engines: for configuration security best practices, such as only allowing trusted registries.
* runtime scanners: to detect and report runtime security events.
* VM-based container sandboxing: for stronger data plane isolation.
While proper security is required independently of tenancy models, not having essential security controls like [pod security](/docs/concepts/security/pod-security-standards/) in a shared cluster provides attackers with means to compromise tenancy models and possibly access sensitive information across tenants increasing the overall risk profile.
## Summary
A 2020 CNCF survey showed that production Kubernetes usage has increased by over 300% since 2016. As an increasing number of Kubernetes workloads move to production, organizations are looking for ways to share Kubernetes resources across teams for agility and cost savings.
The **namespaces as a service** tenancy model allows sharing clusters and hence enables resource efficiencies. However, it requires proper security configurations and has limitations as all tenants share the same cluster-wide resources.
The **clusters as a service** tenancy model addresses these limitations, but with higher management and resource overhead.
The **control planes as a service** model provides a way to share resources of a single Kubernetes cluster and also let tenants manage their own cluster-wide resources. Sharing worker node resources increases resource effeciencies, but also exposes cross tenant security and isolation concerns that exist for shared clusters.
In many cases, organizations will use multiple tenancy models to address different use cases and as different product and development teams will have varying needs. Following security and management best practices, such as applying [Pod Security Standards](/docs/concepts/security/pod-security-standards/) and not using the `default` namespace, makes it easer to switch from one model to another.
The [Kubernetes Multi-Tenancy Working Group](https://github.com/kubernetes-sigs/multi-tenancy) has created several projects like [Hierarchical Namespaces Controller](https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/hnc), [Virtual Cluster](https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster) / [CAPI Nested](https://github.com/kubernetes-sigs/cluster-api-provider-nested), and [Multi-Tenancy Benchmarks](https://github.com/kubernetes-sigs/multi-tenancy/tree/master/benchmarks) to make it easier to provision and manage multi-tenancy models.
If you are interested in multi-tenancy topics, or would like to share your use cases, please join us in an upcoming [community meeting](https://github.com/kubernetes/community/blob/master/wg-multitenancy/README.md) or reach out on the *wg-multitenancy channel* on the [Kubernetes slack](https://slack.k8s.io/).
The CSI Volume Health Monitoring feature, originally introduced in 1.19 has undergone a large update for the 1.21 release.
## Why add Volume Health Monitoring to Kubernetes?
Without Volume Health Monitoring, Kubernetes has no knowledge of the state of the underlying volumes of a storage system after a PVC is provisioned and used by a Pod. Many things could happen to the underlying storage system after a volume is provisioned in Kubernetes. For example, the volume could be deleted by accident outside of Kubernetes, the disk that the volume resides on could fail, it could be out of capacity, the disk may be degraded which affects its performance, and so on. Even when the volume is mounted on a pod and used by an application, there could be problems later on such as read/write I/O errors, file system corruption, accidental unmounting of the volume outside of Kubernetes, etc. It is very hard to debug and detect root causes when something happened like this.
Volume health monitoring can be very beneficial to Kubernetes users. It can communicate with the CSI driver to retrieve errors detected by the underlying storage system. PVC events can be reported up to the user to take action. For example, if the volume is out of capacity, they could request a volume expansion to get more space.
## What is Volume Health Monitoring?
CSI Volume Health Monitoring allows CSI Drivers to detect abnormal volume conditions from the underlying storage systems and report them as events on PVCs or Pods.
The Kubernetes components that monitor the volumes and report events with volume health information include the following:
* Kubelet, in addition to gathering the existing volume stats will watch the volume health of the PVCs on that node. If a PVC has an abnormal health condition, an event will be reported on the pod object using the PVC. If multiple pods are using the same PVC, events will be reported on all pods using that PVC.
* An [External Volume Health Monitor Controller](https://github.com/kubernetes-csi/external-health-monitor) watches volume health of the PVCs and reports events on the PVCs.
Note that the node side volume health monitoring logic was an external agent when this feature was first introduced in the Kubernetes 1.19 release. In Kubernetes 1.21, the node side volume health monitoring logic was moved from the external agent into the Kubelet, to avoid making duplicate CSI function calls. With this change in 1.21, a new alpha [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `CSIVolumeHealth` was introduced for the volume health monitoring logic in Kubelet.
Currently the Volume Health Monitoring feature is informational only as it only reports abnormal volume health events on PVCs or Pods. Users will need to check these events and manually fix the problems. This feature serves as a stepping stone towards programmatic detection and resolution of volume health issues by Kubernetes in the future.
## How do I use Volume Health on my Kubernetes Cluster?
To use the Volume Health feature, first make sure the CSI driver you are using supports this feature. Refer to this [CSI drivers doc](https://kubernetes-csi.github.io/docs/drivers.html) to find out which CSI drivers support this feature.
To enable Volume Health Monitoring from the node side, the alpha feature gate `CSIVolumeHealth` needs to be enabled.
If a CSI driver supports the Volume Health Monitoring feature from the controller side, events regarding abnormal volume conditions will be recorded on PVCs.
If a CSI driver supports the Volume Health Monitoring feature from the controller side, user can also get events regarding node failures if the `enable-node-watcher` flag is set to true when deploying the External Health Monitor Controller. When a node failure event is detected, an event will be reported on the PVC to indicate that pods using this PVC are on a failed node.
If a CSI driver supports the Volume Health Monitoring feature from the node side, events regarding abnormal volume conditions will be recorded on pods using the PVCs.
## As a storage vendor, how do I add support for volume health to my CSI driver?
Volume Health Monitoring includes two parts:
* An External Volume Health Monitoring Controller monitors volume health from the controller side.
* Kubelet monitors volume health from the node side.
For details, see the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md) and the [Kubernetes-CSI Driver Developer Guide](https://kubernetes-csi.github.io/docs/volume-health-monitor.html).
There is a sample implementation for volume health in [CSI host path driver](https://github.com/kubernetes-csi/csi-driver-host-path).
### Controller Side Volume Health Monitoring
To learn how to deploy the External Volume Health Monitoring controller, see [CSI external-health-monitor-controller](https://kubernetes-csi.github.io/docs/external-health-monitor-controller.html) in the CSI documentation.
The External Health Monitor Controller calls either `ListVolumes` or `ControllerGetVolume` CSI RPC and reports VolumeConditionAbnormal events with messages on PVCs if abnormal volume conditions are detected. Only CSI drivers with `LIST_VOLUMES` and `VOLUME_CONDITION` controller capability or `GET_VOLUME` and `VOLUME_CONDITION` controller capability support Volume Health Monitoring in the external controller.
To implement the volume health feature from the controller side, a CSI driver **must** add support for the new controller capabilities.
If a CSI driver supports `LIST_VOLUMES` and `VOLUME_CONDITION` controller capabilities, it **must** implement controller RPC `ListVolumes` and report the volume condition in the response.
If a CSI driver supports `GET_VOLUME` and `VOLUME_CONDITION` controller capability, it **must** implement controller PRC `ControllerGetVolume` and report the volume condition in the response.
If a CSI driver supports `LIST_VOLUMES`, `GET_VOLUME`, and `VOLUME_CONDITION` controller capabilities, only `ListVolumes` CSI RPC will be invoked by the External Health Monitor Controller.
### Node Side Volume Health Monitoring
Kubelet calls `NodeGetVolumeStats` CSI RPC and reports VolumeConditionAbnormal events with messages on Pods if abnormal volume conditions are detected. Only CSI drivers with `VOLUME_CONDITION` node capability support Volume Health Monitoring in Kubelet.
To implement the volume health feature from the node side, a CSI driver **must** add support for the new node capabilities.
If a CSI driver supports `VOLUME_CONDITION` node capability, it **must** report the volume condition in node RPC `NodeGetVoumeStats`.
## What’s next?
Depending on feedback and adoption, the Kubernetes team plans to push the CSI volume health implementation to beta in either 1.22 or 1.23.
We are also exploring how to use volume health information for programmatic detection and automatic reconcile in Kubernetes.
## How can I learn more?
To learn the design details for Volume Health Monitoring, read the [Volume Health Monitor](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1432-volume-health-monitor) enhancement proposal.
The Volume Health Monitor controller source code is at [https://github.com/kubernetes-csi/external-health-monitor](https://github.com/kubernetes-csi/external-health-monitor).
There are also more details about volume health checks in the [Container Storage Interface Documentation](https://kubernetes-csi.github.io/docs/).
## How do I get involved?
The [Kubernetes Slack channel #csi](https://kubernetes.slack.com/messages/csi) and any of the [standard SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and the CSI team.
We offer a huge thank you to the contributors who helped release this feature in 1.21. We want to thank Yuquan Ren ([NickrenREN](https://github.com/nickrenren)) who implemented the initial volume health monitor controller and agent in the external health monitor repo, thank Ran Xu ([fengzixu](https://github.com/fengzixu)) who moved the volume health monitoring logic from the external agent to Kubelet in 1.21, and we offer special thanks to the following people for their insightful reviews: David Ashpole ([dashpole](https://github.com/dashpole)), Michelle Au ([msau42](https://github.com/msau42)), David Eads ([deads2k](https://github.com/deads2k)), Elana Hashman ([ehashman](https://github.com/ehashman)), Seth Jennings ([sjenning](https://github.com/sjenning)), and Jiawei Wang ([Jiawei0227](https://github.com/Jiawei0227)).
Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG). We’re rapidly growing and always welcome new contributors.
Authors: Matt Fenwick (Synopsys), Jay Vyas (VMWare), Ricardo Katz, Amim Knabben (Loadsmart), Douglas Schilling Landgraf (Red Hat), Christopher Tomkins (Tigera)
Special thanks to Tim Hockin and Bowie Du (Google), Dan Winship and Antonio Ojea (Red Hat),
Casey Davenport and Shaun Crampton (Tigera), and Abhishek Raut and Antonin Bas (VMware) for
being supportive of this work, and working with us to resolve issues in different Container Network Interfaces (CNIs) over time.
A brief conversation around "node local" Network Policies in April of 2020 inspired the creation of a NetworkPolicy subproject from SIG Network. It became clear that as a community,
we need a rock-solid story around how to do pod network security on Kubernetes, and this story needed a community around it, so as to grow the cultural adoption of enterprise security patterns in K8s.
In this post we'll discuss:
- Why we created a subproject for [Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
- How we changed the Kubernetes e2e framework to `visualize` NetworkPolicy implementation of your CNI provider
- The initial results of our comprehensive NetworkPolicy conformance validator, _Cyclonus_, built around these principles
- Improvements subproject contributors have made to the NetworkPolicy user experience
## Why we created a subproject for NetworkPolicies
In April of 2020 it was becoming clear that many CNIs were emerging, and many vendors
implement these CNIs in subtly different ways. Users were beginning to express a little bit
of confusion around how to implement policies for different scenarios, and asking for new features.
It was clear that we needed to begin unifying the way we think about Network Policies
in Kubernetes, to avoid API fragmentation and unnecessary complexity.
For example:
- In order to be flexible to the user’s environment, Calico as a CNI provider can be run using IPIP or VXLAN mode, or without encapsulation overhead. CNIs such as Antrea
and Cilium offer similar configuration options as well.
- Some CNI plugins offer iptables for NetworkPolicies amongst other options, whereas other CNIs use a completely
different technology stack (for example, the Antrea project uses Open vSwitch rules).
- Some CNI plugins only implement a subset of the Kubernetes NetworkPolicy API, and some a superset. For example, certain plugins don't support the
ability to target a named port; others don't work with certain IP address types, and there are diverging semantics for similar policy types.
- Some CNI plugins combine with OTHER CNI plugins in order to implement NetworkPolicies (canal), some CNI's might mix implementations (multus), and some clouds do routing separately from NetworkPolicy implementation.
Although this complexity is to some extent necessary to support different environments, end-users find that they need to follow a multistep process to implement Network Policies to secure their applications:
- Confirm that their network plugin supports NetworkPolicies (some don't, such as Flannel)
- Confirm that their cluster's network plugin supports the specific NetworkPolicy features that they are interested in (again, the named port or port range examples come to mind here)
- Confirm that their application's Network Policy definitions are doing the right thing
- Find out the nuances of a vendor's implementation of policy, and check whether or not that implementation has a CNI neutral implementation (which is sometimes adequate for users)
The NetworkPolicy project in upstream Kubernetes aims at providing a community where
people can learn about, and contribute to, the Kubernetes NetworkPolicy API and the surrounding ecosystem.
## The First step: A validation framework for NetworkPolicies that was intuitive to use and understand
The Kubernetes end to end suite has always had NetworkPolicy tests, but these weren't
run in CI, and the way they were implemented didn't provide holistic, easily consumable
information about how a policy was working in a cluster.
This is because the original tests didn't provide any kind of visual summary of connectivity
across a cluster. We thus initially set out to make it easy to confirm CNI support for NetworkPolicies by
making the end to end tests (which are often used by administrators or users to diagnose cluster conformance) easy to interpret.
To solve the problem of confirming that CNIs support the basic features most users care about
for a policy, we built a new NetworkPolicy validation tool into the Kubernetes e2e
framework which allows for visual inspection of policies and their effect on a standard set of pods in a cluster.
For example, take the following test output. We found a bug in
[OVN Kubernetes](https://github.com/ovn-org/ovn-kubernetes/issues/1782). This bug has now been resolved. With this tool the bug was really
easy to characterize, wherein certain policies caused a state-modification that,
later on, caused traffic to incorrectly be blocked (even after all Network Policies were deleted from the cluster).
This is the network policy for the test in question:
```yaml
metadata:
creationTimestamp: null
name: allow-ingress-port-80
spec:
ingress:
- ports:
- port: serve-80-tcp
podSelector: {}
```
These are the expected connectivity results. The test setup is 9 pods (3 namespaces: x, y, and z;
and 3 pods in each namespace: a, b, and c); each pod runs a server on the same port and protocol
that can be reached through HTTP calls in the absence of network policies. Connectivity is verified
by using the [agnhost](https://github.com/kubernetes/kubernetes/tree/master/test/images/agnhost) network utility to issue HTTP calls on a port and protocol that other pods are
expected to be serving. A test scenario first
runs a connectivity check to ensure that each pod can reach each other pod, for 81 (= 9 x 9) data
points. This is the "control". Then perturbations are applied, depending on the test scenario:
policies are created, updated, and deleted; labels are added and removed from pods and namespaces,
and so on. After each change, the connectivity matrix is recollected and compared to the expected
connectivity.
These results give a visual indication of connectivity in a simple matrix. Going down the leftmost column is the "source"
pod, or the pod issuing the request; going across the topmost row is the "destination" pod, or the pod
receiving the request. A `.` means that the connection was allowed; an `X` means the connection was
blocked. For example:
```
Nov 4 16:58:43.449: INFO: expected:
- x/a x/b x/c y/a y/b y/c z/a z/b z/c
x/a . . . . . . . . .
x/b . . . . . . . . .
x/c . . . . . . . . .
y/a . . . . . . . . .
y/b . . . . . . . . .
y/c . . . . . . . . .
z/a . . . . . . . . .
z/b . . . . . . . . .
z/c . . . . . . . . .
```
Below are the observed connectivity results in the case of the OVN Kubernetes bug. Notice how the top three rows indicate that
all requests from namespace x regardless of pod and destination were blocked. Since these
experimental results do not match the expected results, a failure will be reported. Note
how the specific pattern of failure provides clear insight into the nature of the problem --
since all requests from a specific namespace fail, we have a clear clue to start our
investigation.
```
Nov 4 16:58:43.449: INFO: observed:
- x/a x/b x/c y/a y/b y/c z/a z/b z/c
x/a X X X X X X X X X
x/b X X X X X X X X X
x/c X X X X X X X X X
y/a . . . . . . . . .
y/b . . . . . . . . .
y/c . . . . . . . . .
z/a . . . . . . . . .
z/b . . . . . . . . .
z/c . . . . . . . . .
```
This was one of our earliest wins in the Network Policy group, as we were able to
identify and work with the OVN Kubernetes group to fix a bug in egress policy processing.
However, even though this tool has made it easy to validate roughly 30 common scenarios,
it doesn't validate *all* Network Policy scenarios - because there are an enormous number of possible
permutations that one might create (technically, we might say this number is
infinite given that there's an infinite number of possible namespace/pod/port/protocol variations one can create).
Once these tests were in play, we worked with the Upstream SIG Network and SIG Testing communities
(thanks to Antonio Ojea and Ben Elder) to put a testgrid Network Policy job in place. This job
continuously runs the entire suite of Network Policy tests against
[GCE with Calico as a Network Policy provider](https://testgrid.k8s.io/sig-network-gce#presubmit-network-policies,%20google-gce).
Part of our role as a subproject is to help make sure that, when these tests break, we can help triage them effectively.
## Cyclonus: The next step towards Network Policy conformance {#cyclonus}
Around the time that we were finishing the validation work, it became clear from the community that,
in general, we needed to solve the overall problem of testing ALL possible Network Policy implementations.
For example, a KEP was recently written which introduced the concept of micro versioning to
Network Policies to accommodate [describing this at the API level](https://github.com/kubernetes/enhancements/pull/2137/files), by Dan Winship.
In response to this increasingly obvious need to comprehensively evaluate Network
Policy implementations from all vendors, Matt Fenwick decided to evolve our approach to Network Policy validation again by creating Cyclonus.
Cyclonus is a comprehensive Network Policy fuzzing tool which verifies a CNI provider
against hundreds of different Network Policy scenarios, by defining similar truth table/policy
combinations as demonstrated in the end to end tests, while also providing a hierarchical
representation of policy "categories". We've found some interesting nuances and issues
in almost every CNI we've tested so far, and have even contributed some fixes back.
To perform a Cyclonus validation run, you create a Job manifest similar to:
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: cyclonus
spec:
template:
spec:
restartPolicy: Never
containers:
- command:
- ./cyclonus
- generate
- --perturbation-wait-seconds=15
- --server-protocol=tcp,udp
name: cyclonus
imagePullPolicy: IfNotPresent
image: mfenwick100/cyclonus:latest
serviceAccount: cyclonus
```
Cyclonus outputs a report of all the test cases it will run:
```
test cases to run by tag:
- target: 6
- peer-ipblock: 4
- udp: 16
- delete-pod: 1
- conflict: 16
- multi-port/protocol: 14
- ingress: 51
- all-pods: 14
- egress: 51
- all-namespaces: 10
- sctp: 10
- port: 56
- miscellaneous: 22
- direction: 100
- multi-peer: 0
- any-port-protocol: 2
- set-namespace-labels: 1
- upstream-e2e: 0
- allow-all: 6
- namespaces-by-label: 6
- deny-all: 10
- pathological: 6
- action: 6
- rule: 30
- policy-namespace: 4
- example: 0
- tcp: 16
- target-namespace: 3
- named-port: 24
- update-policy: 1
- any-peer: 2
- target-pod-selector: 3
- IP-block-with-except: 2
- pods-by-label: 6
- numbered-port: 28
- protocol: 42
- peer-pods: 20
- create-policy: 2
- policy-stack: 0
- any-port: 14
- delete-namespace: 1
- delete-policy: 1
- create-pod: 1
- IP-block-no-except: 2
- create-namespace: 1
- set-pod-labels: 1
testing 112 cases
```
Note that Cyclonus tags its tests based on the type of policy being created, because
the policies themselves are auto-generated, and thus have no meaningful names to be recognized by.
For each test, Cyclonus outputs a truth table, which is again similar to that of the
Both Cyclonus and the e2e tests use the same strategy to validate a Network Policy - probing pods over TCP or UDP, with
SCTP support available as well for CNIs that support it (such as Calico).
As examples of how we use Cyclonus to help make CNI implementations better from a Network Policy perspective, you can see the following issues:
- [Antrea: NetworkPolicy: unable to allow ingress by CIDR](https://github.com/vmware-tanzu/antrea/issues/1764)
- [Calico: default missing protocol to TCP; don't let single port overwrite all ports](https://github.com/projectcalico/libcalico-go/pull/1373)
- [Cilium: Egress Network Policy allows traffic that should be denied](https://github.com/cilium/cilium/issues/14678)
The good news is that Antrea and Calico have already merged fixes for all the issues found and other CNI providers are working on it,
with the support of SIG Network and the Network Policy subproject.
Are you interested in verifying NetworkPolicy functionality on your cluster?
(if you care about security or offer multi-tenant SaaS, you should be)
If so, you can run the upstream end to end tests, or Cyclonus, or both.
- If you're just getting started with NetworkPolicies and want to simply
verify the "common" NetworkPolicy cases that most CNIs should be
implementing correctly, in a way that is quick to diagnose, then you're
better off running the e2e tests only.
- If you are deeply curious about your CNI provider's NetworkPolicy
implementation, and want to verify it: use Cyclonus.
- If you want to test *hundreds* of policies, and evaluate your CNI plugin
for comprehensive functionality, for deep discovery of potential security
holes: use Cyclonus, and also consider running end-to-end cluster tests.
- If you're thinking of getting involved with the upstream NetworkPolicy efforts:
use Cyclonus, and read at least an outline of which e2e tests are relevant.
## Where to start with NetworkPolicy testing?
- Cyclonus is easy to run on your cluster, check out the [instructions on github](https://github.com/mattfenwick/cyclonus#run-as-a-kubernetes-job),
and determine whether *your* specific CNI configuration is fully conformant to the hundreds of different
Kubernetes Network Policy API constructs.
- Alternatively, you can use a tool like [sonobuoy](https://github.com/vmware-tanzu/sonobuoy)
to run the existing E2E tests in Kubernetes, with the `--ginkgo.focus=NetworkPolicy` flag.
Make sure that you use the K8s conformance image for K8s 1.21 or above (for example, by using the `--kube-conformance-image-version v1.21.0` flag),
as older images will not have the *new* Network Policy tests in them.
## Improvements to the NetworkPolicy API and user experience
In addition to cleaning up the validation story for CNI plugins that implement NetworkPolicies,
subproject contributors have also spent some time improving the Kubernetes NetworkPolicy API for a few commonly requested features.
After months of deliberation, we eventually settled on a few core areas for improvement:
- Port Range policies: We now allow you to specify a *range* of ports for a policy.
This allows users interested in scenarios like FTP or virtualization to enable advanced policies.
The port range option for network policies will be available to use in Kubernetes 1.21.
Read more in [targeting a range of ports](/docs/concepts/services-networking/network-policies/#targeting-a-range-of-ports).
- Namespace as name policies: Allowing users in Kubernetes >= 1.21 to target namespaces using names,
when building Network Policy objects. This was done in collaboration with Jordan Liggitt and Tim Hockin on the API Machinery side.
This change allowed us to improve the Network Policy user experience without actually
changing the API! For more details, you can read
[Automatic labelling](/docs/concepts/overview/working-with-objects/namespaces/#automatic-labelling) in the page about Namespaces.
The TL,DR; is that for Kubernetes 1.21 and later, **all namespaces** have the following label added by default:
```
kubernetes.io/metadata.name: <name-of-namespace>
```
This means you can write a namespace policy against this namespace, even if you can't edit its labels.
For example, this policy, will 'just work', without needing to run a command such as `kubectl edit namespace`.
In fact, it will even work if you can't edit or view this namespace's data at all, because of the magic of API server defaulting.
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
# Allow inbound traffic to Pods labelled role=db, in the namespace 'default'
# provided that the source is a Pod in the namespace 'my-namespace'
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: my-namespace
```
## Results
In our tests, we found that:
- Antrea and Calico are at a point where they support all of cyclonus's scenarios, modulo a few very minor tweaks which we've made.
- Cilium also conformed to the majority of the policies, outside known features that aren't fully supported (for example, related to the way Cilium deals with pod CIDR policies).
If you are a CNI provider and interested in helping us to do a better job curating large tests of network policies, please reach out! We are continuing to curate the Network Policy conformance results from Cyclonus [here](https://raw.githubusercontent.com/K8sbykeshed/cyclonus-artifacts/), but
we are not capable of maintaining all of the subtleties in NetworkPolicy testing data on our own. For now, we use github actions and Kind to test in CI.
## The Future
We're also working on some improvements for the future of Network Policies, including:
- Fully qualified Domain policies: The Google Cloud team created a prototype (which
we are really excited about) of [FQDN policies](https://github.com/GoogleCloudPlatform/gke-fqdnnetworkpolicies-golang).
This tool uses the Network Policy API to enforce policies against L7 URLs, by finding
their IPs and blocking them proactively when requests are made.
- Cluster Administrative policies: We're working hard at enabling *administrative* or
*cluster scoped* Network Policies for the future. These are being presented iteratively to the NetworkPolicy subproject.
You can read about them here in [Cluster Scoped Network Policy](https://docs.google.com/presentation/d/1Jk86jtS3TcGAugVSM_I4Yds5ukXFJ4F1ZCvxN5v2BaY/).
The Network Policy subproject meets on mondays at 4PM EST. For details, check out the
[SIG Network community repo](https://github.com/kubernetes/community/tree/master/sig-network). We'd love
to hang out with you, hack on stuff, and help you adopt K8s Network Policies for your cluster wherever possible.
### A quick note on User Feedback
We've gotten a lot of ideas and feedback from users on Network Policies. A lot of people have interesting ideas about Network Policies,
but we've found that as a subproject, very few people were deeply interested in implementing these ideas to the full extent.
Almost every change to the NetworkPolicy API includes weeks or months of discussion to cover different cases, and ensure no CVEs are being introduced. Thus, long term ownership
is the biggest impediment in improving the NetworkPolicy user experience for us, over time.
- We've documented a lot of the history of the Network Policy dialogue [here](https://github.com/jayunit100/network-policy-subproject/blob/master/history.md).
- We've also taken a poll of users, for what they'd like to see in the Network Policy API [here](https://github.com/jayunit100/network-policy-subproject/blob/master/p0_user_stories.md).
We encourage anyone to provide us with feedback, but our most pressing issues right now
involve finding *long term owners to help us drive changes*.
This doesn't require a lot of technical knowledge, but rather, just a long term commitment to helping us stay organized, do paperwork,
and iterate through the many stages of the K8s feature process. If you want to help us and get involved, please reach out on the SIG Network mailing list, or in the SIG Network room in the k8s.io slack channel!
Anyone can put an oar in the water and help make NetworkPolices better!
title: 'Annotating Kubernetes Services for Humans'
date: 2021-04-20
slug: annotating-k8s-for-humans
---
**Author:** Richard Li, Ambassador Labs
Have you ever been asked to troubleshoot a failing Kubernetes service and struggled to find basic information about the service such as the source repository and owner?
One of the problems as Kubernetes applications grow is the proliferation of services. As the number of services grows, developers start to specialize working with specific services. When it comes to troubleshooting, however, developers need to be able to find the source, understand the service and dependencies, and chat with the owning team for any service.
## Human service discovery
Troubleshooting always begins with information gathering. While much attention has been paid to centralizing machine data (e.g., logs, metrics), much less attention has been given to the human aspect of service discovery. Who owns a particular service? What Slack channel does the team work on? Where is the source for the service? What issues are currently known and being tracked?
## Kubernetes annotations
Kubernetes annotations are designed to solve exactly this problem. Oft-overlooked, Kubernetes annotations are designed to add metadata to Kubernetes objects. The Kubernetes documentation says annotations can “attach arbitrary non-identifying metadata to objects.” This means that annotations should be used for attaching metadata that is external to Kubernetes (i.e., metadata that Kubernetes won’t use to identify objects. As such, annotations can contain any type of data. This is a contrast to labels, which are designed for uses internal to Kubernetes. As such, label structure and values are [constrained](/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) so they can be efficiently used by Kubernetes.
## Kubernetes annotations in action
Here is an example. Imagine you have a Kubernetes service for quoting, called the quote service. You can do the following:
```
kubectl annotate service quote a8r.io/owner=”@sally”
```
In this example, we've just added an annotation called `a8r.io/owner` with the value of @sally. Now, we can use `kubectl describe` to get the information.
```
Name: quote
Namespace: default
Labels: <none>
Annotations: a8r.io/owner: @sally
Selector: app=quote
Type: ClusterIP
IP: 10.109.142.131
Port: http 80/TCP
TargetPort: 8080/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
```
If you’re practicing GitOps (and you should be!) you’ll want to code these values directly into your Kubernetes manifest, e.g.,
```yaml
apiVersion: v1
kind: Service
metadata:
name: quote
annotations:
a8r.io/owner: “@sally”
spec:
ports:
- name: http
port: 80
targetPort: 8080
selector:
app: quote
```
## A Convention for Annotations
Adopting a common convention for annotations ensures consistency and understandability. Typically, you’ll want to attach the annotation to the service object, as services are the high-level resource that maps most clearly to a team’s responsibility. Namespacing your annotations is also very important. Here is one set of conventions, documented at [a8r.io](https://a8r.io), and reproduced below:
{{< table caption="Annotation convention for human-readable services">}}
| `a8r.io/description` | Unstructured text description of the service for humans. |
| `a8r.io/owner` | SSO username (GitHub), email address (linked to GitHub account), or unstructured owner description. |
| `a8r.io/chat` | Slack channel, or link to external chat system. |
| `a8r.io/bugs` | Link to external bug tracker. |
| `a8r.io/logs` | Link to external log viewer. |
| `a8r.io/documentation` | Link to external project documentation. |
| `a8r.io/repository` | Link to external VCS repository. |
| `a8r.io/support` | Link to external support center. |
| `a8r.io/runbook` | Link to external project runbook. |
| `a8r.io/incidents` | Link to external incident dashboard. |
| `a8r.io/uptime` | Link to external uptime dashboard. |
| `a8r.io/performance` | Link to external performance dashboard. |
| `a8r.io/dependencies` | Unstructured text describing the service dependencies for humans. |
## Visualizing annotations: Service Catalogs
As the number of microservices and annotations proliferate, running `kubectl describe` can get tedious. Moreover, using `kubectl describe` requires every developer to have some direct access to the Kubernetes cluster. Over the past few years, service catalogs have gained greater visibility in the Kubernetes ecosystem. Popularized by tools such as [Shopify's ServicesDB](https://shopify.engineering/scaling-mobile-development-by-treating-apps-as-services) and [Spotify's System Z](https://dzone.com/articles/modeling-microservices-at-spotify-with-petter-mari), service catalogs are internally-facing developer portals that present critical information about microservices.
Note that these service catalogs should not be confused with the [Kubernetes Service Catalog project](https://svc-cat.io/). Built on the Open Service Broker API, the Kubernetes Service Catalog enables Kubernetes operators to plug in different services (e.g., databases) to their cluster.
## Annotate your services now and thank yourself later
Much like implementing observability within microservice systems, you often don’t realize that you need human service discovery until it’s too late. Don't wait until something is on fire in production to start wishing you had implemented better metrics and also documented how to get in touch with the part of your organization that looks after it.
There's enormous benefits to building an effective “version 0” service: a [_dancing skeleton_](https://containerjournal.com/topics/container-management/dancing-skeleton-apis-and-microservices/) application with a thin slice of complete functionality that can be deployed to production with a minimal yet effective continuous delivery pipeline.
Adding service annotations should be an essential part of your “version 0” for all of your services. Add them now, and you’ll thank yourself later.
**Authors:** David Porter (Google), Mrunal Patel (Red Hat), and Tim Bannister (The Scale Factory)
Graceful node shutdown, beta in 1.21, enables kubelet to gracefully evict pods during a node shutdown.
Kubernetes is a distributed system and as such we need to be prepared for inevitable failures — nodes will fail, containers might crash or be restarted, and - ideally - your workloads will be able to withstand these catastrophic events.
One of the common classes of issues are workload failures on node shutdown or restart. The best practice prior to bringing your node down is to [safely drain and cordon your node](/docs/tasks/administer-cluster/safely-drain-node/). This will ensure that all pods running on this node can safely be evicted. An eviction will ensure your pods can follow the expected [pod termination lifecycle](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) meaning receiving a SIGTERM in your container and/or running `preStopHooks`.
Prior to Kubernetes 1.20 (when graceful node shutdown was introduced as an alpha feature), safe node draining was not easy: it required users to manually take action and drain the node beforehand. If someone or something shut down your node without draining it first, most likely your pods would not be safely evicted from your node and shutdown abruptly. Other services talking to those pods might see errors due to the pods exiting abruptly. Some examples of this situation may be caused by a reboot due to security patches or preemption of short lived cloud compute instances.
Kubernetes 1.21 brings graceful node shutdown to beta. Graceful node shutdown gives you more control over some of those unexpected shutdown situations. With graceful node shutdown, the kubelet is aware of underlying system shutdown events and can propagate these events to pods, ensuring containers can shut down as gracefully as possible. This gives the containers a chance to checkpoint their state or release back any resources they are holding.
Note, that for the best availability, even with graceful node shutdown, you should still design your deployments to be resilient to node failures.
## How does it work?
On Linux, your system can shut down in many different situations. For example:
* A user or script running `shutdown -h now` or `systemctl poweroff` or `systemctl reboot`.
* Physically pressing a power button on the machine.
* Stopping a VM instance on a cloud provider, e.g. `gcloud compute instances stop` on GCP.
* A Preemptible VM or Spot Instance that your cloud provider can terminate unexpectedly, but with a brief warning.
Many of these situations can be unexpected and there is no guarantee that a cluster administrator drained the node prior to these events. With the graceful node shutdown feature, kubelet uses a systemd mechanism called ["Inhibitor Locks"](https://www.freedesktop.org/wiki/Software/systemd/inhibit) to allow draining in most cases. Using Inhibitor Locks, kubelet instructs systemd to postpone system shutdown for a specified duration, giving a chance for the node to drain and evict pods on the system.
Kubelet makes use of this mechanism to ensure your pods will be terminated cleanly. When the kubelet starts, it acquires a systemd delay-type inhibitor lock. When the system is about to shut down, the kubelet can delay that shutdown for a configurable, short duration utilizing the delay-type inhibitor lock it acquired earlier. This gives your pods extra time to terminate. As a result, even during unexpected shutdowns, your application will receive a SIGTERM, [preStop hooks](/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) will execute, and kubelet will properly update `Ready` node condition and respective pod statuses to the api-server.
For example, on a node with graceful node shutdown enabled, you can see that the inhibitor lock is taken by the kubelet:
```
kubelet-node ~ # systemd-inhibit --list
Who: kubelet (UID 0/root, PID 1515/kubelet)
What: shutdown
Why: Kubelet needs time to handle node shutdown
Mode: delay
1 inhibitors listed.
```
One important consideration we took when designing this feature is that not all pods are created equal. For example, some of the pods running on a node such as a logging related daemonset should stay running as long as possible to capture important logs during the shutdown itself. As a result, pods are split into two categories: "regular" and "critical". [Critical pods](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/#marking-pod-as-critical) are those that have `priorityClassName` set to `system-cluster-critical` or `system-node-critical`; all other pods are considered regular.
In our example, the logging DaemonSet would run as a critical pod. During the graceful node shutdown, regular pods are terminated first, followed by critical pods. As an example, this would allow a critical pod associated with a logging daemonset to continue functioning, and collecting logs during the termination of regular pods.
We will evaluate during the beta phase if we need more flexibility for different pod priority classes and add support if needed, please let us know if you have some scenarios in mind.
## How do I use it?
Graceful node shutdown is controlled with the `GracefulNodeShutdown` [feature gate](/docs/reference/command-line-tools-reference/feature-gates) and is enabled by default in Kubernetes 1.21.
You can configure the graceful node shutdown behavior using two kubelet configuration options: `ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods`. To configure these options, you edit the kubelet configuration file that is passed to kubelet via the `--config` flag; for more details, refer to [Set kubelet parameters via a configuration file](/docs/tasks/administer-cluster/kubelet-config-file/).
During a shutdown, kubelet terminates pods in two phases. You can configure how long each of these phases lasts.
1. Terminate regular pods running on the node.
2. Terminate critical pods running on the node.
The settings that control the duration of shutdown are:
* `ShutdownGracePeriod`
* Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and critical pods.
* `ShutdownGracePeriodCriticalPods`
* Specifies the duration used to terminate critical pods during a node shutdown. This should be less than `ShutdownGracePeriod`.
For example, if `ShutdownGracePeriod=30s`, and `ShutdownGracePeriodCriticalPods=10s`, kubelet will delay the node shutdown by 30 seconds. During this time, the first 20 seconds (30-10) would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating critical pods.
Note that by default, both configuration options described above, `ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are set to zero, so you will need to configure them as appropriate for your environment to activate graceful node shutdown functionality.
## How can I learn more?
* Read the [documentation](/docs/concepts/architecture/nodes/#graceful-node-shutdown)
* Read the enhancement proposal, [KEP 2000](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown)
* View the [code](https://github.com/kubernetes/kubernetes/tree/release-1.21/pkg/kubelet/nodeshutdown)
## How do I get involved?
Your feedback is always welcome! SIG Node meets regularly and can be reached via [Slack](https://slack.k8s.io) (channel `#sig-node`), or the SIG's [mailing list](https://github.com/kubernetes/community/tree/master/sig-node#contact)
**Authors:** Mark Church (Google), Harry Bagdi (Kong), Daneyon Hanson (Red Hat), Nick Young (VMware), Manuel Zapf (Traefik Labs)
The Ingress resource is one of the many Kubernetes success stories. It created a [diverse ecosystem of Ingress controllers](/docs/concepts/services-networking/ingress-controllers/) which were used across hundreds of thousands of clusters in a standardized and consistent way. This standardization helped users adopt Kubernetes. However, five years after the creation of Ingress, there are signs of fragmentation into different but [strikingly similar CRDs](https://dave.cheney.net/paste/ingress-is-dead-long-live-ingressroute.pdf) and [overloaded annotations](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/). The same portability that made Ingress pervasive also limited its future.
It was at Kubecon 2019 San Diego when a passionate group of contributors gathered to discuss the [evolution of Ingress](https://static.sched.com/hosted_files/kccncna19/a5/Kubecon%20San%20Diego%202019%20-%20Evolving%20the%20Kubernetes%20Ingress%20APIs%20to%20GA%20and%20Beyond%20%5BPUBLIC%5D.pdf). The discussion overflowed to the hotel lobby across the street and what came out of it would later be known as the [Gateway API](https://gateway-api.sigs.k8s.io). This discussion was based on a few key assumptions:
1. The API standards underlying route matching, traffic management, and service exposure are commoditized and provide little value to their implementers and users as custom APIs
2. It’s possible to represent L4/L7 routing and traffic management through common core API resources
3. It’s possible to provide extensibility for more complex capabilities in a way that does not sacrifice the user experience of the core API
## Introducing the Gateway API
This led to design principles that allow the Gateway API to improve upon Ingress:
- **Expressiveness** - In addition to HTTP host/path matching and TLS, Gateway API can express capabilities like HTTP header manipulation, traffic weighting & mirroring, TCP/UDP routing, and other capabilities that were only possible in Ingress through custom annotations.
- **Role-oriented design** - The API resource model reflects the separation of responsibilities that is common in routing and Kubernetes service networking.
- **Extensibility** - The resources allow arbitrary configuration attachment at various layers within the API. This makes granular customization possible at the most appropriate places.
- **Flexible conformance** - The Gateway API defines varying conformance levels - core (mandatory support), extended (portable if supported), and custom (no portability guarantee), known together as [flexible conformance](https://gateway-api.sigs.k8s.io/concepts/guidelines/#conformance). This promotes a highly portable core API (like Ingress) that still gives flexibility for Gateway controller implementers.
### What does the Gateway API look like?
The Gateway API introduces a few new resource types:
- **[GatewayClasses](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.GatewayClass)** are cluster-scoped resources that act as templates to explicitly define behavior for Gateways derived from them. This is similar in concept to StorageClasses, but for networking data-planes.
- **[Gateways](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.Gateway)** are the deployed instances of GatewayClasses. They are the logical representation of the data-plane which performs routing, which may be in-cluster proxies, hardware LBs, or cloud LBs.
- **Routes** are not a single resource, but represent many different protocol-specific Route resources. The [HTTPRoute](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.HTTPRoute) has matching, filtering, and routing rules that get applied to Gateways that can process HTTP and HTTPS traffic. Similarly, there are [TCPRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.TCPRoute), [UDPRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.UDPRoute), and [TLSRoutes](https://gateway-api.sigs.k8s.io/references/spec/#networking.x-k8s.io/v1alpha1.TLSRoute) which also have protocol-specific semantics. This model also allows the Gateway API to incrementally expand its protocol support in the future.
### Gateway Controller Implementations
The good news is that although Gateway is in [Alpha](https://github.com/kubernetes-sigs/gateway-api/releases), there are already several [Gateway controller implementations](https://gateway-api.sigs.k8s.io/references/implementations/) that you can run. Since it’s a standardized spec, the following example could be run on any of them and should function the exact same way. Check out [getting started](https://gateway-api.sigs.k8s.io/guides/getting-started/) to see how to install and use one of these Gateway controllers.
## Getting Hands-on with the Gateway API
In the following example, we’ll demonstrate the relationships between the different API Resources and walk you through a common use case:
* Team foo has their app deployed in the foo Namespace. They need to control the routing logic for the different pages of their app.
* Team bar is running in the bar Namespace. They want to be able to do blue-green rollouts of their application to reduce risk.
* The platform team is responsible for managing the load balancer and network security of all the apps in the Kubernetes cluster.
The following foo-route does path matching to various Services in the foo Namespace and also has a default route to a 404 server. This exposes foo-auth and foo-home Services via `foo.example.com/login` and `foo.example.com/home` respectively.:
```yaml
kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
name: foo-route
namespace: foo
labels:
gateway: external-https-prod
spec:
hostnames:
- "foo.example.com"
rules:
- matches:
- path:
type: Prefix
value: /login
forwardTo:
- serviceName: foo-auth
port: 8080
- matches:
- path:
type: Prefix
value: /home
forwardTo:
- serviceName: foo-home
port: 8080
- matches:
- path:
type: Prefix
value: /
forwardTo:
- serviceName: foo-404
port: 8080
```
The bar team, operating in the bar Namespace of the same Kubernetes cluster, also wishes to expose their application to the internet, but they also want to control their own canary and blue-green rollouts. The following HTTPRoute is configured for the following behavior:
* For traffic to `bar.example.com`:
* Send 90% of the traffic to bar-v1
* Send 10% of the traffic to bar-v2
* For traffic to `bar.example.com` with the HTTP header `env: canary`:
* Send all the traffic to bar-v2
```yaml
kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
name: bar-route
namespace: bar
labels:
gateway: external-https-prod
spec:
hostnames:
- "bar.example.com"
rules:
- forwardTo:
- serviceName: bar-v1
port: 8080
weight: 90
- serviceName: bar-v2
port: 8080
weight: 10
- matches:
- headers:
values:
env: canary
forwardTo:
- serviceName: bar-v2
port: 8080
```
### Route and Gateway Binding
So we have two HTTPRoutes matching and routing traffic to different Services. You might be wondering, where are these Services accessible? Through which networks or IPs are they exposed?
How Routes are exposed to clients is governed by [Route binding](https://gateway-api.sigs.k8s.io/concepts/api-overview/#route-binding), which describes how Routes and Gateways create a bidirectional relationship between each other. When Routes are bound to a Gateway it means their collective routing rules are configured on the underlying load balancers or proxies and the Routes are accessible through the Gateway. Thus, a Gateway is a logical representation of a networking data plane that can be configured through Routes.
### Administrative Delegation
The split between Gateway and Route resources allows the cluster administrator to delegate some of the routing configuration to individual teams while still retaining centralized control. The following Gateway resource exposes HTTPS on port 443 and terminates all traffic on the port with a certificate controlled by the cluster administrator.
```yaml
kind: Gateway
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
name: prod-web
spec:
gatewayClassName: acme-lb
listeners:
- protocol: HTTPS
port: 443
routes:
kind: HTTPRoute
selector:
matchLabels:
gateway: external-https-prod
namespaces:
from: All
tls:
certificateRef:
name: admin-controlled-cert
```
The following HTTPRoute shows how the Route can ensure it matches the Gateway's selector via it’s `kind` (HTTPRoute) and resource labels (`gateway=external-https-prod`).
```yaml
# Matches the required kind selector on the Gateway
kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
name: foo-route
namespace: foo-ns
labels:
# Matches the required label selector on the Gateway
gateway: external-https-prod
...
```
### Role Oriented Design
When you put it all together, you have a single load balancing infrastructure that can be safely shared by multiple teams. The Gateway API is not only a more expressive API for advanced routing, but is also a role-oriented API, designed for multi-tenant infrastructure. Its extensibility ensures that it will evolve for future use-cases while preserving portability. Ultimately these characteristics will allow the Gateway API to adapt to different organizational models and implementations well into the future.
### Try it out and get involved
There are many resources to check out to learn more.
* Check out the [user guides](https://gateway-api.sigs.k8s.io/guides/getting-started/) to see what use-cases can be addressed.
* Try out one of the [existing Gateway controllers ](https://gateway-api.sigs.k8s.io/references/implementations/)
* Or [get involved](https://gateway-api.sigs.k8s.io/contributing/community/) and help design and influence the future of Kubernetes service networking!
**Authors**: Han Kang (Google), Elana Hashman (Red Hat)
Kubernetes 1.21 marks the graduation of the metrics stability framework and along with it, the first officially supported stable metrics. Not only do stable metrics come with supportability guarantees, the metrics stability framework brings escape hatches that you can use if you encounter problematic metrics.
See the list of [stable Kubernetes metrics here](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml)
### What are stable metrics and why do we need them?
A stable metric is one which, from a consumption point of view, can be reliably consumed across a number of Kubernetes versions without risk of ingestion failure.
Metrics stability is an ongoing community concern. Cluster monitoring infrastructure often assumes the stability of some control plane metrics, so we have introduced a mechanism for versioning metrics as a proper API, with stability guarantees around a formal metrics deprecation process.
### What are the stability levels for metrics?
Metrics can currently have one of two stability levels: alpha or stable.
_Alpha metrics_ have no stability guarantees; as such they can be modified or deleted at any time. At this time, all Kubernetes metrics implicitly fall into this category.
_Stable metrics_ can be guaranteed to not change, except that the metric may become marked deprecated for a future Kubernetes version. By not change, we mean three things:
1. the metric itself will not be deleted or renamed
2. the type of metric will not be modified
3. no labels can be added or removed from this metric
From an ingestion point of view, it is backwards-compatible to add or remove possible values for labels which already do exist, but not labels themselves. Therefore, adding or removing values from an existing label is permitted. Stable metrics can also be marked as deprecated for a future Kubernetes version, since this is tracked in a metadata field and does not actually change the metric itself.
Removing or adding labels from stable metrics is not permitted. In order to add or remove a label from an existing stable metric, one would have to introduce a new metric and deprecate the stable one; otherwise this would violate compatibility agreements.
#### How are metrics deprecated?
While deprecation policies only affect stability guarantees for stable metrics (and not alpha ones), deprecation information may be optionally provided on alpha metrics to help component owners inform users of future intent and assist with transition plans.
A stable metric undergoing the deprecation process signals that the metric will eventually be deleted. The metrics deprecation lifecycle looks roughly like this (with each stage representing a Kubernetes release):
_Deprecated metrics_ have the same stability guarantees of their stable counterparts. If a stable metric is deprecated, then a deprecated stable metric is guaranteed to not change. When deprecating a stable metric, a future Kubernetes release is specified as the point from which the metric will be considered deprecated.
Deprecated metrics will have their description text prefixed with a deprecation notice string “(Deprecated from x.y)” and a warning log will be emitted during metric registration, in the spirit of the official Kubernetes deprecation policy.
Like their stable metric counterparts, deprecated metrics will be automatically registered to the metrics endpoint. On a subsequent release (when the metric's deprecatedVersion is equal to _current\_kubernetes\_version - 4_)), a deprecated metric will become a _hidden_ metric. _Hidden metrics_ are not automatically registered, and hence are hidden by default from end users. These hidden metrics can be explicitly re-enabled for one release after they reach the hidden state, to provide a migration path for cluster operators.
#### As an owner of a Kubernetes component, how do I add stable metrics?
During metric instantiation, stability can be specified by setting the metadata field, StabilityLevel, to “Stable”. When a StabilityLevel is not explicitly set, metrics default to “Alpha” stability. Note that metrics which have fields determined at runtime cannot be marked as Stable. Stable metrics will be detected during static analysis during the pre-commit phase, and must be reviewed by sig-instrumentation.
```golang
var metricDefinition = kubemetrics.CounterOpts{
Name: "some_metric",
Help: "some description",
StabilityLevel: kubemetrics.STABLE,
}
```
For more examples of setting metrics stability and deprecation, see the [Metrics Stability KEP](http://bit.ly/metrics-stability).
### How do I get involved?
This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together.
We offer a huge thank you to all the contributors in Kubernetes community who helped review the design and implementation of the project, including but not limited to the following:
- Han Kang (logicalhan)
- Frederic Branczyk (brancz)
- Marek Siarkowicz (serathius)
- Elana Hashman (ehashman)
- Solly Ross (DirectXMan12)
- Stefan Schimanski (sttts)
- David Ashpole (dashpole)
- Yuchen Zhou (yoyinzyc)
- Yu Yi (erain)
If you’re interested in getting involved with the design and development of instrumentation or any part of the Kubernetes metrics system, join the [Kubernetes Instrumentation Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-instrumentation). We’re rapidly growing and always welcome new contributors.
Deleting objects in Kubernetes can be challenging. You may think you’ve deleted something, only to find it still persists. While issuing a `kubectl delete` command and hoping for the best might work for day-to-day operations, understanding how Kubernetes `delete` commands operate will help you understand why some objects linger after deletion.
In this post, I’ll look at:
- What properties of a resource govern deletion
- How finalizers and owner references impact object deletion
- How the propagation policy can be used to change the order of deletions
- How deletion works, with examples
For simplicity, all examples will use ConfigMaps and basic shell commands to demonstrate the process. We’ll explore how the commands work and discuss repercussions and results from using them in practice.
## The basic `delete`
Kubernetes has several different commands you can use that allow you to create, read, update, and delete objects. For the purpose of this blog post, we’ll focus on four `kubectl` commands: `create`, `get`, `patch`, and `delete`.
Here are examples of the basic `kubectl delete` command:
```
kubectl create configmap mymap
configmap/mymap created
```
```
kubectl get configmap/mymap
NAME DATA AGE
mymap 0 12s
```
```
kubectl delete configmap/mymap
configmap "mymap" deleted
```
```
kubectl get configmap/mymap
Error from server (NotFound): configmaps "mymap" not found
```
Shell commands preceded by `$` are followed by their output. You can see that we begin with a `kubectl create configmap mymap`, which will create the empty configmap `mymap`. Next, we need to `get` the configmap to prove it exists. We can then delete that configmap. Attempting to `get` it again produces an HTTP 404 error, which means the configmap is not found.
The state diagram for the basic `delete` command is very simple:
{{<figure width="495" src="/images/blog/2021-05-14-using-finalizers-to-control-deletion/state-diagram-delete.png" caption="State diagram for delete">}}
Although this operation is straightforward, other factors may interfere with the deletion, including finalizers and owner references.
## Understanding Finalizers
When it comes to understanding resource deletion in Kubernetes, knowledge of how finalizers work is helpful and can help you understand why some objects don’t get deleted.
Finalizers are keys on resources that signal pre-delete operations. They control the garbage collection on resources, and are designed to alert controllers what cleanup operations to perform prior to removing a resource. However, they don’t necessarily name code that should be executed; finalizers on resources are basically just lists of keys much like annotations. Like annotations, they can be manipulated.
Some common finalizers you’ve likely encountered are:
- `kubernetes.io/pv-protection`
- `kubernetes.io/pvc-protection`
The finalizers above are used on volumes to prevent accidental deletion. Similarly, some finalizers can be used to prevent deletion of any resource but are not managed by any controller.
Below with a custom configmap, which has no properties but contains a finalizer:
```
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: mymap
finalizers:
- kubernetes
EOF
```
The configmap resource controller doesn't understand what to do with the `kubernetes` finalizer key. I term these “dead” finalizers for configmaps as it is normally used on namespaces. Here’s what happen upon attempting to delete the configmap:
```
kubectl delete configmap/mymap &
configmap "mymap" deleted
jobs
[1]+ Running kubectl delete configmap/mymap
```
Kubernetes will report back that the object has been deleted, however, it hasn’t been deleted in a traditional sense. Rather, it’s in the process of deletion. When we attempt to `get` that object again, we discover the object has been modified to include the deletion timestamp.
In short, what’s happened is that the object was updated, not deleted. That’s because Kubernetes saw that the object contained finalizers and put it into a read-only state. The deletion timestamp signals that the object can only be read, with the exception of removing the finalizer key updates. In other words, the deletion will not be complete until we edit the object and remove the finalizer.
Here's a demonstration of using the `patch` command to remove finalizers. If we want to delete an object, we can simply patch it on the command line to remove the finalizers. In this way, the deletion that was running in the background will complete and the object will be deleted. When we attempt to `get` that configmap, it will be gone.
Error from server (NotFound): configmaps "mymap" not found
```
Here's a state diagram for finalization:
{{<figure width="617" src="/images/blog/2021-05-14-using-finalizers-to-control-deletion/state-diagram-finalize.png" caption="State diagram for finalize">}}
So, if you attempt to delete an object that has a finalizer on it, it will remain in finalization until the controller has removed the finalizer keys or the finalizers are removed using Kubectl. Once that finalizer list is empty, the object can actually be reclaimed by Kubernetes and put into a queue to be deleted from the registry.
## Owner References
Owner references describe how groups of objects are related. They are properties on resources that specify the relationship to one another, so entire trees of resources can be deleted.
Finalizer rules are processed when there are owner references. An owner reference consists of a name and a UID. Owner references link resources within the same namespace, and it also needs a UID for that reference to work. Pods typically have owner references to the owning replica set. So, when deployments or stateful sets are deleted, then the child replica sets and pods are deleted in the process.
Here are some examples of owner references and how they work. In the first example, we create a parent object first, then the child. The result is a very simple configmap that contains an owner reference to its parent:
```
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: mymap-parent
EOF
CM_UID=$(kubectl get configmap mymap-parent -o jsonpath="{.metadata.uid}")
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: mymap-child
ownerReferences:
- apiVersion: v1
kind: ConfigMap
name: mymap-parent
uid: $CM_UID
EOF
```
Deleting the child object when an owner reference is involved does not delete the parent:
```
kubectl get configmap
NAME DATA AGE
mymap-child 0 12m4s
mymap-parent 0 12m4s
kubectl delete configmap/mymap-child
configmap "mymap-child" deleted
kubectl get configmap
NAME DATA AGE
mymap-parent 0 12m10s
```
In this example, we re-created the parent-child configmaps from above. Now, when deleting from the parent (instead of the child) with an owner reference from the child to the parent, when we `get` the configmaps, none are in the namespace:
```
kubectl get configmap
NAME DATA AGE
mymap-child 0 10m2s
mymap-parent 0 10m2s
kubectl delete configmap/mymap-parent
configmap "mymap-parent" deleted
kubectl get configmap
No resources found in default namespace.
```
To sum things up, when there's an override owner reference from a child to a parent, deleting the parent deletes the children automatically. This is called `cascade`. The default for cascade is `true`, however, you can use the --cascade=false option for `kubectl delete` to delete an object and orphan its children.
In the following example, there is a parent and a child. Notice the owner references are still included. If I delete the parent using --cascade=false, the parent is deleted but the child still exists:
The --cascade option links to the propagation policy in the API, which allows you to change the order in which objects are deleted within a tree. In the following example uses API access to craft a custom delete API call with the background propagation policy:
Note that the propagation policy cannot be specified on the command line using kubectl. You have to specify it using a custom API call. Simply create a proxy, so you have access to the API server from the client, and execute a `curl` command with just a URL to execute that `delete` command.
There are three different options for the propagation policy:
- `Foreground`: Children are deleted before the parent (post-order)
- `Background`: Parent is deleted before the children (pre-order)
- `Orphan`: Owner references are ignored
Keep in mind that when you delete an object and owner references have been specified, finalizers will be honored in the process. This can result in trees of objects persisting, and you end up with a partial deletion. At that point, you have to look at any existing owner references on your objects, as well as any finalizers, to understand what’s happening.
## Forcing a Deletion of a Namespace
There's one situation that may require forcing finalization for a namespace. If you've deleted a namespace and you've cleaned out all of the objects under it, but the namespace still exists, deletion can be forced by updating the namespace subresource, `finalize`. This informs the namespace controller that it needs to remove the finalizer from the namespace and perform any cleanup:
```
cat <<EOF | curl -X PUT \
localhost:8080/api/v1/namespaces/test/finalize \
-H "Content-Type: application/json" \
--data-binary @-
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "test"
},
"spec": {
"finalizers": null
}
}
EOF
```
This should be done with caution as it may delete the namespace only and leave orphan objects within the, now non-exiting, namespace - a confusing state for Kubernetes. If this happens, the namespace can be re-created manually and sometimes the orphaned objects will re-appear under the just-created namespace which will allow manual cleanup and recovery.
## Key Takeaways
As these examples demonstrate, finalizers can get in the way of deleting resources in Kubernetes, especially when there are parent-child relationships between objects. Often, there is a reason for adding a finalizer into the code, so you should always investigate before manually deleting it. Owner references allow you to specify and remove trees of resources, although finalizers will be honored in the process. Finally, the propagation policy can be used to specify the order of deletion via a custom API call, giving you control over how objects are deleted. Now that you know a little more about how deletions work in Kubernetes, we recommend you try it out on your own, using a test cluster.
{{< youtube class="youtube-quote-sm" id="F7-ZxWwf4sY" title="Clean Up Your Room! What Does It Mean to Delete Something in K8s">}}
has details on the schedule for the v1.22 release.
## API removals for Kubernetes v1.22 {#api-changes}
The **v1.22** release will stop serving the API versions we've listed immediately below.
These are all beta APIs that were previously deprecated in favor of newer and more stable
API versions.
<!-- sorted by API group -->
* Beta versions of the `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` API (the **admissionregistration.k8s.io/v1beta1** API versions)
* The beta `CustomResourceDefinition` API (**apiextensions.k8s.io/v1beta1**)
* The beta `APIService` API (**apiregistration.k8s.io/v1beta1**)
* The beta `TokenReview` API (**authentication.k8s.io/v1beta1**)
* Beta API versions of `SubjectAccessReview`, `LocalSubjectAccessReview`, `SelfSubjectAccessReview` (API versions from **authorization.k8s.io/v1beta1**)
* The beta `CertificateSigningRequest` API (**certificates.k8s.io/v1beta1**)
* The beta `Lease` API (**coordination.k8s.io/v1beta1**)
* All beta `Ingress` APIs (the **extensions/v1beta1** and **networking.k8s.io/v1beta1** API versions)
The Kubernetes documentation covers these
[API removals for v1.22](/docs/reference/using-api/deprecation-guide/#v1-22) and explains
how each of those APIs change between beta and stable.
## What to do
We're going to run through each of the resources that are affected by these removals
annotation, plan to switch to using the `.spec.ingressClassName` field instead.
On any cluster running Kubernetes v1.19 or later, you can use the v1 API to
retrieve or update existing Ingress objects, even if they were created using an
older API version.
When you convert an Ingress to the v1 API, you should review each rule in that Ingress.
Older Ingresses use the legacy `ImplementationSpecific` path type. Instead of `ImplementationSpecific`, switch [path matching](/docs/concepts/services-networking/ingress/#path-types) to either `Prefix` or `Exact`. One of the benefits of moving to these alternative path types is that it becomes easier to migrate between different Ingress classes.
**ⓘ** As well as upgrading _your_ own use of the Ingress API as a client, make sure that
every ingress controller that you use is compatible with the v1 Ingress API.
Are you interested in learning about what [SIG Usability](https://github.com/kubernetes/community/tree/master/sig-usability) does and how you can get involved? Well, you're at the right place. SIG Usability is all about making Kubernetes more accessible to new folks, and its main activity is conducting user research for the community. In this blog, we have summarized our conversation with [Gaby Moreno](https://twitter.com/morengab), who walks us through the various aspects of being a part of the SIG and shares some insights about how others can get involved.
Gaby is a co-lead for SIG Usability. She works as a Product Designer at IBM and enjoys working on the user experience of open, hybrid cloud technologies like Kubernetes, OpenShift, Terraform, and Cloud Foundry.
## A summary of our conversation
### Q. Could you tell us a little about what SIG Usability does?
A. SIG Usability at a high level started because there was no dedicated user experience team for Kubernetes. The extent of SIG Usability is focussed on the end-client ease of use of the Kubernetes project. The main activity is user research for the community, which includes speaking to Kubernetes users.
This covers points like user experience and accessibility. The objectives of the SIG are to guarantee that the Kubernetes project is maximally usable by people of a wide range of foundations and capacities, such as incorporating internationalization and ensuring the openness of documentation.
### Q. Why should new and existing contributors consider joining SIG Usability?
A. There are plenty of territories where new contributors can begin. For example:
- User research projects, where people can help understand the usability of the end-user experiences, including error messages, end-to-end tasks, etc.
- Accessibility guidelines for Kubernetes community artifacts, examples include: internationalization of documentation, color choices for people with color blindness, ensuring compatibility with screen reader technology, user interface design for core components with user interfaces, and more.
### Q. What do you do to help new contributors get started?
A. New contributors can get started by shadowing one of the user interviews, going through user interview transcripts, analyzing them, and designing surveys.
SIG Usability is also open to new project ideas. If you have an idea, we’ll do what we can to support it. There are regular SIG Meetings where people can ask their questions live. These meetings are also recorded for those who may not be able to attend. As always, you can reach out to us on Slack as well.
### Q. What does the survey include?
A. In simple terms, the survey gathers information about how people use Kubernetes, such as trends in learning to deploy a new system, error messages they receive, and workflows.
One of our goals is to standardize the responses accordingly. The ultimate goal is to analyze survey responses for important user stories whose needs aren't being met.
### Q. Are there any particular skills you’d like to recruit for? What skills are contributors to SIG Usability likely to learn?
A. Although contributing to SIG Usability does not have any pre-requisites as such, experience with user research, qualitative research, or prior experience with how to conduct an interview would be great plus points. Quantitative research, like survey design and screening, is also helpful and something that we expect contributors to learn.
### Q. What are you getting positive feedback on, and what’s coming up next for SIG Usability?
A. We have had new members joining and coming to monthly meetings regularly and showing interests in becoming a contributor and helping the community. We have also had a lot of people reach out to us via Slack showcasing their interest in the SIG.
Currently, we are focused on finishing the study mentioned in our [talk](https://www.youtube.com/watch?v=Byn0N_ZstE0), also our project for this year. We are always happy to have new contributors join us.
### Q: Any closing thoughts/resources you’d like to share?
A. We love meeting new contributors and assisting them in investigating different Kubernetes project spaces. We will work with and team up with other SIGs to facilitate engaging with end-users, running studies, and help them integrate accessible design practices into their development practices.
SIG Usability hosted a [KubeCon talk](https://www.youtube.com/watch?v=Byn0N_ZstE0) about studying Kubernetes users' experiences. The talk focuses on updates to the user study projects, understanding who is using Kubernetes, what they are trying to achieve, how the project is addressing their needs, and where we need to improve the project and the client experience. Join the SIG's update to find out about the most recent research results, what the plans are for the forthcoming year, and how to get involved in the upstream usability team as a contributor!
title: "Kubernetes Release Cadence Change: Here’s What You Need To Know"
date: 2021-07-20
slug: new-kubernetes-release-cadence
---
**Authors**: Celeste Horgan, Adolfo García Veytia, James Laverack, Jeremy Rickard
On April 23, 2021, the Release Team merged a Kubernetes Enhancement Proposal (KEP) changing the Kubernetes release cycle from four releases a year (once a quarter) to three releases a year.
This blog post provides a high level overview about what this means for the Kubernetes community's contributors and maintainers.
## What's changing and when
Starting with the [Kubernetes 1.22 release](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.22), a lightweight policy will drive the creation of each release schedule. This policy states:
* The first Kubernetes release of a calendar year should start at the second or third
week of January to provide people more time for contributors coming back from the
end of year holidays.
* The last Kubernetes release of a calendar year should be finished by the middle of
December.
* A Kubernetes release cycle has a length of approximately 15 weeks.
* The week of KubeCon + CloudNativeCon is not considered a 'working week' for SIG Release. The Release Team will not hold meetings or make decisions in this period.
* An explicit SIG Release break of at least two weeks between each cycle will
be enforced.
As a result, Kubernetes will follow a three releases per year cadence. Kubernetes 1.23 will be the final release of the 2021 calendar year. This new policy results in a very predictable release schedule, allowing us to forecast upcoming release dates:
*Proposed Kubernetes Release Schedule for the remainder of 2021*
| Week Number in Year | Release Number | Release Week | Note |
| Week Number in Year | Release Number | Release Week | Note |
| -------- | -------- | -------- | -------- |
| 1 | 1.24 | 1 (January 03) | |
| 15 | 1.24 | 15 (April 12) | |
| 17 | 1.25 | 1 (April 26) | KubeCon + CloudNativeCon EU likely to occur |
| 32 | 1.25 | 15 (August 09) | |
| 34 | 1.26 | 1 (August 22 | KubeCon + CloudNativeCon NA likely to occur |
| 49 | 1.26 | 14 (December 06) |
These proposed dates reflect only the start and end dates, and they are subject to change. The Release Team will select dates for enhancement freeze, code freeze, and other milestones at the start of each release. For more information on these milestones, please refer to the [release phases](https://www.k8s.dev/resources/release/#phases) documentation. Feedback from prior releases will feed into this process.
## What this means for end users
The major change end users will experience is a slower release cadence and a slower rate of enhancement graduation. Kubernetes release artifacts, release notes, and all other aspects of any given release will stay the same.
Prior to this change an enhancement could graduate from alpha to stable in 9 months. With the change in cadence, this will stretch to 12 months. Additionally, graduation of features over the last few releases has in some part been driven by release team activities.
With fewer releases, users can expect to see the rate of feature graduation slow. Users can also expect releases to contain a larger number of enhancements that they need to be aware of during upgrades. However, with fewer releases to consume per year, it's intended that end user organizations will spend less time on upgrades and gain more time on supporting their Kubernetes clusters. It also means that Kubernetes releases are in support for a slightly longer period of time, so bug fixes and security patches will be available for releases for a longer period of time.
## What this means for Kubernetes contributors
With a lower release cadence, contributors have more time for project enhancements, feature development, planning, and testing. A slower release cadence also provides more room for maintaining their mental health, preparing for events like KubeCon + CloudNativeCon or work on downstream integrations.
## Why we decided to change the release cadence
The Kubernetes 1.19 cycle was far longer than usual. SIG Release extended it to lessen the burden on both Kubernetes contributors and end users due the COVID-19 pandemic. Following this extended release, the Kubernetes 1.20 release became the third, and final, release for 2020.
As the Kubernetes project matures, the number of enhancements per cycle grows, along with the burden on contributors, the Release Engineering team. Downstream consumers and integrators also face increased challenges keeping up with [ever more feature-packed releases](https://kubernetes.io/blog/2021/04/08/kubernetes-1-21-release-announcement/). A wider project adoption means the complexity of supporting a rapidly evolving platform affects a bigger downstream chain of consumers.
Changing the release cadence from four to three releases per year balances a variety of factors for stakeholders: while it's not strictly an LTS policy, consumers and integrators will get longer support terms for each minor version as the extended release cycles lead to the [previous three releases being supported](https://kubernetes.io/blog/2020/08/31/kubernetes-1-19-feature-one-year-support/) for a longer period. Contributors get more time to [mature enhancements](https://www.cncf.io/blog/2021/04/12/enhancing-the-kubernetes-enhancements-process/) and [get them ready for production](https://github.com/kubernetes/community/blob/master/sig-architecture/production-readiness.md).
Finally, the management overhead for SIG Release and the Release Engineering team diminishes allowing the team to spend more time on improving the quality of the software releases and the tooling that drives them.
## How you can help
Join the [discussion](https://github.com/kubernetes/sig-release/discussions/1566) about communicating future release dates and be sure to be on the lookout for post release surveys.
## Where you can find out more
- Read the KEP [here](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/2572-release-cadence)
- Join the [kubernetes-dev](https://groups.google.com/g/kubernetes-dev) mailing list
- Join [Kubernetes Slack](https://slack.k8s.io) and follow the #announcements channel
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
PI-Victor
