Merge remote-tracking branch 'upstream/main' into dev-1.24
This commit is contained in:
Nate W
@@ -3,9 +3,11 @@ layout: blog
|
||||
title: "Dockershim Deprecation FAQ"
|
||||
date: 2020-12-02
|
||||
slug: dockershim-faq
|
||||
aliases: [ '/dockershim' ]
|
||||
---
|
||||
|
||||
|
||||
_**Update**: There is a [newer version](/blog/2022/02/17/dockershim-faq/) of this article available._
|
||||
|
||||
This document goes over some frequently asked questions regarding the Dockershim
|
||||
deprecation announced as a part of the Kubernetes v1.20 release. For more detail
|
||||
on the deprecation of Docker as a container runtime for Kubernetes kubelets, and
|
||||
|
||||
@@ -101,4 +101,4 @@ questions regardless of experience level or complexity! Our goal is to make sure
|
||||
everyone is educated as much as possible on the upcoming changes. We hope
|
||||
this has answered most of your questions and soothed some anxieties! ❤️
|
||||
|
||||
Looking for more answers? Check out our accompanying [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/).
|
||||
Looking for more answers? Check out our accompanying [Dockershim Removal FAQ](/blog/2022/02/17/dockershim-faq/) _(updated February 2022)_.
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
layout: blog
|
||||
title: 'SIG Node CI Subproject Celebrates Two Years of Test Improvements'
|
||||
date: 2022-02-16
|
||||
slug: sig-node-ci-subproject-celebrates
|
||||
canonicalUrl: https://www.kubernetes.dev/blog/2022/02/16/sig-node-ci-subproject-celebrates-two-years-of-test-improvements/
|
||||
---
|
||||
|
||||
**Authors:** Sergey Kanzhelev (Google), Elana Hashman (Red Hat)
|
||||
|
||||
Ensuring the reliability of SIG Node upstream code is a continuous effort
|
||||
that takes a lot of behind-the-scenes effort from many contributors.
|
||||
There are frequent releases of Kubernetes, base operating systems,
|
||||
container runtimes, and test infrastructure that result in a complex matrix that
|
||||
requires attention and steady investment to "keep the lights on."
|
||||
In May 2020, the Kubernetes node special interest group ("SIG Node") organized a new
|
||||
subproject for continuous integration (CI) for node-related code and tests. Since its
|
||||
inauguration, the SIG Node CI subproject has run a weekly meeting, and even the full hour
|
||||
is often not enough to complete triage of all bugs, test-related PRs and issues, and discuss all
|
||||
related ongoing work within the subgroup.
|
||||
|
||||
Over the past two years, we've fixed merge-blocking and release-blocking tests, reducing time to merge Kubernetes contributors' pull requests thanks to reduced test flakes. When we started, Node test jobs only passed 42% of the time, and through our efforts, we now ensure a consistent >90% job pass rate. We've closed 144 test failure issues and merged 176 pull requests just in kubernetes/kubernetes. And we've helped subproject participants ascend the Kubernetes contributor ladder, with 3 new org members, 6 new reviewers, and 2 new approvers.
|
||||
|
||||
The Node CI subproject is an approachable first stop to help new contributors
|
||||
get started with SIG Node. There is a low barrier to entry for new contributors
|
||||
to address high-impact bugs and test fixes, although there is a long
|
||||
road before contributors can climb the entire contributor ladder:
|
||||
it took over a year to establish two new approvers for the group.
|
||||
The complexity of all the different components that power Kubernetes nodes
|
||||
and its test infrastructure requires a sustained investment over a long period
|
||||
for developers to deeply understand the entire system,
|
||||
both at high and low levels of detail.
|
||||
|
||||
We have several regular contributors at our meetings, however; our reviewers
|
||||
and approvers pool is still small. It is our goal to continue to grow
|
||||
contributors to ensure a sustainable distribution of work
|
||||
that does not just fall to a few key approvers.
|
||||
|
||||
It's not always obvious how subprojects within SIGs are formed, operate,
|
||||
and work. Each is unique to its sponsoring SIG and tailored to the projects
|
||||
that the group is intended to support. As a group that has welcomed many
|
||||
first-time SIG Node contributors, we'd like to share some of the details and
|
||||
accomplishments over the past two years,
|
||||
helping to demystify our inner workings and celebrate the hard work
|
||||
of all our dedicated contributors!
|
||||
|
||||
## Timeline
|
||||
|
||||
***May 2020.*** SIG Node CI group was formed on May 11, 2020, with more than
|
||||
[30 volunteers](https://docs.google.com/document/d/1fb-ugvgdSVIkkuJ388_nhp2pBTy_4HEVg5848Xy7n5U/edit#bookmark=id.vsb8pqnf4gib)
|
||||
signed up, to improve SIG Node CI signal and overall observability.
|
||||
Victor Pickard focused on getting
|
||||
[testgrid jobs](https://testgrid.k8s.io/sig-node) passing
|
||||
when Ning Liao suggested forming a group around this effort and came up with
|
||||
the [original group charter document](https://docs.google.com/document/d/1yS-XoUl6GjZdjrwxInEZVHhxxLXlTIX2CeWOARmD8tY/edit#heading=h.te6sgum6s8uf).
|
||||
The SIG Node chairs sponsored group creation with Victor as a subproject lead.
|
||||
Sergey Kanzhelev joined Victor shortly after as a co-lead.
|
||||
|
||||
At the kick-off meeting, we discussed which tests to concentrate on fixing first
|
||||
and discussed merge-blocking and release-blocking tests, many of which were failing due
|
||||
to infrastructure issues or buggy test code.
|
||||
|
||||
The subproject launched weekly hour-long meetings to discuss ongoing work
|
||||
discussion and triage.
|
||||
|
||||
***June 2020.*** Morgan Bauer, Karan Goel, and Jorge Alarcon Ochoa were
|
||||
recognized as reviewers for the SIG Node CI group for their contributions,
|
||||
helping significantly with the early stages of the subproject.
|
||||
David Porter and Roy Yang also joined the SIG test failures GitHub team.
|
||||
|
||||
***August 2020.*** All merge-blocking and release-blocking tests were passing,
|
||||
with some flakes. However, only 42% of all SIG Node test jobs were green, as there
|
||||
were many flakes and failing tests.
|
||||
|
||||
***October 2020.*** Amim Knabben becomes a Kubernetes org member for his
|
||||
contributions to the subproject.
|
||||
|
||||
***January 2021.*** With healthy presubmit and critical periodic jobs passing,
|
||||
the subproject discussed its goal for cleaning up the rest of periodic tests
|
||||
and ensuring they passed without flakes.
|
||||
|
||||
Elana Hashman joined the subproject, stepping up to help lead it after
|
||||
Victor's departure.
|
||||
|
||||
***February 2021.*** Artyom Lukianov becomes a Kubernetes org member for his
|
||||
contributions to the subproject.
|
||||
|
||||
***August 2021.*** After SIG Node successfully ran a [bug scrub](https://groups.google.com/g/kubernetes-dev/c/w2ghO4ihje0/m/VeEql1LJBAAJ)
|
||||
to clean up its bug backlog, the scope of the meeting was extended to
|
||||
include bug triage to increase overall reliability, anticipating issues
|
||||
before they affect the CI signal.
|
||||
|
||||
Subproject leads Elana Hashman and Sergey Kanzhelev are both recognized as
|
||||
approvers on all node test code, supported by SIG Node and SIG Testing.
|
||||
|
||||
***September 2021.*** After significant deflaking progress with serial tests in
|
||||
the 1.22 release spearheaded by Francesco Romani, the subproject set a goal
|
||||
for getting the serial job fully passing by the 1.23 release date.
|
||||
|
||||
Mike Miranda becomes a Kubernetes org member for his contributions
|
||||
to the subproject.
|
||||
|
||||
***November 2021.*** Throughout 2021, SIG Node had no merge or
|
||||
release-blocking test failures. Many flaky tests from past releases are removed
|
||||
from release-blocking dashboards as they had been fully cleaned up.
|
||||
|
||||
Danielle Lancashire was recognized as a reviewer for SIG Node's subgroup, test code.
|
||||
|
||||
The final node serial tests were completely fixed. The serial tests consist of
|
||||
many disruptive and slow tests which tend to be flakey and are hard
|
||||
to troubleshoot. By the 1.23 release freeze, the last serial tests were
|
||||
fixed and the job was passing without flakes.
|
||||
|
||||
[](https://kubernetes.slack.com/archives/C0BP8PW9G/p1638211041322900)
|
||||
|
||||
The 1.23 release got a special shout out for the tests quality and CI signal.
|
||||
The SIG Node CI subproject was proud to have helped contribute to such
|
||||
a high-quality release, in part due to our efforts in identifying
|
||||
and fixing flakes in Node and beyond.
|
||||
|
||||
[](https://kubernetes.slack.com/archives/C92G08FGD/p1637175755023200)
|
||||
|
||||
***December 2021.*** An estimated 90% of test jobs were passing at the time of
|
||||
the 1.23 release (up from 42% in August 2020).
|
||||
|
||||
Dockershim code was removed from Kubernetes. This affected nearly half of SIG Node's
|
||||
test jobs and the SIG Node CI subproject reacted quickly and retargeted all the
|
||||
tests. SIG Node was the first SIG to complete test migrations off dockershim,
|
||||
providing examples for other affected SIGs. The vast majority of new jobs passed
|
||||
at the time of introduction without further fixes required. The [effort of
|
||||
removing dockershim](https://k8s.io/dockershim)) from Kubernetes is ongoing.
|
||||
There are still some wrinkles from the dockershim removal as we uncover more
|
||||
dependencies on dockershim, but we plan to stabilize all test jobs
|
||||
by the 1.24 release.
|
||||
|
||||
## Statistics
|
||||
|
||||
Our regular meeting attendees and subproject participants for the past few months:
|
||||
|
||||
- Aditi Sharma
|
||||
- Artyom Lukianov
|
||||
- Arnaud Meukam
|
||||
- Danielle Lancashire
|
||||
- David Porter
|
||||
- Davanum Srinivas
|
||||
- Elana Hashman
|
||||
- Francesco Romani
|
||||
- Matthias Bertschy
|
||||
- Mike Miranda
|
||||
- Paco Xu
|
||||
- Peter Hunt
|
||||
- Ruiwen Zhao
|
||||
- Ryan Phillips
|
||||
- Sergey Kanzhelev
|
||||
- Skyler Clark
|
||||
- Swati Sehgal
|
||||
- Wenjun Wu
|
||||
|
||||
The [kubernetes/test-infra](https://github.com/kubernetes/test-infra/) source code repository contains test definitions. The number of
|
||||
Node PRs just in that repository:
|
||||
- 2020 PRs (since May): [183](https://github.com/kubernetes/test-infra/pulls?q=is%3Apr+is%3Aclosed+label%3Asig%2Fnode+created%3A2020-05-01..2020-12-31+-author%3Ak8s-infra-ci-robot+)
|
||||
- 2021 PRs: [264](https://github.com/kubernetes/test-infra/pulls?q=is%3Apr+is%3Aclosed+label%3Asig%2Fnode+created%3A2021-01-01..2021-12-31+-author%3Ak8s-infra-ci-robot+)
|
||||
|
||||
Triaged issues and PRs on CI board (including triaging away from the subgroup scope):
|
||||
|
||||
- 2020 (since May): [132](https://github.com/issues?q=project%3Akubernetes%2F43+created%3A2020-05-01..2020-12-31)
|
||||
- 2021: [532](https://github.com/issues?q=project%3Akubernetes%2F43+created%3A2021-01-01..2021-12-31+)
|
||||
|
||||
## Future
|
||||
|
||||
Just "keeping the lights on" is a bold task and we are committed to improving this experience.
|
||||
We are working to simplify the triage and review processes for SIG Node.
|
||||
|
||||
Specifically, we are working on better test organization, naming,
|
||||
and tracking:
|
||||
|
||||
- https://github.com/kubernetes/enhancements/pull/3042
|
||||
- https://github.com/kubernetes/test-infra/issues/24641
|
||||
- [Kubernetes SIG-Node CI Testgrid Tracker](https://docs.google.com/spreadsheets/d/1IwONkeXSc2SG_EQMYGRSkfiSWNk8yWLpVhPm-LOTbGM/edit#gid=0)
|
||||
|
||||
We are also constantly making progress on improved tests debuggability and de-flaking.
|
||||
|
||||
If any of this interests you, we'd love for you to join us!
|
||||
There's plenty to learn in debugging test failures, and it will help you gain
|
||||
familiarity with the code that SIG Node maintains.
|
||||
|
||||
You can always find information about the group on the
|
||||
[SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) page.
|
||||
We give group updates at our maintainer track sessions, such as
|
||||
[KubeCon + CloudNativeCon Europe 2021](https://kccnceu2021.sched.com/event/iE8E/kubernetes-sig-node-intro-and-deep-dive-elana-hashman-red-hat-sergey-kanzhelev-google) and
|
||||
[KubeCon + CloudNative North America 2021](https://kccncna2021.sched.com/event/lV9D/kubenetes-sig-node-intro-and-deep-dive-elana-hashman-derek-carr-red-hat-sergey-kanzhelev-dawn-chen-google?iframe=no&w=100%&sidebar=yes&bg=no).
|
||||
Join us in our mission to keep the kubelet and other SIG Node components reliable and ensure smooth and uneventful releases!
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
@@ -0,0 +1,205 @@
|
||||
---
|
||||
layout: blog
|
||||
title: "Updated: Dockershim Removal FAQ"
|
||||
date: 2022-02-17
|
||||
slug: dockershim-faq
|
||||
aliases: [ '/dockershim' ]
|
||||
---
|
||||
|
||||
**This is an update to the original [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/) article,
|
||||
published in late 2020.**
|
||||
|
||||
This document goes over some frequently asked questions regarding the
|
||||
deprecation and removal of _dockershim_, that was
|
||||
[announced](/blog/2020/12/08/kubernetes-1-20-release-announcement/)
|
||||
as a part of the Kubernetes v1.20 release. For more detail
|
||||
on what that means, check out the blog post
|
||||
[Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/).
|
||||
|
||||
Also, you can read [check whether dockershim removal affects you](/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/)
|
||||
to determine how much impact the removal of dockershim would have for you
|
||||
or for your organization.
|
||||
|
||||
As the Kubernetes 1.24 release has become imminent, we've been working hard to try to make this a smooth transition.
|
||||
|
||||
- We've written a blog post detailing our [commitment and next steps](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/).
|
||||
- We believe there are no major blockers to migration to [other container runtimes](/docs/setup/production-environment/container-runtimes/#container-runtimes).
|
||||
- There is also a [Migrating from dockershim](/docs/tasks/administer-cluster/migrating-from-dockershim/) guide available.
|
||||
- We've also created a page to list
|
||||
[articles on dockershim removal and on using CRI-compatible runtimes](/docs/reference/node/topics-on-dockershim-and-cri-compatible-runtimes/).
|
||||
That list includes some of the already mentioned docs, and also covers selected external sources
|
||||
(including vendor guides).
|
||||
|
||||
### Why is the dockershim being removed from Kubernetes?
|
||||
|
||||
Early versions of Kubernetes only worked with a specific container runtime:
|
||||
Docker Engine. Later, Kubernetes added support for working with other container runtimes.
|
||||
The CRI standard was [created](/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) to
|
||||
enable interoperability between orchestrators (like Kubernetes) and many different container
|
||||
runtimes.
|
||||
Docker Engine doesn't implement that interface (CRI), so the Kubernetes project created
|
||||
special code to help with the transition, and made that _dockershim_ code part of Kubernetes
|
||||
itself.
|
||||
|
||||
The dockershim code was always intended to be a temporary solution (hence the name: shim).
|
||||
You can read more about the community discussion and planning in the
|
||||
[Dockershim Removal Kubernetes Enhancement Proposal][drkep].
|
||||
In fact, maintaining dockershim had become a heavy burden on the Kubernetes maintainers.
|
||||
|
||||
Additionally, features that were largely incompatible with the dockershim, such
|
||||
as cgroups v2 and user namespaces are being implemented in these newer CRI
|
||||
runtimes. Removing support for the dockershim will allow further development in
|
||||
those areas.
|
||||
|
||||
[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2221-remove-dockershim
|
||||
|
||||
### Can I still use Docker Engine in Kubernetes 1.23?
|
||||
|
||||
Yes, the only thing changed in 1.20 is a single warning log printed at [kubelet]
|
||||
startup if using Docker Engine as the runtime. You'll see this warning in all versions up to 1.23. The dockershim removal occurs in Kubernetes 1.24.
|
||||
|
||||
[kubelet]: /docs/reference/command-line-tools-reference/kubelet/
|
||||
|
||||
### When will dockershim be removed?
|
||||
|
||||
Given the impact of this change, we are using an extended deprecation timeline.
|
||||
Removal of dockershim is scheduled for Kubernetes v1.24, see [Dockershim Removal Kubernetes Enhancement Proposal][drkep].
|
||||
The Kubernetes project will be working closely with vendors and other ecosystem groups to ensure
|
||||
a smooth transition and will evaluate things as the situation evolves.
|
||||
|
||||
### Can I still use Docker Engine as my container runtime?
|
||||
|
||||
First off, if you use Docker on your own PC to develop or test containers: nothing changes.
|
||||
You can still use Docker locally no matter what container runtime(s) you use for your
|
||||
Kubernetes clusters. Containers make this kind of interoperability possible.
|
||||
|
||||
Mirantis and Docker have [committed][mirantis] to maintaining a replacement adapter for
|
||||
Docker Engine, and to maintain that adapter even after the in-tree dockershim is removed
|
||||
from Kubernetes. The replacement adapter is named [`cri-dockerd`](https://github.com/Mirantis/cri-dockerd).
|
||||
|
||||
[mirantis]: https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/
|
||||
|
||||
### Will my existing container images still work?
|
||||
|
||||
Yes, the images produced from `docker build` will work with all CRI implementations.
|
||||
All your existing images will still work exactly the same.
|
||||
|
||||
#### What about private images?
|
||||
|
||||
Yes. All CRI runtimes support the same pull secrets configuration used in
|
||||
Kubernetes, either via the PodSpec or ServiceAccount.
|
||||
|
||||
### Are Docker and containers the same thing?
|
||||
|
||||
Docker popularized the Linux containers pattern and has been instrumental in
|
||||
developing the underlying technology, however containers in Linux have existed
|
||||
for a long time. The container ecosystem has grown to be much broader than just
|
||||
Docker. Standards like OCI and CRI have helped many tools grow and thrive in our
|
||||
ecosystem, some replacing aspects of Docker while others enhance existing
|
||||
functionality.
|
||||
|
||||
### Are there examples of folks using other runtimes in production today?
|
||||
|
||||
All Kubernetes project produced artifacts (Kubernetes binaries) are validated
|
||||
with each release.
|
||||
|
||||
Additionally, the [kind] project has been using containerd for some time and has
|
||||
seen an improvement in stability for its use case. Kind and containerd are leveraged
|
||||
multiple times every day to validate any changes to the Kubernetes codebase. Other
|
||||
related projects follow a similar pattern as well, demonstrating the stability and
|
||||
usability of other container runtimes. As an example, OpenShift 4.x has been
|
||||
using the [CRI-O] runtime in production since June 2019.
|
||||
|
||||
For other examples and references you can look at the adopters of containerd and
|
||||
CRI-O, two container runtimes under the Cloud Native Computing Foundation ([CNCF]).
|
||||
|
||||
- [containerd](https://github.com/containerd/containerd/blob/master/ADOPTERS.md)
|
||||
- [CRI-O](https://github.com/cri-o/cri-o/blob/master/ADOPTERS.md)
|
||||
|
||||
[CRI-O]: https://cri-o.io/
|
||||
[kind]: https://kind.sigs.k8s.io/
|
||||
[CNCF]: https://cncf.io
|
||||
|
||||
### People keep referencing OCI, what is that?
|
||||
|
||||
OCI stands for the [Open Container Initiative], which standardized many of the
|
||||
interfaces between container tools and technologies. They maintain a standard
|
||||
specification for packaging container images (OCI image-spec) and running containers
|
||||
(OCI runtime-spec). They also maintain an actual implementation of the runtime-spec
|
||||
in the form of [runc], which is the underlying default runtime for both
|
||||
[containerd] and [CRI-O]. The CRI builds on these low-level specifications to
|
||||
provide an end-to-end standard for managing containers.
|
||||
|
||||
[Open Container Initiative]: https://opencontainers.org/about/overview/
|
||||
[runc]: https://github.com/opencontainers/runc
|
||||
[containerd]: https://containerd.io/
|
||||
|
||||
### Which CRI implementation should I use?
|
||||
|
||||
That’s a complex question and it depends on a lot of factors. If Docker is
|
||||
working for you, moving to containerd should be a relatively easy swap and
|
||||
will have strictly better performance and less overhead. However, we encourage you
|
||||
to explore all the options from the [CNCF landscape] in case another would be an
|
||||
even better fit for your environment.
|
||||
|
||||
[CNCF landscape]: https://landscape.cncf.io/card-mode?category=container-runtime&grouping=category
|
||||
|
||||
### What should I look out for when changing CRI implementations?
|
||||
|
||||
While the underlying containerization code is the same between Docker and most
|
||||
CRIs (including containerd), there are a few differences around the edges. Some
|
||||
common things to consider when migrating are:
|
||||
|
||||
- Logging configuration
|
||||
- Runtime resource limitations
|
||||
- Node provisioning scripts that call docker or use docker via it's control socket
|
||||
- Kubectl plugins that require docker CLI or the control socket
|
||||
- Tools from the Kubernetes project that require direct access to Docker Engine
|
||||
(for example: the deprecated `kube-imagepuller` tool)
|
||||
- Configuration of functionality like `registry-mirrors` and insecure registries
|
||||
- Other support scripts or daemons that expect Docker Engine to be available and are run
|
||||
outside of Kubernetes (for example, monitoring or security agents)
|
||||
- GPUs or special hardware and how they integrate with your runtime and Kubernetes
|
||||
|
||||
If you use Kubernetes resource requests/limits or file-based log collection
|
||||
DaemonSets then they will continue to work the same, but if you’ve customized
|
||||
your `dockerd` configuration, you’ll need to adapt that for your new container
|
||||
runtime where possible.
|
||||
|
||||
Another thing to look out for is anything expecting to run for system maintenance
|
||||
or nested inside a container when building images will no longer work. For the
|
||||
former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the
|
||||
latter you can use newer container build options like [img], [buildah],
|
||||
[kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker.
|
||||
|
||||
[cr]: https://github.com/kubernetes-sigs/cri-tools
|
||||
[img]: https://github.com/genuinetools/img
|
||||
[buildah]: https://github.com/containers/buildah
|
||||
[kaniko]: https://github.com/GoogleContainerTools/kaniko
|
||||
[buildkit-cli-for-kubectl]: https://github.com/vmware-tanzu/buildkit-cli-for-kubectl
|
||||
|
||||
For containerd, you can start with their [documentation] to see what configuration
|
||||
options are available as you migrate things over.
|
||||
|
||||
[documentation]: https://github.com/containerd/cri/blob/master/docs/registry.md
|
||||
|
||||
For instructions on how to use containerd and CRI-O with Kubernetes, see the
|
||||
Kubernetes documentation on [Container Runtimes]
|
||||
|
||||
[Container Runtimes]: /docs/setup/production-environment/container-runtimes/
|
||||
|
||||
### What if I have more questions?
|
||||
|
||||
If you use a vendor-supported Kubernetes distribution, you can ask them about
|
||||
upgrade plans for their products. For end-user questions, please post them
|
||||
to our end user community forum: https://discuss.kubernetes.io/.
|
||||
|
||||
You can also check out the excellent blog post
|
||||
[Wait, Docker is deprecated in Kubernetes now?][dep] a more in-depth technical
|
||||
discussion of the changes.
|
||||
|
||||
[dep]: https://dev.to/inductor/wait-docker-is-deprecated-in-kubernetes-now-what-do-i-do-e4m
|
||||
|
||||
### Can I have a hug?
|
||||
|
||||
Yes, we're still giving hugs as requested. 🤗🤗🤗
|
||||
@@ -21,7 +21,6 @@ This page lists some of the available add-ons and links to their respective inst
|
||||
* [Canal](https://github.com/tigera/canal/tree/master/k8s-install) unites Flannel and Calico, providing networking and network policy.
|
||||
* [Cilium](https://github.com/cilium/cilium) is a L3 network and network policy plugin that can enforce HTTP/API/L7 policies transparently. Both routing and overlay/encapsulation mode are supported, and it can work on top of other CNI plugins.
|
||||
* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, Romana, or Weave.
|
||||
* [Contiv](https://contiv.github.io) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](https://github.com/contiv). The [installer](https://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options.
|
||||
* [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is an open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads.
|
||||
* [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) is an overlay network provider that can be used with Kubernetes.
|
||||
* [Knitter](https://github.com/ZTE/Knitter/) is a plugin to support multiple network interfaces in a Kubernetes pod.
|
||||
|
||||
@@ -66,7 +66,7 @@ with `--tracing-config-file=<path-to-config>`. This is an example config that re
|
||||
spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:
|
||||
|
||||
```yaml
|
||||
apiVersion: apiserver.config.k8s.io/v1beta1
|
||||
apiVersion: apiserver.config.k8s.io/v1alpha1
|
||||
kind: TracingConfiguration
|
||||
# default value
|
||||
#endpoint: localhost:4317
|
||||
@@ -74,7 +74,7 @@ samplingRatePerMillion: 100
|
||||
```
|
||||
|
||||
For more information about the `TracingConfiguration` struct, see
|
||||
[API server config API (v1beta1)](/docs/reference/config-api/apiserver-config.v1beta1/#apiserver-k8s-io-v1beta1-TracingConfiguration).
|
||||
[API server config API (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/#apiserver-k8s-io-v1alpha1-TracingConfiguration).
|
||||
|
||||
## Stability
|
||||
|
||||
|
||||
@@ -231,7 +231,7 @@ The kubelet reports the resource usage of a Pod as part of the Pod
|
||||
|
||||
If optional [tools for monitoring](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)
|
||||
are available in your cluster, then Pod resource usage can be retrieved either
|
||||
from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#the-metrics-api)
|
||||
from the [Metrics API](/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-api)
|
||||
directly or from your monitoring tools.
|
||||
|
||||
## Local ephemeral storage
|
||||
|
||||
@@ -111,6 +111,7 @@ Operator.
|
||||
{{% thirdparty-content %}}
|
||||
|
||||
* [Charmed Operator Framework](https://juju.is/)
|
||||
* [Kopf](https://github.com/nolar/kopf) (Kubernetes Operator Pythonic Framework)
|
||||
* [kubebuilder](https://book.kubebuilder.io/)
|
||||
* [KubeOps](https://buehler.github.io/dotnet-operator-sdk/) (.NET operator SDK)
|
||||
* [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator)
|
||||
|
||||
@@ -63,7 +63,7 @@ One way to create a Deployment using a `.yaml` file like the one above is to use
|
||||
in the `kubectl` command-line interface, passing the `.yaml` file as an argument. Here's an example:
|
||||
|
||||
```shell
|
||||
kubectl apply -f https://k8s.io/examples/application/deployment.yaml --record
|
||||
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
|
||||
```
|
||||
|
||||
The output is similar to this:
|
||||
|
||||
@@ -339,7 +339,7 @@ If that has happened, or you suspect that it might have, you can retry expansion
|
||||
size that is within the capacity limits of underlying storage provider. You can monitor status of resize operation by watching `.status.resizeStatus` and events on the PVC.
|
||||
|
||||
Note that,
|
||||
although you can a specify a lower amount of storage than what was requested previously,
|
||||
although you can specify a lower amount of storage than what was requested previously,
|
||||
the new value must still be higher than `.status.capacity`.
|
||||
Kubernetes does not support shrinking a PVC to less than its current size.
|
||||
{{% /tab %}}
|
||||
|
||||
@@ -49,7 +49,7 @@ metadata:
|
||||
name: standard
|
||||
provisioner: kubernetes.io/aws-ebs
|
||||
parameters:
|
||||
type: gp2
|
||||
type: gp3
|
||||
reclaimPolicy: Retain
|
||||
allowVolumeExpansion: true
|
||||
mountOptions:
|
||||
@@ -271,9 +271,9 @@ parameters:
|
||||
fsType: ext4
|
||||
```
|
||||
|
||||
* `type`: `io1`, `gp2`, `sc1`, `st1`. See
|
||||
* `type`: `io1`, `gp2`, `gp3`, `sc1`, `st1`. See
|
||||
[AWS docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)
|
||||
for details. Default: `gp2`.
|
||||
for details. Default: `gp3`.
|
||||
* `zone` (Deprecated): AWS zone. If neither `zone` nor `zones` is specified, volumes are
|
||||
generally round-robin-ed across all active zones where Kubernetes cluster
|
||||
has a node. `zone` and `zones` parameters must not be used at the same time.
|
||||
|
||||
@@ -86,6 +86,7 @@ Responsibilities for New Contributor Ambassadors include:
|
||||
- Mentoring new contributors through their first few PRs to the docs repo.
|
||||
- Helping new contributors create the more complex PRs they need to become Kubernetes members.
|
||||
- [Sponsoring contributors](/docs/contribute/advanced/#sponsor-a-new-contributor) on their path to becoming Kubernetes members.
|
||||
- Hosting a monthly meeting to help and mentor new contributors.
|
||||
|
||||
Current New Contributor Ambassadors are announced at each SIG-Docs meeting and in the [Kubernetes #sig-docs channel](https://kubernetes.slack.com).
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ cards:
|
||||
title: K8s Release Notes
|
||||
description: If you are installing Kubernetes or upgrading to the newest version, refer to the current release notes.
|
||||
button: "Download Kubernetes"
|
||||
button_path: "/docs/setup/release/notes"
|
||||
button_path: "/releases/download"
|
||||
- name: about
|
||||
title: About the documentation
|
||||
description: This website contains documentation for the current and previous 4 versions of Kubernetes.
|
||||
|
||||
@@ -74,6 +74,7 @@ by the API server in a RESTful way though they are essential for a user or an
|
||||
operator to use or manage a cluster.
|
||||
|
||||
|
||||
* [kube-apiserver configuration (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/)
|
||||
* [kube-apiserver configuration (v1)](/docs/reference/config-api/apiserver-config.v1/)
|
||||
* [kube-apiserver encryption (v1)](/docs/reference/config-api/apiserver-encryption.v1/)
|
||||
* [kubelet configuration (v1alpha1)](/docs/reference/config-api/kubelet-config.v1alpha1/) and
|
||||
|
||||
@@ -401,7 +401,7 @@ different Kubernetes components.
|
||||
| `PodShareProcessNamespace` | `true` | Beta | 1.12 | 1.16 |
|
||||
| `PodShareProcessNamespace` | `true` | GA | 1.17 | - |
|
||||
| `RequestManagement` | `false` | Alpha | 1.15 | 1.16 |
|
||||
| `RequestManagement` | - | Derecated | 1.17 | - |
|
||||
| `RequestManagement` | - | Deprecated | 1.17 | - |
|
||||
| `ResourceLimitsPriorityFunction` | `false` | Alpha | 1.9 | 1.18 |
|
||||
| `ResourceLimitsPriorityFunction` | - | Deprecated | 1.19 | - |
|
||||
| `ResourceQuotaScopeSelectors` | `false` | Alpha | 1.11 | 1.11 |
|
||||
@@ -816,7 +816,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
|
||||
and gracefully terminate pods running on the node. See
|
||||
[Graceful Node Shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown)
|
||||
for more details.
|
||||
= `GracefulNodeShutdownBasedOnPodPriority`: Enables the kubelet to check Pod priorities
|
||||
- `GracefulNodeShutdownBasedOnPodPriority`: Enables the kubelet to check Pod priorities
|
||||
when shutting down a node gracefully.
|
||||
- `GRPCContainerProbe`: Enables the gRPC probe method for {Liveness,Readiness,Startup}Probe. See [Configure Liveness, Readiness and Startup Probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe).
|
||||
- `HonorPVReclaimPolicy`: Honor persistent volume reclaim policy when it is `Delete` irrespective of PV-PVC deletion ordering.
|
||||
|
||||
@@ -946,7 +946,7 @@ kube-apiserver [flags]
|
||||
<td colspan="2">--service-account-key-file strings</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key is provided</p></td>
|
||||
<td></td><td style="line-height: 130%; word-wrap: break-word;"><p>File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key-file is provided</p></td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
|
||||
@@ -0,0 +1,375 @@
|
||||
---
|
||||
title: kube-apiserver Configuration (v1alpha1)
|
||||
content_type: tool-reference
|
||||
package: apiserver.k8s.io/v1alpha1
|
||||
auto_generated: true
|
||||
---
|
||||
<p>Package v1alpha1 is the v1alpha1 version of the API.</p>
|
||||
|
||||
|
||||
## Resource Types
|
||||
|
||||
|
||||
- [AdmissionConfiguration](#apiserver-k8s-io-v1alpha1-AdmissionConfiguration)
|
||||
- [EgressSelectorConfiguration](#apiserver-k8s-io-v1alpha1-EgressSelectorConfiguration)
|
||||
- [TracingConfiguration](#apiserver-k8s-io-v1alpha1-TracingConfiguration)
|
||||
|
||||
|
||||
|
||||
## `AdmissionConfiguration` {#apiserver-k8s-io-v1alpha1-AdmissionConfiguration}
|
||||
|
||||
|
||||
|
||||
<p>AdmissionConfiguration provides versioned configuration for admission controllers.</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
<tr><td><code>apiVersion</code><br/>string</td><td><code>apiserver.k8s.io/v1alpha1</code></td></tr>
|
||||
<tr><td><code>kind</code><br/>string</td><td><code>AdmissionConfiguration</code></td></tr>
|
||||
|
||||
|
||||
<tr><td><code>plugins</code><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-AdmissionPluginConfiguration"><code>[]AdmissionPluginConfiguration</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>Plugins allows specifying a configuration per admission control plugin.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `EgressSelectorConfiguration` {#apiserver-k8s-io-v1alpha1-EgressSelectorConfiguration}
|
||||
|
||||
|
||||
|
||||
<p>EgressSelectorConfiguration provides versioned configuration for egress selector clients.</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
<tr><td><code>apiVersion</code><br/>string</td><td><code>apiserver.k8s.io/v1alpha1</code></td></tr>
|
||||
<tr><td><code>kind</code><br/>string</td><td><code>EgressSelectorConfiguration</code></td></tr>
|
||||
|
||||
|
||||
<tr><td><code>egressSelections</code> <B>[Required]</B><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-EgressSelection"><code>[]EgressSelection</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>connectionServices contains a list of egress selection client configurations</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `TracingConfiguration` {#apiserver-k8s-io-v1alpha1-TracingConfiguration}
|
||||
|
||||
|
||||
|
||||
<p>TracingConfiguration provides versioned configuration for tracing clients.</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
<tr><td><code>apiVersion</code><br/>string</td><td><code>apiserver.k8s.io/v1alpha1</code></td></tr>
|
||||
<tr><td><code>kind</code><br/>string</td><td><code>TracingConfiguration</code></td></tr>
|
||||
|
||||
|
||||
<tr><td><code>endpoint</code><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>Endpoint of the collector that's running on the control-plane node.
|
||||
The APIServer uses the egressType ControlPlane when sending data to the collector.
|
||||
The syntax is defined in https://github.com/grpc/grpc/blob/master/doc/naming.md.
|
||||
Defaults to the otlpgrpc default, localhost:4317
|
||||
The connection is insecure, and does not support TLS.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>samplingRatePerMillion</code><br/>
|
||||
<code>int32</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>SamplingRatePerMillion is the number of samples to collect per million spans.
|
||||
Defaults to 0.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `AdmissionPluginConfiguration` {#apiserver-k8s-io-v1alpha1-AdmissionPluginConfiguration}
|
||||
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [AdmissionConfiguration](#apiserver-k8s-io-v1alpha1-AdmissionConfiguration)
|
||||
|
||||
|
||||
<p>AdmissionPluginConfiguration provides the configuration for a single plug-in.</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
|
||||
<tr><td><code>name</code> <B>[Required]</B><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>Name is the name of the admission controller.
|
||||
It must match the registered admission plugin name.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>path</code><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>Path is the path to a configuration file that contains the plugin's
|
||||
configuration</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>configuration</code><br/>
|
||||
<a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime#Unknown"><code>k8s.io/apimachinery/pkg/runtime.Unknown</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>Configuration is an embedded configuration object to be used as the plugin's
|
||||
configuration. If present, it will be used instead of the path to the configuration file.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `Connection` {#apiserver-k8s-io-v1alpha1-Connection}
|
||||
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [EgressSelection](#apiserver-k8s-io-v1alpha1-EgressSelection)
|
||||
|
||||
|
||||
<p>Connection provides the configuration for a single egress selection client.</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
|
||||
<tr><td><code>proxyProtocol</code> <B>[Required]</B><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-ProtocolType"><code>ProtocolType</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>Protocol is the protocol used to connect from client to the konnectivity server.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>transport</code><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-Transport"><code>Transport</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>Transport defines the transport configurations we use to dial to the konnectivity server.
|
||||
This is required if ProxyProtocol is HTTPConnect or GRPC.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `EgressSelection` {#apiserver-k8s-io-v1alpha1-EgressSelection}
|
||||
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [EgressSelectorConfiguration](#apiserver-k8s-io-v1alpha1-EgressSelectorConfiguration)
|
||||
|
||||
|
||||
<p>EgressSelection provides the configuration for a single egress selection client.</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
|
||||
<tr><td><code>name</code> <B>[Required]</B><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>name is the name of the egress selection.
|
||||
Currently supported values are "controlplane", "master", "etcd" and "cluster"
|
||||
The "master" egress selector is deprecated in favor of "controlplane"</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>connection</code> <B>[Required]</B><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-Connection"><code>Connection</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>connection is the exact information used to configure the egress selection</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `ProtocolType` {#apiserver-k8s-io-v1alpha1-ProtocolType}
|
||||
|
||||
(Alias of `string`)
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [Connection](#apiserver-k8s-io-v1alpha1-Connection)
|
||||
|
||||
|
||||
<p>ProtocolType is a set of valid values for Connection.ProtocolType</p>
|
||||
|
||||
|
||||
|
||||
|
||||
## `TCPTransport` {#apiserver-k8s-io-v1alpha1-TCPTransport}
|
||||
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [Transport](#apiserver-k8s-io-v1alpha1-Transport)
|
||||
|
||||
|
||||
<p>TCPTransport provides the information to connect to konnectivity server via TCP</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
|
||||
<tr><td><code>url</code> <B>[Required]</B><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>URL is the location of the konnectivity server to connect to.
|
||||
As an example it might be "https://127.0.0.1:8131"</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>tlsConfig</code><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-TLSConfig"><code>TLSConfig</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>TLSConfig is the config needed to use TLS when connecting to konnectivity server</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `TLSConfig` {#apiserver-k8s-io-v1alpha1-TLSConfig}
|
||||
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [TCPTransport](#apiserver-k8s-io-v1alpha1-TCPTransport)
|
||||
|
||||
|
||||
<p>TLSConfig provides the authentication information to connect to konnectivity server
|
||||
Only used with TCPTransport</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
|
||||
<tr><td><code>caBundle</code><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>caBundle is the file location of the CA to be used to determine trust with the konnectivity server.
|
||||
Must be absent/empty if TCPTransport.URL is prefixed with http://
|
||||
If absent while TCPTransport.URL is prefixed with https://, default to system trust roots.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>clientKey</code><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>clientKey is the file location of the client key to be used in mtls handshakes with the konnectivity server.
|
||||
Must be absent/empty if TCPTransport.URL is prefixed with http://
|
||||
Must be configured if TCPTransport.URL is prefixed with https://</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>clientCert</code><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>clientCert is the file location of the client certificate to be used in mtls handshakes with the konnectivity server.
|
||||
Must be absent/empty if TCPTransport.URL is prefixed with http://
|
||||
Must be configured if TCPTransport.URL is prefixed with https://</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `Transport` {#apiserver-k8s-io-v1alpha1-Transport}
|
||||
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [Connection](#apiserver-k8s-io-v1alpha1-Connection)
|
||||
|
||||
|
||||
<p>Transport defines the transport configurations we use to dial to the konnectivity server</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
|
||||
<tr><td><code>tcp</code><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-TCPTransport"><code>TCPTransport</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>TCP is the TCP configuration for communicating with the konnectivity server via TCP
|
||||
ProxyProtocol of GRPC is not supported with TCP transport at the moment
|
||||
Requires at least one of TCP or UDS to be set</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr><td><code>uds</code><br/>
|
||||
<a href="#apiserver-k8s-io-v1alpha1-UDSTransport"><code>UDSTransport</code></a>
|
||||
</td>
|
||||
<td>
|
||||
<p>UDS is the UDS configuration for communicating with the konnectivity server via UDS
|
||||
Requires at least one of TCP or UDS to be set</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## `UDSTransport` {#apiserver-k8s-io-v1alpha1-UDSTransport}
|
||||
|
||||
|
||||
**Appears in:**
|
||||
|
||||
- [Transport](#apiserver-k8s-io-v1alpha1-Transport)
|
||||
|
||||
|
||||
<p>UDSTransport provides the information to connect to konnectivity server via UDS</p>
|
||||
|
||||
|
||||
<table class="table">
|
||||
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
|
||||
<tbody>
|
||||
|
||||
|
||||
<tr><td><code>udsName</code> <B>[Required]</B><br/>
|
||||
<code>string</code>
|
||||
</td>
|
||||
<td>
|
||||
<p>UDSName is the name of the unix domain socket to connect to konnectivity server
|
||||
This does not use a unix:// prefix. (Eg: /etc/srv/kubernetes/konnectivity-server/konnectivity-server.socket)</p>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -72,6 +72,14 @@ This annotation is a best guess at why something was changed.
|
||||
|
||||
It is populated when adding `--record` to a `kubectl` command that may change an object.
|
||||
|
||||
## kubernetes.io/description {#description}
|
||||
|
||||
Example: `kubernetes.io/description: "Description of K8s object."`
|
||||
|
||||
Used on: All Objects
|
||||
|
||||
This annotation is used for describing specific behaviour of given object.
|
||||
|
||||
## controller.kubernetes.io/pod-deletion-cost {#pod-deletion-cost}
|
||||
|
||||
Example: `controller.kubernetes.io/pod-deletion-cost=10`
|
||||
@@ -464,4 +472,4 @@ This annotation has been deprecated since Kubernetes v1.19 and will become non-f
|
||||
The tutorial [Restrict a Container's Syscalls with seccomp](/docs/tutorials/clusters/seccomp/) takes
|
||||
you through the steps you follow to apply a seccomp profile to a Pod or to one of
|
||||
its containers. That tutorial covers the supported mechanism for configuring seccomp in Kubernetes,
|
||||
based on setting `securityContext` within the Pod's `.spec`.
|
||||
based on setting `securityContext` within the Pod's `.spec`.
|
||||
|
||||
@@ -12,7 +12,7 @@ with that removal.
|
||||
|
||||
## Kubernetes project
|
||||
|
||||
* Kubernetes blog: [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/) (originally published 2020/12/02)
|
||||
* Kubernetes blog: [Dockershim Removal FAQ](/blog/2022/02/17/dockershim-faq/) (originally published 2022/02/17)
|
||||
|
||||
* Kubernetes blog: [Kubernetes is Moving on From Dockershim: Commitments and Next Steps](/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/) (published 2022/01/07)
|
||||
|
||||
|
||||
@@ -327,6 +327,17 @@ In a cluster that includes Windows nodes, you can use the following types of Ser
|
||||
* `LoadBalancer`
|
||||
* `ExternalName`
|
||||
|
||||
{{< warning >}}
|
||||
There are known issue with NodePort services on overlay networking, if the target destination node is running Windows Server 2022.
|
||||
To avoid the issue entirely, you can configure the service with `externalTrafficPolicy: Local`.
|
||||
|
||||
There are known issues with pod to pod connectivity on l2bridge network on Windows Server 2022 with KB5005619 or higher installed.
|
||||
To workaround the issue and restore pod-pod connectivity, you can disable the WinDSR feature in kube-proxy.
|
||||
|
||||
These issues require OS fixes.
|
||||
Please follow https://github.com/microsoft/Windows-Containers/issues/204 for updates.
|
||||
{{< /warning >}}
|
||||
|
||||
Windows container networking differs in some important ways from Linux networking.
|
||||
The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) provides
|
||||
additional details and background.
|
||||
|
||||
+1
-1
@@ -16,7 +16,7 @@ A Kubernetes cluster can be divided into namespaces. Once you have a namespace t
|
||||
has a default memory
|
||||
[limit](/docs/concepts/configuration/manage-resources-containers/#requests-and-limits),
|
||||
and you then try to create a Pod with a container that does not specify its own memory
|
||||
limit its own memory limit, then the
|
||||
limit, then the
|
||||
{{< glossary_tooltip text="control plane" term_id="control-plane" >}} assigns the default
|
||||
memory limit to that container.
|
||||
|
||||
|
||||
@@ -11,7 +11,8 @@ dockershim to other container runtimes.
|
||||
|
||||
Since the announcement of [dockershim deprecation](/blog/2020/12/08/kubernetes-1-20-release-announcement/#dockershim-deprecation)
|
||||
in Kubernetes 1.20, there were questions on how this will affect various workloads and Kubernetes
|
||||
installations. You can find this blog post useful to understand the problem better: [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/)
|
||||
installations. Our [Dockershim Removal FAQ](/blog/2022/02/17/dockershim-faq/) is there to help you
|
||||
to understand the problem better.
|
||||
|
||||
It is recommended to migrate from dockershim to alternative container runtimes.
|
||||
Check out [container runtimes](/docs/setup/production-environment/container-runtimes/)
|
||||
|
||||
@@ -2,75 +2,208 @@
|
||||
reviewers:
|
||||
- fgrzadkowski
|
||||
- piosz
|
||||
title: Resource metrics pipeline
|
||||
title: Resource metrics pipeline
|
||||
content_type: concept
|
||||
---
|
||||
|
||||
<!-- overview -->
|
||||
|
||||
Resource usage metrics, such as container CPU and memory usage,
|
||||
are available in Kubernetes through the Metrics API. These metrics can be accessed either directly
|
||||
by the user with the `kubectl top` command, or by a controller in the cluster, for example
|
||||
Horizontal Pod Autoscaler, to make decisions.
|
||||
For Kubernetes, the _Metrics API_ offers a basic set of metrics to support automatic scaling and similar use cases.
|
||||
This API makes information available about resource usage for node and pod, including metrics for CPU and memory.
|
||||
If you deploy the Metrics API into your cluster, clients of the Kubernetes API can then query for this information, and
|
||||
you can use Kubernetes' access control mechanisms to manage permissions to do so.
|
||||
|
||||
<!-- body -->
|
||||
The [HorizontalPodAutoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/) (HPA) and [VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#readme) (VPA) use data from the metrics API to adjust workload replicas and resources to meet customer demand.
|
||||
|
||||
## The Metrics API
|
||||
|
||||
Through the Metrics API, you can get the amount of resource currently used
|
||||
by a given node or a given pod. This API doesn't store the metric values,
|
||||
so it's not possible, for example, to get the amount of resources used by a
|
||||
given node 10 minutes ago.
|
||||
|
||||
The API is no different from any other API:
|
||||
|
||||
- it is discoverable through the same endpoint as the other Kubernetes APIs under the path: `/apis/metrics.k8s.io/`
|
||||
- it offers the same security, scalability, and reliability guarantees
|
||||
|
||||
The API is defined in [k8s.io/metrics](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1beta1/types.go)
|
||||
repository. You can find more information about the API there.
|
||||
You can also view the resource metrics using the [`kubectl top`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#top) command.
|
||||
|
||||
{{< note >}}
|
||||
The API requires the metrics server to be deployed in the cluster. Otherwise it will be not available.
|
||||
The Metrics API, and the metrics pipeline that it enables, only offers the minimum
|
||||
CPU and memory metrics to enable automatic scaling using HPA and / or VPA.
|
||||
If you would like to provide a more complete set of metrics, you can complement
|
||||
the simpler Metrics API by deploying a second
|
||||
[metrics pipeline](/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipeline)
|
||||
that uses the _Custom Metrics API_.
|
||||
{{< /note >}}
|
||||
|
||||
## Measuring Resource Usage
|
||||
|
||||
Figure 1 illustrates the architecture of the resource metrics pipeline.
|
||||
|
||||
{{< mermaid >}}
|
||||
flowchart RL
|
||||
subgraph cluster[Cluster]
|
||||
direction RL
|
||||
S[ <br><br> ]
|
||||
A[Metrics-<br>Server]
|
||||
subgraph B[Nodes]
|
||||
direction TB
|
||||
D[cAdvisor] --> C[kubelet]
|
||||
E[Container<br>runtime] --> D
|
||||
E1[Container<br>runtime] --> D
|
||||
P[pod data] -.- C
|
||||
end
|
||||
L[API<br>server]
|
||||
W[HPA]
|
||||
C ---->|Summary<br>API| A -->|metrics<br>API| L --> W
|
||||
end
|
||||
L ---> K[kubectl<br>top]
|
||||
classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000;
|
||||
class W,B,P,K,cluster,D,E,E1 box
|
||||
classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
|
||||
class S spacewhite
|
||||
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff;
|
||||
class A,L,C k8s
|
||||
{{< /mermaid >}}
|
||||
|
||||
Figure 1. Resource Metrics Pipeline
|
||||
|
||||
The architecture components, from right to left in the figure, consist of the following:
|
||||
|
||||
* [cAdvisor](https://github.com/google/cadvisor): Daemon for collecting, aggregating and exposing container metrics included in Kubelet.
|
||||
* [kubelet](/docs/concepts/overview/components/#kubelet): Node agent for managing container resources. Resource metrics are accessible using the `/metrics/resource` and `/stats` kubelet API endpoints.
|
||||
* [Summary API](#summary-api-source): API provided by the kubelet for discovering and retrieving per-node summarized stats available through the `/stats` endpoint.
|
||||
* [metrics-server](#metrics-server): Cluster addon component that collects and aggregates resource metrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and by the `kubectl top` command. Metrics Server is a reference implementation of the Metrics API.
|
||||
* [Metrics API](#metrics-api): Kubernetes API supporting access to CPU and memory used for workload autoscaling. To make this work in your cluster, you need an API extension server that provides the Metrics API.
|
||||
|
||||
{{< note >}}
|
||||
cAdvisor supports reading metrics from cgroups, which works with typical container runtimes on Linux.
|
||||
If you use a container runtime that uses another resource isolation mechanism, for example virtualization, then that container runtime must support [CRI Container Metrics](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/cri-container-stats.md) in order for metrics to be available to the kubelet.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
<!-- body -->
|
||||
|
||||
## Metrics API
|
||||
|
||||
The metrics-server implements the Metrics API. This API allows you to access CPU and memory usage for the nodes and pods in your cluster. Its primary role is to feed resource usage metrics to K8s autoscaler components.
|
||||
|
||||
Here is an example of the Metrics API request for a `minikube` node piped through `jq` for easier reading:
|
||||
```shell
|
||||
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/minikube" | jq '.'
|
||||
```
|
||||
|
||||
Here is the same API call using `curl`:
|
||||
```shell
|
||||
curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube
|
||||
```
|
||||
Sample reply:
|
||||
```json
|
||||
{
|
||||
"kind": "NodeMetrics",
|
||||
"apiVersion": "metrics.k8s.io/v1beta1",
|
||||
"metadata": {
|
||||
"name": "minikube",
|
||||
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/minikube",
|
||||
"creationTimestamp": "2022-01-27T18:48:43Z"
|
||||
},
|
||||
"timestamp": "2022-01-27T18:48:33Z",
|
||||
"window": "30s",
|
||||
"usage": {
|
||||
"cpu": "487558164n",
|
||||
"memory": "732212Ki"
|
||||
}
|
||||
}
|
||||
```
|
||||
Here is an example of the Metrics API request for a `kube-scheduler-minikube` pod contained in the `kube-system` namespace and piped through `jq` for easier reading:
|
||||
|
||||
```shell
|
||||
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq '.'
|
||||
```
|
||||
Here is the same API call using `curl`:
|
||||
```shell
|
||||
curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube
|
||||
```
|
||||
Sample reply:
|
||||
```json
|
||||
{
|
||||
"kind": "PodMetrics",
|
||||
"apiVersion": "metrics.k8s.io/v1beta1",
|
||||
"metadata": {
|
||||
"name": "kube-scheduler-minikube",
|
||||
"namespace": "kube-system",
|
||||
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube",
|
||||
"creationTimestamp": "2022-01-27T19:25:00Z"
|
||||
},
|
||||
"timestamp": "2022-01-27T19:24:31Z",
|
||||
"window": "30s",
|
||||
"containers": [
|
||||
{
|
||||
"name": "kube-scheduler",
|
||||
"usage": {
|
||||
"cpu": "9559630n",
|
||||
"memory": "22244Ki"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The Metrics API is defined in the [k8s.io/metrics](https://github.com/kubernetes/metrics) repository. You must enable the [API aggregation layer](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) and register an [APIService](/docs/reference/kubernetes-api/cluster-resources/api-service-v1/) for the `metrics.k8s.io` API.
|
||||
|
||||
To learn more about the Metrics API, see [resource metrics API design](https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md), the [metrics-server repository](https://github.com/kubernetes-sigs/metrics-server) and the [resource metrics API](https://github.com/kubernetes/metrics#resource-metrics-api).
|
||||
|
||||
|
||||
{{< note >}} You must deploy the metrics-server or alternative adapter that serves the Metrics API to be able to access it. {{< /note >}}
|
||||
|
||||
## Measuring resource usage
|
||||
|
||||
### CPU
|
||||
|
||||
CPU is reported as the average usage, in
|
||||
[CPU cores](/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu),
|
||||
over a period of time. This value is derived by taking a rate over a cumulative CPU counter
|
||||
provided by the kernel (in both Linux and Windows kernels).
|
||||
The kubelet chooses the window for the rate calculation.
|
||||
CPU is reported as the average core usage measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers, and 1 hyper-thread on bare-metal Intel processors.
|
||||
|
||||
This value is derived by taking a rate over a cumulative CPU counter provided by the kernel (in both Linux and Windows kernels). The time window used to calculate CPU is shown under window field in Metrics API.
|
||||
|
||||
To learn more about how Kubernetes allocates and measures CPU resources, see [meaning of CPU](/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu).
|
||||
|
||||
### Memory
|
||||
|
||||
Memory is reported as the working set, in bytes, at the instant the metric was collected.
|
||||
In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure.
|
||||
However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate.
|
||||
It includes all anonymous (non-file-backed) memory since Kubernetes does not support swap.
|
||||
The metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim such pages.
|
||||
Memory is reported as the working set, measured in bytes, at the instant the metric was collected.
|
||||
|
||||
In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure. However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate.
|
||||
|
||||
The Kubernetes model for a container's working set expects that the container runtime counts anonymous memory associated with the container in question. The working set metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim pages.
|
||||
|
||||
To learn more about how Kubernetes allocates and measures memory resources, see [meaning of memory](/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory).
|
||||
|
||||
## Metrics Server
|
||||
|
||||
[Metrics Server](https://github.com/kubernetes-sigs/metrics-server) is a cluster-wide aggregator of resource usage data.
|
||||
By default, it is deployed in clusters created by `kube-up.sh` script
|
||||
as a Deployment object. If you use a different Kubernetes setup mechanism, you can deploy it using the provided
|
||||
[deployment components.yaml](https://github.com/kubernetes-sigs/metrics-server/releases) file.
|
||||
The metrics-server fetches resource metrics from the kubelets and exposes them in the Kubernetes API server through the Metrics API for use by the HPA and VPA. You can also view these metrics using the `kubectl top` command.
|
||||
|
||||
Metrics Server collects metrics from the Summary API, exposed by
|
||||
[Kubelet](/docs/reference/command-line-tools-reference/kubelet/) on each node, and is registered with the main API server via
|
||||
[Kubernetes aggregator](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/).
|
||||
The metrics-server uses the Kubernetes API to track nodes and pods in your cluster. The metrics-server queries each node over HTTP to fetch metrics. The metrics-server also builds an internal view of pod metadata, and keeps a cache of pod health. That cached pod health information is available via the extension API that the metrics-server makes available.
|
||||
|
||||
Learn more about the metrics server in
|
||||
[the design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md).
|
||||
For example with an HPA query, the metrics-server needs to identify which pods fulfill the label selectors in the deployment.
|
||||
|
||||
### Summary API Source
|
||||
The [Kubelet](/docs/reference/command-line-tools-reference/kubelet/) gathers stats at node, volume, pod and container level, and emits their statistics in
|
||||
The metrics-server calls the [kubelet](/docs/reference/command-line-tools-reference/kubelet/) API to collect metrics from each node. Depending on the metrics-server version it uses:
|
||||
* Metrics resource endpoint `/metrics/resource` in version v0.6.0+ or
|
||||
* Summary API endpoint `/stats/summary` in older versions
|
||||
|
||||
|
||||
To learn more about the metrics-server, see the [metrics-server repository](https://github.com/kubernetes-sigs/metrics-server).
|
||||
|
||||
You can also check out the following:
|
||||
|
||||
* [metrics-server design](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md)
|
||||
* [metrics-server FAQ](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md)
|
||||
* [metrics-server known issues](https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md)
|
||||
* [metrics-server releases](https://github.com/kubernetes-sigs/metrics-server/releases)
|
||||
* [Horizontal Pod Autoscaling](/docs/tasks/run-application/horizontal-pod-autoscale/)
|
||||
|
||||
### Summary API source
|
||||
|
||||
The [kubelet](/docs/reference/command-line-tools-reference/kubelet/) gathers stats at the node, volume, pod and container level, and emits this information in
|
||||
the [Summary API](https://github.com/kubernetes/kubernetes/blob/7d309e0104fedb57280b261e5677d919cb2a0e2d/staging/src/k8s.io/kubelet/pkg/apis/stats/v1alpha1/types.go)
|
||||
for consumers to read.
|
||||
|
||||
Pre-1.23, these resources have been primarily gathered from [cAdvisor](https://github.com/google/cadvisor). However, in 1.23 with the
|
||||
introduction of the `PodAndContainerStatsFromCRI` FeatureGate, container and pod level stats can be gathered by the CRI implementation.
|
||||
Note: this also requires support from the CRI implementations (containerd >= 1.6.0, CRI-O >= 1.23.0).
|
||||
Here is an example of a Summary API request for a `minikube` node:
|
||||
|
||||
|
||||
```shell
|
||||
kubectl get --raw "/api/v1/nodes/minikube/proxy/stats/summary"
|
||||
```
|
||||
Here is the same API call using `curl`:
|
||||
```shell
|
||||
curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary
|
||||
```
|
||||
{{< note >}}
|
||||
The summary API `/stats/summary` endpoint will be replaced by the `/metrics/resource` endpoint beginning with metrics-server 0.6.x.
|
||||
{{< /note >}}
|
||||
Reference in New Issue
Block a user