diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 88d163d67c..dd4ecb79e3 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,4 +1,4 @@ - +- [为文档做贡献](#为文档做贡献) +- [README.md 本地化](#readmemd-本地化) + + @@ -46,7 +53,7 @@ Before you start, install the dependencies. Clone the repository and navigate to --> 开始前，先安装这些依赖。克隆本仓库并进入对应目录： -``` +```bash git clone https://github.com/kubernetes/website.git cd website ``` @@ -57,7 +64,7 @@ The Kubernetes website uses the [Docsy Hugo theme](https://github.com/google/doc Kubernetes 网站使用的是 [Docsy Hugo 主题](https://github.com/google/docsy#readme)。 即使你打算在容器中运行网站，我们也强烈建议你通过运行以下命令来引入子模块和其他开发依赖项： -``` +```bash # pull in the Docsy submodule git submodule update --init --recursive --depth 1 ``` @@ -72,15 +79,23 @@ To build the site in a container, run the following to build the container image 要在容器中构建网站，请通过以下命令来构建容器镜像并运行： -``` +```bash make container-image make container-serve ``` -启动浏览器，打开 http://localhost:1313 来查看网站。 +如果您看到错误，这可能意味着 hugo 容器没有足够的可用计算资源。 +要解决这个问题，请增加机器（[MacOSX](https://docs.docker.com/docker-for-mac/#resources) +和 [Windows](https://docs.docker.com/docker-for-windows/#resources)）上 +Docker 允许的 CPU 和内存使用量。 + + +启动浏览器，打开 来查看网站。 当你对源文件作出修改时，Hugo 会更新网站并强制浏览器执行刷新操作。 上述命令会在端口 1313 上启动本地 Hugo 服务器。 -启动浏览器，打开 http://localhost:1313 来查看网站。 +启动浏览器，打开 来查看网站。 当你对源文件作出修改时，Hugo 会更新网站并强制浏览器执行刷新操作。 + +## 构建 API 参考页面 + + +位于 `content/en/docs/reference/kubernetes-api` 的 API 参考页面是根据 Swagger 规范构建的，使用 。 + +要更新新 Kubernetes 版本的参考页面，请执行以下步骤： + + +1. 拉取 `api-ref-generator` 子模块： + + ```bash + git submodule update --init --recursive --depth 1 + ``` + + +2. 更新 Swagger 规范： + + ```bash + curl 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json' > api-ref-assets/api/swagger.json + ``` + + +3. 在 `api-ref-assets/config/` 中，调整文件 `toc.yaml` 和 `fields.yaml` 以反映新版本的变化。 + + +4. 接下来，构建页面： + + ```bash + make api-reference + ``` + + + 您可以通过从容器映像创建和提供站点来在本地测试结果： + + ```bash + make container-image + make container-serve + ``` + + + 在 Web 浏览器中，打开 查看 API 参考。 + + +5. 当所有新的更改都反映到配置文件 `toc.yaml` 和 `fields.yaml` 中时，使用新生成的 API 参考页面创建一个 Pull Request。 + ## 故障排除 @@ -135,18 +216,24 @@ If you run `make serve` on macOS and receive the following error: 如果在 macOS 上运行 `make serve` 收到以下错误： -``` +```bash ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files make: *** [serve] Error 1 ``` + 试着查看一下当前打开文件数的限制： `launchctl limit maxfiles` -然后运行以下命令（参考https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c）： + +然后运行以下命令（参考 ）： -``` +```shell #!/bin/sh # These are the original gist links, linking to my gists now. @@ -165,6 +252,9 @@ sudo chown root:wheel /Library/LaunchDaemons/limit.maxproc.plist sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist ``` + 这适用于 Catalina 和 Mojave macOS。 # 参与 SIG Docs 工作 @@ -184,20 +275,21 @@ You can also reach the maintainers of this project at: 你也可以通过以下渠道联系本项目的维护人员： -- [Slack](https://kubernetes.slack.com/messages/sig-docs) [加入Slack](https://slack.k8s.io/) +- [Slack](https://kubernetes.slack.com/messages/sig-docs) + - [获得此 Slack 的邀请](https://slack.k8s.io/) - [邮件列表](https://groups.google.com/forum/#!forum/kubernetes-sig-docs) # 为文档做贡献 你也可以点击屏幕右上方区域的 **Fork** 按钮，在你自己的 GitHub -账号下创建本仓库的拷贝。此拷贝被称作 *fork*。 +账号下创建本仓库的拷贝。此拷贝被称作 _fork_。 你可以在自己的拷贝中任意地修改文档，并在你已准备好将所作修改提交给我们时， 在你自己的拷贝下创建一个拉取请求（Pull Request），以便让我们知道。 @@ -208,7 +300,7 @@ Once your pull request is created, a Kubernetes reviewer will take responsibilit 还要提醒的一点，有时可能会有不止一个 Kubernetes 评审人为你提供反馈意见。 有时候，某个评审人的意见和另一个最初被指派的评审人的意见不同。 @@ -220,17 +312,65 @@ Furthermore, in some cases, one of your reviewers might ask for a technical revi 有关为 Kubernetes 文档做出贡献的更多信息，请参阅： -* [贡献 Kubernetes 文档](https://kubernetes.io/docs/contribute/) -* [页面内容类型](https://kubernetes.io/docs/contribute/style/page-content-types/) -* [文档风格指南](https://kubernetes.io/docs/contribute/style/style-guide/) -* [本地化 Kubernetes 文档](https://kubernetes.io/docs/contribute/localization/) +- [贡献 Kubernetes 文档](https://kubernetes.io/docs/contribute/) +- [页面内容类型](https://kubernetes.io/docs/contribute/style/page-content-types/) +- [文档风格指南](https://kubernetes.io/docs/contribute/style/style-guide/) +- [本地化 Kubernetes 文档](https://kubernetes.io/docs/contribute/localization/) + + +### 新贡献者大使 + + +如果您在贡献时需要帮助，[新贡献者大使](https://kubernetes.io/docs/contribute/advanced/#serve-as-a-new-contributor-ambassador)是一个很好的联系人。 +这些是 SIG Docs 批准者，其职责包括指导新贡献者并帮助他们完成最初的几个拉取请求。 +联系新贡献者大使的最佳地点是 [Kubernetes Slack](https://slack.k8s.io/)。 +SIG Docs 的当前新贡献者大使： + + +| 姓名 | Slack | GitHub | +| -------------------------- | -------------------------- | -------------------------- | +| Arsh Sharma | @arsh | @RinkiyaKeDad | + + +## `README.md` 本地化 + + +| 语言 | 语言 | +| -------------------------- | -------------------------- | +| [中文](README-zh.md) | [韩语](README-ko.md) | +| [法语](README-fr.md) | [波兰语](README-pl.md) | +| [德语](README-de.md) | [葡萄牙语](README-pt.md) | +| [印地语](README-hi.md) | [俄语](README-ru.md) | +| [印尼语](README-id.md) | [西班牙语](README-es.md) | +| [意大利语](README-it.md) | [乌克兰语](README-uk.md) | +| [日语](README-ja.md) | [越南语](README-vi.md) | # 中文本地化 @@ -241,19 +381,19 @@ For more information about contributing to the Kubernetes documentation, see: * [Slack channel](https://kubernetes.slack.com/messages/kubernetes-docs-zh) -# 行为准则 +## 行为准则 参与 Kubernetes 社区受 [CNCF 行为准则](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) 约束。 -# 感谢！ +## 感谢你 Kubernetes 因为社区的参与而蓬勃发展，感谢您对我们网站和文档的贡献！ diff --git a/README.md b/README.md index 005452ef06..f037e0426b 100644 --- a/README.md +++ b/README.md @@ -36,10 +36,10 @@ git submodule update --init --recursive --depth 1 ## Running the website using a container -To build the site in a container, run the following to build the container image and run it: +To build the site in a container, run the following: ```bash -make container-image +# You can set $CONTAINER_ENGINE to the name of any Docker-like container tool make container-serve ``` @@ -189,7 +189,7 @@ If you need help at any point when contributing, the [New Contributor Ambassador ## Code of conduct -Participation in the Kubernetes community is governed by the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). +Participation in the Kubernetes community is governed by the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md). ## Thank you diff --git a/assets/scss/_custom.scss b/assets/scss/_custom.scss index 1ebe8c81fa..db0263d991 100644 --- a/assets/scss/_custom.scss +++ b/assets/scss/_custom.scss @@ -634,12 +634,12 @@ body.td-documentation { a { color: inherit; - border-bottom: 1px solid #fff; + text-decoration: underline; } a:hover { color: inherit; - border-bottom: none; + text-decoration: initial; } } @@ -648,6 +648,9 @@ body.td-documentation { } #announcement { + // default background is blue; overrides are possible + color: #fff; + .announcement-main { margin-left: auto; margin-right: auto; @@ -660,9 +663,8 @@ body.td-documentation { } - /* always white */ h1, h2, h3, h4, h5, h6, p * { - color: #ffffff; + color: inherit; /* defaults to white */ background: transparent; img.event-logo { diff --git a/cloudbuild.yaml b/cloudbuild.yaml index 5039818482..542d58016c 100644 --- a/cloudbuild.yaml +++ b/cloudbuild.yaml @@ -9,17 +9,20 @@ options: steps: # It's fine to bump the tag to a recent version, as needed - name: "gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20210917-12df099d55" - entrypoint: make + entrypoint: 'bash' env: - DOCKER_CLI_EXPERIMENTAL=enabled - TAG=$_GIT_TAG - BASE_REF=$_PULL_BASE_REF args: - - container-image + - -c + - | + gcloud auth configure-docker \ + && make container-push substitutions: # _GIT_TAG will be filled with a git-based tag for the image, of the form vYYYYMMDD-hash, and # can be used as a substitution _GIT_TAG: "12345" # _PULL_BASE_REF will contain the ref that was pushed to to trigger this build - - # a branch like 'master' or 'release-0.2', or a tag like 'v0.2'. - _PULL_BASE_REF: "master" + # a branch like 'main' or 'release-0.2', or a tag like 'v0.2'. + _PULL_BASE_REF: "main" diff --git a/config.toml b/config.toml index ade9eb83f0..a153bdf11b 100644 --- a/config.toml +++ b/config.toml @@ -122,7 +122,7 @@ id = "UA-00000000-0" [params] copyright_k8s = "The Kubernetes Authors" -copyright_linux = "Copyright © 2020 The Linux Foundation ®." +copyright_linux = "Copyright © 2020 The Linux Foundation ®." # privacy_policy = "https://policies.google.com/privacy" @@ -155,10 +155,10 @@ githubWebsiteRaw = "raw.githubusercontent.com/kubernetes/website" # GitHub repository link for editing a page and opening issues. github_repo = "https://github.com/kubernetes/website" -#Searching +# Searching k8s_search = true -#The following search parameters are specific to Docsy's implementation. Kubernetes implementes its own search-related partials and scripts. +# The following search parameters are specific to Docsy's implementation. Kubernetes implementes its own search-related partials and scripts. # Google Custom Search Engine ID. Remove or comment out to disable search. #gcs_engine_id = "011737558837375720776:fsdu1nryfng" @@ -221,11 +221,11 @@ sidebar_menu_compact = false sidebar_menu_foldable = true # https://github.com/gohugoio/hugo/issues/8918#issuecomment-903314696 sidebar_cache_limit = 1 -# Set to true to disable breadcrumb navigation. +# Set to true to disable breadcrumb navigation. breadcrumb_disable = false -# Set to true to hide the sidebar search box (the top nav search box will still be displayed if search is enabled) +# Set to true to hide the sidebar search box (the top nav search box will still be displayed if search is enabled) sidebar_search_disable = false -# Set to false if you don't want to display a logo (/assets/icons/logo.svg) in the top nav bar +# Set to false if you don't want to display a logo (/assets/icons/logo.svg) in the top nav bar navbar_logo = true # Set to true to disable the About link in the site footer footer_about_disable = false @@ -246,50 +246,50 @@ no = 'Sorry to hear that. Please }} As our contributing community grows in great numbers, with more than 16,000 contributors this year across 150+ GitHub repositories, it’s important to provide face to face connections for our large distributed teams to have opportunities for collaboration and learning. In [Contributor Experience], our methodology with planning events is a lot like our documentation; we build from personas -- interests, skills, and motivators to name a few. This way we ensure there is valuable content and learning for everyone. @@ -28,13 +28,14 @@ We build the contributor summits around you: These personas combined with ample feedback from previous events, produce the altogether experience that welcomed over 600 contributors in Copenhagen (May), Shanghai(November), and Seattle(December) in 2018. Seattle's event drew over 300+ contributors, equal to Shanghai and Copenhagen combined, for the 6th contributor event in Kubernetes history. In true Kubernetes fashion, we expect another record breaking year of attendance. We've pre-ordered 900+ [contributor patches], a tradition, and we are looking forward to giving them to you! -With that said... +With that said… + **Save the Dates:** Barcelona: May 19th (evening) and 20th (all day) Shanghai: June 24th (all day) San Diego: November 18th, 19th, and activities in KubeCon/CloudNativeCon week -In an effort of continual improvement, here's what to expect from us this year: +In an effort of continual improvement, here's what to expect from us this year: * Large new contributor workshops and contributor socials at all three events expected to break previous attendance records * A multiple track event in San Diego for all contributor types including workshops, birds of a feather, lightning talks and more @@ -42,7 +43,8 @@ In an effort of continual improvement, here's what to expect from us this year: * [An event website]! * Follow along with updates: kubernetes-dev@googlegroups.com is our main communication hub as always; however, we will also blog here, our [Thursday Kubernetes Community Meeting], [twitter], SIG meetings, event site, discuss.kubernetes.io, and #contributor-summit on Slack. * Opportunities to get involved: We still have 2019 roles available! -Reach out to Contributor Experience via community@kubernetes.io, stop by a Wednesday SIG update meeting, or catch us on Slack (#sig-contribex). +Reach out to Contributor Experience via community@kubernetes.io, stop by a Wednesday SIG update meeting, or catch us on Slack (#sig-contribex). + {{
}} @@ -51,11 +53,11 @@ Reach out to Contributor Experience via community@kubernetes.io, stop by a Wedne Our 2018 crew 🥁 Jorge Castro, Paris Pittman, Bob Killen, Jeff Sica, Megan Lehn, Guinevere Saenger, Josh Berkus, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Lindsey Tulloch, Zach Corleissen, Tim Pepper, Ihor Dvoretskyi, Nancy Mohamed, Chris Short, Mario Loria, Jason DeTiberus, Sahdev Zala, Mithra Raja -And an introduction to our 2019 crew (a thanks in advance ;) )... -Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, Guinevere Saenger, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Rui Chen, Tim Pepper, Ihor Dvoretskyi, Dawn Foster +And an introduction to our 2019 crew (a thanks in advance ;) )… +Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, Guinevere Saenger, Noah Abrahams, Yang Li, Xiangpeng Zhao, Puja Abbassi, Rui Chen, Tim Pepper, Ihor Dvoretskyi, Dawn Foster -## Relive Seattle Contributor Summit +## Relive Seattle Contributor Summit 📈 80% growth rate since the Austin 2017 December event @@ -81,15 +83,11 @@ Jonas Rosland, Josh Berkus, Paris Pittman, Jorge Castro, Bob Killen, Deb Giles, 📸 Pictures (special thanks to [rdodev]) -Garage Pic -Reg Desk - {{
}} “I love Contrib Summit! The intros and deep dives during KubeCon were a great extension of Contrib Summit. Y'all did an excellent job in the morning to level set expectations and prime everyone.” -- julianv “great work! really useful and fun!” - coffeepac -[click here]: https://events.linuxfoundation.org/events/contributor-summit-europe-2019/ [Contributor Experience]: https://github.com/kubernetes/community/tree/master/sig-contributor-experience [Subproject OWNERs]: https://github.com/kubernetes/community/blob/master/community-membership.md [Chair or Tech Lead]: https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance.md diff --git a/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md b/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md index 448309ba3a..e021bbf1db 100644 --- a/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md +++ b/content/en/blog/_posts/2019-03-25-1-14-release-announcement.md @@ -2,6 +2,7 @@ title: 'Kubernetes 1.14: Production-level support for Windows Nodes, Kubectl Updates, Persistent Local Volumes GA' date: 2019-03-25 slug: kubernetes-1-14-release-announcement +evergreen: true --- **Authors:** The 1.14 [Release Team](https://bit.ly/k8s114-team) diff --git a/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md b/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md index 41c5fbcf36..bf8417673b 100644 --- a/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md +++ b/content/en/blog/_posts/2019-05-02-kubecon-diversity-lunch-and-hack.md @@ -2,6 +2,7 @@ title: "Join us for the 2019 KubeCon Diversity Lunch & Hack" date: 2019-05-02 slug: kubecon-diversity-lunch-and-hack +evergreen: false --- **Authors:** Kiran Oliver, Podcast Producer, The New Stack @@ -36,4 +37,4 @@ To make this all possible, we need you. Yes, you, to register. As much as we lov We look forward to seeing you! -_Special thanks to [Leah Petersen](https://www.linkedin.com/in/leahstunts/), [Sarah Conway](https://www.linkedin.com/in/sarah-conway-6166151/) and [Paris Pittman](https://www.linkedin.com/in/parispittman/) for their help in editing this post._ +_Special thanks to [Leah Petersen](https://www.linkedin.com/in/leahstunts/), [Sarah Conway](https://www.linkedin.com/in/sarah-conway-6166151/) and [Paris Pittman](https://www.linkedin.com/in/parispittman/) for their help in editing this post._ diff --git a/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md b/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md index 3561203548..83dfb6dd50 100644 --- a/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md +++ b/content/en/blog/_posts/2019-06-19-kubernetes-1-15-release-announcement.md @@ -3,6 +3,7 @@ layout: blog title: "Kubernetes 1.15: Extensibility and Continuous Improvement" date: 2019-06-19 slug: kubernetes-1-15-release-announcement +evergreen: true --- **Authors:** The 1.15 [Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.15/release_team.md) diff --git a/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md b/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md index 350fbb1735..62ea9c8337 100644 --- a/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md +++ b/content/en/blog/_posts/2019-09-24-san-diego-contributor-summit.md @@ -3,19 +3,18 @@ layout: blog title: "Contributor Summit San Diego Registration Open!" date: 2019-09-24 slug: san-diego-contributor-summit +evergreen: true --- -**Authors: Paris Pittman (Google), Jeffrey Sica (Red Hat), Jonas Rosland (VMware)** - - +**Authors:** Paris Pittman (Google), Jeffrey Sica (Red Hat), Jonas Rosland (VMware) [Contributor Summit San Diego 2019 Event Page] -Registration is now open and in record time, we’ve hit capacity for the -*new contributor workshop* session of the event! Waitlist is now available. +In record time, we’ve hit capacity for the *new contributor workshop* session of +the event! **Sunday, November 17** Evening Contributor Celebration: -[QuartYard]* +[QuartYard]† Address: 1301 Market Street, San Diego, CA 92101 Time: 6:00PM - 9:00PM @@ -68,7 +67,7 @@ Check out past blogs on [persona building around our events] and the [Barcelona ![Group Picture in 2018](/images/blog/2019-09-24-san-diego-contributor-summit/IMG_2588.JPG) -*=QuartYard has a huge stage! Want to perform something in front of your contributor peers? Reach out to us! community@kubernetes.io +†=QuartYard has a huge stage! Want to perform something in front of your contributor peers? Reach out to us! community@kubernetes.io diff --git a/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md b/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md index e78b6a0ec7..f0f2835513 100644 --- a/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md +++ b/content/en/blog/_posts/2019-10-10-contributor-summit-san-diego-schedule.md @@ -5,13 +5,7 @@ date: 2019-10-10 slug: contributor-summit-san-diego-schedule --- - -Authors: Josh Berkus (Red Hat), Paris Pittman (Google), Jonas Rosland (VMware) - -tl;dr A week ago we announced that [registration is open][reg] for the contributor -summit , and we're now live with [the full Contributor Summit schedule!][schedule] -Grab your spot while tickets are still available. There is currently a waitlist -for new contributor workshop. ([Register here!][reg]) +**Authors:** Josh Berkus (Red Hat), Paris Pittman (Google), Jonas Rosland (VMware) There are many great sessions planned for the Contributor Summit, spread across five rooms of current contributor content in addition to the new contributor @@ -32,7 +26,7 @@ While the schedule contains difficult decisions in every timeslot, we've picked a few below to give you a taste of what you'll hear, see, and participate in, at the summit: -* **[Vision]**: SIG-Architecture will be sharing their vision of where we're going +* **[Vision]**: SIG Architecture will be sharing their vision of where we're going with Kubernetes development for the next year and beyond. * **[Security]**: Tim Allclair and CJ Cullen will present on the current state of Kubernetes security. In another security talk, Vallery Lancey will lead a @@ -47,7 +41,7 @@ the summit: one, or at least pass one. * **[End Users]**: Several end users from the CNCF partner ecosystem, invited by Cheryl Hung, will hold a Q&A with contributors to strengthen our feedback loop. -* **[Docs]**: As always, SIG-Docs will run a three-hour contributing-to-documentation +* **[Docs]**: As always, SIG Docs will run a three-hour contributing-to-documentation workshop. We're also giving out awards to contributors who distinguished themselves in 2019, diff --git a/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md b/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md index ae05bd8b9c..11e3ccc412 100644 --- a/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md +++ b/content/en/blog/_posts/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced.md @@ -5,11 +5,28 @@ date: 2020-02-18 slug: Contributor-Summit-Amsterdam-Schedule-Announced --- -**Authors:** Jeffrey Sica (Red Hat), Amanda Katona (VMware) +**Authors:** Jeffrey Sica (Red Hat), Amanda Katona (VMware) -tl;dr [Registration is open](https://events.linuxfoundation.org/kubernetes-contributor-summit-europe/) and the [schedule is live](https://kcseu2020.sched.com/) so register now and we’ll see you in Amsterdam! +![Contributor Summit](/images/blog/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced/contribsummit.jpg) -## Kubernetes Contributor Summit +Hello everyone and Happy 2020! It’s hard to believe that KubeCon EU 2020 is less than six weeks away, and with that another contributor summit! This year we have the pleasure of being in Amsterdam in early spring, so be sure to pack some warmer clothing. This summit looks to be exciting with a lot of fantastic community-driven content. We received **26** submissions from the CFP. From that, the events team selected **12** sessions. Each of the sessions falls into one of four categories: + +* Community +* Contributor Improvement +* Sustainability +* In-depth Technical + +On top of the presentations, there will be a dedicated Docs Sprint as well as the New Contributor Workshop 101 and 201 Sessions. All told, we will have five separate rooms of content throughout the day on Monday. Please **[see the full schedule](https://kcseu2020.sched.com/)** to see what sessions you’d be interested in. We hope between the content provided and the inevitable hallway track, everyone has a fun and enriching experience. + +Speaking of fun, the social Sunday night should be a blast! We’re hosting this summit’s social close to the conference center, at [ZuidPool](https://www.zuid-pool.nl/en/). There will be games, bingo, and unconference sign-up throughout the evening. It should be a relaxed way to kick off the week. + +[~Registration is open~](https://events.linuxfoundation.org/kubernetes-contributor-summit-europe/)! Space is limited so it’s always a good idea to register early. + +If you have any questions, reach out to the [Amsterdam Team](https://github.com/kubernetes/community/tree/master/events/2020/03-contributor-summit#team) on Slack in the [#contributor-summit](https://kubernetes.slack.com/archives/C7J893413) channel. + +Hope to see you there! + +## Kubernetes Contributor Summit schedule **Sunday, March 29, 2020** @@ -25,21 +42,3 @@ tl;dr [Registration is open](https://events.linuxfoundation.org/kubernetes-contr - Address: [Europaplein 24, 1078 GZ Amsterdam, Netherlands](https://www.google.com/search?q=kubecon+amsterdam+2020&oq=kubecon+amste&aqs=chrome.0.35i39j69i57j0l4j69i61l2.3957j1j4&sourceid=chrome&ie=UTF-8&ibp=htl;events&rciv=evn&sa=X&ved=2ahUKEwiZoLvQ0dvnAhVST6wKHScBBZ8Q5bwDMAB6BAgSEAE#) - Time: 09:00 - 17:00 (Breakfast at 08:00) -![Contributor Summit](/images/blog/2020-02-18-Contributor-Summit-Amsterdam-Schedule-Announced/contribsummit.jpg) - -Hello everyone and Happy 2020! It’s hard to believe that KubeCon EU 2020 is less than six weeks away, and with that another contributor summit! This year we have the pleasure of being in Amsterdam in early spring, so be sure to pack some warmer clothing. This summit looks to be exciting with a lot of fantastic community-driven content. We received **26** submissions from the CFP. From that, the events team selected **12** sessions. Each of the sessions falls into one of four categories: - -* Community -* Contributor Improvement -* Sustainability -* In-depth Technical - -On top of the presentations, there will be a dedicated Docs Sprint as well as the New Contributor Workshop 101 and 201 Sessions. All told, we will have five separate rooms of content throughout the day on Monday. Please **[see the full schedule](https://kcseu2020.sched.com/)** to see what sessions you’d be interested in. We hope between the content provided and the inevitable hallway track, everyone has a fun and enriching experience. - -Speaking of fun, the social Sunday night should be a blast! We’re hosting this summit’s social close to the conference center, at [ZuidPool](https://www.zuid-pool.nl/en/). There will be games, bingo, and unconference sign-up throughout the evening. It should be a relaxed way to kick off the week. - -[Registration is open](https://events.linuxfoundation.org/kubernetes-contributor-summit-europe/)! Space is limited so it’s always a good idea to register early. - -If you have any questions, reach out to the [Amsterdam Team](https://github.com/kubernetes/community/tree/master/events/2020/03-contributor-summit#team) on Slack in the [#contributor-summit](https://kubernetes.slack.com/archives/C7J893413) channel. - -Hope to see you there! diff --git a/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md b/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md index 1996b6201e..10d96ec351 100644 --- a/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md +++ b/content/en/blog/_posts/2020-03-04-Contributor-Summit-Delayed.md @@ -3,13 +3,14 @@ layout: blog title: Contributor Summit Amsterdam Postponed date: 2020-03-04 slug: Contributor-Summit-Delayed +evergreen: true --- -**Authors:** Dawn Foster (VMware), Jorge Castro (VMware) +**Authors:** Dawn Foster (VMware), Jorge Castro (VMware) The CNCF has announced that [KubeCon + CloudNativeCon EU has been delayed](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/attend/novel-coronavirus-update/) until July/August of 2020. As a result the Contributor Summit planning team is weighing options for how to proceed. Here’s the current plan: - There will be an in-person Contributor Summit as planned when KubeCon + CloudNativeCon is rescheduled. -- We are looking at options for having additional virtual contributor activities in the meantime. +- We are looking at options for having additional virtual contributor activities in the meantime. We will communicate via this blog and the usual communications channels on the final plan. Please bear with us as we adapt when we get more information. Thank you for being patient as the team pivots to bring you a great Contributor Summit! \ No newline at end of file diff --git a/content/en/blog/_posts/2020-12-02-dockershim-faq.md b/content/en/blog/_posts/2020-12-02-dockershim-faq.md index c0cb2ee83a..4711207b1c 100644 --- a/content/en/blog/_posts/2020-12-02-dockershim-faq.md +++ b/content/en/blog/_posts/2020-12-02-dockershim-faq.md @@ -155,7 +155,7 @@ runtime where possible. Another thing to look out for is anything expecting to run for system maintenance or nested inside a container when building images will no longer work. For the -former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the +former, you can use the [`crictl`][cr] tool as a drop-in replacement (see [mapping from dockercli to crictl](/docs/reference/tools/map-crictl-dockercli/)) and for the latter you can use newer container build options like [img], [buildah], [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker. diff --git a/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md b/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md index a361c4d0be..c868b1bd5c 100644 --- a/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md +++ b/content/en/blog/_posts/2021-05-14-using-finalizers-to-control-deletion.md @@ -108,7 +108,7 @@ metadata: uid: 93a37fed-23e3-45e8-b6ee-b2521db81638 ``` -In short, what’s happened is that the object was updated, not deleted. That’s because Kubernetes saw that the object contained finalizers and put it into a read-only state. The deletion timestamp signals that the object can only be read, with the exception of removing the finalizer key updates. In other words, the deletion will not be complete until we edit the object and remove the finalizer. +In short, what’s happened is that the object was updated, not deleted. That’s because Kubernetes saw that the object contained finalizers and blocked removal of the object from etcd. The deletion timestamp signals that deletion was requested, but the deletion will not be complete until we edit the object and remove the finalizer. Here's a demonstration of using the `patch` command to remove finalizers. If we want to delete an object, we can simply patch it on the command line to remove the finalizers. In this way, the deletion that was running in the background will complete and the object will be deleted. When we attempt to `get` that configmap, it will be gone. diff --git a/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md b/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md index e569c7a015..64d47ae657 100644 --- a/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md +++ b/content/en/blog/_posts/2021-09-13-read-write-once-pod-access-mode-alpha.md @@ -255,7 +255,7 @@ The minimum required versions are: ## What’s next? -As part of the beta graduation for this feature, SIG Storage plans to update the Kubenetes scheduler to support pod preemption in relation to ReadWriteOncePod storage. +As part of the beta graduation for this feature, SIG Storage plans to update the Kubernetes scheduler to support pod preemption in relation to ReadWriteOncePod storage. This means if two pods request a PersistentVolumeClaim with ReadWriteOncePod, the pod with highest priority will gain access to the PersistentVolumeClaim and any pod with lower priority will be preempted from the node and be unable to access the PersistentVolumeClaim. ## How can I learn more? diff --git a/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md b/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md index 50d66ec966..74e61b117d 100644 --- a/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md +++ b/content/en/blog/_posts/2022-02-17-updated-dockershim-faq.md @@ -175,7 +175,7 @@ runtime where possible. Another thing to look out for is anything expecting to run for system maintenance or nested inside a container when building images will no longer work. For the former, you can use the [`crictl`][cr] tool as a drop-in replacement (see -[mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#mapping-from-docker-cli-to-crictl)) +[mapping from docker cli to crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/#mapping-from-docker-cli-to-crictl)) and for the latter you can use newer container build options like [img], [buildah], [kaniko], or [buildkit-cli-for-kubectl] that don’t require Docker. diff --git a/content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md b/content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md index 999c3c514e..7bc79b38d1 100644 --- a/content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md +++ b/content/en/blog/_posts/2022-04-07-Kubernetes-1-24-removals-and-deprecations.md @@ -69,11 +69,11 @@ been deprecated. These removals have been superseded by newer, stable/generally * [Dynamic kubelet configuration](https://github.com/kubernetes/enhancements/issues/281): `DynamicKubeletConfig` is used to enable the dynamic configuration of the kubelet. The `DynamicKubeletConfig` flag was deprecated in Kubernetes 1.22. In v1.24, this feature gate will be removed from the kubelet. See [Reconfigure kubelet](/docs/tasks/administer-cluster/reconfigure-kubelet/). Refer to the ["Dynamic kubelet config is removed" KEP](https://github.com/kubernetes/enhancements/issues/281) for more information. * [Dynamic log sanitization](https://github.com/kubernetes/kubernetes/pull/107207): The experimental dynamic log sanitization feature is deprecated and will be removed in v1.24. This feature introduced a logging filter that could be applied to all Kubernetes system components logs to prevent various types of sensitive information from leaking via logs. Refer to [KEP-1753: Kubernetes system components logs sanitization](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#deprecation) for more information and an [alternative approach](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1753-logs-sanitization#alternatives=). -* In-tree provisioner to CSI driver migration: This applies to a number of in-tree plugins, including [Portworx](https://github.com/kubernetes/enhancements/issues/2589). Refer to the [In-tree Storage Plugin to CSI Migration Design Doc](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/csi-migration.md#background-and-motivations) for more information. +* In-tree provisioner to CSI driver migration: This applies to a number of in-tree plugins, including [Portworx](https://github.com/kubernetes/enhancements/issues/2589). Refer to the [In-tree Storage Plugin to CSI Migration Design Doc](https://git.k8s.io/design-proposals-archive/storage/csi-migration.md#background-and-motivations) for more information. * [Removing Dockershim from kubelet](https://github.com/kubernetes/enhancements/issues/2221): the Container Runtime Interface (CRI) for Docker (i.e. Dockershim) is currently a built-in container runtime in the kubelet code base. It was deprecated in v1.20. As of v1.24, the kubelet will no longer have dockershim. Check out this blog on [what you need to do be ready for v1.24](/blog/2022/03/31/ready-for-dockershim-removal/). * [Storage capacity tracking for pod scheduling](https://github.com/kubernetes/enhancements/issues/1472): The CSIStorageCapacity API supports exposing currently available storage capacity via CSIStorageCapacity objects and enhances scheduling of pods that use CSI volumes with late binding. In v1.24, the CSIStorageCapacity API will be stable. The API graduating to stable initates the deprecation of the v1beta1 CSIStorageCapacity API. Refer to the [Storage Capacity Constraints for Pod Scheduling KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1472-storage-capacity-tracking) for more information. * [The `master` label is no longer present on kubeadm control plane nodes](https://github.com/kubernetes/kubernetes/pull/107533). For new clusters, the label 'node-role.kubernetes.io/master' will no longer be added to control plane nodes, only the label 'node-role.kubernetes.io/control-plane' will be added. For more information, refer to [KEP-2067: Rename the kubeadm "master" label and taint](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint). -* [VolumeSnapshot v1beta1 CRD will be removed](https://github.com/kubernetes/enhancements/issues/177). Volume snapshot and restore functionality for Kubernetes and the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, entered beta in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.21 and is now unsupported. Refer to [KEP-177: CSI Snapshot](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot#kep-177-csi-snapshot) and [kubernetes-csi/external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v4.1.0) for more information. +* [VolumeSnapshot v1beta1 CRD will be removed](https://github.com/kubernetes/enhancements/issues/177). Volume snapshot and restore functionality for Kubernetes and the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, moved to GA in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.20 and will become unsupported with the v1.24 release. Refer to [KEP-177: CSI Snapshot](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot#kep-177-csi-snapshot) and the [Volume Snapshot GA blog](/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/) blog article for more information. ## What to do diff --git a/content/en/blog/_posts/2022-05-03-kubernetes-release-1.24.md b/content/en/blog/_posts/2022-05-03-kubernetes-release-1.24.md index abc2ddcb9d..f29c6c92fe 100644 --- a/content/en/blog/_posts/2022-05-03-kubernetes-release-1.24.md +++ b/content/en/blog/_posts/2022-05-03-kubernetes-release-1.24.md @@ -5,8 +5,6 @@ date: 2022-05-03 slug: kubernetes-1-24-release-announcement --- -# Kubernetes 1.24 - **Authors**: [Kubernetes 1.24 Release Team](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.24/release-team.md) We are excited to announce the release of Kubernetes 1.24, the first release of 2022! @@ -120,6 +118,13 @@ With containerd v1.6.0–v1.6.3, if you do not upgrade the CNI plugins and/or declare the CNI config version, you might encounter the following "Incompatible CNI versions" or "Failed to destroy network for sandbox" error conditions. +## CSI Snapshot + +_This information was added after initial publication._ + +[VolumeSnapshot v1beta1 CRD has been removed](https://github.com/kubernetes/enhancements/issues/177). +Volume snapshot and restore functionality for Kubernetes and the Container Storage Interface (CSI), which provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers, moved to GA in v1.20. VolumeSnapshot v1beta1 was deprecated in v1.20 and is now unsupported. Refer to [KEP-177: CSI Snapshot](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot#kep-177-csi-snapshot) and [Volume Snapshot GA blog](https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/) for more information. + ## Other Updates ### Graduations to Stable @@ -202,7 +207,7 @@ all the stargazers out there. ✨ ### Ecosystem Updates -* KubeCon + CloudNativeCon Europe 2022 will take place in Valencia, Spain, from 16 – 20 May 2022! You can find more information about the conference and registration on the [event site](https://events.linuxfoundation.org/archive/2021/kubecon-cloudnativecon-europe/). +* KubeCon + CloudNativeCon Europe 2022 will take place in Valencia, Spain, from 16 – 20 May 2022! You can find more information about the conference and registration on the [event site](https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/). * In the [2021 Cloud Native Survey](https://www.cncf.io/announcements/2022/02/10/cncf-sees-record-kubernetes-and-container-adoption-in-2021-cloud-native-survey/), the CNCF saw record Kubernetes and container adoption. Take a look at the [results of the survey](https://www.cncf.io/reports/cncf-annual-survey-2021/). * The [Linux Foundation](https://www.linuxfoundation.org/) and [The Cloud Native Computing Foundation](https://www.cncf.io/) (CNCF) announced the availability of a new [Cloud Native Developer Bootcamp](https://training.linuxfoundation.org/training/cloudnativedev-bootcamp/?utm_source=lftraining&utm_medium=pr&utm_campaign=clouddevbc0322) to provide participants with the knowledge and skills to design, build, and deploy cloud native applications. Check out the [announcement](https://www.cncf.io/announcements/2022/03/15/new-cloud-native-developer-bootcamp-provides-a-clear-path-to-cloud-native-careers/) to learn more. @@ -213,7 +218,7 @@ aggregates a number of interesting data points related to the velocity of Kubern sub-projects. This includes everything from individual contributions to the number of companies that are contributing, and is an illustration of the depth and breadth of effort that goes into evolving this ecosystem. -In the v1.24 release cycle, which [ran for 17 weeks](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24) (January 10 to May 3), we saw contributions from [1029 companies](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=v1.23.0%20-%20now&var-metric=contributions) and [1179 individuals](https://k8s.devstats.cncf.io/d/66/developer-activity-counts-by-companies?orgId=1&var-period_name=v1.23.0%20-%20now&var-metric=contributions&var-repogroup_name=Kubernetes&var-country_name=All&var-companies=All&var-repo_name=kubernetes%2Fkubernetes). +In the v1.24 release cycle, which [ran for 17 weeks](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24) (January 10 to May 3), we saw contributions from [1029 companies](https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var-period_name=v1.23.0%20-%20v1.24.0&var-metric=contributions) and [1179 individuals](https://k8s.devstats.cncf.io/d/66/developer-activity-counts-by-companies?orgId=1&var-period_name=v1.23.0%20-%20v1.24.0&var-metric=contributions&var-repogroup_name=Kubernetes&var-country_name=All&var-companies=All&var-repo_name=kubernetes%2Fkubernetes). ## Upcoming Release Webinar diff --git a/content/en/blog/_posts/2022-05-05-volume-expansion-ga.md b/content/en/blog/_posts/2022-05-05-volume-expansion-ga.md index c97d50ea31..c823ae8a2c 100644 --- a/content/en/blog/_posts/2022-05-05-volume-expansion-ga.md +++ b/content/en/blog/_posts/2022-05-05-volume-expansion-ga.md @@ -49,7 +49,7 @@ Not every volume type however is expandable by default. Some volume types such a must have capability `EXPAND_VOLUME` in controller or node service (or both if appropriate). Please refer to documentation of your CSI driver, to find out if it supports volume expansion. -Please refer to volume expansion documentation for intree volume types which support volume expansion - [Expanding Persistent Volumes](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims) +Please refer to volume expansion documentation for intree volume types which support volume expansion - [Expanding Persistent Volumes](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims). In general to provide some degree of control over volumes that can be expanded, only dynamically provisioned PVCs whose storage class has `allowVolumeExpansion` parameter set to `true` are expandable. @@ -85,7 +85,7 @@ provider to find out - what mode of volume expansion it supports. When volume expansion was introduced as an alpha feature, Kubernetes only supported offline filesystem expansion on the node and hence required users to restart their pods for file system resizing to finish. -his behaviour has been changed and Kubernetes tries its best to fulfil any resize request regardless +His behaviour has been changed and Kubernetes tries its best to fulfil any resize request regardless of whether the underlying PersistentVolume volume is online or offline. If your storage provider supports online expansion then no Pod restart should be necessary for volume expansion to finish. diff --git a/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md b/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md index 35d6838f51..2bb85059e3 100644 --- a/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md +++ b/content/en/blog/_posts/2022-05-06-storage-capacity-GA/index.md @@ -1,6 +1,6 @@ --- layout: blog -title: "Storage Capacity Tracking reaches GA in Kubernetes 1.24" +title: "Kubernetes 1.24: Storage Capacity Tracking Now Generally Available" date: 2022-05-06 slug: storage-capacity-ga --- diff --git a/content/en/blog/_posts/2022-05-13-grpc-probes-in-beta.md b/content/en/blog/_posts/2022-05-13-grpc-probes-in-beta.md new file mode 100644 index 0000000000..5ff495410b --- /dev/null +++ b/content/en/blog/_posts/2022-05-13-grpc-probes-in-beta.md @@ -0,0 +1,209 @@ +--- +layout: blog +title: "Kubernetes 1.24: gRPC container probes in beta" +date: 2022-05-13 +slug: grpc-probes-now-in-beta +--- + +**Author**: Sergey Kanzhelev (Google) + + +With Kubernetes 1.24 the gRPC probes functionality entered beta and is available by default. +Now you can configure startup, liveness, and readiness probes for your gRPC app +without exposing any HTTP endpoint, nor do you need an executable. Kubernetes can natively connect to your your workload via gRPC and query its status. + +## Some history + +It's useful to let the system managing your workload check that the app is +healthy, has started OK, and whether the app considers itself good to accept +traffic. Before the gRPC support was added, Kubernetes already allowed you to +check for health based on running an executable from inside the container image, +by making an HTTP request, or by checking whether a TCP connection succeeded. + +For most apps, those checks are enough. If your app provides a gRPC endpoint +for a health (or readiness) check, it is easy +to repurpose the `exec` probe to use it for gRPC health checking. +In the blog article [Health checking gRPC servers on Kubernetes](/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/), +Ahmet Alp Balkan described how you can do that — a mechanism that still works today. + +There is a commonly used tool to enable this that was [created](https://github.com/grpc-ecosystem/grpc-health-probe/commit/2df4478982e95c9a57d5fe3f555667f4365c025d) +on August 21, 2018, and with +the first release at [Sep 19, 2018](https://github.com/grpc-ecosystem/grpc-health-probe/releases/tag/v0.1.0-alpha.1). + +This approach for gRPC apps health checking is very popular. There are [3,626 Dockerfiles](https://github.com/search?l=Dockerfile&q=grpc_health_probe&type=code) +with the `grpc_health_probe` and [6,621 yaml](https://github.com/search?l=YAML&q=grpc_health_probe&type=Code) files that are discovered with the +basic search on GitHub (at the moment of writing). This is good indication of the tool popularity +and the need to support this natively. + +Kubernetes v1.23 introduced an alpha-quality implementation of native support for +querying a workload status using gRPC. Because it was an alpha feature, +this was disabled by default for the v1.23 release. + +## Using the feature + +We built gRPC health checking in similar way with other probes and believe +it will be [easy to use](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) +if you are familiar with other probe types in Kubernetes. +The natively supported health probe has many benefits over the workaround involving `grpc_health_probe` executable. + +With the native gRPC support you don't need to download and carry `10MB` of an additional executable with your image. +Exec probes are generally slower than a gRPC call as they require instantiating a new process to run an executable. +It also makes the checks less sensible for edge cases when the pod is running at maximum resources and has troubles +instantiating new processes. + +There are a few limitations though. Since configuring a client certificate for probes is hard, +services that require client authentication are not supported. The built-in probes are also +not checking the server certificates and ignore related problems. + +Built-in checks also cannot be configured to ignore certain types of errors +(`grpc_health_probe` returns different exit codes for different errors), +and cannot be "chained" to run the health check on multiple services in a single probe. + +But all these limitations are quite standard for gRPC and there are easy workarounds +for those. + +## Try it for yourself + +### Cluster-level setup + +You can try this feature today. To try native gRPC probes, you can spin up a Kubernetes cluster +yourself with the `GRPCContainerProbe` feature gate enabled, there are many [tools available](/docs/tasks/tools/). + +Since the feature gate `GRPCContainerProbe` is enabled by default in 1.24, +many vendors will have this functionality working out of the box. +So you may just create an 1.24 cluster on platform of your choice. Some vendors +allow to enable alpha features on 1.23 clusters. + +For example, at the moment of writing, you can spin up the test cluster on GKE for a quick test. +Other vendors may also have similar capabilities, especially if you +are reading this blog post long after the Kubernetes 1.24 release. + +On GKE use the following command (note, version is `1.23` and `enable-kubernetes-alpha` are specified). + +```shell +gcloud container clusters create test-grpc \ + --enable-kubernetes-alpha \ + --no-enable-autorepair \ + --no-enable-autoupgrade \ + --release-channel=rapid \ + --cluster-version=1.23 +``` + +You will also need to configure `kubectl` to access the cluster: + +```shell +gcloud container clusters get-credentials test-grpc +``` + +### Trying the feature out + +Let's create the pod to test how gRPC probes work. For this test we will use the `agnhost` image. +This is a k8s maintained image with that can be used for all sorts of workload testing. +For example, it has a useful [grpc-health-checking](https://github.com/kubernetes/kubernetes/blob/b2c5bd2a278288b5ef19e25bf7413ecb872577a4/test/images/agnhost/README.md#grpc-health-checking) module +that exposes two ports - one is serving health checking service, +another - http port to react on commands `make-serving` and `make-not-serving`. + +Here is an example pod definition. It starts the `grpc-health-checking` module, +exposes ports `5000` and `8080`, and configures gRPC readiness probe: + +``` yaml +--- +apiVersion: v1 +kind: Pod +metadata: + name: test-grpc +spec: + containers: + - name: agnhost + image: k8s.gcr.io/e2e-test-images/agnhost:2.35 + command: ["/agnhost", "grpc-health-checking"] + ports: + - containerPort: 5000 + - containerPort: 8080 + readinessProbe: + grpc: + port: 5000 +``` + +If the file called `test.yaml`, you can create the pod and check it's status. +The pod will be in ready state as indicated by the snippet of the output. + +```shell +kubectl apply -f test.yaml +kubectl describe test-grpc +``` + +The output will contain something like this: + +``` +Conditions: + Type Status + Initialized True + Ready True + ContainersReady True + PodScheduled True +``` + +Now let's change the health checking endpoint status to NOT_SERVING. +In order to call the http port of the Pod, let's create a port forward: + +```shell +kubectl port-forward test-grpc 8080:8080 +``` + +You can `curl` to call the command... + +```shell +curl http://localhost:8080/make-not-serving +``` + +... and in a few seconds the port status will switch to not ready. + +```shell +kubectl describe pod test-grpc +``` + +The output now will have: + +``` +Conditions: + Type Status + Initialized True + Ready False + ContainersReady False + PodScheduled True + +... + + Warning Unhealthy 2s (x6 over 42s) kubelet Readiness probe failed: service unhealthy (responded with "NOT_SERVING") +``` + +Once it is switched back, in about one second the Pod will get back to ready status: + +``` bsh +curl http://localhost:8080/make-serving +kubectl describe test-grpc +``` + +The output indicates that the Pod went back to being `Ready`: + +``` +Conditions: + Type Status + Initialized True + Ready True + ContainersReady True + PodScheduled True +``` + +This new built-in gRPC health probing on Kubernetes makes implementing a health-check via gRPC +much easier than the older approach that relied on using a separate `exec` probe. Read through +the official +[documentation](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) +to learn more and provide feedback before the feature will be promoted to GA. + +## Summary + +Kubernetes is a popular workload orchestration platform and we add features based on feedback and demand. +Features like gRPC probes support is a minor improvement that will make life of many app developers +easier and apps more resilient. Try it today and give feedback, before the feature went into GA. diff --git a/content/en/blog/_posts/2022-05-16-volume-populators-beta.md b/content/en/blog/_posts/2022-05-16-volume-populators-beta.md index 8585014f7a..4558f07eae 100644 --- a/content/en/blog/_posts/2022-05-16-volume-populators-beta.md +++ b/content/en/blog/_posts/2022-05-16-volume-populators-beta.md @@ -8,11 +8,11 @@ slug: volume-populators-beta **Author:** Ben Swartzlander (NetApp) -The volume populators feature is now two releases old and entering beta! The `AnyVolumeDataSouce` feature +The volume populators feature is now two releases old and entering beta! The `AnyVolumeDataSource` feature gate defaults to enabled in Kubernetes v1.24, which means that users can specify any custom resource as the data source of a PVC. -An [earlier blog article](/blog/2021/08/30-volume-populators-redesigned/) detailed how the +An [earlier blog article](/blog/2021/08/30/volume-populators-redesigned/) detailed how the volume populators feature works. In short, a cluster administrator can install a CRD and associated populator controller in the cluster, and any user who can create instances of the CR can create pre-populated volumes by taking advantage of the populator. diff --git a/content/en/blog/_posts/2022-05-18-prevent-unauthorised-volume-mode-conversion.md b/content/en/blog/_posts/2022-05-18-prevent-unauthorised-volume-mode-conversion.md new file mode 100644 index 0000000000..920d578d01 --- /dev/null +++ b/content/en/blog/_posts/2022-05-18-prevent-unauthorised-volume-mode-conversion.md @@ -0,0 +1,117 @@ +--- +layout: blog +title: 'Kubernetes 1.24: Prevent unauthorised volume mode conversion' +date: 2022-05-18 +slug: prevent-unauthorised-volume-mode-conversion-alpha +--- + +**Author:** Raunak Pradip Shah (Mirantis) + +Kubernetes v1.24 introduces a new alpha-level feature that prevents unauthorised users +from modifying the volume mode of a [`PersistentVolumeClaim`](/docs/concepts/storage/persistent-volumes/) created from an +existing [`VolumeSnapshot`](/docs/concepts/storage/volume-snapshots/) in the Kubernetes cluster. + + + +### The problem + +The [Volume Mode](/docs/concepts/storage/persistent-volumes/#volume-mode) determines whether a volume +is formatted into a filesystem or presented as a raw block device. + +Users can leverage the `VolumeSnapshot` feature, which has been stable since Kubernetes v1.20, +to create a `PersistentVolumeClaim` (shortened as PVC) from an existing `VolumeSnapshot` in +the Kubernetes cluster. The PVC spec includes a `dataSource` field, which can point to an +existing `VolumeSnapshot` instance. +Visit [Create a PersistentVolumeClaim from a Volume Snapshot](/docs/concepts/storage/persistent-volumes/#create-persistent-volume-claim-from-volume-snapshot) for more details. + +When leveraging the above capability, there is no logic that validates whether the mode of the +original volume, whose snapshot was taken, matches the mode of the newly created volume. + +This presents a security gap that allows malicious users to potentially exploit an +as-yet-unknown vulnerability in the host operating system. + +Many popular storage backup vendors convert the volume mode during the course of a +backup operation, for efficiency purposes, which prevents Kubernetes from blocking +the operation completely and presents a challenge in distinguishing trusted +users from malicious ones. + +### Preventing unauthorised users from converting the volume mode + +In this context, an authorised user is one who has access rights to perform `Update` +or `Patch` operations on `VolumeSnapshotContents`, which is a cluster-level resource. +It is upto the cluster administrator to provide these rights only to trusted users +or applications, like backup vendors. + +If the alpha feature is [enabled](https://kubernetes-csi.github.io/docs/) in +`snapshot-controller`, `snapshot-validation-webhook` and `external-provisioner`, +then unauthorised users will not be allowed to modify the volume mode of a PVC +when it is being created from a `VolumeSnapshot`. + +To convert the volume mode, an authorised user must do the following: + +1. Identify the `VolumeSnapshot` that is to be used as the data source for a newly +created PVC in the given namespace. +2. Identify the `VolumeSnapshotContent` bound to the above `VolumeSnapshot`. + + ```shell + kubectl get volumesnapshot -n + ``` + +3. Add the annotation [`snapshot.storage.kubernetes.io/allowVolumeModeChange`](/docs/reference/labels-annotations-taints/#snapshot-storage-kubernetes-io-allowvolumemodechange) +to the `VolumeSnapshotContent`. + +4. This annotation can be added either via software or manually by the authorised +user. The `VolumeSnapshotContent` annotation must look like following manifest fragment: + + ```yaml + kind: VolumeSnapshotContent + metadata: + annotations: + - snapshot.storage.kubernetes.io/allowVolumeModeChange: "true" + ... + ``` + +**Note**: For pre-provisioned `VolumeSnapshotContents`, you must take an extra +step of setting `spec.sourceVolumeMode` field to either `Filesystem` or `Block`, +depending on the mode of the volume from which this snapshot was taken. + +An example is shown below: + + ```yaml + apiVersion: snapshot.storage.k8s.io/v1 + kind: VolumeSnapshotContent + metadata: + annotations: + - snapshot.storage.kubernetes.io/allowVolumeModeChange: "true" + name: new-snapshot-content-test + spec: + deletionPolicy: Delete + driver: hostpath.csi.k8s.io + source: + snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002 + sourceVolumeMode: Filesystem + volumeSnapshotRef: + name: new-snapshot-test + namespace: default + ``` + +Repeat steps 1 to 3 for all `VolumeSnapshotContents` whose volume mode needs to be +converted during a backup or restore operation. + +If the annotation shown in step 4 above is present on a `VolumeSnapshotContent` +object, Kubernetes will not prevent the volume mode from being converted. +Users should keep this in mind before they attempt to add the annotation +to any `VolumeSnapshotContent`. + + +### What's next + +[Enable this feature](https://kubernetes-csi.github.io/docs/) and let us know +what you think! + +We hope this feature causes no disruption to existing workflows while preventing +malicious users from exploiting security vulnerabilities in their clusters. + +For any queries or issues, join [Kubernetes on Slack](https://slack.k8s.io/) and +create a thread in the #sig-storage channel. Alternately, create an issue in the +CSI external-snapshotter [repository](https://github.com/kubernetes-csi/external-snapshotter). \ No newline at end of file diff --git a/content/en/blog/_posts/2022-05-20-non-graceful-node-shutdown.md b/content/en/blog/_posts/2022-05-20-non-graceful-node-shutdown.md new file mode 100644 index 0000000000..f8f4876285 --- /dev/null +++ b/content/en/blog/_posts/2022-05-20-non-graceful-node-shutdown.md @@ -0,0 +1,96 @@ +--- +layout: blog +title: "Kubernetes 1.24: Introducing Non-Graceful Node Shutdown Alpha" +date: 2022-05-20 +slug: kubernetes-1-24-non-graceful-node-shutdown-alpha +--- + +**Authors** Xing Yang and Yassine Tijani (VMware) + +Kubernetes v1.24 introduces alpha support for [Non-Graceful Node Shutdown](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2268-non-graceful-shutdown). This feature allows stateful workloads to failover to a different node after the original node is shutdown or in a non-recoverable state such as hardware failure or broken OS. + +## How is this different from Graceful Node Shutdown + +You might have heard about the [Graceful Node Shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown) capability of Kubernetes, +and are wondering how the Non-Graceful Node Shutdown feature is different from that. Graceful Node Shutdown +allows Kubernetes to detect when a node is shutting down cleanly, and handles that situation appropriately. +A Node Shutdown can be "graceful" only if the node shutdown action can be detected by the kubelet ahead +of the actual shutdown. However, there are cases where a node shutdown action may not be detected by +the kubelet. This could happen either because the shutdown command does not trigger the systemd inhibitor +locks mechanism that kubelet relies upon, or because of a configuration error +(the `ShutdownGracePeriod` and `ShutdownGracePeriodCriticalPods` are not configured properly). + +Graceful node shutdown relies on Linux-specific support. The kubelet does not watch for upcoming +shutdowns on Windows nodes (this may change in a future Kubernetes release). + +When a node is shutdown but without the kubelet detecting it, pods on that node +also shut down ungracefully. For stateless apps, that's often not a problem (a ReplicaSet adds a new pod once +the cluster detects that the affected node or pod has failed). For stateful apps, the story is more complicated. +If you use a StatefulSet and have a pod from that StatefulSet on a node that fails uncleanly, that affected pod +will be marked as terminating; the StatefulSet cannot create a replacement pod because the pod +still exists in the cluster. +As a result, the application running on the StatefulSet may be degraded or even offline. If the original, shut +down node comes up again, the kubelet on that original node reports in, deletes the existing pods, and +the control plane makes a replacement pod for that StatefulSet on a different running node. +If the original node has failed and does not come up, those stateful pods would be stuck in a +terminating status on that failed node indefinitely. + +``` +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +web-0 1/1 Running 0 100m 10.244.2.4 k8s-node-876-1639279816 +web-1 1/1 Terminating 0 100m 10.244.1.3 k8s-node-433-1639279804 +``` + +## Try out the new non-graceful shutdown handling + +To use the non-graceful node shutdown handling, you must enable the `NodeOutOfServiceVolumeDetach` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the `kube-controller-manager` +component. + +In the case of a node shutdown, you can manually taint that node as out of service. You should make certain that +the node is truly shutdown (not in the middle of restarting) before you add that taint. You could add that +taint following a shutdown that the kubelet did not detect and handle in advance; another case where you +can use that taint is when the node is in a non-recoverable state due to a hardware failure or a broken OS. +The values you set for that taint can be `node.kubernetes.io/out-of-service=nodeshutdown: "NoExecute"` +or `node.kubernetes.io/out-of-service=nodeshutdown:" NoSchedule"`. +Provided you have enabled the feature gate mentioned earlier, setting the out-of-service taint on a Node +means that pods on the node will be deleted unless if there are matching tolerations on the pods. +Persistent volumes attached to the shutdown node will be detached, and for StatefulSets, replacement pods will +be created successfully on a different running node. + +``` +$ kubectl taint nodes node.kubernetes.io/out-of-service=nodeshutdown:NoExecute + +$ kubectl get pod -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +web-0 1/1 Running 0 150m 10.244.2.4 k8s-node-876-1639279816 +web-1 1/1 Running 0 10m 10.244.1.7 k8s-node-433-1639279804 +``` + +Note: Before applying the out-of-service taint, you **must** verify that a node is already in shutdown or power off state (not in the middle of restarting), either because the user intentionally shut it down or the node is down due to hardware failures, OS issues, etc. + +Once all the workload pods that are linked to the out-of-service node are moved to a new running node, and the shutdown node has been recovered, you should remove +that taint on the affected node after the node is recovered. +If you know that the node will not return to service, you could instead delete the node from the cluster. + +## What’s next? + +Depending on feedback and adoption, the Kubernetes team plans to push the Non-Graceful Node Shutdown implementation to Beta in either 1.25 or 1.26. + +This feature requires a user to manually add a taint to the node to trigger workloads failover and remove the taint after the node is recovered. In the future, we plan to find ways to automatically detect and fence nodes that are shutdown/failed and automatically failover workloads to another node. + +## How can I learn more? + +Check out the [documentation](/docs/concepts/architecture/nodes/#non-graceful-node-shutdown) +for non-graceful node shutdown. + +## How to get involved? + +This feature has a long story. Yassine Tijani ([yastij](https://github.com/yastij)) started the KEP more than two years ago. Xing Yang ([xing-yang](https://github.com/xing-yang)) continued to drive the effort. There were many discussions among SIG Storage, SIG Node, and API reviewers to nail down the design details. Ashutosh Kumar ([sonasingh46](https://github.com/sonasingh46)) did most of the implementation and brought it to Alpha in Kubernetes 1.24. + +We want to thank the following people for their insightful reviews: Tim Hockin ([thockin](https://github.com/thockin)) for his guidance on the design, Jing Xu ([jingxu97](https://github.com/jingxu97)), Hemant Kumar ([gnufied](https://github.com/gnufied)), and Michelle Au ([msau42](https://github.com/msau42)) for reviews from SIG Storage side, and Mrunal Patel ([mrunalp](https://github.com/mrunalp)), David Porter ([bobbypage](https://github.com/bobbypage)), Derek Carr ([derekwaynecarr](https://github.com/derekwaynecarr)), and Danielle Endocrimes ([endocrimes](https://github.com/endocrimes)) for reviews from SIG Node side. + +There are many people who have helped review the design and implementation along the way. We want to thank everyone who has contributed to this effort including the about 30 people who have reviewed the [KEP](https://github.com/kubernetes/enhancements/pull/1116) and implementation over the last couple of years. + +This feature is a collaboration between SIG Storage and SIG Node. For those interested in getting involved with the design and development of any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG). For those interested in getting involved with the design and development of the components that support the controlled interactions between pods and host resources, join the [Kubernetes Node SIG](https://github.com/kubernetes/community/tree/master/sig-node). diff --git a/content/en/blog/_posts/2022-05-23-service-ip-dynamic-and-static-allocation.md b/content/en/blog/_posts/2022-05-23-service-ip-dynamic-and-static-allocation.md new file mode 100644 index 0000000000..c17453a6de --- /dev/null +++ b/content/en/blog/_posts/2022-05-23-service-ip-dynamic-and-static-allocation.md @@ -0,0 +1,137 @@ +--- +layout: blog +title: "Kubernetes 1.24: Avoid Collisions Assigning IP Addresses to Services" +date: 2022-05-23 +slug: service-ip-dynamic-and-static-allocation +--- + +**Author:** Antonio Ojea (Red Hat) + + +In Kubernetes, [Services](/docs/concepts/services-networking/service/) are an abstract way to expose +an application running on a set of Pods. Services +can have a cluster-scoped virtual IP address (using a Service of `type: ClusterIP`). +Clients can connect using that virtual IP address, and Kubernetes then load-balances traffic to that +Service across the different backing Pods. + +## How Service ClusterIPs are allocated? + +A Service `ClusterIP` can be assigned: + +_dynamically_ +: the cluster's control plane automatically picks a free IP address from within the configured IP range for `type: ClusterIP` Services. + +_statically_ +: you specify an IP address of your choice, from within the configured IP range for Services. + +Across your whole cluster, every Service `ClusterIP` must be unique. +Trying to create a Service with a specific `ClusterIP` that has already +been allocated will return an error. + +## Why do you need to reserve Service Cluster IPs? + +Sometimes you may want to have Services running in well-known IP addresses, so other components and +users in the cluster can use them. + +The best example is the DNS Service for the cluster. Some Kubernetes installers assign the 10th address from +the Service IP range to the DNS service. Assuming you configured your cluster with Service IP range +10.96.0.0/16 and you want your DNS Service IP to be 10.96.0.10, you'd have to create a Service like +this: + +```yaml +apiVersion: v1 +kind: Service +metadata: + labels: + k8s-app: kube-dns + kubernetes.io/cluster-service: "true" + kubernetes.io/name: CoreDNS + name: kube-dns + namespace: kube-system +spec: + clusterIP: 10.96.0.10 + ports: + - name: dns + port: 53 + protocol: UDP + targetPort: 53 + - name: dns-tcp + port: 53 + protocol: TCP + targetPort: 53 + selector: + k8s-app: kube-dns + type: ClusterIP +``` + +but as I explained before, the IP address 10.96.0.10 has not been reserved; if other Services are created +before or in parallel with dynamic allocation, there is a chance they can allocate this IP, hence, +you will not be able to create the DNS Service because it will fail with a conflict error. + +## How can you avoid Service ClusterIP conflicts? {#avoid-ClusterIP-conflict} + +In Kubernetes 1.24, you can enable a new feature gate `ServiceIPStaticSubrange`. +Turning this on allows you to use a different IP +allocation strategy for Services, reducing the risk of collision. + +The `ClusterIP` range will be divided, based on the formula `min(max(16, cidrSize / 16), 256)`, +described as _never less than 16 or more than 256 with a graduated step between them_. + +Dynamic IP assignment will use the upper band by default, once this has been exhausted it will +use the lower range. This will allow users to use static allocations on the lower band with a low +risk of collision. + +Examples: + +#### Service IP CIDR block: 10.96.0.0/24 + +Range Size: 28 - 2 = 254 +Band Offset: `min(max(16, 256/16), 256)` = `min(16, 256)` = 16 +Static band start: 10.96.0.1 +Static band end: 10.96.0.16 +Range end: 10.96.0.254 + +{{< mermaid >}} +pie showData + title 10.96.0.0/24 + "Static" : 16 + "Dynamic" : 238 +{{< /mermaid >}} + +#### Service IP CIDR block: 10.96.0.0/20 + +Range Size: 212 - 2 = 4094 +Band Offset: `min(max(16, 4096/16), 256)` = `min(256, 256)` = 256 +Static band start: 10.96.0.1 +Static band end: 10.96.1.0 +Range end: 10.96.15.254 + +{{< mermaid >}} +pie showData + title 10.96.0.0/20 + "Static" : 256 + "Dynamic" : 3838 +{{< /mermaid >}} + +#### Service IP CIDR block: 10.96.0.0/16 + +Range Size: 216 - 2 = 65534 +Band Offset: `min(max(16, 65536/16), 256)` = `min(4096, 256)` = 256 +Static band start: 10.96.0.1 +Static band ends: 10.96.1.0 +Range end: 10.96.255.254 + +{{< mermaid >}} +pie showData + title 10.96.0.0/16 + "Static" : 256 + "Dynamic" : 65278 +{{< /mermaid >}} + +## Get involved with SIG Network + +The current SIG-Network [KEPs](https://github.com/orgs/kubernetes/projects/10) and [issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Fnetwork) on GitHub illustrate the SIG’s areas of emphasis. + +[SIG Network meetings](https://github.com/kubernetes/community/tree/master/sig-network) are a friendly, welcoming venue for you to connect with the community and share your ideas. +Looking forward to hearing from you! + diff --git a/content/en/blog/_posts/2022-05-25-contextual-logging/index.md b/content/en/blog/_posts/2022-05-25-contextual-logging/index.md new file mode 100644 index 0000000000..2d5ef5c4c7 --- /dev/null +++ b/content/en/blog/_posts/2022-05-25-contextual-logging/index.md @@ -0,0 +1,251 @@ +--- +layout: blog +title: "Contextual Logging in Kubernetes 1.24" +date: 2022-05-25 +slug: contextual-logging +canonicalUrl: https://kubernetes.dev/blog/2022/05/25/contextual-logging/ +--- + + **Authors:** Patrick Ohly (Intel) + +The [Structured Logging Working +Group](https://github.com/kubernetes/community/blob/master/wg-structured-logging/README.md) +has added new capabilities to the logging infrastructure in Kubernetes +1.24. This blog post explains how developers can take advantage of those to +make log output more useful and how they can get involved with improving Kubernetes. + +## Structured logging + +The goal of [structured +logging](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/1602-structured-logging/README.md) +is to replace C-style formatting and the resulting opaque log strings with log +entries that have a well-defined syntax for storing message and parameters +separately, for example as a JSON struct. + +When using the traditional klog text output format for structured log calls, +strings were originally printed with `\n` escape sequences, except when +embedded inside a struct. For structs, log entries could still span multiple +lines, with no clean way to split the log stream into individual entries: + +``` +I1112 14:06:35.783529 328441 structured_logging.go:51] "using InfoS" longData={Name:long Data:Multiple +lines +with quite a bit +of text. internal:0} +I1112 14:06:35.783549 328441 structured_logging.go:52] "using InfoS with\nthe message across multiple lines" int=1 stringData="long: Multiple\nlines\nwith quite a bit\nof text." str="another value" +``` + +Now, the `<` and `>` markers along with indentation are used to ensure that splitting at a +klog header at the start of a line is reliable and the resulting output is human-readable: + +``` +I1126 10:31:50.378204 121736 structured_logging.go:59] "using InfoS" longData=< + {Name:long Data:Multiple + lines + with quite a bit + of text. internal:0} + > +I1126 10:31:50.378228 121736 structured_logging.go:60] "using InfoS with\nthe message across multiple lines" int=1 stringData=< + long: Multiple + lines + with quite a bit + of text. + > str="another value" +``` + +Note that the log message itself is printed with quoting. It is meant to be a +fixed string that identifies a log entry, so newlines should be avoided there. + +Before Kubernetes 1.24, some log calls in kube-scheduler still used `klog.Info` +for multi-line strings to avoid the unreadable output. Now all log calls have +been updated to support structured logging. + +## Contextual logging + +[Contextual logging](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/3077-contextual-logging/README.md) +is based on the [go-logr API](https://github.com/go-logr/logr#a-minimal-logging-api-for-go). The key +idea is that libraries are passed a logger instance by their caller and use +that for logging instead of accessing a global logger. The binary decides about +the logging implementation, not the libraries. The go-logr API is designed +around structured logging and supports attaching additional information to a +logger. + +This enables additional use cases: + +- The caller can attach additional information to a logger: + - [`WithName`](https://pkg.go.dev/github.com/go-logr/logr#Logger.WithName) adds a prefix + - [`WithValues`](https://pkg.go.dev/github.com/go-logr/logr#Logger.WithValues) adds key/value pairs + + When passing this extended logger into a function and a function uses it + instead of the global logger, the additional information is + then included in all log entries, without having to modify the code that + generates the log entries. This is useful in highly parallel applications + where it can become hard to identify all log entries for a certain operation + because the output from different operations gets interleaved. + +- When running unit tests, log output can be associated with the current test. + Then when a test fails, only the log output of the failed test gets shown + by `go test`. That output can also be more verbose by default because it + will not get shown for successful tests. Tests can be run in parallel + without interleaving their output. + +One of the design decisions for contextual logging was to allow attaching a +logger as value to a `context.Context`. Since the logger encapsulates all +aspects of the intended logging for the call, it is *part* of the context and +not just *using* it. A practical advantage is that many APIs already have a +`ctx` parameter or adding one has additional advantages, like being able to get +rid of `context.TODO()` calls inside the functions. + +Another decision was to not break compatibility with klog v2: + +- Libraries that use the traditional klog logging calls in a binary that has + set up contextual logging will work and log through the logging backend + chosen by the binary. However, such log output will not include the + additional information and will not work well in unit tests, so libraries + should be modified to support contextual logging. The [migration guide](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/migration-to-structured-logging.md) + for structured logging has been extended to also cover contextual logging. + +- When a library supports contextual logging and retrieves a logger from its + context, it will still work in a binary that does not initialize contextual + logging because it will get a logger that logs through klog. + +In Kubernetes 1.24, contextual logging is a new alpha feature with +`ContextualLogging` as feature gate. When disabled (the default), the new klog +API calls for contextual logging (see below) become no-ops to avoid performance +or functional regressions. + +No Kubernetes component has been converted yet. An [example program](https://github.com/kubernetes/kubernetes/blob/v1.24.0-beta.0/staging/src/k8s.io/component-base/logs/example/cmd/logger.go) +in the Kubernetes repository demonstrates how to enable contextual logging in a +binary and how the output depends on the binary's parameters: + +```console +$ cd $GOPATH/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/example/cmd/ +$ go run . --help +... + --feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are: + AllAlpha=true|false (ALPHA - default=false) + AllBeta=true|false (BETA - default=false) + ContextualLogging=true|false (ALPHA - default=false) +$ go run . --feature-gates ContextualLogging=true +... +I0404 18:00:02.916429 451895 logger.go:94] "example/myname: runtime" foo="bar" duration="1m0s" +I0404 18:00:02.916447 451895 logger.go:95] "example: another runtime" foo="bar" duration="1m0s" +``` + +The `example` prefix and `foo="bar"` were added by the caller of the function +which logs the `runtime` message and `duration="1m0s"` value. + +The sample code for klog includes an +[example](https://github.com/kubernetes/klog/blob/v2.60.1/ktesting/example/example_test.go) +for a unit test with per-test output. + +## klog enhancements + +### Contextual logging API + +The following calls manage the lookup of a logger: + +[`FromContext`](https://pkg.go.dev/k8s.io/klog/v2#FromContext) +: from a `context` parameter, with fallback to the global logger + +[`Background`](https://pkg.go.dev/k8s.io/klog/v2#Background) +: the global fallback, with no intention to support contextual logging + +[`TODO`](https://pkg.go.dev/k8s.io/klog/v2#TODO) +: the global fallback, but only as a temporary solution until the function gets extended to accept + a logger through its parameters + +[`SetLoggerWithOptions`](https://pkg.go.dev/k8s.io/klog/v2#SetLoggerWithOptions) +: changes the fallback logger; when called with [`ContextualLogger(true)`](https://pkg.go.dev/k8s.io/klog/v2#ContextualLogger), + the logger is ready to be called directly, in which case logging will be done + without going through klog + +To support the feature gate mechanism in Kubernetes, klog has wrapper calls for +the corresponding go-logr calls and a global boolean controlling their behavior: + +- [`LoggerWithName`](https://pkg.go.dev/k8s.io/klog/v2#LoggerWithName) +- [`LoggerWithValues`](https://pkg.go.dev/k8s.io/klog/v2#LoggerWithValues) +- [`NewContext`](https://pkg.go.dev/k8s.io/klog/v2#NewContext) +- [`EnableContextualLogging`](https://pkg.go.dev/k8s.io/klog/v2#EnableContextualLogging) + +Usage of those functions in Kubernetes code is enforced with a linter +check. The klog default for contextual logging is to enable the functionality +because it is considered stable in klog. It is only in Kubernetes binaries +where that default gets overridden and (in some binaries) controlled via the +`--feature-gate` parameter. + +### ktesting logger + +The new [ktesting](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/ktesting) package +implements logging through `testing.T` using klog's text output format. It has +a [single API call](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/ktesting#NewTestContext) for +instrumenting a test case and [support for command line flags](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/ktesting/init). + +### klogr + +[`klog/klogr`](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/klogr) continues to be +supported and it's default behavior is unchanged: it formats structured log +entries using its own, custom format and prints the result via klog. + +However, this usage is discouraged because that format is neither +machine-readable (in contrast to real JSON output as produced by zapr, the +go-logr implementation used by Kubernetes) nor human-friendly (in contrast to +the klog text format). + +Instead, a klogr instance should be created with +[`WithFormat(FormatKlog)`](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/klogr#WithFormat) +which chooses the klog text format. A simpler construction method with the same +result is the new +[`klog.NewKlogr`](https://pkg.go.dev/k8s.io/klog/v2#NewKlogr). That is the +logger that klog returns as fallback when nothing else is configured. + +### Reusable output test + +A lot of go-logr implementations have very similar unit tests where they check +the result of certain log calls. If a developer didn't know about certain +caveats like for example a `String` function that panics when called, then it +is likely that both the handling of such caveats and the unit test are missing. + +[`klog.test`](https://pkg.go.dev/k8s.io/klog/v2@v2.60.1/test) is a reusable set +of test cases that can be applied to a go-logr implementation. + +### Output flushing + +klog used to start a goroutine unconditionally during `init` which flushed +buffered data at a hard-coded interval. Now that goroutine is only started on +demand (i.e. when writing to files with buffering) and can be controlled with +[`StopFlushDaemon`](https://pkg.go.dev/k8s.io/klog/v2#StopFlushDaemon) and +[`StartFlushDaemon`](https://pkg.go.dev/k8s.io/klog/v2#StartFlushDaemon). + +When a go-logr implementation buffers data, flushing that data can be +integrated into [`klog.Flush`](https://pkg.go.dev/k8s.io/klog/v2#Flush) by +registering the logger with the +[`FlushLogger`](https://pkg.go.dev/k8s.io/klog/v2#FlushLogger) option. + +### Various other changes + +For a description of all other enhancements see in the [release notes](https://github.com/kubernetes/klog/releases). + +## logcheck + +Originally designed as a linter for structured log calls, the + [`logcheck`](https://github.com/kubernetes/klog/tree/788efcdee1e9be0bfbe5b076343d447314f2377e/hack/tools/logcheck) +tool has been enhanced to support also contextual logging and traditional klog +log calls. These enhanced checks already found bugs in Kubernetes, like calling +`klog.Info` instead of `klog.Infof` with a format string and parameters. + +It can be included as a plugin in a `golangci-lint` invocation, which is how +[Kubernetes uses it now](https://github.com/kubernetes/kubernetes/commit/17e3c555c5115f8c9176bae10ba45baa04d23a7b), +or get invoked stand-alone. + +We are in the process of [moving the tool](https://github.com/kubernetes/klog/issues/312) into a new repository because it isn't +really related to klog and its releases should be tracked and tagged properly. + +## Next steps + +The [Structured Logging WG](https://github.com/kubernetes/community/tree/master/wg-structured-logging) +is always looking for new contributors. The migration +away from C-style logging is now going to target structured, contextual logging +in one step to reduce the overall code churn and number of PRs. Changing log +calls is good first contribution to Kubernetes and an opportunity to get to +know code in various different areas. diff --git a/content/en/blog/_posts/2022-05-27-maxunavailable-for-statefulset.md b/content/en/blog/_posts/2022-05-27-maxunavailable-for-statefulset.md new file mode 100644 index 0000000000..aa6257eb3e --- /dev/null +++ b/content/en/blog/_posts/2022-05-27-maxunavailable-for-statefulset.md @@ -0,0 +1,148 @@ +--- +layout: blog +title: 'Kubernetes 1.24: Maximum Unavailable Replicas for StatefulSet' +date: 2022-05-27 +slug: maxunavailable-for-statefulset +--- + +**Author:** Mayank Kumar (Salesforce) + +Kubernetes [StatefulSets](/docs/concepts/workloads/controllers/statefulset/), since their introduction in +1.5 and becoming stable in 1.9, have been widely used to run stateful applications. They provide stable pod identity, persistent +per pod storage and ordered graceful deployment, scaling and rolling updates. You can think of StatefulSet as the atomic building +block for running complex stateful applications. As the use of Kubernetes has grown, so has the number of scenarios requiring +StatefulSets. Many of these scenarios, require faster rolling updates than the currently supported one-pod-at-a-time updates, in the +case where you're using the `OrderedReady` Pod management policy for a StatefulSet. + + +Here are some examples: + +- I am using a StatefulSet to orchestrate a multi-instance, cache based application where the size of the cache is large. The cache + starts cold and requires some siginificant amount of time before the container can start. There could be more initial startup tasks + that are required. A RollingUpdate on this StatefulSet would take a lot of time before the application is fully updated. If the + StatefulSet supported updating more than one pod at a time, it would result in a much faster update. + +- My stateful application is composed of leaders and followers or one writer and multiple readers. I have multiple readers or + followers and my application can tolerate multiple pods going down at the same time. I want to update this application more than + one pod at a time so that i get the new updates rolled out quickly, especially if the number of instances of my application are + large. Note that my application still requires unique identity per pod. + + +In order to support such scenarios, Kubernetes 1.24 includes a new alpha feature to help. Before you can use the new feature you must +enable the `MaxUnavailableStatefulSet` feature flag. Once you enable that, you can specify a new field called `maxUnavailable`, part +of the `spec` for a StatefulSet. For example: + +``` +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: web + namespace: default +spec: + podManagementPolicy: OrderedReady # you must set OrderedReady + replicas: 5 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - image: k8s.gcr.io/nginx-slim:0.8 + imagePullPolicy: IfNotPresent + name: nginx + updateStrategy: + rollingUpdate: + maxUnavailable: 2 # this is the new alpha field, whose default value is 1 + partition: 0 + type: RollingUpdate +``` + +If you enable the new feature and you don't specify a value for `maxUnavailable` in a StatefulSet, Kubernetes applies a default +`maxUnavailable: 1`. This matches the behavior you would see if you don't enable the new feature. + +I'll run through a scenario based on that example manifest to demonstrate how this feature works. I will deploy a StatefulSet that +has 5 replicas, with `maxUnavailable` set to 2 and `partition` set to 0. + +I can trigger a rolling update by changing the image to `k8s.gcr.io/nginx-slim:0.9`. Once I initiate the rolling update, I can +watch the pods update 2 at a time as the current value of maxUnavailable is 2. The below output shows a span of time and is not +complete. The maxUnavailable can be an absolute number (for example, 2) or a percentage of desired Pods (for example, 10%). The +absolute number is calculated from percentage by rounding down. +``` +kubectl get pods --watch +``` + +``` +NAME READY STATUS RESTARTS AGE +web-0 1/1 Running 0 85s +web-1 1/1 Running 0 2m6s +web-2 1/1 Running 0 106s +web-3 1/1 Running 0 2m47s +web-4 1/1 Running 0 2m27s +web-4 1/1 Terminating 0 5m43s ----> start terminating 4 +web-3 1/1 Terminating 0 6m3s ----> start terminating 3 +web-3 0/1 Terminating 0 6m7s +web-3 0/1 Pending 0 0s +web-3 0/1 Pending 0 0s +web-4 0/1 Terminating 0 5m48s +web-4 0/1 Terminating 0 5m48s +web-3 0/1 ContainerCreating 0 2s +web-3 1/1 Running 0 2s +web-4 0/1 Pending 0 0s +web-4 0/1 Pending 0 0s +web-4 0/1 ContainerCreating 0 0s +web-4 1/1 Running 0 1s +web-2 1/1 Terminating 0 5m46s ----> start terminating 2 (only after both 4 and 3 are running) +web-1 1/1 Terminating 0 6m6s ----> start terminating 1 +web-2 0/1 Terminating 0 5m47s +web-1 0/1 Terminating 0 6m7s +web-1 0/1 Pending 0 0s +web-1 0/1 Pending 0 0s +web-1 0/1 ContainerCreating 0 1s +web-1 1/1 Running 0 2s +web-2 0/1 Pending 0 0s +web-2 0/1 Pending 0 0s +web-2 0/1 ContainerCreating 0 0s +web-2 1/1 Running 0 1s +web-0 1/1 Terminating 0 6m6s ----> start terminating 0 (only after 2 and 1 are running) +web-0 0/1 Terminating 0 6m7s +web-0 0/1 Pending 0 0s +web-0 0/1 Pending 0 0s +web-0 0/1 ContainerCreating 0 0s +web-0 1/1 Running 0 1s +``` +Note that as soon as the rolling update starts, both 4 and 3 (the two highest ordinal pods) start terminating at the same time. Pods +with ordinal 4 and 3 may become ready at their own pace. As soon as both pods 4 and 3 are ready, pods 2 and 1 start terminating at the +same time. When pods 2 and 1 are both running and ready, pod 0 starts terminating. + +In Kubernetes, updates to StatefulSets follow a strict ordering when updating Pods. In this example, the update starts at replica 4, then +replica 3, then replica 2, and so on, one pod at a time. When going one pod at a time, its not possible for 3 to be running and ready +before 4. When `maxUnavailable` is more than 1 (in the example scenario I set `maxUnavailable` to 2), it is possible that replica 3 becomes +ready and running before replica 4 is ready—and that is ok. If you're a developer and you set `maxUnavailable` to more than 1, you should +know that this outcome is possible and you must ensure that your application is able to handle such ordering issues that occur +if any. When you set `maxUnavailable` greater than 1, the ordering is guaranteed in between each batch of pods being updated. That guarantee +means that pods in update batch 2 (replicas 2 and 1) cannot start updating until the pods from batch 0 (replicas 4 and 3) are ready. + +Although Kubernetes refers to these as _replicas_, your stateful application may have a different view and each pod of the StatefulSet may +be holding completely different data than other pods. The important thing here is that updates to StatefulSets happen in batches, and you can +now have a batch size larger than 1 (as an alpha feature). + +Also note, that the above behavior is with `podManagementPolicy: OrderedReady`. If you defined a StatefulSet as `podManagementPolicy: Parallel`, +not only `maxUnavailable` number of replicas are terminated at the same time; `maxUnavailable` number of replicas start in `ContainerCreating` +phase at the same time as well. This is called bursting. + +So, now you may have a lot of questions about:- +- What is the behavior when you set `podManagementPolicy: Parallel`? +- What is the behavior when `partition` to a value other than `0`? + +It might be better to try and see it for yourself. This is an alpha feature, and the Kubernetes contributors are looking for feedback on this feature. Did +this help you achieve your stateful scenarios Did you find a bug or do you think the behavior as implemented is not intuitive or can +break applications or catch them by surprise? Please [open an issue](https://github.com/kubernetes/kubernetes/issues) to let us know. + +## Further reading and next steps {#next-steps} +- [Maximum unavailable Pods](/docs/concepts/workloads/controllers/statefulset/#maximum-unavailable-pods) +- [KEP for MaxUnavailable for StatefulSet](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/961-maxunavailable-for-statefulset) +- [Implementation](https://github.com/kubernetes/kubernetes/pull/82162/files) +- [Enhancement Tracking Issue](https://github.com/kubernetes/enhancements/issues/961) diff --git a/content/en/blog/_posts/2022-06-01-annual-report-2021.md b/content/en/blog/_posts/2022-06-01-annual-report-2021.md new file mode 100644 index 0000000000..e0a4130357 --- /dev/null +++ b/content/en/blog/_posts/2022-06-01-annual-report-2021.md @@ -0,0 +1,19 @@ +--- +layout: blog +title: "Annual Report Summary 2021" +date: 2022-06-01 +slug: annual-report-summary-2021 +--- + +**Author:** Paris Pittman (Steering Committee) + +Last year, we published our first [Annual Report Summary](/blog/2021/06/28/announcing-kubernetes-community-group-annual-reports/) for 2020 and it's already time for our second edition! + +[2021 Annual Report Summary](https://www.cncf.io/reports/kubernetes-annual-report-2021/) + +This summary reflects the work that has been done in 2021 and the initiatives on deck for the rest of 2022. Please forward to organizations and indidviduals participating in upstream activities, planning cloud native strategies, and/or those looking to help out. To find a specific community group's complete report, go to the [kubernetes/community repo](https://github.com/kubernetes/community) under the groups folder. Example: [sig-api-machinery/annual-report-2021.md](https://github.com/kubernetes/community/blob/master/sig-api-machinery/annual-report-2021.md) + +You’ll see that this report summary is a growth area in itself. It takes us roughly 6 months to prepare and execute, which isn’t helpful or valuable to anyone as a fast moving project with short and long term needs. How can we make this better? Provide your feedback here: https://github.com/kubernetes/steering/issues/242 + +Reference: +[Annual Report Documentation](https://github.com/kubernetes/community/blob/master/committee-steering/governance/annual-reports.md) diff --git a/content/en/community/code-of-conduct.md b/content/en/community/code-of-conduct.md index a66b0572bf..84c95370a3 100644 --- a/content/en/community/code-of-conduct.md +++ b/content/en/community/code-of-conduct.md @@ -8,9 +8,9 @@ community_styles_migrated: true

Kubernetes follows the -CNCF Code of Conduct. +CNCF Code of Conduct. The text of the CNCF CoC is replicated below, as of -commit 214585e. +commit 71b12a2. If you notice that this is out of date, please file an issue.

diff --git a/content/en/community/static/cncf-code-of-conduct.md b/content/en/community/static/cncf-code-of-conduct.md index d07444c418..fb3202b24a 100644 --- a/content/en/community/static/cncf-code-of-conduct.md +++ b/content/en/community/static/cncf-code-of-conduct.md @@ -1,45 +1,72 @@ -## CNCF Community Code of Conduct v1.0 + https://github.com/cncf/foundation/blob/main/code-of-conduct.md --> +## CNCF Community Code of Conduct v1.1 ### Contributor Code of Conduct -As contributors and maintainers of this project, and in the interest of fostering +As contributors and maintainers in the CNCF community, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities. -We are committed to making participation in this project a harassment-free experience for -everyone, regardless of level of experience, gender, gender identity and expression, +We are committed to making participation in the CNCF community a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality. -Examples of unacceptable behavior by participants include: +## Scope -* The use of sexualized language or imagery -* Personal attacks -* Trolling or insulting/derogatory comments +This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. + +### CNCF Events + +CNCF events, or events run by the Linux Foundation with professional events staff, are governed by the Linux Foundation [Events Code of Conduct](https://events.linuxfoundation.org/code-of-conduct/) available on the event page. This is designed to be used in conjunction with the CNCF Code of Conduct. + +## Our Standards + +Examples of behavior that contributes to a positive environment include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment -* Publishing other's private information, such as physical or electronic addresses, - without explicit permission -* Other unethical or unprofessional conduct. +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting -Project maintainers have the right and responsibility to remove, edit, or reject -comments, commits, code, wiki edits, issues, and other contributions that are not -aligned to this Code of Conduct. By adopting this Code of Conduct, project maintainers -commit themselves to fairly and consistently applying these principles to every aspect -of managing this project. Project maintainers who do not follow or enforce the Code of +Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. +By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect +of managing this project. +Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team. -This code of conduct applies both within project spaces and in public spaces -when an individual is representing the project or its community. +## Reporting -Instances of abusive, harassing, or otherwise unacceptable behavior in Kubernetes may be reported by contacting the [Kubernetes Code of Conduct Committee](https://git.k8s.io/community/committee-code-of-conduct) via . For other projects, please contact a CNCF project maintainer or our mediator, Mishi Choudhary . +For incidents occurring in the Kubernetes community, contact the [Kubernetes Code of Conduct Committee](https://git.k8s.io/community/committee-code-of-conduct) via . You can expect a response within three business days. + +For other projects, please contact the CNCF staff via . You can expect a response within three business days. + +In matters that require an outside mediator, CNCF has retained Mishi Choudhary (mishi@linux.com). Use of an outside mediator can be requested when reporting or used at CNCF staff's discretion. In general, contacting directly is preferred. + + +## Enforcement + +The Kubernetes project's [Code of Conduct Committee](https://github.com/kubernetes/community/tree/master/committee-code-of-conduct) enforces code of conduct issues. For all other projects, the CNCF enforces code of conduct issues. + +Both bodies try to resolve incidents without punishment, but may remove people from the project or CNCF communities at their discretion. + +## Acknowledgements This Code of Conduct is adapted from the Contributor Covenant -(https://contributor-covenant.org), version 1.2.0, available at -https://contributor-covenant.org/version/1/2/0/ - -### CNCF Events Code of Conduct - -CNCF events are governed by the Linux Foundation [Code of Conduct](https://events.linuxfoundation.org/code-of-conduct/) available on the event page. This is designed to be compatible with the above policy and also includes more details on responding to incidents. +(http://contributor-covenant.org), version 2.0 available at +http://contributor-covenant.org/version/2/0/code_of_conduct/ \ No newline at end of file diff --git a/content/en/community/static/community-values.md b/content/en/community/static/community-values.md index f6469a3e61..6fd1a1a06c 100644 --- a/content/en/community/static/community-values.md +++ b/content/en/community/static/community-values.md @@ -3,26 +3,26 @@ # Kubernetes Community Values -Kubernetes Community culture is frequently cited as a substantial contributor to the meteoric rise of this Open Source project. Below are the distilled values which have evolved over the last many years in our community pushing our project and peers toward constant improvement. +Kubernetes Community culture contributes substantially to the project's success. The following values have evolved over time, pushing our project and peers toward constant improvement. ## Distribution is better than centralization -The scale of the Kubernetes project is only viable through high-trust and high-visibility distribution of work, which includes delegation of authority, decision making, technical design, code ownership, and documentation. Distributed asynchronous ownership, collaboration, communication and decision making are the cornerstone of our world-wide community. +The scale of the Kubernetes project is only viable through high-trust and high-visibility distribution of work, which includes delegation of authority, decision making, technical design, code ownership, and documentation. Distributed asynchronous ownership, collaboration, communication and decision making are the cornerstones of our world-wide community. ## Community over product or company -We are here as a community first, our allegiance is to the intentional stewardship of the Kubernetes project for the benefit of all its members and users everywhere. We support working together publicly for the common goal of a vibrant interoperable ecosystem providing an excellent experience for our users. Individuals gain status through work, companies gain status through their commitments to support this community and fund the resources necessary for the project to operate. +We are here as a community first. Our allegiance is to the intentional stewardship of the Kubernetes project for the benefit of all its members and users everywhere. We support working together publicly for the common goal of a vibrant interoperable ecosystem, providing an excellent experience for our users. Individuals gain status through work. Companies gain status through their commitments to support this community and fund the resources necessary for the project to operate. ## Automation over process -Large projects have a lot of less exciting, yet, hard work. We value time spent automating repetitive work more highly than toil. Where that work cannot be automated, it is our culture to recognize and reward all types of contributions. However, heroism is not sustainable. +Large projects have a lot of hard yet less exciting work. We value time spent automating repetitive work more highly than toil. Where work cannot be automated, our culture recognizes and rewards all types of contributions while recognizing that heroism is not sustainable. ## Inclusive is better than exclusive -Broadly successful and useful technology requires different perspectives and skill sets which can only be heard in a welcoming and respectful environment. Community membership is a privilege, not a right. Community Leadership is earned through effort, scope, quality, quantity, and duration of contributions. Our community shows respect for the time and effort put into a discussion regardless of where a contributor is on their growth path. +Broadly successful and useful technologies require different perspectives and skill sets, which can only be heard in a welcoming and respectful environment. Community membership is a privilege, not a right. Community members earn leadership through effort, scope, quality, quantity, and duration of contributions. Our community respects the time and effort put into a discussion, regardless of where a contributor is on their growth path. ## Evolution is better than stagnation -Openness to new ideas and studied technological evolution make Kubernetes a stronger project. Continual improvement, servant leadership, mentorship and respect are the foundations of the Kubernetes project culture. It is the duty for leaders in the Kubernetes community to find, sponsor, and promote new community members. Leaders should expect to step aside. Community members should expect to step up. +Openness to new ideas and studied technological evolution make Kubernetes a stronger project. Continual improvement, servant leadership, mentorship, and respect are the foundations of Kubernetes culture. Kubernetes community leaders have a duty to find, sponsor, and promote new community members. Leaders should expect to step aside. Community members should expect to step up. **"Culture eats strategy for breakfast." --Peter Drucker** diff --git a/content/en/community/values.md b/content/en/community/values.md index 675e93c865..2974dc1e43 100644 --- a/content/en/community/values.md +++ b/content/en/community/values.md @@ -9,7 +9,15 @@ community_styles_migrated: true sitemap: priority: 0.1 --- +
+

+This page is a replicated version of +Kubernetes Community Values, as of +commit 5c64274. +If you notice that this is out of date, please +file an issue. +

{{< include "/static/community-values.md" >}}
diff --git a/content/en/docs/concepts/architecture/control-plane-node-communication.md b/content/en/docs/concepts/architecture/control-plane-node-communication.md index a4814aab4b..df384800e9 100644 --- a/content/en/docs/concepts/architecture/control-plane-node-communication.md +++ b/content/en/docs/concepts/architecture/control-plane-node-communication.md @@ -2,7 +2,7 @@ reviewers: - dchen1107 - liggitt -title: Control Plane-Node Communication +title: Communication between Nodes and the Control Plane content_type: concept weight: 20 aliases: @@ -11,62 +11,109 @@ aliases: -This document catalogs the communication paths between the control plane (apiserver) and the Kubernetes cluster. The intent is to allow users to customize their installation to harden the network configuration such that the cluster can be run on an untrusted network (or on fully public IPs on a cloud provider). - - +This document catalogs the communication paths between the API server and the Kubernetes cluster. +The intent is to allow users to customize their installation to harden the network configuration +such that the cluster can be run on an untrusted network (or on fully public IPs on a cloud +provider). ## Node to Control Plane -Kubernetes has a "hub-and-spoke" API pattern. All API usage from nodes (or the pods they run) terminates at the apiserver. None of the other control plane components are designed to expose remote services. The apiserver is configured to listen for remote connections on a secure HTTPS port (typically 443) with one or more forms of client [authentication](/docs/reference/access-authn-authz/authentication/) enabled. -One or more forms of [authorization](/docs/reference/access-authn-authz/authorization/) should be enabled, especially if [anonymous requests](/docs/reference/access-authn-authz/authentication/#anonymous-requests) or [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens) are allowed. -Nodes should be provisioned with the public root certificate for the cluster such that they can connect securely to the apiserver along with valid client credentials. A good approach is that the client credentials provided to the kubelet are in the form of a client certificate. See [kubelet TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) for automated provisioning of kubelet client certificates. +Kubernetes has a "hub-and-spoke" API pattern. All API usage from nodes (or the pods they run) +terminates at the API server. None of the other control plane components are designed to expose +remote services. The API server is configured to listen for remote connections on a secure HTTPS +port (typically 443) with one or more forms of client +[authentication](/docs/reference/access-authn-authz/authentication/) enabled. +One or more forms of [authorization](/docs/reference/access-authn-authz/authorization/) should be +enabled, especially if [anonymous requests](/docs/reference/access-authn-authz/authentication/#anonymous-requests) +or [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens) +are allowed. -Pods that wish to connect to the apiserver can do so securely by leveraging a service account so that Kubernetes will automatically inject the public root certificate and a valid bearer token into the pod when it is instantiated. -The `kubernetes` service (in `default` namespace) is configured with a virtual IP address that is redirected (via kube-proxy) to the HTTPS endpoint on the apiserver. +Nodes should be provisioned with the public root certificate for the cluster such that they can +connect securely to the API server along with valid client credentials. A good approach is that the +client credentials provided to the kubelet are in the form of a client certificate. See +[kubelet TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) +for automated provisioning of kubelet client certificates. -The control plane components also communicate with the cluster apiserver over the secure port. +Pods that wish to connect to the API server can do so securely by leveraging a service account so +that Kubernetes will automatically inject the public root certificate and a valid bearer token +into the pod when it is instantiated. +The `kubernetes` service (in `default` namespace) is configured with a virtual IP address that is +redirected (via `kube-proxy`) to the HTTPS endpoint on the API server. -As a result, the default operating mode for connections from the nodes and pods running on the nodes to the control plane is secured by default and can run over untrusted and/or public networks. +The control plane components also communicate with the API server over the secure port. -## Control Plane to node +As a result, the default operating mode for connections from the nodes and pods running on the +nodes to the control plane is secured by default and can run over untrusted and/or public +networks. -There are two primary communication paths from the control plane (apiserver) to the nodes. The first is from the apiserver to the kubelet process which runs on each node in the cluster. The second is from the apiserver to any node, pod, or service through the apiserver's proxy functionality. +## Control plane to node -### apiserver to kubelet +There are two primary communication paths from the control plane (the API server) to the nodes. +The first is from the API server to the kubelet process which runs on each node in the cluster. +The second is from the API server to any node, pod, or service through the API server's _proxy_ +functionality. -The connections from the apiserver to the kubelet are used for: +### API server to kubelet + +The connections from the API server to the kubelet are used for: * Fetching logs for pods. -* Attaching (through kubectl) to running pods. +* Attaching (usually through `kubectl`) to running pods. * Providing the kubelet's port-forwarding functionality. -These connections terminate at the kubelet's HTTPS endpoint. By default, the apiserver does not verify the kubelet's serving certificate, which makes the connection subject to man-in-the-middle attacks and **unsafe** to run over untrusted and/or public networks. +These connections terminate at the kubelet's HTTPS endpoint. By default, the API server does not +verify the kubelet's serving certificate, which makes the connection subject to man-in-the-middle +attacks and **unsafe** to run over untrusted and/or public networks. -To verify this connection, use the `--kubelet-certificate-authority` flag to provide the apiserver with a root certificate bundle to use to verify the kubelet's serving certificate. +To verify this connection, use the `--kubelet-certificate-authority` flag to provide the API +server with a root certificate bundle to use to verify the kubelet's serving certificate. -If that is not possible, use [SSH tunneling](#ssh-tunnels) between the apiserver and kubelet if required to avoid connecting over an +If that is not possible, use [SSH tunneling](#ssh-tunnels) between the API server and kubelet if +required to avoid connecting over an untrusted or public network. -Finally, [Kubelet authentication and/or authorization](/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/) should be enabled to secure the kubelet API. -### apiserver to nodes, pods, and services +Finally, [Kubelet authentication and/or authorization](/docs/reference/access-authn-authz/kubelet-authn-authz/) +should be enabled to secure the kubelet API. -The connections from the apiserver to a node, pod, or service default to plain HTTP connections and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS connection by prefixing `https:` to the node, pod, or service name in the API URL, but they will not validate the certificate provided by the HTTPS endpoint nor provide client credentials. So while the connection will be encrypted, it will not provide any guarantees of integrity. These connections **are not currently safe** to run over untrusted or public networks. +### API server to nodes, pods, and services + +The connections from the API server to a node, pod, or service default to plain HTTP connections +and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS +connection by prefixing `https:` to the node, pod, or service name in the API URL, but they will +not validate the certificate provided by the HTTPS endpoint nor provide client credentials. So +while the connection will be encrypted, it will not provide any guarantees of integrity. These +connections **are not currently safe** to run over untrusted or public networks. ### SSH tunnels -Kubernetes supports SSH tunnels to protect the control plane to nodes communication paths. In this configuration, the apiserver initiates an SSH tunnel to each node in the cluster (connecting to the ssh server listening on port 22) and passes all traffic destined for a kubelet, node, pod, or service through the tunnel. -This tunnel ensures that the traffic is not exposed outside of the network in which the nodes are running. +Kubernetes supports SSH tunnels to protect the control plane to nodes communication paths. In this +configuration, the API server initiates an SSH tunnel to each node in the cluster (connecting to +the SSH server listening on port 22) and passes all traffic destined for a kubelet, node, pod, or +service through the tunnel. +This tunnel ensures that the traffic is not exposed outside of the network in which the nodes are +running. -SSH tunnels are currently deprecated, so you shouldn't opt to use them unless you know what you are doing. The Konnectivity service is a replacement for this communication channel. +{{< note >}} +SSH tunnels are currently deprecated, so you shouldn't opt to use them unless you know what you +are doing. The [Konnectivity service](#konnectivity-service) is a replacement for this +communication channel. +{{< /note >}} ### Konnectivity service {{< feature-state for_k8s_version="v1.18" state="beta" >}} -As a replacement to the SSH tunnels, the Konnectivity service provides TCP level proxy for the control plane to cluster communication. The Konnectivity service consists of two parts: the Konnectivity server in the control plane network and the Konnectivity agents in the nodes network. The Konnectivity agents initiate connections to the Konnectivity server and maintain the network connections. -After enabling the Konnectivity service, all control plane to nodes traffic goes through these connections. +As a replacement to the SSH tunnels, the Konnectivity service provides TCP level proxy for the +control plane to cluster communication. The Konnectivity service consists of two parts: the +Konnectivity server in the control plane network and the Konnectivity agents in the nodes network. +The Konnectivity agents initiate connections to the Konnectivity server and maintain the network +connections. +After enabling the Konnectivity service, all control plane to nodes traffic goes through these +connections. + +Follow the [Konnectivity service task](/docs/tasks/extend-kubernetes/setup-konnectivity/) to set +up the Konnectivity service in your cluster. -Follow the [Konnectivity service task](/docs/tasks/extend-kubernetes/setup-konnectivity/) to set up the Konnectivity service in your cluster. diff --git a/content/en/docs/concepts/architecture/nodes.md b/content/en/docs/concepts/architecture/nodes.md index 39d229a897..2321fc6474 100644 --- a/content/en/docs/concepts/architecture/nodes.md +++ b/content/en/docs/concepts/architecture/nodes.md @@ -458,7 +458,7 @@ Message: Pod was terminated in response to imminent node shutdown. {{< feature-state state="alpha" for_k8s_version="v1.24" >}} -A node shutdown action may not be detected by kubelet's Node Shutdown Mananger, +A node shutdown action may not be detected by kubelet's Node Shutdown Manager, either because the command does not trigger the inhibitor locks mechanism used by kubelet or because of a user error, i.e., the ShutdownGracePeriod and ShutdownGracePeriodCriticalPods are not configured properly. Please refer to above @@ -654,7 +654,7 @@ see [KEP-2400](https://github.com/kubernetes/enhancements/issues/2400) and its * Learn about the [components](/docs/concepts/overview/components/#node-components) that make up a node. * Read the [API definition for Node](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#node-v1-core). -* Read the [Node](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md#the-kubernetes-node) +* Read the [Node](https://git.k8s.io/design-proposals-archive/architecture/architecture.md#the-kubernetes-node) section of the architecture design document. * Read about [taints and tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/). diff --git a/content/en/docs/concepts/cluster-administration/_index.md b/content/en/docs/concepts/cluster-administration/_index.md index 7e5827a6f7..ace5297b33 100644 --- a/content/en/docs/concepts/cluster-administration/_index.md +++ b/content/en/docs/concepts/cluster-administration/_index.md @@ -63,8 +63,8 @@ Before choosing a guide, here are some considerations: ### Securing the kubelet * [Control Plane-Node communication](/docs/concepts/architecture/control-plane-node-communication/) - * [TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) - * [Kubelet authentication/authorization](/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/) + * [TLS bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) + * [Kubelet authentication/authorization](/docs/reference/acess-authn-authz/kubelet-authn-authz/) ## Optional Cluster Services diff --git a/content/en/docs/concepts/cluster-administration/addons.md b/content/en/docs/concepts/cluster-administration/addons.md index 20626f2ff4..5f7df077c9 100644 --- a/content/en/docs/concepts/cluster-administration/addons.md +++ b/content/en/docs/concepts/cluster-administration/addons.md @@ -18,19 +18,19 @@ This page lists some of the available add-ons and links to their respective inst * [ACI](https://www.github.com/noironetworks/aci-containers) provides integrated container networking and network security with Cisco ACI. * [Antrea](https://antrea.io/) operates at Layer 3/4 to provide networking and security services for Kubernetes, leveraging Open vSwitch as the networking data plane. * [Calico](https://docs.projectcalico.org/latest/introduction/) is a networking and network policy provider. Calico supports a flexible set of networking options so you can choose the most efficient option for your situation, including non-overlay and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts, pods, and (if using Istio & Envoy) applications at the service mesh layer. -* [Canal](https://github.com/tigera/canal/tree/master/k8s-install) unites Flannel and Calico, providing networking and network policy. +* [Canal](https://projectcalico.docs.tigera.io/getting-started/kubernetes/flannel/flannel) unites Flannel and Calico, providing networking and network policy. * [Cilium](https://github.com/cilium/cilium) is a L3 network and network policy plugin that can enforce HTTP/API/L7 policies transparently. Both routing and overlay/encapsulation mode are supported, and it can work on top of other CNI plugins. -* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, Romana, or Weave. +* [CNI-Genie](https://github.com/cni-genie/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, or Weave. * [Contiv](https://contivpp.io/) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](https://github.com/contiv). The [installer](https://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options. * [Contrail](https://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is an open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads. * [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) is an overlay network provider that can be used with Kubernetes. * [Knitter](https://github.com/ZTE/Knitter/) is a plugin to support multiple network interfaces in a Kubernetes pod. -* Multus is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes. +* [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes. * [OVN-Kubernetes](https://github.com/ovn-org/ovn-kubernetes/) is a networking provider for Kubernetes based on [OVN (Open Virtual Network)](https://github.com/ovn-org/ovn/), a virtual networking implementation that came out of the Open vSwitch (OVS) project. OVN-Kubernetes provides an overlay based networking implementation for Kubernetes, including an OVS based implementation of load balancing and network policy. -* [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) is OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC), Multiple OVN overlay networking, dynamic subnet creation, dynamic creation of virtual networks, VLAN Provider network, Direct provider network and pluggable with other Multi-network plugins, ideal for edge based cloud native workloads in Multi-cluster networking -* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift. +* [OVN4NFV-K8S-Plugin](https://github.com/opnfv/ovn4nfv-k8s-plugin) is OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC), Multiple OVN overlay networking, dynamic subnet creation, dynamic creation of virtual networks, VLAN Provider network, Direct provider network and pluggable with other Multi-network plugins, ideal for edge based cloud native workloads in Multi-cluster networking. +* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/index.html) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift. * [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) is an SDN platform that provides policy-based networking between Kubernetes Pods and non-Kubernetes environments with visibility and security monitoring. -* **Romana** is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize). +* [Romana](https://github.com/romana) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy](/docs/concepts/services-networking/network-policies/) API. * [Weave Net](https://www.weave.works/docs/net/latest/kubernetes/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database. ## Service Discovery diff --git a/content/en/docs/concepts/cluster-administration/flow-control.md b/content/en/docs/concepts/cluster-administration/flow-control.md index 9e8f2a7923..40af92cd5b 100644 --- a/content/en/docs/concepts/cluster-administration/flow-control.md +++ b/content/en/docs/concepts/cluster-administration/flow-control.md @@ -174,7 +174,7 @@ to balance progress between request flows. The queuing configuration allows tuning the fair queuing algorithm for a priority level. Details of the algorithm can be read in the -[enhancement proposal](#whats-next), but in short: +[enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness), but in short: * Increasing `queues` reduces the rate of collisions between different flows, at the cost of increased memory usage. A value of 1 here effectively disables the @@ -331,7 +331,7 @@ Thus, in a situation with a mixture of servers of different versions there may be thrashing as long as different servers have different opinions of the proper content of these objects. -Each `kube-apiserver` makes an inital maintenance pass over the +Each `kube-apiserver` makes an initial maintenance pass over the mandatory and suggested configuration objects, and after that does periodic maintenance (once per minute) of those objects. diff --git a/content/en/docs/concepts/cluster-administration/manage-deployment.md b/content/en/docs/concepts/cluster-administration/manage-deployment.md index c09e59f1df..c90715da09 100644 --- a/content/en/docs/concepts/cluster-administration/manage-deployment.md +++ b/content/en/docs/concepts/cluster-administration/manage-deployment.md @@ -461,7 +461,7 @@ That's it! The Deployment will declaratively update the deployed nginx applicati ## {{% heading "whatsnext" %}} -- Learn about [how to use `kubectl` for application introspection and debugging](/docs/tasks/debug-application-cluster/debug-application-introspection/). +- Learn about [how to use `kubectl` for application introspection and debugging](/docs/tasks/debug/debug-application/debug-running-pod/). - See [Configuration Best Practices and Tips](/docs/concepts/configuration/overview/). diff --git a/content/en/docs/concepts/cluster-administration/networking.md b/content/en/docs/concepts/cluster-administration/networking.md index 9fed36c2fd..b780ef15ca 100644 --- a/content/en/docs/concepts/cluster-administration/networking.md +++ b/content/en/docs/concepts/cluster-administration/networking.md @@ -203,4 +203,4 @@ to run, and in both cases, the network provides one IP address per pod - as is s The early design of the networking model and its rationale, and some future plans are described in more detail in the -[networking design document](https://git.k8s.io/community/contributors/design-proposals/network/networking.md). +[networking design document](https://git.k8s.io/design-proposals-archive/network/networking.md). diff --git a/content/en/docs/concepts/configuration/configmap.md b/content/en/docs/concepts/configuration/configmap.md index eb48d8e501..4f5ace4348 100644 --- a/content/en/docs/concepts/configuration/configmap.md +++ b/content/en/docs/concepts/configuration/configmap.md @@ -279,5 +279,6 @@ to the deleted ConfigMap, it is recommended to recreate these pods. * Read about [Secrets](/docs/concepts/configuration/secret/). * Read [Configure a Pod to Use a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/). +* Read about [changing a ConfigMap (or any other Kubernetes object)](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/) * Read [The Twelve-Factor App](https://12factor.net/) to understand the motivation for separating code from configuration. diff --git a/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md b/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md index 713592cf98..87c4737ad7 100644 --- a/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md +++ b/content/en/docs/concepts/configuration/organize-cluster-access-kubeconfig.md @@ -18,7 +18,7 @@ It does not mean that there is a file named `kubeconfig`. {{< /note >}} {{< warning >}} -Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. +Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. If you must use an untrusted kubeconfig file, inspect it carefully first, much as you would a shell script. {{< /warning>}} @@ -53,7 +53,7 @@ clusters and namespaces. A *context* element in a kubeconfig file is used to group access parameters under a convenient name. Each context has three parameters: cluster, namespace, and user. By default, the `kubectl` command-line tool uses parameters from -the *current context* to communicate with the cluster. +the *current context* to communicate with the cluster. To choose the current context: ``` @@ -150,16 +150,16 @@ are stored absolutely. ## Proxy -You can configure `kubectl` to use proxy by setting `proxy-url` in the kubeconfig file, like: +You can configure `kubectl` to use a proxy per cluster using `proxy-url` in your kubeconfig file, like this: ```yaml apiVersion: v1 kind: Config -proxy-url: https://proxy.host:3128 - clusters: - cluster: + proxy-url: http://proxy.example.org:3128 + server: https://k8s.example.org/k8s/clusters/c-xxyyzz name: development users: @@ -168,7 +168,6 @@ users: contexts: - context: name: development - ``` diff --git a/content/en/docs/concepts/configuration/secret.md b/content/en/docs/concepts/configuration/secret.md index d9611439a4..f83372532f 100644 --- a/content/en/docs/concepts/configuration/secret.md +++ b/content/en/docs/concepts/configuration/secret.md @@ -247,6 +247,8 @@ You can still [manually create](/docs/tasks/configure-pod-container/configure-se a service account token Secret; for example, if you need a token that never expires. However, using the [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) subresource to obtain a token to access the API is recommended instead. +You can use the [`kubectl create token`](/docs/reference/generated/kubectl/kubectl-commands#-em-token-em-) +command to obtain a token from the `TokenRequest` API. {{< /note >}} #### Projection of Secret keys to specific paths @@ -886,15 +888,30 @@ In this case, `0` means you have created an empty Secret. ### Service account token Secrets A `kubernetes.io/service-account-token` type of Secret is used to store a -token that identifies a +token credential that identifies a {{< glossary_tooltip text="service account" term_id="service-account" >}}. + +Since 1.22, this type of Secret is no longer used to mount credentials into Pods, +and obtaining tokens via the [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) +API is recommended instead of using service account token Secret objects. +Tokens obtained from the `TokenRequest` API are more secure than ones stored in Secret objects, +because they have a bounded lifetime and are not readable by other API clients. +You can use the [`kubectl create token`](/docs/reference/generated/kubectl/kubectl-commands#-em-token-em-) +command to obtain a token from the `TokenRequest` API. + +You should only create a service account token Secret object +if you can't use the `TokenRequest` API to obtain a token, +and the security exposure of persisting a non-expiring token credential +in a readable API object is acceptable to you. + When using this Secret type, you need to ensure that the `kubernetes.io/service-account.name` annotation is set to an existing -service account name. A Kubernetes -{{< glossary_tooltip text="controller" term_id="controller" >}} fills in some -other fields such as the `kubernetes.io/service-account.uid` annotation, and the -`token` key in the `data` field, which is set to contain an authentication -token. +service account name. If you are creating both the ServiceAccount and +the Secret objects, you should create the ServiceAccount object first. + +After the Secret is created, a Kubernetes {{< glossary_tooltip text="controller" term_id="controller" >}} +fills in some other fields such as the `kubernetes.io/service-account.uid` annotation, and the +`token` key in the `data` field, which is populated with an authentication token. The following example configuration declares a service account token Secret: @@ -911,20 +928,14 @@ data: extra: YmFyCg== ``` -When creating a `Pod`, Kubernetes automatically finds or creates a service account -Secret and then automatically modifies your Pod to use this Secret. The service account -token Secret contains credentials for accessing the Kubernetes API. - -The automatic creation and use of API credentials can be disabled or -overridden if desired. However, if all you need to do is securely access the -API server, this is the recommended workflow. +After creating the Secret, wait for Kubernetes to populate the `token` key in the `data` field. See the [ServiceAccount](/docs/tasks/configure-pod-container/configure-service-account/) documentation for more information on how service accounts work. You can also check the `automountServiceAccountToken` field and the `serviceAccountName` field of the [`Pod`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core) -for information on referencing service account from Pods. +for information on referencing service account credentials from within Pods. ### Docker config Secrets @@ -982,7 +993,7 @@ kubectl create secret docker-registry secret-tiger-docker \ ``` That command creates a Secret of type `kubernetes.io/dockerconfigjson`. -If you dump the `.data.dockercfgjson` field from that new Secret and then +If you dump the `.data.dockerconfigjson` field from that new Secret and then decode it from base64: ```shell @@ -1291,7 +1302,7 @@ on that node. - When deploying applications that interact with the Secret API, you should limit access using [authorization policies](/docs/reference/access-authn-authz/authorization/) such as - [RBAC]( /docs/reference/access-authn-authz/rbac/). + [RBAC](/docs/reference/access-authn-authz/rbac/). - In the Kubernetes API, `watch` and `list` requests for Secrets within a namespace are extremely powerful capabilities. Avoid granting this access where feasible, since listing Secrets allows the clients to inspect the values of every Secret in that @@ -1310,7 +1321,7 @@ have access to run a Pod that then exposes the Secret. - When deploying applications that interact with the Secret API, you should limit access using [authorization policies](/docs/reference/access-authn-authz/authorization/) such as - [RBAC]( /docs/reference/access-authn-authz/rbac/). + [RBAC](/docs/reference/access-authn-authz/rbac/). - In the API server, objects (including Secrets) are persisted into {{< glossary_tooltip term_id="etcd" >}}; therefore: - only allow cluster admistrators to access etcd (this includes read-only access); diff --git a/content/en/docs/concepts/configuration/windows-resource-management.md b/content/en/docs/concepts/configuration/windows-resource-management.md index 6593caa5fb..955fea194a 100644 --- a/content/en/docs/concepts/configuration/windows-resource-management.md +++ b/content/en/docs/concepts/configuration/windows-resource-management.md @@ -32,52 +32,48 @@ host, and thus privileged containers are not available on Windows. Containers cannot assume an identity from the host because the Security Account Manager (SAM) is separate. -## Memory reservations {#resource-management-memory} +## Memory management {#resource-management-memory} Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. -Windows nodes do not overcommit memory for processes running in containers. The +Windows nodes do not overcommit memory for processes. The net effect is that Windows won't reach out of memory conditions the same way Linux does, and processes page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance. -You can place bounds on memory use for workloads using the kubelet -parameters `--kubelet-reserve` and/or `--system-reserve`; these account -for memory usage on the node (outside of containers), and reduce -[NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable). -As you deploy workloads, set resource limits on containers. This also subtracts from -`NodeAllocatable` and prevents the scheduler from adding more pods once a node is full. +## CPU management {#resource-management-cpu} -{{< note >}} -When you set memory resource limits for Windows containers, you should either set a -limit and leave the memory request unspecified, or set the request equal to the limit. -{{< /note >}} +Windows can limit the amount of CPU time allocated for different processes but cannot +guarantee a minimum amount of CPU time. -On Windows, good practice to avoid over-provisioning is to configure the kubelet -with a system reserved memory of at least 2GiB to account for Windows, Kubernetes -and container runtime overheads. - -## CPU reservations {#resource-management-cpu} - -To account for CPU use by the operating system, the container runtime, and by -Kubernetes host processes such as the kubelet, you can (and should) reserve a -percentage of total CPU. You should determine this CPU reservation taking account of -to the number of CPU cores available on the node. To decide on the CPU percentage to -reserve, identify the maximum pod density for each node and monitor the CPU usage of -the system services running there, then choose a value that meets your workload needs. - -You can place bounds on CPU usage for workloads using the -kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to -account for CPU usage on the node (outside of containers). -This reduces `NodeAllocatable`. -The cluster-wide scheduler then takes this reservation into account when determining -pod placement. - -On Windows, the kubelet supports a command-line flag to set the priority of the +On Windows, the kubelet supports a command-line flag to set the +[scheduling priority](https://docs.microsoft.com/windows/win32/procthread/scheduling-priorities) of the kubelet process: `--windows-priorityclass`. This flag allows the kubelet process to get more CPU time slices when compared to other processes running on the Windows host. More information on the allowable values and their meaning is available at [Windows Priority Classes](https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities#priority-class). To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to `ABOVE_NORMAL_PRIORITY_CLASS` or above. + +## Resource reservation {#resource-reservation} + +To account for memory and CPU used by the operating system, the container runtime, and by +Kubernetes host processes such as the kubelet, you can (and should) reserve +memory and CPU resources with the `--kube-reserved` and/or `--system-reserved` kubelet flags. +On Windows these values are only used to calculate the node's +[allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) resources. + +{{< caution >}} +As you deploy workloads, set resource memory and CPU limits on containers. +This also subtracts from `NodeAllocatable` and helps the cluster-wide scheduler in determining which pods to place on which nodes. + +Scheduling pods without limits may over-provision the Windows nodes and in extreme +cases can cause the nodes to become unhealthy. +{{< /caution >}} + +On Windows, a good practice is to reserve at least 2GiB of memory. + +To determine how much CPU to reserve, +identify the maximum pod density for each node and monitor the CPU usage of +the system services running there, then choose a value that meets your workload needs. diff --git a/content/en/docs/concepts/containers/runtime-class.md b/content/en/docs/concepts/containers/runtime-class.md index 38a74cf186..6366ee0551 100644 --- a/content/en/docs/concepts/containers/runtime-class.md +++ b/content/en/docs/concepts/containers/runtime-class.md @@ -97,7 +97,7 @@ spec: This will instruct the kubelet to use the named RuntimeClass to run this pod. If the named RuntimeClass does not exist, or the CRI cannot run the corresponding handler, the pod will enter the `Failed` terminal [phase](/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase). Look for a -corresponding [event](/docs/tasks/debug-application-cluster/debug-application-introspection/) for an +corresponding [event](/docs/tasks/debug/debug-application/debug-running-pod/) for an error message. If no `runtimeClassName` is specified, the default RuntimeHandler will be used, which is equivalent diff --git a/content/en/docs/concepts/extend-kubernetes/_index.md b/content/en/docs/concepts/extend-kubernetes/_index.md index 3cf3eb1f7a..c50010a39f 100644 --- a/content/en/docs/concepts/extend-kubernetes/_index.md +++ b/content/en/docs/concepts/extend-kubernetes/_index.md @@ -160,7 +160,7 @@ After a request is authorized, if it is a write operation, it also goes through ### Storage Plugins -[Flex Volumes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/flexvolume-deployment.md +[Flex Volumes](https://git.k8s.io/design-proposals-archive/storage/flexvolume-deployment.md ) allow users to mount volume types without built-in support by having the Kubelet call a Binary Plugin to mount the volume. @@ -191,7 +191,7 @@ This is a significant undertaking, and almost all Kubernetes users find they do not need to modify the scheduler. The scheduler also supports a -[webhook](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md) +[webhook](https://git.k8s.io/design-proposals-archive/scheduling/scheduler_extender.md) that permits a webhook backend (scheduler extension) to filter and prioritize the nodes chosen for a pod. diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 6785dccdac..6f265f57f3 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -8,7 +8,7 @@ weight: 20 {{< feature-state for_k8s_version="v1.10" state="beta" >}} -Kubernetes provides a [device plugin framework](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md) +Kubernetes provides a [device plugin framework](https://git.k8s.io/design-proposals-archive/resource-management/device-plugin.md) that you can use to advertise system hardware resources to the {{< glossary_tooltip term_id="kubelet" >}}. diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md index dc3940d5e9..647111b375 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.md @@ -14,6 +14,8 @@ weight: 10 Kubernetes {{< skew currentVersion >}} supports [Container Network Interface](https://github.com/containernetworking/cni) (CNI) plugins for cluster networking. You must use a CNI plugin that is compatible with your cluster and that suits your needs. Different plugins are available (both open- and closed- source) in the wider Kubernetes ecosystem. +A CNI plugin is required to implement the [Kubernetes network model](/docs/concepts/services-networking/#the-kubernetes-network-model). + You must use a CNI plugin that is compatible with the [v0.4.0](https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md) or later releases of the CNI specification. The Kubernetes project recommends using a plugin that is @@ -24,26 +26,37 @@ CNI specification (plugins can be compatible with multiple spec versions). ## Installation -A CNI plugin is required to implement the [Kubernetes network model](/docs/concepts/services-networking/#the-kubernetes-network-model). The CRI manages its own CNI plugins. There are two Kubelet command line parameters to keep in mind when using plugins: +A Container Runtime, in the networking context, is a daemon on a node configured to provide CRI Services for kubelet. In particular, the Container Runtime must be configured to load the CNI plugins required to implement the Kubernetes network model. -* `cni-bin-dir`: Kubelet probes this directory for plugins on startup -* `network-plugin`: The network plugin to use from `cni-bin-dir`. It must match the name reported by a plugin probed from the plugin directory. For CNI plugins, this is `cni`. +{{< note >}} +Prior to Kubernetes 1.24, the CNI plugins could also be managed by the kubelet using the `cni-bin-dir` and `network-plugin` command-line parameters. +These command-line parameters were removed in Kubernetes 1.24, with management of the CNI no longer in scope for kubelet. + +See [Troubleshooting CNI plugin-related errors](/docs/tasks/administer-cluster/migrating-from-dockershim/troubleshooting-cni-plugin-related-errors/) +if you are facing issues following the removal of dockershim. +{{< /note >}} + +For specific information about how a Container Runtime manages the CNI plugins, see the documentation for that Container Runtime, for example: +- [containerd](https://github.com/containerd/containerd/blob/main/script/setup/install-cni) +- [CRI-O](https://github.com/cri-o/cri-o/blob/main/contrib/cni/README.md) + +For specific information about how to install and manage a CNI plugin, see the documentation for that plugin or [networking provider](/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model). ## Network Plugin Requirements -Besides providing the [`NetworkPlugin` interface](https://github.com/kubernetes/kubernetes/tree/{{< param "fullversion" >}}/pkg/kubelet/dockershim/network/plugins.go) to configure and clean up pod networking, the plugin may also need specific support for kube-proxy. The iptables proxy obviously depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. If the plugin does not use a Linux bridge (but instead something like Open vSwitch or some other mechanism) it should ensure container traffic is appropriately routed for the proxy. +For plugin developers and users who regularly build or deploy Kubernetes, the plugin may also need specific configuration to support kube-proxy. +The iptables proxy depends on iptables, and the plugin may need to ensure that container traffic is made available to iptables. +For example, if the plugin connects containers to a Linux bridge, the plugin must set the `net/bridge/bridge-nf-call-iptables` sysctl to `1` to ensure that the iptables proxy functions correctly. +If the plugin does not use a Linux bridge, but uses something like Open vSwitch or some other mechanism instead, it should ensure container traffic is appropriately routed for the proxy. -By default if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy. +By default, if no kubelet network plugin is specified, the `noop` plugin is used, which sets `net/bridge/bridge-nf-call-iptables=1` to ensure simple configurations (like Docker with a bridge) work correctly with the iptables proxy. -### CNI +### Loopback CNI -The CNI plugin is selected by passing Kubelet the `--network-plugin=cni` command-line option. Kubelet reads a file from `--cni-conf-dir` (default `/etc/cni/net.d`) and uses the CNI configuration from that file to set up each pod's network. The CNI configuration file must match the [CNI specification](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration), and any required CNI plugins referenced by the configuration must be present in `--cni-bin-dir` (default `/opt/cni/bin`). +In addition to the CNI plugin installed on the nodes for implementing the Kubernetes network model, Kubernetes also requires the container runtimes to provide a loopback interface `lo`, which is used for each sandbox (pod sandboxes, vm sandboxes, ...). +Implementing the loopback interface can be accomplished by re-using the [CNI loopback plugin.](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) or by developing your own code to achieve this (see [this example from CRI-O](https://github.com/cri-o/ocicni/blob/release-1.24/pkg/ocicni/util_linux.go#L91)). -If there are multiple CNI configuration files in the directory, the kubelet uses the configuration file that comes first by name in lexicographic order. - -In addition to the CNI plugin specified by the configuration file, Kubernetes requires the standard CNI [`lo`](https://github.com/containernetworking/plugins/blob/master/plugins/main/loopback/loopback.go) plugin, at minimum version 0.2.0 - -#### Support hostPort +### Support hostPort The CNI networking plugin supports `hostPort`. You can use the official [portmap](https://github.com/containernetworking/plugins/tree/master/plugins/meta/portmap) plugin offered by the CNI plugin team or use your own plugin with portMapping functionality. @@ -80,7 +93,7 @@ For example: } ``` -#### Support traffic shaping +### Support traffic shaping **Experimental Feature** @@ -132,8 +145,4 @@ metadata: ... ``` -## Usage Summary - -* `--network-plugin=cni` specifies that we use the `cni` network plugin with actual CNI plugin binaries located in `--cni-bin-dir` (default `/opt/cni/bin`) and CNI plugin configuration located in `--cni-conf-dir` (default `/etc/cni/net.d`). - ## {{% heading "whatsnext" %}} diff --git a/content/en/docs/concepts/extend-kubernetes/operator.md b/content/en/docs/concepts/extend-kubernetes/operator.md index 96a17ada96..133814e98a 100644 --- a/content/en/docs/concepts/extend-kubernetes/operator.md +++ b/content/en/docs/concepts/extend-kubernetes/operator.md @@ -111,7 +111,9 @@ Operator. {{% thirdparty-content %}} * [Charmed Operator Framework](https://juju.is/) +* [Java Operator SDK](https://github.com/java-operator-sdk/java-operator-sdk) * [Kopf](https://github.com/nolar/kopf) (Kubernetes Operator Pythonic Framework) +* [kube-rs](https://kube.rs/) (Rust) * [kubebuilder](https://book.kubebuilder.io/) * [KubeOps](https://buehler.github.io/dotnet-operator-sdk/) (.NET operator SDK) * [KUDO](https://kudo.dev/) (Kubernetes Universal Declarative Operator) diff --git a/content/en/docs/concepts/overview/kubernetes-api.md b/content/en/docs/concepts/overview/kubernetes-api.md index 3c0bba3adb..b72d1aab1f 100644 --- a/content/en/docs/concepts/overview/kubernetes-api.md +++ b/content/en/docs/concepts/overview/kubernetes-api.md @@ -76,7 +76,7 @@ request headers as follows: Kubernetes implements an alternative Protobuf based serialization format that is primarily intended for intra-cluster communication. For more information -about this format, see the [Kubernetes Protobuf serialization](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/protobuf.md) design proposal and the +about this format, see the [Kubernetes Protobuf serialization](https://git.k8s.io/design-proposals-archive/api-machinery/protobuf.md) design proposal and the Interface Definition Language (IDL) files for each schema located in the Go packages that define the API objects. diff --git a/content/en/docs/concepts/overview/working-with-objects/names.md b/content/en/docs/concepts/overview/working-with-objects/names.md index 9bafb1584c..7b6b380e35 100644 --- a/content/en/docs/concepts/overview/working-with-objects/names.md +++ b/content/en/docs/concepts/overview/working-with-objects/names.md @@ -100,4 +100,4 @@ UUIDs are standardized as ISO/IEC 9834-8 and as ITU-T X.667. ## {{% heading "whatsnext" %}} * Read about [labels](/docs/concepts/overview/working-with-objects/labels/) in Kubernetes. -* See the [Identifiers and Names in Kubernetes](https://git.k8s.io/community/contributors/design-proposals/architecture/identifiers.md) design document. +* See the [Identifiers and Names in Kubernetes](https://git.k8s.io/design-proposals-archive/architecture/identifiers.md) design document. diff --git a/content/en/docs/concepts/policy/limit-range.md b/content/en/docs/concepts/policy/limit-range.md index 8158b78437..f5a7e66cac 100644 --- a/content/en/docs/concepts/policy/limit-range.md +++ b/content/en/docs/concepts/policy/limit-range.md @@ -53,7 +53,7 @@ Neither contention nor changes to a LimitRange will affect already created resou ## {{% heading "whatsnext" %}} -Refer to the [LimitRanger design document](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) for more information. +Refer to the [LimitRanger design document](https://git.k8s.io/design-proposals-archive/resource-management/admission_control_limit_range.md) for more information. For examples on using limits, see: diff --git a/content/en/docs/concepts/policy/resource-quotas.md b/content/en/docs/concepts/policy/resource-quotas.md index 0e88ca3433..8d9490b828 100644 --- a/content/en/docs/concepts/policy/resource-quotas.md +++ b/content/en/docs/concepts/policy/resource-quotas.md @@ -22,8 +22,7 @@ be consumed by resources in that namespace. Resource quotas work like this: -- Different teams work in different namespaces. Currently this is voluntary, but - support for making this mandatory via ACLs is planned. +- Different teams work in different namespaces. This can be enforced with [RBAC](/docs/reference/access-authn-authz/rbac/). - The administrator creates one ResourceQuota for each namespace. @@ -698,7 +697,7 @@ and it is to be created in a namespace other than `kube-system`. ## {{% heading "whatsnext" %}} -- See [ResourceQuota design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_resource_quota.md) for more information. +- See [ResourceQuota design doc](https://git.k8s.io/design-proposals-archive/resource-management/admission_control_resource_quota.md) for more information. - See a [detailed example for how to use resource quota](/docs/tasks/administer-cluster/quota-api-object/). -- Read [Quota support for priority class design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/pod-priority-resourcequota.md). +- Read [Quota support for priority class design doc](https://git.k8s.io/design-proposals-archive/scheduling/pod-priority-resourcequota.md). - See [LimitedResources](https://github.com/kubernetes/kubernetes/pull/36765) diff --git a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md index 067e3d60ed..8d3cc1831c 100644 --- a/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md +++ b/content/en/docs/concepts/scheduling-eviction/assign-pod-node.md @@ -302,9 +302,8 @@ the Pod onto a node that is in the same zone as one or more Pods with the label `topology.kubernetes.io/zone=R` label if there are other nodes in the same zone currently running Pods with the `Security=S2` Pod label. -See the -[design doc](https://git.k8s.io/community/contributors/design-proposals/scheduling/podaffinity.md) -for many more examples of Pod affinity and anti-affinity. +To get yourself more familiar with the examples of Pod affinity and anti-affinity, +refer to the [design proposal](https://git.k8s.io/design-proposals-archive/scheduling/podaffinity.md). You can use the `In`, `NotIn`, `Exists` and `DoesNotExist` values in the `operator` field for Pod affinity and anti-affinity. @@ -472,8 +471,8 @@ The above Pod will only run on the node `kube-01`. ## {{% heading "whatsnext" %}} * Read more about [taints and tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/) . -* Read the design docs for [node affinity](https://git.k8s.io/community/contributors/design-proposals/scheduling/nodeaffinity.md) - and for [inter-pod affinity/anti-affinity](https://git.k8s.io/community/contributors/design-proposals/scheduling/podaffinity.md). +* Read the design docs for [node affinity](https://git.k8s.io/design-proposals-archive/scheduling/nodeaffinity.md) + and for [inter-pod affinity/anti-affinity](https://git.k8s.io/design-proposals-archive/scheduling/podaffinity.md). * Learn about how the [topology manager](/docs/tasks/administer-cluster/topology-manager/) takes part in node-level resource allocation decisions. * Learn how to use [nodeSelector](/docs/tasks/configure-pod-container/assign-pods-nodes/). diff --git a/content/en/docs/concepts/scheduling-eviction/pod-overhead.md b/content/en/docs/concepts/scheduling-eviction/pod-overhead.md index 72b160653e..2610ea80ad 100644 --- a/content/en/docs/concepts/scheduling-eviction/pod-overhead.md +++ b/content/en/docs/concepts/scheduling-eviction/pod-overhead.md @@ -97,7 +97,7 @@ The output is: map[cpu:250m memory:120Mi] ``` -If a ResourceQuota is defined, the sum of container requests as well as the +If a [ResourceQuota](/docs/concepts/policy/resource-quotas/) is defined, the sum of container requests as well as the `overhead` field are counted. When the kube-scheduler is deciding which node should run a new Pod, the scheduler considers that Pod's diff --git a/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md b/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md index a81d9904ac..951d3f273d 100644 --- a/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md +++ b/content/en/docs/concepts/scheduling-eviction/resource-bin-packing.md @@ -3,70 +3,100 @@ reviewers: - bsalamat - k82cn - ahg-g -title: Resource Bin Packing for Extended Resources +title: Resource Bin Packing content_type: concept weight: 80 --- -{{< feature-state for_k8s_version="v1.16" state="alpha" >}} - -The kube-scheduler can be configured to enable bin packing of resources along -with extended resources using `RequestedToCapacityRatioResourceAllocation` -priority function. Priority functions can be used to fine-tune the -kube-scheduler as per custom needs. +In the [scheduling-plugin](/docs/reference/scheduling/config/#scheduling-plugins) `NodeResourcesFit` of kube-scheduler, there are two +scoring strategies that support the bin packing of resources: `MostAllocated` and `RequestedToCapacityRatio`. -## Enabling Bin Packing using RequestedToCapacityRatioResourceAllocation +## Enabling bin packing using MostAllocated strategy +The `MostAllocated` strategy scores the nodes based on the utilization of resources, favoring the ones with higher allocation. +For each resource type, you can set a weight to modify its influence in the node score. -Kubernetes allows the users to specify the resources along with weights for +To set the `MostAllocated` strategy for the `NodeResourcesFit` plugin, use a +[scheduler configuration](/docs/reference/scheduling/config) similar to the following: + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1beta3 +kind: KubeSchedulerConfiguration +profiles: +- pluginConfig: + - args: + scoringStrategy: + resources: + - name: cpu + weight: 1 + - name: memory + weight: 1 + - name: intel.com/foo + weight: 3 + - name: intel.com/bar + weight: 3 + type: MostAllocated + name: NodeResourcesFit +``` + +To learn more about other parameters and their default configuration, see the API documentation for +[`NodeResourcesFitArgs`](/docs/reference/config-api/kube-scheduler-config.v1beta3/#kubescheduler-config-k8s-io-v1beta3-NodeResourcesFitArgs). + +## Enabling bin packing using RequestedToCapacityRatio + +The `RequestedToCapacityRatio` strategy allows the users to specify the resources along with weights for each resource to score nodes based on the request to capacity ratio. This allows users to bin pack extended resources by using appropriate parameters -and improves the utilization of scarce resources in large clusters. The -behavior of the `RequestedToCapacityRatioResourceAllocation` priority function -can be controlled by a configuration option called `RequestedToCapacityRatioArgs`. -This argument consists of two parameters `shape` and `resources`. The `shape` +to improve the utilization of scarce resources in large clusters. It favors nodes according to a +configured function of the allocated resources. The behavior of the `RequestedToCapacityRatio` in +the `NodeResourcesFit` score function can be controlled by the +[scoringStrategy](/docs/reference/config-api/kube-scheduler-config.v1beta3/#kubescheduler-config-k8s-io-v1beta3-ScoringStrategy) field. +Within the `scoringStrategy` field, you can configure two parameters: `requestedToCapacityRatioParam` and +`resources`. The `shape` in `requestedToCapacityRatioParam` parameter allows the user to tune the function as least requested or most requested based on `utilization` and `score` values. The `resources` parameter consists of `name` of the resource to be considered during scoring and `weight` specify the weight of each resource. Below is an example configuration that sets -`requestedToCapacityRatioArguments` to bin packing behavior for extended -resources `intel.com/foo` and `intel.com/bar`. +the bin packing behavior for extended resources `intel.com/foo` and `intel.com/bar` +using the `requestedToCapacityRatio` field. ```yaml apiVersion: kubescheduler.config.k8s.io/v1beta3 kind: KubeSchedulerConfiguration profiles: -# ... - pluginConfig: - - name: RequestedToCapacityRatio - args: - shape: - - utilization: 0 - score: 10 - - utilization: 100 - score: 0 - resources: - - name: intel.com/foo - weight: 3 - - name: intel.com/bar - weight: 5 +- pluginConfig: + - args: + scoringStrategy: + resources: + - name: intel.com/foo + weight: 3 + - name: intel.com/bar + weight: 3 + requestedToCapacityRatioParam: + shape: + - utilization: 0 + score: 0 + - utilization: 100 + score: 10 + type: RequestedToCapacityRatio + name: NodeResourcesFit ``` Referencing the `KubeSchedulerConfiguration` file with the kube-scheduler flag `--config=/path/to/config/file` will pass the configuration to the scheduler. -**This feature is disabled by default** +To learn more about other parameters and their default configuration, see the API documentation for +[`NodeResourcesFitArgs`](/docs/reference/config-api/kube-scheduler-config.v1beta3/#kubescheduler-config-k8s-io-v1beta3-NodeResourcesFitArgs). -### Tuning the Priority Function +### Tuning the score function -`shape` is used to specify the behavior of the -`RequestedToCapacityRatioPriority` function. +`shape` is used to specify the behavior of the `RequestedToCapacityRatio` function. ```yaml shape: diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index 030f28e7d1..ff7718a1f3 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -15,8 +15,7 @@ is a property of {{< glossary_tooltip text="Pods" term_id="pod" >}} that *attrac a set of {{< glossary_tooltip text="nodes" term_id="node" >}} (either as a preference or a hard requirement). _Taints_ are the opposite -- they allow a node to repel a set of pods. -_Tolerations_ are applied to pods, and allow (but do not require) the pods to schedule -onto nodes with matching taints. +_Tolerations_ are applied to pods. Tolerations allow the scheduler to schedule pods with matching taints. Tolerations allow scheduling but don't guarantee scheduling: the scheduler also [evaluates other parameters](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/) as part of its function. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this diff --git a/content/en/docs/concepts/security/pod-security-admission.md b/content/en/docs/concepts/security/pod-security-admission.md index 1e452ea6af..60f87c04d9 100644 --- a/content/en/docs/concepts/security/pod-security-admission.md +++ b/content/en/docs/concepts/security/pod-security-admission.md @@ -37,8 +37,8 @@ To use this mechanism, your cluster must enforce Pod Security admission. ### Built-in Pod Security admission enforcement -In Kubernetes v{{< skew currentVersion >}}, the `PodSecurity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) -is a beta feature and is enabled by default. You must have this feature gate enabled. +From Kubernetes v1.23, the `PodSecurity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is a beta feature and is enabled by default. +This page is part of the documentation for Kubernetes v{{< skew currentVersion >}}. If you are running a different version of Kubernetes, consult the documentation for that release. ### Alternative: installing the `PodSecurity` admission webhook {#webhook} @@ -102,7 +102,7 @@ For each mode, there are two labels that determine the policy used: pod-security.kubernetes.io/: # Optional: per-mode version label that can be used to pin the policy to the -# version that shipped with a given Kubernetes minor version (for example v{{< skew latestVersion >}}). +# version that shipped with a given Kubernetes minor version (for example v{{< skew currentVersion >}}). # # MODE must be one of `enforce`, `audit`, or `warn`. # VERSION must be a valid Kubernetes minor version, or `latest`. diff --git a/content/en/docs/concepts/security/pod-security-policy.md b/content/en/docs/concepts/security/pod-security-policy.md index 30b0c8bab0..cc0acc410d 100644 --- a/content/en/docs/concepts/security/pod-security-policy.md +++ b/content/en/docs/concepts/security/pod-security-policy.md @@ -658,7 +658,7 @@ added. Capabilities listed in `RequiredDropCapabilities` must not be included in **DefaultAddCapabilities** - The capabilities which are added to containers by default, in addition to the runtime defaults. See the -the documentation for your container runtime for information on working with Linux capabilities. +documentation for your container runtime for information on working with Linux capabilities. ### SELinux diff --git a/content/en/docs/concepts/security/pod-security-standards.md b/content/en/docs/concepts/security/pod-security-standards.md index 393468ac74..47e93d3e9e 100644 --- a/content/en/docs/concepts/security/pod-security-standards.md +++ b/content/en/docs/concepts/security/pod-security-standards.md @@ -29,10 +29,9 @@ This guide outlines the requirements of each policy. **The _Privileged_ policy is purposely-open, and entirely unrestricted.** This type of policy is typically aimed at system- and infrastructure-level workloads managed by privileged, trusted users. -The Privileged policy is defined by an absence of restrictions. For allow-by-default enforcement -mechanisms (such as gatekeeper), the Privileged policy may be an absence of applied constraints -rather than an instantiated profile. In contrast, for a deny-by-default mechanism (such as Pod -Security Policy) the Privileged policy should enable all controls (disable all restrictions). +The Privileged policy is defined by an absence of restrictions. Allow-by-default +mechanisms (such as gatekeeper) may be Privileged by default. In contrast, for a deny-by-default mechanism (such as Pod +Security Policy) the Privileged policy should disable all restrictions. ### Baseline @@ -58,7 +57,7 @@ fail validation. HostProcess -

Windows pods offer the ability to run HostProcess containers which enables privileged access to the Windows node. Privileged access to the host is disallowed in the baseline policy. HostProcess pods are an alpha feature as of Kubernetes v1.22.

+

Windows pods offer the ability to run HostProcess containers which enables privileged access to the Windows node. Privileged access to the host is disallowed in the baseline policy. {{< feature-state for_k8s_version="v1.23" state="beta" >}}

Restricted Fields

  • spec.securityContext.windowsOptions.hostProcess
    • @@ -458,6 +457,16 @@ of individual policies are not defined here. - {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}} - {{< example file="policy/restricted-psp.yaml" >}}Restricted{{< /example >}} +### Alternatives + +{{% thirdparty-content %}} + +Other alternatives for enforcing policies are being developed in the Kubernetes ecosystem, such as: +- [Kubewarden](https://github.com/kubewarden) +- [Kyverno](https://kyverno.io/policies/pod-security/) +- [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) + + ## FAQ ### Why isn't there a profile between privileged and baseline? @@ -481,14 +490,6 @@ as well as other related parameters outside the Security Context. As of July 202 [Pod Security Policies](/docs/concepts/security/pod-security-policy/) are deprecated in favor of the built-in [Pod Security Admission Controller](/docs/concepts/security/pod-security-admission/). -{{% thirdparty-content %}} - -Other alternatives for enforcing security profiles are being developed in the Kubernetes -ecosystem, such as: -- [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper). -- [Kubewarden](https://github.com/kubewarden). -- [Kyverno](https://kyverno.io/policies/pod-security/). - ### What profiles should I apply to my Windows Pods? Windows in Kubernetes has some limitations and differentiators from standard Linux-based diff --git a/content/en/docs/concepts/security/rbac-good-practices.md b/content/en/docs/concepts/security/rbac-good-practices.md new file mode 100644 index 0000000000..7361c3a163 --- /dev/null +++ b/content/en/docs/concepts/security/rbac-good-practices.md @@ -0,0 +1,179 @@ +--- +reviewers: +title: Role Based Access Control Good Practices +description: > + Principles and practices for good RBAC design for cluster operators. +content_type: concept +--- + + + +Kubernetes {{< glossary_tooltip text="RBAC" term_id="rbac" >}} is a key security control +to ensure that cluster users and workloads have only the access to resources required to +execute their roles. It is important to ensure that, when designing permissions for cluster +users, the cluster administrator understands the areas where privilge escalation could occur, +to reduce the risk of excessive access leading to security incidents. + +The good practices laid out here should be read in conjunction with the general [RBAC documentation](/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update). + + + +## General good practice + +### Least privilege + +Ideally minimal RBAC rights should be assigned to users and service accounts. Only permissions +explicitly required for their operation should be used. Whilst each cluster will be different, +some general rules that can be applied are : + + - Assign permissions at the namespace level where possible. Use RoleBindings as opposed to + ClusterRoleBindings to give users rights only within a specific namespace. + - Avoid providing wildcard permissions when possible, especially to all resources. + As Kubernetes is an extensible system, providing wildcard access gives rights + not just to all object types presently in the cluster, but also to all future object types + which are created in the future. + - Administrators should not use `cluster-admin` accounts except where specifically needed. + Providing a low privileged account with [impersonation rights](/docs/reference/access-authn-authz/authentication/#user-impersonation) + can avoid accidental modification of cluster resources. + - Avoid adding users to the `system:masters` group. Any user who is a member of this group + bypasses all RBAC rights checks and will always have unrestricted superuser access, which cannot be + revoked by removing RoleBindings or ClusterRoleBindings. As an aside, if a cluster is + using an authorization webhook, membership of this group also bypasses that webhook (requests + from users who are members of that group are never sent to the webhook) + +### Minimize distribution of privileged tokens + +Ideally, pods shouldn't be assigned service accounts that have been granted powerful permissions (for example, any of the rights listed under +[privilege escalation risks](#privilege-escalation-risks)). +In cases where a workload requires powerful permissions, consider the following practices: + + - Limit the number of nodes running powerful pods. Ensure that any DaemonSets you run + are necessary and are run with least privilege to limit the blast radius of container escapes. + - Avoid running powerful pods alongside untrusted or publicly-exposed ones. Consider using + [Taints and Toleration](/docs/concepts/scheduling-eviction/taint-and-toleration/), [NodeAffinity](/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity), or [PodAntiAffinity](/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity) to ensure + pods don't run alongside untrusted or less-trusted Pods. Pay especial attention to + situations where less-trustworthy Pods are not meeting the **Restricted** Pod Security Standard. + +### Hardening + +Kubernetes defaults to providing access which may not be required in every cluster. Reviewing +the RBAC rights provided by default can provide opportunities for security hardening. +In general, changes should not be made to rights provided to `system:` accounts some options +to harden cluster rights exist: + +- Review bindings for the `system:unauthenticated` group and remove where possible, as this gives + access to anyone who can contact the API server at a network level. +- Avoid the default auto-mounting of service account tokens by setting + `automountServiceAccountToken: false`. For more details, see + [using default service account token](/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server). + Setting this value for a Pod will overwrite the service account setting, workloads + which require service account tokens can still mount them. + +### Periodic review + +It is vital to periodically review the Kubernetes RBAC settings for redundant entries and +possible privilege escalations. +If an attacker is able to create a user account with the same name as a deleted user, +they can automatically inherit all the rights of the deleted user, especially the +rights assigned to that user. + +## Kubernetes RBAC - privilege escalation risks {#privilege-escalation-risks} + +Within Kubernetes RBAC there are a number of privileges which, if granted, can allow a user or a service account +to escalate their privileges in the cluster or affect systems outside the cluster. + +This section is intended to provide visibility of the areas where cluster operators +should take care, to ensure that they do not inadvertently allow for more access to clusters than intended. + +### Listing secrets + +It is generally clear that allowing `get` access on Secrets will allow a user to read their contents. +It is also important to note that `list` and `watch` access also effectively allow for users to reveal the Secret contents. +For example, when a List response is returned (for example, via `kubectl get secrets -A -o yaml`), the response +includes the contents of all Secrets. + +### Workload creation + +Users who are able to create workloads (either Pods, or +[workload resources](/docs/concepts/workloads/controllers/) that manage Pods) will +be able to gain access to the underlying node unless restrictions based on the Kubernetes +[Pod Security Standards](/docs/concepts/security/pod-security-standards/) are in place. + +Users who can run privileged Pods can use that access to gain node access and potentially to +further elevate their privileges. Where you do not fully trust a user or other principal +with the ability to create suitably secure and isolated Pods, you should enforce either the +**Baseline** or **Restricted** Pod Security Standard. +You can use [Pod Security admission](/docs/concepts/security/pod-security-admission/) +or other (third party) mechanisms to implement that enforcement. + +You can also use the deprecated [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/) mechanism +to restrict users' abilities to create privileged Pods (N.B. PodSecurityPolicy is scheduled for removal +in version 1.25). + +Creating a workload in a namespace also grants indirect access to Secrets in that namespace. +Creating a pod in kube-system or a similarly privileged namespace can grant a user access to +Secrets they would not have through RBAC directly. + +### Persistent volume creation + +As noted in the [PodSecurityPolicy](/docs/concepts/policy/pod-security-policy/#volumes-and-file-systems) documentation, access to create PersistentVolumes can allow for escalation of access to the underlying host. Where access to persistent storage is required trusted administrators should create +PersistentVolumes, and constrained users should use PersistentVolumeClaims to access that storage. + +### Access to `proxy` subresource of Nodes + +Users with access to the proxy sub-resource of node objects have rights to the Kubelet API, +which allows for command execution on every pod on the node(s) which they have rights to. +This access bypasses audit logging and admission control, so care should be taken before +granting rights to this resource. + +### Escalate verb + +Generally the RBAC system prevents users from creating clusterroles with more rights than +they possess. The exception to this is the `escalate` verb. As noted in the [RBAC documentation](/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update), +users with this right can effectively escalate their privileges. + +### Bind verb + +Similar to the `escalate` verb, granting users this right allows for bypass of Kubernetes +in-built protections against privilege escalation, allowing users to create bindings to +roles with rights they do not already have. + +### Impersonate verb + +This verb allows users to impersonate and gain the rights of other users in the cluster. +Care should be taken when granting it, to ensure that excessive permissions cannot be gained +via one of the impersonated accounts. + +### CSRs and certificate issuing + +The CSR API allows for users with `create` rights to CSRs and `update` rights on `certificatesigningrequests/approval` +where the signer is `kubernetes.io/kube-apiserver-client` to create new client certificates +which allow users to authenticate to the cluster. Those client certificates can have arbitrary +names including duplicates of Kubernetes system components. This will effectively allow for privilege escalation. + +### Token request + +Users with `create` rights on `serviceaccounts/token` can create TokenRequests to issue +tokens for existing service accounts. + +### Control admission webhooks + +Users with control over `validatingwebhookconfigurations` or `mutatingwebhookconfigurations` +can control webhooks that can read any object admitted to the cluster, and in the case of +mutating webhooks, also mutate admitted objects. + + +## Kubernetes RBAC - denial of service risks {#denial-of-service-risks} + +### Object creation denial-of-service {#object-creation-dos} +Users who have rights to create objects in a cluster may be able to create sufficient large +objects to create a denial of service condition either based on the size or number of objects, as discussed in +[etcd used by Kubernetes is vulnerable to OOM attack](https://github.com/kubernetes/kubernetes/issues/107325). This may be +specifically relevant in multi-tenant clusters if semi-trusted or untrusted users +are allowed limited access to a system. + +One option for mitigation of this issue would be to use [resource quotas](/docs/concepts/policy/resource-quotas/#object-count-quota) +to limit the quantity of objects which can be created. + +## {{% heading "whatsnext" %}} +* To learn more about RBAC, see the [RBAC documentation](/docs/reference/access-authn-authz/rbac/). diff --git a/content/en/docs/concepts/security/windows-security.md b/content/en/docs/concepts/security/windows-security.md index 1341f38c59..2126bdddaa 100644 --- a/content/en/docs/concepts/security/windows-security.md +++ b/content/en/docs/concepts/security/windows-security.md @@ -51,5 +51,5 @@ Windows containers can also run as Active Directory identities by utilizing [Gro Linux-specific pod security context mechanisms (such as SELinux, AppArmor, Seccomp, or custom POSIX capabilities) are not supported on Windows nodes. -Privileged containers are [not supported](#compatibility-v1-pod-spec-containers-securitycontext) on Windows. +Privileged containers are [not supported](/docs/concepts/windows/intro/#compatibility-v1-pod-spec-containers-securitycontext) on Windows. Instead [HostProcess containers](/docs/tasks/configure-pod-container/create-hostprocess-pod) can be used on Windows to perform many of the tasks performed by privileged containers on Linux. diff --git a/content/en/docs/concepts/services-networking/_index.md b/content/en/docs/concepts/services-networking/_index.md index ab1b784658..b4f7861075 100644 --- a/content/en/docs/concepts/services-networking/_index.md +++ b/content/en/docs/concepts/services-networking/_index.md @@ -7,26 +7,25 @@ description: > ## The Kubernetes network model -Every [`Pod`](/docs/concepts/workloads/pods/) gets its own IP address. +Every [`Pod`](/docs/concepts/workloads/pods/) in a cluster gets its own unique cluster-wide IP address. This means you do not need to explicitly create links between `Pods` and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where `Pods` can be treated much like VMs or physical hosts from the perspectives of port allocation, -naming, service discovery, [load balancing](/docs/concepts/services-networking/ingress/#load-balancing), application configuration, -and migration. +naming, service discovery, [load balancing](/docs/concepts/services-networking/ingress/#load-balancing), +application configuration, and migration. Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies): - * pods on a [node](/docs/concepts/architecture/nodes/) can communicate with all pods on all nodes without NAT + * pods can communicate with all other pods on any other [node](/docs/concepts/architecture/nodes/) + without NAT * agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node Note: For those platforms that support `Pods` running in the host network (e.g. -Linux): - - * pods in the host network of a node can communicate with all pods on all - nodes without NAT +Linux), when pods are attached to the host network of a node they can still communicate +with all pods on all nodes without NAT. This model is not only less complex overall, but it is principally compatible with the desire for Kubernetes to enable low-friction porting of apps from VMs diff --git a/content/en/docs/concepts/services-networking/dns-pod-service.md b/content/en/docs/concepts/services-networking/dns-pod-service.md index 9ca11a3457..939269f8ec 100644 --- a/content/en/docs/concepts/services-networking/dns-pod-service.md +++ b/content/en/docs/concepts/services-networking/dns-pod-service.md @@ -8,8 +8,8 @@ weight: 20 --- -Kubernetes creates DNS records for services and pods. You can contact -services with consistent DNS names instead of IP addresses. +Kubernetes creates DNS records for Services and Pods. You can contact +Services with consistent DNS names instead of IP addresses. @@ -25,20 +25,20 @@ Pod's own namespace and the cluster's default domain. ### Namespaces of Services -A DNS query may return different results based on the namespace of the pod making -it. DNS queries that don't specify a namespace are limited to the pod's -namespace. Access services in other namespaces by specifying it in the DNS query. +A DNS query may return different results based on the namespace of the Pod making +it. DNS queries that don't specify a namespace are limited to the Pod's +namespace. Access Services in other namespaces by specifying it in the DNS query. -For example, consider a pod in a `test` namespace. A `data` service is in +For example, consider a Pod in a `test` namespace. A `data` Service is in the `prod` namespace. -A query for `data` returns no results, because it uses the pod's `test` namespace. +A query for `data` returns no results, because it uses the Pod's `test` namespace. A query for `data.prod` returns the intended result, because it specifies the namespace. -DNS queries may be expanded using the pod's `/etc/resolv.conf`. Kubelet -sets this file for each pod. For example, a query for just `data` may be +DNS queries may be expanded using the Pod's `/etc/resolv.conf`. Kubelet +sets this file for each Pod. For example, a query for just `data` may be expanded to `data.test.svc.cluster.local`. The values of the `search` option are used to expand queries. To learn more about DNS queries, see [the `resolv.conf` manual page.](https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html) @@ -49,7 +49,7 @@ search .svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` -In summary, a pod in the _test_ namespace can successfully resolve either +In summary, a Pod in the _test_ namespace can successfully resolve either `data.prod` or `data.prod.svc.cluster.local`. ### DNS Records @@ -70,14 +70,14 @@ For more up-to-date specification, see ### A/AAAA records "Normal" (not headless) Services are assigned a DNS A or AAAA record, -depending on the IP family of the service, for a name of the form +depending on the IP family of the Service, for a name of the form `my-svc.my-namespace.svc.cluster-domain.example`. This resolves to the cluster IP of the Service. "Headless" (without a cluster IP) Services are also assigned a DNS A or AAAA record, -depending on the IP family of the service, for a name of the form +depending on the IP family of the Service, for a name of the form `my-svc.my-namespace.svc.cluster-domain.example`. Unlike normal -Services, this resolves to the set of IPs of the pods selected by the Service. +Services, this resolves to the set of IPs of the Pods selected by the Service. Clients are expected to consume the set or else use standard round-robin selection from the set. @@ -87,36 +87,36 @@ SRV Records are created for named ports that are part of normal or [Headless Services](/docs/concepts/services-networking/service/#headless-services). For each named port, the SRV record would have the form `_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster-domain.example`. -For a regular service, this resolves to the port number and the domain name: +For a regular Service, this resolves to the port number and the domain name: `my-svc.my-namespace.svc.cluster-domain.example`. -For a headless service, this resolves to multiple answers, one for each pod -that is backing the service, and contains the port number and the domain name of the pod +For a headless Service, this resolves to multiple answers, one for each Pod +that is backing the Service, and contains the port number and the domain name of the Pod of the form `auto-generated-name.my-svc.my-namespace.svc.cluster-domain.example`. ## Pods ### A/AAAA records -In general a pod has the following DNS resolution: +In general a Pod has the following DNS resolution: `pod-ip-address.my-namespace.pod.cluster-domain.example`. -For example, if a pod in the `default` namespace has the IP address 172.17.0.3, +For example, if a Pod in the `default` namespace has the IP address 172.17.0.3, and the domain name for your cluster is `cluster.local`, then the Pod has a DNS name: `172-17-0-3.default.pod.cluster.local`. -Any pods exposed by a Service have the following DNS resolution available: +Any Pods exposed by a Service have the following DNS resolution available: `pod-ip-address.service-name.my-namespace.svc.cluster-domain.example`. ### Pod's hostname and subdomain fields -Currently when a pod is created, its hostname is the Pod's `metadata.name` value. +Currently when a Pod is created, its hostname is the Pod's `metadata.name` value. The Pod spec has an optional `hostname` field, which can be used to specify the Pod's hostname. When specified, it takes precedence over the Pod's name to be -the hostname of the pod. For example, given a Pod with `hostname` set to +the hostname of the Pod. For example, given a Pod with `hostname` set to "`my-host`", the Pod will have its hostname set to "`my-host`". The Pod spec also has an optional `subdomain` field which can be used to specify @@ -173,14 +173,14 @@ spec: name: busybox ``` -If there exists a headless service in the same namespace as the pod and with +If there exists a headless Service in the same namespace as the Pod and with the same name as the subdomain, the cluster's DNS Server also returns an A or AAAA record for the Pod's fully qualified hostname. For example, given a Pod with the hostname set to "`busybox-1`" and the subdomain set to "`default-subdomain`", and a headless Service named "`default-subdomain`" in -the same namespace, the pod will see its own FQDN as +the same namespace, the Pod will see its own FQDN as "`busybox-1.default-subdomain.my-namespace.svc.cluster-domain.example`". DNS serves an -A or AAAA record at that name, pointing to the Pod's IP. Both pods "`busybox1`" and +A or AAAA record at that name, pointing to the Pod's IP. Both Pods "`busybox1`" and "`busybox2`" can have their distinct A or AAAA records. The Endpoints object can specify the `hostname` for any endpoint addresses, @@ -189,7 +189,7 @@ along with its IP. {{< note >}} Because A or AAAA records are not created for Pod names, `hostname` is required for the Pod's A or AAAA record to be created. A Pod with no `hostname` but with `subdomain` will only create the -A or AAAA record for the headless service (`default-subdomain.my-namespace.svc.cluster-domain.example`), +A or AAAA record for the headless Service (`default-subdomain.my-namespace.svc.cluster-domain.example`), pointing to the Pod's IP address. Also, Pod needs to become ready in order to have a record unless `publishNotReadyAddresses=True` is set on the Service. {{< /note >}} @@ -205,17 +205,17 @@ When you set `setHostnameAsFQDN: true` in the Pod spec, the kubelet writes the P {{< note >}} In Linux, the hostname field of the kernel (the `nodename` field of `struct utsname`) is limited to 64 characters. -If a Pod enables this feature and its FQDN is longer than 64 character, it will fail to start. The Pod will remain in `Pending` status (`ContainerCreating` as seen by `kubectl`) generating error events, such as Failed to construct FQDN from pod hostname and cluster domain, FQDN `long-FQDN` is too long (64 characters is the max, 70 characters requested). One way of improving user experience for this scenario is to create an [admission webhook controller](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) to control FQDN size when users create top level objects, for example, Deployment. +If a Pod enables this feature and its FQDN is longer than 64 character, it will fail to start. The Pod will remain in `Pending` status (`ContainerCreating` as seen by `kubectl`) generating error events, such as Failed to construct FQDN from Pod hostname and cluster domain, FQDN `long-FQDN` is too long (64 characters is the max, 70 characters requested). One way of improving user experience for this scenario is to create an [admission webhook controller](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) to control FQDN size when users create top level objects, for example, Deployment. {{< /note >}} ### Pod's DNS Policy -DNS policies can be set on a per-pod basis. Currently Kubernetes supports the -following pod-specific DNS policies. These policies are specified in the +DNS policies can be set on a per-Pod basis. Currently Kubernetes supports the +following Pod-specific DNS policies. These policies are specified in the `dnsPolicy` field of a Pod Spec. - "`Default`": The Pod inherits the name resolution configuration from the node - that the pods run on. + that the Pods run on. See [related discussion](/docs/tasks/administer-cluster/dns-custom-nameservers) for more details. - "`ClusterFirst`": Any DNS query that does not match the configured cluster @@ -226,6 +226,7 @@ following pod-specific DNS policies. These policies are specified in the for details on how DNS queries are handled in those cases. - "`ClusterFirstWithHostNet`": For Pods running with hostNetwork, you should explicitly set its DNS policy "`ClusterFirstWithHostNet`". + - Note: This is not supported on Windows. See [below](#dns-windows) for details - "`None`": It allows a Pod to ignore DNS settings from the Kubernetes environment. All DNS settings are supposed to be provided using the `dnsConfig` field in the Pod Spec. @@ -306,7 +307,7 @@ For IPv6 setup, search path and name server should be setup like this: kubectl exec -it dns-example -- cat /etc/resolv.conf ``` The output is similar to this: -```shell +``` nameserver fd00:79:30::a search default.svc.cluster-domain.example svc.cluster-domain.example cluster-domain.example options ndots:5 @@ -323,8 +324,25 @@ If the feature gate `ExpandedDNSConfig` is enabled for the kube-apiserver and the kubelet, it is allowed for Kubernetes to have at most 32 search domains and a list of search domains of up to 2048 characters. -## {{% heading "whatsnext" %}} +## DNS resolution on Windows nodes {#dns-windows} +- ClusterFirstWithHostNet is not supported for Pods that run on Windows nodes. + Windows treats all names with a `.` as a FQDN and skips FQDN resolution. +- On Windows, there are multiple DNS resolvers that can be used. As these come with + slightly different behaviors, using the + [`Resolve-DNSName`](https://docs.microsoft.com/powershell/module/dnsclient/resolve-dnsname) + powershell cmdlet for name query resolutions is recommended. +- On Linux, you have a DNS suffix list, which is used after resolution of a name as fully + qualified has failed. + On Windows, you can only have 1 DNS suffix, which is the DNS suffix associated with that + Pod's namespace (example: `mydns.svc.cluster.local`). Windows can resolve FQDNs, Services, + or network name which can be resolved with this single suffix. For example, a Pod spawned + in the `default` namespace, will have the DNS suffix `default.svc.cluster.local`. + Inside a Windows Pod, you can resolve both `kubernetes.default.svc.cluster.local` + and `kubernetes`, but not the partially qualified names (`kubernetes.default` or + `kubernetes.default.svc`). + +## {{% heading "whatsnext" %}} For guidance on administering DNS configurations, check [Configure DNS Service](/docs/tasks/administer-cluster/dns-custom-nameservers/) diff --git a/content/en/docs/concepts/services-networking/dual-stack.md b/content/en/docs/concepts/services-networking/dual-stack.md index a120df4392..921d69e8fb 100644 --- a/content/en/docs/concepts/services-networking/dual-stack.md +++ b/content/en/docs/concepts/services-networking/dual-stack.md @@ -1,16 +1,15 @@ --- -reviewers: -- lachie83 -- khenidak -- aramase -- bridgetkromhout title: IPv4/IPv6 dual-stack feature: title: IPv4/IPv6 dual-stack description: > Allocation of IPv4 and IPv6 addresses to Pods and Services - content_type: concept +reviewers: + - lachie83 + - khenidak + - aramase + - bridgetkromhout weight: 70 --- @@ -18,11 +17,11 @@ weight: 70 {{< feature-state for_k8s_version="v1.23" state="stable" >}} -IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to {{< glossary_tooltip text="Pods" term_id="pod" >}} and {{< glossary_tooltip text="Services" term_id="service" >}}. - -IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in 1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses. - +IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to +{{< glossary_tooltip text="Pods" term_id="pod" >}} and {{< glossary_tooltip text="Services" term_id="service" >}}. +IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in +1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses. @@ -30,68 +29,78 @@ IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluste IPv4/IPv6 dual-stack on your Kubernetes cluster provides the following features: - * Dual-stack Pod networking (a single IPv4 and IPv6 address assignment per Pod) - * IPv4 and IPv6 enabled Services - * Pod off-cluster egress routing (eg. the Internet) via both IPv4 and IPv6 interfaces +* Dual-stack Pod networking (a single IPv4 and IPv6 address assignment per Pod) +* IPv4 and IPv6 enabled Services +* Pod off-cluster egress routing (eg. the Internet) via both IPv4 and IPv6 interfaces ## Prerequisites The following prerequisites are needed in order to utilize IPv4/IPv6 dual-stack Kubernetes clusters: - * Kubernetes 1.20 or later - For information about using dual-stack services with earlier - Kubernetes versions, refer to the documentation for that version - of Kubernetes. - * Provider support for dual-stack networking (Cloud provider or otherwise must be able to provide Kubernetes nodes with routable IPv4/IPv6 network interfaces) - * A [network plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) that supports dual-stack networking. +* Kubernetes 1.20 or later + + For information about using dual-stack services with earlier + Kubernetes versions, refer to the documentation for that version + of Kubernetes. + +* Provider support for dual-stack networking (Cloud provider or otherwise must be able to provide + Kubernetes nodes with routable IPv4/IPv6 network interfaces) +* A [network plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) that + supports dual-stack networking. ## Configure IPv4/IPv6 dual-stack To configure IPv4/IPv6 dual-stack, set dual-stack cluster network assignments: - * kube-apiserver: - * `--service-cluster-ip-range=,` - * kube-controller-manager: - * `--cluster-cidr=,` - * `--service-cluster-ip-range=,` - * `--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6` defaults to /24 for IPv4 and /64 for IPv6 - * kube-proxy: - * `--cluster-cidr=,` - * kubelet: - * when there is no `--cloud-provider` the administrator can pass a comma-separated pair - of IP addresses via `--node-ip` to manually configure dual-stack `.status.addresses` - for that Node. - If a Pod runs on that node in HostNetwork mode, the Pod reports these IP addresses in its - `.status.podIPs` field. - All `podIPs` in a node match the IP family preference defined by the - `.status.addresses` field for that Node. +* kube-apiserver: + * `--service-cluster-ip-range=,` +* kube-controller-manager: + * `--cluster-cidr=,` + * `--service-cluster-ip-range=,` + * `--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6` defaults to /24 for IPv4 and /64 for IPv6 +* kube-proxy: + * `--cluster-cidr=,` +* kubelet: + * when there is no `--cloud-provider` the administrator can pass a comma-separated pair of IP + addresses via `--node-ip` to manually configure dual-stack `.status.addresses` for that Node. + If a Pod runs on that node in HostNetwork mode, the Pod reports these IP addresses in its + `.status.podIPs` field. + All `podIPs` in a node match the IP family preference defined by the `.status.addresses` + field for that Node. {{< note >}} An example of an IPv4 CIDR: `10.244.0.0/16` (though you would supply your own address range) -An example of an IPv6 CIDR: `fdXY:IJKL:MNOP:15::/64` (this shows the format but is not a valid address - see [RFC 4193](https://tools.ietf.org/html/rfc4193)) - +An example of an IPv6 CIDR: `fdXY:IJKL:MNOP:15::/64` (this shows the format but is not a valid +address - see [RFC 4193](https://tools.ietf.org/html/rfc4193)) {{< /note >}} ## Services You can create {{< glossary_tooltip text="Services" term_id="service" >}} which can use IPv4, IPv6, or both. -The address family of a Service defaults to the address family of the first service cluster IP range (configured via the `--service-cluster-ip-range` flag to the kube-apiserver). +The address family of a Service defaults to the address family of the first service cluster IP +range (configured via the `--service-cluster-ip-range` flag to the kube-apiserver). When you define a Service you can optionally configure it as dual stack. To specify the behavior you want, you set the `.spec.ipFamilyPolicy` field to one of the following values: -* `SingleStack`: Single-stack service. The control plane allocates a cluster IP for the Service, using the first configured service cluster IP range. +* `SingleStack`: Single-stack service. The control plane allocates a cluster IP for the Service, + using the first configured service cluster IP range. * `PreferDualStack`: * Allocates IPv4 and IPv6 cluster IPs for the Service. * `RequireDualStack`: Allocates Service `.spec.ClusterIPs` from both IPv4 and IPv6 address ranges. - * Selects the `.spec.ClusterIP` from the list of `.spec.ClusterIPs` based on the address family of the first element in the `.spec.ipFamilies` array. + * Selects the `.spec.ClusterIP` from the list of `.spec.ClusterIPs` based on the address family + of the first element in the `.spec.ipFamilies` array. -If you would like to define which IP family to use for single stack or define the order of IP families for dual-stack, you can choose the address families by setting an optional field, `.spec.ipFamilies`, on the Service. +If you would like to define which IP family to use for single stack or define the order of IP +families for dual-stack, you can choose the address families by setting an optional field, +`.spec.ipFamilies`, on the Service. {{< note >}} -The `.spec.ipFamilies` field is immutable because the `.spec.ClusterIP` cannot be reallocated on a Service that already exists. If you want to change `.spec.ipFamilies`, delete and recreate the Service. +The `.spec.ipFamilies` field is immutable because the `.spec.ClusterIP` cannot be reallocated on a +Service that already exists. If you want to change `.spec.ipFamilies`, delete and recreate the +Service. {{< /note >}} You can set `.spec.ipFamilies` to any of the following array values: @@ -109,139 +118,197 @@ These examples demonstrate the behavior of various dual-stack Service configurat #### Dual-stack options on new Services -1. This Service specification does not explicitly define `.spec.ipFamilyPolicy`. When you create this Service, Kubernetes assigns a cluster IP for the Service from the first configured `service-cluster-ip-range` and sets the `.spec.ipFamilyPolicy` to `SingleStack`. ([Services without selectors](/docs/concepts/services-networking/service/#services-without-selectors) and [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors will behave in this same way.) +1. This Service specification does not explicitly define `.spec.ipFamilyPolicy`. When you create + this Service, Kubernetes assigns a cluster IP for the Service from the first configured + `service-cluster-ip-range` and sets the `.spec.ipFamilyPolicy` to `SingleStack`. ([Services + without selectors](/docs/concepts/services-networking/service/#services-without-selectors) and + [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors + will behave in this same way.) -{{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} -1. This Service specification explicitly defines `PreferDualStack` in `.spec.ipFamilyPolicy`. When you create this Service on a dual-stack cluster, Kubernetes assigns both IPv4 and IPv6 addresses for the service. The control plane updates the `.spec` for the Service to record the IP address assignments. The field `.spec.ClusterIPs` is the primary field, and contains both assigned IP addresses; `.spec.ClusterIP` is a secondary field with its value calculated from `.spec.ClusterIPs`. +1. This Service specification explicitly defines `PreferDualStack` in `.spec.ipFamilyPolicy`. When + you create this Service on a dual-stack cluster, Kubernetes assigns both IPv4 and IPv6 + addresses for the service. The control plane updates the `.spec` for the Service to record the IP + address assignments. The field `.spec.ClusterIPs` is the primary field, and contains both assigned + IP addresses; `.spec.ClusterIP` is a secondary field with its value calculated from + `.spec.ClusterIPs`. - * For the `.spec.ClusterIP` field, the control plane records the IP address that is from the same address family as the first service cluster IP range. - * On a single-stack cluster, the `.spec.ClusterIPs` and `.spec.ClusterIP` fields both only list one address. - * On a cluster with dual-stack enabled, specifying `RequireDualStack` in `.spec.ipFamilyPolicy` behaves the same as `PreferDualStack`. + * For the `.spec.ClusterIP` field, the control plane records the IP address that is from the + same address family as the first service cluster IP range. + * On a single-stack cluster, the `.spec.ClusterIPs` and `.spec.ClusterIP` fields both only list + one address. + * On a cluster with dual-stack enabled, specifying `RequireDualStack` in `.spec.ipFamilyPolicy` + behaves the same as `PreferDualStack`. -{{< codenew file="service/networking/dual-stack-preferred-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-preferred-svc.yaml" >}} -1. This Service specification explicitly defines `IPv6` and `IPv4` in `.spec.ipFamilies` as well as defining `PreferDualStack` in `.spec.ipFamilyPolicy`. When Kubernetes assigns an IPv6 and IPv4 address in `.spec.ClusterIPs`, `.spec.ClusterIP` is set to the IPv6 address because that is the first element in the `.spec.ClusterIPs` array, overriding the default. +1. This Service specification explicitly defines `IPv6` and `IPv4` in `.spec.ipFamilies` as well + as defining `PreferDualStack` in `.spec.ipFamilyPolicy`. When Kubernetes assigns an IPv6 and + IPv4 address in `.spec.ClusterIPs`, `.spec.ClusterIP` is set to the IPv6 address because that is + the first element in the `.spec.ClusterIPs` array, overriding the default. -{{< codenew file="service/networking/dual-stack-preferred-ipfamilies-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-preferred-ipfamilies-svc.yaml" >}} #### Dual-stack defaults on existing Services -These examples demonstrate the default behavior when dual-stack is newly enabled on a cluster where Services already exist. (Upgrading an existing cluster to 1.21 or beyond will enable dual-stack.) +These examples demonstrate the default behavior when dual-stack is newly enabled on a cluster +where Services already exist. (Upgrading an existing cluster to 1.21 or beyond will enable +dual-stack.) -1. When dual-stack is enabled on a cluster, existing Services (whether `IPv4` or `IPv6`) are configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set `.spec.ipFamilies` to the address family of the existing Service. The existing Service cluster IP will be stored in `.spec.ClusterIPs`. +1. When dual-stack is enabled on a cluster, existing Services (whether `IPv4` or `IPv6`) are + configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set + `.spec.ipFamilies` to the address family of the existing Service. The existing Service cluster IP + will be stored in `.spec.ClusterIPs`. -{{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} You can validate this behavior by using kubectl to inspect an existing service. -```shell -kubectl get svc my-service -o yaml -``` + ```shell + kubectl get svc my-service -o yaml + ``` -```yaml -apiVersion: v1 -kind: Service -metadata: - labels: - app: MyApp - name: my-service -spec: - clusterIP: 10.0.197.123 - clusterIPs: - - 10.0.197.123 - ipFamilies: - - IPv4 - ipFamilyPolicy: SingleStack - ports: - - port: 80 - protocol: TCP - targetPort: 80 - selector: - app: MyApp - type: ClusterIP -status: - loadBalancer: {} -``` + ```yaml + apiVersion: v1 + kind: Service + metadata: + labels: + app: MyApp + name: my-service + spec: + clusterIP: 10.0.197.123 + clusterIPs: + - 10.0.197.123 + ipFamilies: + - IPv4 + ipFamilyPolicy: SingleStack + ports: + - port: 80 + protocol: TCP + targetPort: 80 + selector: + app: MyApp + type: ClusterIP + status: + loadBalancer: {} + ``` -1. When dual-stack is enabled on a cluster, existing [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors are configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set `.spec.ipFamilies` to the address family of the first service cluster IP range (configured via the `--service-cluster-ip-range` flag to the kube-apiserver) even though `.spec.ClusterIP` is set to `None`. +1. When dual-stack is enabled on a cluster, existing + [headless Services](/docs/concepts/services-networking/service/#headless-services) with selectors are + configured by the control plane to set `.spec.ipFamilyPolicy` to `SingleStack` and set + `.spec.ipFamilies` to the address family of the first service cluster IP range (configured via the + `--service-cluster-ip-range` flag to the kube-apiserver) even though `.spec.ClusterIP` is set to + `None`. -{{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} + {{< codenew file="service/networking/dual-stack-default-svc.yaml" >}} You can validate this behavior by using kubectl to inspect an existing headless service with selectors. -```shell -kubectl get svc my-service -o yaml -``` + ```shell + kubectl get svc my-service -o yaml + ``` -```yaml -apiVersion: v1 -kind: Service -metadata: - labels: - app: MyApp - name: my-service -spec: - clusterIP: None - clusterIPs: - - None - ipFamilies: - - IPv4 - ipFamilyPolicy: SingleStack - ports: - - port: 80 - protocol: TCP - targetPort: 80 - selector: - app: MyApp -``` + ```yaml + apiVersion: v1 + kind: Service + metadata: + labels: + app: MyApp + name: my-service + spec: + clusterIP: None + clusterIPs: + - None + ipFamilies: + - IPv4 + ipFamilyPolicy: SingleStack + ports: + - port: 80 + protocol: TCP + targetPort: 80 + selector: + app: MyApp + ``` #### Switching Services between single-stack and dual-stack Services can be changed from single-stack to dual-stack and from dual-stack to single-stack. -1. To change a Service from single-stack to dual-stack, change `.spec.ipFamilyPolicy` from `SingleStack` to `PreferDualStack` or `RequireDualStack` as desired. When you change this Service from single-stack to dual-stack, Kubernetes assigns the missing address family so that the Service now has IPv4 and IPv6 addresses. +1. To change a Service from single-stack to dual-stack, change `.spec.ipFamilyPolicy` from + `SingleStack` to `PreferDualStack` or `RequireDualStack` as desired. When you change this + Service from single-stack to dual-stack, Kubernetes assigns the missing address family so that the + Service now has IPv4 and IPv6 addresses. Edit the Service specification updating the `.spec.ipFamilyPolicy` from `SingleStack` to `PreferDualStack`. -Before: -```yaml -spec: - ipFamilyPolicy: SingleStack -``` -After: -```yaml -spec: - ipFamilyPolicy: PreferDualStack -``` + Before: -1. To change a Service from dual-stack to single-stack, change `.spec.ipFamilyPolicy` from `PreferDualStack` or `RequireDualStack` to `SingleStack`. When you change this Service from dual-stack to single-stack, Kubernetes retains only the first element in the `.spec.ClusterIPs` array, and sets `.spec.ClusterIP` to that IP address and sets `.spec.ipFamilies` to the address family of `.spec.ClusterIPs`. + ```yaml + spec: + ipFamilyPolicy: SingleStack + ``` + + After: + + ```yaml + spec: + ipFamilyPolicy: PreferDualStack + ``` + +1. To change a Service from dual-stack to single-stack, change `.spec.ipFamilyPolicy` from + `PreferDualStack` or `RequireDualStack` to `SingleStack`. When you change this Service from + dual-stack to single-stack, Kubernetes retains only the first element in the `.spec.ClusterIPs` + array, and sets `.spec.ClusterIP` to that IP address and sets `.spec.ipFamilies` to the address + family of `.spec.ClusterIPs`. ### Headless Services without selector -For [Headless Services without selectors](/docs/concepts/services-networking/service/#without-selectors) and without `.spec.ipFamilyPolicy` explicitly set, the `.spec.ipFamilyPolicy` field defaults to `RequireDualStack`. +For [Headless Services without selectors](/docs/concepts/services-networking/service/#without-selectors) +and without `.spec.ipFamilyPolicy` explicitly set, the `.spec.ipFamilyPolicy` field defaults to +`RequireDualStack`. ### Service type LoadBalancer To provision a dual-stack load balancer for your Service: - * Set the `.spec.type` field to `LoadBalancer` - * Set `.spec.ipFamilyPolicy` field to `PreferDualStack` or `RequireDualStack` + +* Set the `.spec.type` field to `LoadBalancer` +* Set `.spec.ipFamilyPolicy` field to `PreferDualStack` or `RequireDualStack` {{< note >}} -To use a dual-stack `LoadBalancer` type Service, your cloud provider must support IPv4 and IPv6 load balancers. +To use a dual-stack `LoadBalancer` type Service, your cloud provider must support IPv4 and IPv6 +load balancers. {{< /note >}} ## Egress traffic -If you want to enable egress traffic in order to reach off-cluster destinations (eg. the public Internet) from a Pod that uses non-publicly routable IPv6 addresses, you need to enable the Pod to use a publicly routed IPv6 address via a mechanism such as transparent proxying or IP masquerading. The [ip-masq-agent](https://github.com/kubernetes-sigs/ip-masq-agent) project supports IP masquerading on dual-stack clusters. +If you want to enable egress traffic in order to reach off-cluster destinations (eg. the public +Internet) from a Pod that uses non-publicly routable IPv6 addresses, you need to enable the Pod to +use a publicly routed IPv6 address via a mechanism such as transparent proxying or IP +masquerading. The [ip-masq-agent](https://github.com/kubernetes-sigs/ip-masq-agent) project +supports IP masquerading on dual-stack clusters. {{< note >}} Ensure your {{< glossary_tooltip text="CNI" term_id="cni" >}} provider supports IPv6. {{< /note >}} +## Windows support + +Kubernetes on Windows does not support single-stack "IPv6-only" networking. However, +dual-stack IPv4/IPv6 networking for pods and nodes with single-family services +is supported. + +You can use IPv4/IPv6 dual-stack networking with `l2bridge` networks. + +{{< note >}} +Overlay (VXLAN) networks on Windows **do not** support dual-stack networking. +{{< /note >}} + +You can read more about the different network modes for Windows within the +[Networking on Windows](/docs/concepts/services-networking/windows-networking#network-modes) topic. + ## {{% heading "whatsnext" %}} - * [Validate IPv4/IPv6 dual-stack](/docs/tasks/network/validate-dual-stack) networking -* [Enable dual-stack networking using kubeadm -](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/) +* [Enable dual-stack networking using kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/) + diff --git a/content/en/docs/concepts/services-networking/ingress.md b/content/en/docs/concepts/services-networking/ingress.md index da3ad5c6b4..aebcc800c6 100644 --- a/content/en/docs/concepts/services-networking/ingress.md +++ b/content/en/docs/concepts/services-networking/ingress.md @@ -30,23 +30,8 @@ For clarity, this guide defines the following terms: Traffic routing is controlled by rules defined on the Ingress resource. Here is a simple example where an Ingress sends all its traffic to one Service: -{{< mermaid >}} -graph LR; - client([client])-. Ingress-managed
    load balancer .->ingress[Ingress]; - ingress-->|routing rule|service[Service]; - subgraph cluster - ingress; - service-->pod1[Pod]; - service-->pod2[Pod]; - end - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class ingress,service,pod1,pod2 k8s; - class client plain; - class cluster cluster; -{{}} +{{< figure src="/docs/images/ingress.svg" alt="ingress-diagram" class="diagram-large" caption="Figure. Ingress" link="https://mermaid.live/edit#pako:eNqNkstuwyAQRX8F4U0r2VHqPlSRKqt0UamLqlnaWWAYJygYLB59KMm_Fxcix-qmGwbuXA7DwAEzzQETXKutof0Ovb4vaoUQkwKUu6pi3FwXM_QSHGBt0VFFt8DRU2OWSGrKUUMlVQwMmhVLEV1Vcm9-aUksiuXRaO_CEhkv4WjBfAgG1TrGaLa-iaUw6a0DcwGI-WgOsF7zm-pN881fvRx1UDzeiFq7ghb1kgqFWiElyTjnuXVG74FkbdumefEpuNuRu_4rZ1pqQ7L5fL6YQPaPNiFuywcG9_-ihNyUkm6YSONWkjVNM8WUIyaeOJLO3clTB_KhL8NQDmVe-OJjxgZM5FhFiiFTK5zjDkxHBQ9_4zB4a-x20EGNSZhyaKmXrg7f5hSsvufUwTMXThtMWiot5Jh6p9ffimHijIezaSVoeN0uiqcfMJvf7w" >}} An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting. An [Ingress controller](/docs/concepts/services-networking/ingress-controllers) is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic. @@ -398,25 +383,8 @@ A fanout configuration routes traffic from a single IP address to more than one based on the HTTP URI being requested. An Ingress allows you to keep the number of load balancers down to a minimum. For example, a setup like: -{{< mermaid >}} -graph LR; - client([client])-. Ingress-managed
    load balancer .->ingress[Ingress, 178.91.123.132]; - ingress-->|/foo|service1[Service service1:4200]; - ingress-->|/bar|service2[Service service2:8080]; - subgraph cluster - ingress; - service1-->pod1[Pod]; - service1-->pod2[Pod]; - service2-->pod3[Pod]; - service2-->pod4[Pod]; - end - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s; - class client plain; - class cluster cluster; -{{}} +{{< figure src="/docs/images/ingressFanOut.svg" alt="ingress-fanout-diagram" class="diagram-large" caption="Figure. Ingress Fan Out" link="https://mermaid.live/edit#pako:eNqNUslOwzAQ_RXLvYCUhMQpUFzUUzkgcUBwbHpw4klr4diR7bCo8O8k2FFbFomLPZq3jP00O1xpDpjijWHtFt09zAuFUCUFKHey8vf6NE7QrdoYsDZumGIb4Oi6NAskNeOoZJKpCgxK4oXwrFVgRyi7nCVXWZKRPMlysv5yD6Q4Xryf1Vq_WzDPooJs9egLNDbolKTpT03JzKgh3zWEztJZ0Niu9L-qZGcdmAMfj4cxvWmreba613z9C0B-AMQD-V_AdA-A4j5QZu0SatRKJhSqhZR0wjmPrDP6CeikrutQxy-Cuy2dtq9RpaU2dJKm6fzI5Glmg0VOLio4_5dLjx27hFSC015KJ2VZHtuQvY2fuHcaE43G0MaCREOow_FV5cMxHZ5-oPX75UM5avuXhXuOI9yAaZjg_aLuBl6B3RYaKDDtSw4166QrcKE-emrXcubghgunDaY1kxYizDqnH99UhakzHYykpWD9hjS--fEJoIELqQ" >}} + would require an Ingress such as: @@ -460,25 +428,7 @@ you are using, you may need to create a default-http-backend Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address. -{{< mermaid >}} -graph LR; - client([client])-. Ingress-managed
    load balancer .->ingress[Ingress, 178.91.123.132]; - ingress-->|Host: foo.bar.com|service1[Service service1:80]; - ingress-->|Host: bar.foo.com|service2[Service service2:80]; - subgraph cluster - ingress; - service1-->pod1[Pod]; - service1-->pod2[Pod]; - service2-->pod3[Pod]; - service2-->pod4[Pod]; - end - classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000; - classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; - classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5; - class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s; - class client plain; - class cluster cluster; -{{}} +{{< figure src="/docs/images/ingressNameBased.svg" alt="ingress-namebase-diagram" class="diagram-large" caption="Figure. Ingress Name Based Virtual hosting" link="https://mermaid.live/edit#pako:eNqNkl9PwyAUxb8KYS-atM1Kp05m9qSJJj4Y97jugcLtRqTQAPVPdN_dVlq3qUt8gZt7zvkBN7xjbgRgiteW1Rt0_zjLNUJcSdD-ZBn21WmcoDu9tuBcXDHN1iDQVWHnSBkmUMEU0xwsSuK5DK5l745QejFNLtMkJVmSZmT1Re9NcTz_uDXOU1QakxTMJtxUHw7ss-SQLhehQEODTsdH4l20Q-zFyc84-Y67pghv5apxHuweMuj9eS2_NiJdPhix-kMgvwQShOyYMNkJoEUYM3PuGkpUKyY1KqVSdCSEiJy35gnoqCzLvo5fpPAbOqlfI26UsXQ0Ho9nB5CnqesRGTnncPYvSqsdUvqp9KRdlI6KojjEkB0mnLgjDRONhqENBYm6oXbLV5V1y6S7-l42_LowlIN2uFm_twqOcAW2YlK0H_i9c-bYb6CCHNO2FFCyRvkc53rbWptaMA83QnpjMS2ZchBh1nizeNMcU28bGEzXkrV_pArN7Sc0rBTu" >}} The following Ingress tells the backing load balancer to route requests based on diff --git a/content/en/docs/concepts/services-networking/network-policies.md b/content/en/docs/concepts/services-networking/network-policies.md index dc4e2b76ca..9a97abfb5b 100644 --- a/content/en/docs/concepts/services-networking/network-policies.md +++ b/content/en/docs/concepts/services-networking/network-policies.md @@ -54,7 +54,7 @@ POSTing this to the API server for your cluster will have no effect unless your __Mandatory Fields__: As with all other Kubernetes config, a NetworkPolicy needs `apiVersion`, `kind`, and `metadata` fields. For general information about working with config files, see -[Configure Containers Using a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/), +[Configure a Pod to Use a ConfigMap](/docs/tasks/configure-pod-container/configure-pod-configmap/), and [Object Management](/docs/concepts/overview/working-with-objects/object-management). __spec__: NetworkPolicy [spec](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status) has all the information needed to define a particular network policy in the given namespace. @@ -258,7 +258,7 @@ standardized label to target a specific namespace. ## What you can't do with network policies (at least, not yet) -As of Kubernetes {{< skew latestVersion >}}, the following functionality does not exist in the NetworkPolicy API, but you might be able to implement workarounds using Operating System components (such as SELinux, OpenVSwitch, IPTables, and so on) or Layer 7 technologies (Ingress controllers, Service Mesh implementations) or admission controllers. In case you are new to network security in Kubernetes, its worth noting that the following User Stories cannot (yet) be implemented using the NetworkPolicy API. +As of Kubernetes {{< skew currentVersion >}}, the following functionality does not exist in the NetworkPolicy API, but you might be able to implement workarounds using Operating System components (such as SELinux, OpenVSwitch, IPTables, and so on) or Layer 7 technologies (Ingress controllers, Service Mesh implementations) or admission controllers. In case you are new to network security in Kubernetes, its worth noting that the following User Stories cannot (yet) be implemented using the NetworkPolicy API. - Forcing internal cluster traffic to go through a common gateway (this might be best served with a service mesh or other proxy). - Anything TLS related (use a service mesh or ingress controller for this). diff --git a/content/en/docs/concepts/services-networking/service-traffic-policy.md b/content/en/docs/concepts/services-networking/service-traffic-policy.md index b7a367a4b7..b9abe34b3f 100644 --- a/content/en/docs/concepts/services-networking/service-traffic-policy.md +++ b/content/en/docs/concepts/services-networking/service-traffic-policy.md @@ -60,12 +60,6 @@ considered. When the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `ServiceInternalTrafficPolicy` is enabled, `spec.internalTrafficPolicy` defaults to "Cluster". -## Constraints - -* Service Internal Traffic Policy is not used when `externalTrafficPolicy` is set - to `Local` on a Service. It is possible to use both features in the same cluster - on different Services, just not on the same Service. - ## {{% heading "whatsnext" %}} * Read about [Topology Aware Hints](/docs/concepts/services-networking/topology-aware-hints) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index 679f96b139..ad5af427ec 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -122,7 +122,7 @@ metadata: spec: containers: - name: nginx - image: nginx:11.14.2 + image: nginx:stable ports: - containerPort: 80 name: http-web-svc @@ -192,6 +192,7 @@ where it's running, by adding an Endpoints object manually: apiVersion: v1 kind: Endpoints metadata: + # the name here should match the name of the Service name: my-service subsets: - addresses: @@ -203,6 +204,10 @@ subsets: The name of the Endpoints object must be a valid [DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). +When you create an [Endpoints](/docs/reference/kubernetes-api/service-resources/endpoints-v1/) +object for a Service, you set the name of the new object to be the same as that +of the Service. + {{< note >}} The endpoint IPs _must not_ be: loopback (127.0.0.0/8 for IPv4, ::1/128 for IPv6), or link-local (169.254.0.0/16 and 224.0.0.0/24 for IPv4, fe80::/64 for IPv6). @@ -394,6 +399,10 @@ You can also set the maximum session sticky time by setting `service.spec.sessionAffinityConfig.clientIP.timeoutSeconds` appropriately. (the default value is 10800, which works out to be 3 hours). +{{< note >}} +On Windows, setting the maximum session sticky time for Services is not supported. +{{< /note >}} + ## Multi-Port Services For some Services, you need to expose more than one port. @@ -447,7 +456,7 @@ server will return a 422 HTTP status code to indicate that there's a problem. You can set the `spec.externalTrafficPolicy` field to control how traffic from external sources is routed. Valid values are `Cluster` and `Local`. Set the field to `Cluster` to route external traffic to all ready endpoints -and `Local` to only route to ready node-local endpoints. If the traffic policy is `Local` and there are are no node-local +and `Local` to only route to ready node-local endpoints. If the traffic policy is `Local` and there are no node-local endpoints, the kube-proxy does not forward any traffic for the relevant Service. {{< note >}} @@ -853,6 +862,17 @@ metadata: [...] ``` +{{% /tab %}} +{{% tab name="OCI" %}} + +```yaml +[...] +metadata: + name: my-service + annotations: + service.beta.kubernetes.io/oci-load-balancer-internal: true +[...] +``` {{% /tab %}} {{< /tabs >}} diff --git a/content/en/docs/concepts/services-networking/windows-networking.md b/content/en/docs/concepts/services-networking/windows-networking.md new file mode 100644 index 0000000000..6aa79f0a03 --- /dev/null +++ b/content/en/docs/concepts/services-networking/windows-networking.md @@ -0,0 +1,164 @@ +--- +reviewers: +- aravindhp +- jayunit100 +- jsturtevant +- marosset +title: Networking on Windows +content_type: concept +weight: 75 +--- + + + +Kubernetes supports running nodes on either Linux or Windows. You can mix both kinds of node +within a single cluster. +This page provides an overview to networking specific to the Windows operating system. + + +## Container networking on Windows {#networking} + +Networking for Windows containers is exposed through +[CNI plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/). +Windows containers function similarly to virtual machines in regards to +networking. Each container has a virtual network adapter (vNIC) which is connected +to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the +Host Compute Service (HCS) work together to create containers and attach container +vNICs to networks. HCS is responsible for the management of containers whereas HNS +is responsible for the management of networking resources such as: + +* Virtual networks (including creation of vSwitches) +* Endpoints / vNICs +* Namespaces +* Policies including packet encapsulations, load-balancing rules, ACLs, and NAT rules. + +The Windows HNS and vSwitch implement namespacing and can +create virtual NICs as needed for a pod or container. However, many configurations such +as DNS, routes, and metrics are stored in the Windows registry database rather than as +files inside `/etc`, which is how Linux stores those configurations. The Windows registry for the container +is separate from that of the host, so concepts like mapping `/etc/resolv.conf` from +the host into a container don't have the same effect they would on Linux. These must +be configured using Windows APIs run in the context of that container. Therefore +CNI implementations need to call the HNS instead of relying on file mappings to pass +network details into the pod or container. + +## Network modes + +Windows supports five different networking drivers/modes: L2bridge, L2tunnel, +Overlay (Beta), Transparent, and NAT. In a heterogeneous cluster with Windows and Linux +worker nodes, you need to select a networking solution that is compatible on both +Windows and Linux. The following table lists the out-of-tree plugins are supported on Windows, +with recommendations on when to use each CNI: + +| Network Driver | Description | Container Packet Modifications | Network Plugins | Network Plugin Characteristics | +| -------------- | ----------- | ------------------------------ | --------------- | ------------------------------ | +| L2bridge | Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress. | MAC is rewritten to host MAC, IP may be rewritten to host IP using HNS OutboundNAT policy. | [win-bridge](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-bridge), [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), Flannel host-gateway uses win-bridge | win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires user-defined routes (UDR) for inter-node connectivity. | +| L2Tunnel | This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied. | MAC rewritten, IP visible on the underlay network | [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) | Azure-CNI allows integration of containers with Azure vNET, and allows them to leverage the set of capabilities that [Azure Virtual Network provides](https://azure.microsoft.com/en-us/services/virtual-network/). For example, securely connect to Azure services or use Azure NSGs. See [azure-cni for some examples](https://docs.microsoft.com/azure/aks/concepts-network#azure-cni-advanced-networking) | +| Overlay | Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation. | Encapsulated with an outer header. | [win-overlay](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-overlay), Flannel VXLAN (uses win-overlay) | win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option requires [KB4489899](https://support.microsoft.com/help/4489899) on Windows Server 2019. | +| Transparent (special use case for [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes)) | Requires an external vSwitch. Containers are attached to an external vSwitch which enables intra-pod communication via logical networks (logical switches and routers). | Packet is encapsulated either via [GENEVE](https://datatracker.ietf.org/doc/draft-gross-geneve/) or [STT](https://datatracker.ietf.org/doc/draft-davie-stt/) tunneling to reach pods which are not on the same host.
    Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller.
    NAT is done for north-south communication. | [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes) | [Deploy via ansible](https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib). Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh. | +| NAT (*not used in Kubernetes*) | Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called [WinNAT](https://techcommunity.microsoft.com/t5/virtualization/windows-nat-winnat-capabilities-and-limitations/ba-p/382303) | MAC and IP is rewritten to host MAC/IP. | [nat](https://github.com/Microsoft/windows-container-networking/tree/master/plugins/nat) | Included here for completeness | + +As outlined above, the [Flannel](https://github.com/coreos/flannel) +[CNI plugin](https://github.com/flannel-io/cni-plugin) +is also [supported](https://github.com/flannel-io/cni-plugin#windows-support-experimental) on Windows via the +[VXLAN network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) (**Beta support** ; delegates to win-overlay) +and [host-gateway network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) (stable support; delegates to win-bridge). + +This plugin supports delegating to one of the reference CNI plugins (win-overlay, +win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for +automatic node subnet lease assignment and HNS network creation. This plugin reads +in its own configuration file (cni.conf), and aggregates it with the environment +variables from the FlannelD generated subnet.env file. It then delegates to one of +the reference CNI plugins for network plumbing, and sends the correct configuration +containing the node-assigned subnet to the IPAM plugin (for example: `host-local`). + +For Node, Pod, and Service objects, the following network flows are supported for +TCP/UDP traffic: + +* Pod → Pod (IP) +* Pod → Pod (Name) +* Pod → Service (Cluster IP) +* Pod → Service (PQDN, but only if there are no ".") +* Pod → Service (FQDN) +* Pod → external (IP) +* Pod → external (DNS) +* Node → Pod +* Pod → Node + +## IP address management (IPAM) {#ipam} + +The following IPAM options are supported on Windows: + +* [host-local](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local) +* [azure-vnet-ipam](https://github.com/Azure/azure-container-networking/blob/master/docs/ipam.md) (for azure-cni only) +* [Windows Server IPAM](https://docs.microsoft.com/windows-server/networking/technologies/ipam/ipam-top) (fallback option if no IPAM is set) + +## Load balancing and Services + +A Kubernetes {{< glossary_tooltip text="Service" term_id="service" >}} is an abstraction +that defines a logical set of Pods and a means to access them over a network. +In a cluster that includes Windows nodes, you can use the following types of Service: + +* `NodePort` +* `ClusterIP` +* `LoadBalancer` +* `ExternalName` + +Windows container networking differs in some important ways from Linux networking. +The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) +provides additional details and background. + +On Windows, you can use the following settings to configure Services and load +balancing behavior: + +{{< table caption="Windows Service Settings" >}} +| Feature | Description | Minimum Supported Windows OS build | How to enable | +| ------- | ----------- | -------------------------- | ------------- | +| Session affinity | Ensures that connections from a particular client are passed to the same Pod each time. | Windows Server 2022 | Set `service.spec.sessionAffinity` to "ClientIP" | +| Direct Server Return (DSR) | Load balancing mode where the IP address fixups and the LBNAT occurs at the container vSwitch port directly; service traffic arrives with the source IP set as the originating pod IP. | Windows Server 2019 | Set the following flags in kube-proxy: `--feature-gates="WinDSR=true" --enable-dsr=true` | +| Preserve-Destination | Skips DNAT of service traffic, thereby preserving the virtual IP of the target service in packets reaching the backend Pod. Also disables node-node forwarding. | Windows Server, version 1903 | Set `"preserve-destination": "true"` in service annotations and enable DSR in kube-proxy. | +| IPv4/IPv6 dual-stack networking | Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster | Windows Server 2019 | See [IPv4/IPv6 dual-stack](#ipv4ipv6-dual-stack) | +| Client IP preservation | Ensures that source IP of incoming ingress traffic gets preserved. Also disables node-node forwarding. | Windows Server 2019 | Set `service.spec.externalTrafficPolicy` to "Local" and enable DSR in kube-proxy | +{{< /table >}} + +{{< warning >}} +There are known issue with NodePort Services on overlay networking, if the destination node is running Windows Server 2022. +To avoid the issue entirely, you can configure the service with `externalTrafficPolicy: Local`. + +There are known issues with Pod to Pod connectivity on l2bridge network on Windows Server 2022 with KB5005619 or higher installed. +To workaround the issue and restore Pod to Pod connectivity, you can disable the WinDSR feature in kube-proxy. + +These issues require OS fixes. +Please follow https://github.com/microsoft/Windows-Containers/issues/204 for updates. +{{< /warning >}} + +## Limitations + +The following networking functionality is _not_ supported on Windows nodes: + +* Host networking mode +* Local NodePort access from the node itself (works for other nodes or external clients) +* More than 64 backend pods (or unique destination addresses) for a single Service +* IPv6 communication between Windows pods connected to overlay networks +* Local Traffic Policy in non-DSR mode +* Outbound communication using the ICMP protocol via the `win-overlay`, `win-bridge`, or using the Azure-CNI plugin.\ + Specifically, the Windows data plane ([VFP](https://www.microsoft.com/research/project/azure-virtual-filtering-platform/)) + doesn't support ICMP packet transpositions, and this means: + * ICMP packets directed to destinations within the same network (such as pod to pod communication via ping) + work as expected; + * TCP/UDP packets work as expected; + * ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping) + cannot be transposed and thus will not be routed back to their source; + * Since TCP/UDP packets can still be transposed, you can substitute `ping ` with + `curl ` when debugging connectivity with the outside world. + +Other limitations: + +* Windows reference network plugins win-bridge and win-overlay do not implement + [CNI spec](https://github.com/containernetworking/cni/blob/master/SPEC.md) v0.4.0, + due to a missing `CHECK` implementation. +* The Flannel VXLAN CNI plugin has the following limitations on Windows: + * Node-pod connectivity is only possible for local pods with Flannel v0.12.0 (or higher). + * Flannel is restricted to using VNI 4096 and UDP port 4789. See the official + [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) + backend docs for more details on these parameters. diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index 16f0b8ed3b..66aa8f7f8a 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -540,6 +540,15 @@ In the CLI, the access modes are abbreviated to: * RWX - ReadWriteMany * RWOP - ReadWriteOncePod +{{< note >}} +Kubernetes uses volume access modes to match PersistentVolumeClaims and PersistentVolumes. +In some cases, the volume access modes also constrain where the PersistentVolume can be mounted. +Volume access modes do **not** enforce write protection once the storage has been mounted. +Even if the access modes are specified as ReadWriteOnce, ReadOnlyMany, or ReadWriteMany, they don't set any constraints on the volume. +For example, even if a PersistentVolume is created as ReadOnlyMany, it is no guarantee that it will be read-only. +If the access modes are specified as ReadWriteOncePod, the volume is constrained and can be mounted on only a single Pod. +{{< /note >}} + > __Important!__ A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time. @@ -673,7 +682,7 @@ Claims use [the same convention as volumes](#volume-mode) to indicate the consum ### Resources -Claims, like Pods, can request specific quantities of a resource. In this case, the request is for storage. The same [resource model](https://git.k8s.io/community/contributors/design-proposals/scheduling/resources.md) applies to both volumes and claims. +Claims, like Pods, can request specific quantities of a resource. In this case, the request is for storage. The same [resource model](https://git.k8s.io/design-proposals-archive/scheduling/resources.md) applies to both volumes and claims. ### Selector @@ -1012,7 +1021,7 @@ and need persistent storage, it is recommended that you use the following patter * Learn more about [Creating a PersistentVolume](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolume). * Learn more about [Creating a PersistentVolumeClaim](/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolumeclaim). -* Read the [Persistent Storage design document](https://git.k8s.io/community/contributors/design-proposals/storage/persistent-storage.md). +* Read the [Persistent Storage design document](https://git.k8s.io/design-proposals-archive/storage/persistent-storage.md). ### API references {#reference} diff --git a/content/en/docs/concepts/storage/projected-volumes.md b/content/en/docs/concepts/storage/projected-volumes.md index ed7c31db14..df67132cf5 100644 --- a/content/en/docs/concepts/storage/projected-volumes.md +++ b/content/en/docs/concepts/storage/projected-volumes.md @@ -26,7 +26,7 @@ Currently, the following types of volume sources can be projected: * [`serviceAccountToken`](#serviceaccounttoken) All sources are required to be in the same namespace as the Pod. For more details, -see the [all-in-one volume](https://github.com/kubernetes/design-proposals-archive/blob/main/node/all-in-one-volume.md) design document. +see the [all-in-one volume](https://git.k8s.io/design-proposals-archive/node/all-in-one-volume.md) design document. ### Example configuration with a secret, a downwardAPI, and a configMap {#example-configuration-secret-downwardapi-configmap} diff --git a/content/en/docs/concepts/storage/storage-classes.md b/content/en/docs/concepts/storage/storage-classes.md index 53ee88a2e7..8fda0b2ff3 100644 --- a/content/en/docs/concepts/storage/storage-classes.md +++ b/content/en/docs/concepts/storage/storage-classes.md @@ -87,7 +87,7 @@ for provisioning PVs. This field must be specified. You are not restricted to specifying the "internal" provisioners listed here (whose names are prefixed with "kubernetes.io" and shipped alongside Kubernetes). You can also run and specify external provisioners, -which are independent programs that follow a [specification](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/volume-provisioning.md) +which are independent programs that follow a [specification](https://git.k8s.io/design-proposals-archive/storage/volume-provisioning.md) defined by Kubernetes. Authors of external provisioners have full discretion over where their code lives, how the provisioner is shipped, how it needs to be run, what volume plugin it uses (including Flex), etc. The repository diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index cc9a9565eb..fd994c97fa 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -64,7 +64,9 @@ a different volume. Kubernetes supports several types of volumes. -### awsElasticBlockStore {#awselasticblockstore} +### awsElasticBlockStore (deprecated) {#awselasticblockstore} + +{{< feature-state for_k8s_version="v1.17" state="deprecated" >}} An `awsElasticBlockStore` volume mounts an Amazon Web Services (AWS) [EBS volume](https://aws.amazon.com/ebs/) into your pod. Unlike @@ -115,7 +117,7 @@ spec: fsType: ext4 ``` -If the EBS volume is partitioned, you can supply the optional field `partition: ""` to specify which parition to mount on. +If the EBS volume is partitioned, you can supply the optional field `partition: ""` to specify which partition to mount on. #### AWS EBS CSI migration @@ -135,7 +137,9 @@ beta features must be enabled. To disable the `awsElasticBlockStore` storage plugin from being loaded by the controller manager and the kubelet, set the `InTreePluginAWSUnregister` flag to `true`. -### azureDisk {#azuredisk} +### azureDisk (deprecated) {#azuredisk} + +{{< feature-state for_k8s_version="v1.19" state="deprecated" >}} The `azureDisk` volume type mounts a Microsoft Azure [Data Disk](https://docs.microsoft.com/en-us/azure/aks/csi-storage-drivers) into a pod. @@ -158,7 +162,9 @@ must be installed on the cluster and the `CSIMigration` feature must be enabled. To disable the `azureDisk` storage plugin from being loaded by the controller manager and the kubelet, set the `InTreePluginAzureDiskUnregister` flag to `true`. -### azureFile {#azurefile} +### azureFile (deprecated) {#azurefile} + +{{< feature-state for_k8s_version="v1.21" state="deprecated" >}} The `azureFile` volume type mounts a Microsoft Azure File volume (SMB 2.1 and 3.0) into a pod. @@ -201,7 +207,9 @@ You must have your own Ceph server running with the share exported before you ca See the [CephFS example](https://github.com/kubernetes/examples/tree/master/volumes/cephfs/) for more details. -### cinder +### cinder (deprecated) {#cinder} + +{{< feature-state for_k8s_version="v1.18" state="deprecated" >}} {{< note >}} Kubernetes must be configured with the OpenStack cloud provider. @@ -295,15 +303,17 @@ keyed with `log_level`. ### downwardAPI {#downwardapi} -A `downwardAPI` volume makes downward API data available to applications. -It mounts a directory and writes the requested data in plain text files. +A `downwardAPI` volume makes {{< glossary_tooltip term_id="downward-api" text="downward API" >}} +data available to applications. Within the volume, you can find the exposed +data as read-only files in plain text format. {{< note >}} -A container using the downward API as a [`subPath`](#using-subpath) volume mount will not -receive downward API updates. +A container using the downward API as a [`subPath`](#using-subpath) volume mount does not +receive updates when field values change. {{< /note >}} -See the [downward API example](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) for more details. +See [Expose Pod Information to Containers Through Files](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) +to learn more. ### emptyDir {#emptydir} @@ -390,7 +400,9 @@ You must have your own Flocker installation running before you can use it. See the [Flocker example](https://github.com/kubernetes/examples/tree/master/staging/volumes/flocker) for more details. -### gcePersistentDisk +### gcePersistentDisk (deprecated) {#gcepersistentdisk} + +{{< feature-state for_k8s_version="v1.17" state="deprecated" >}} A `gcePersistentDisk` volume mounts a Google Compute Engine (GCE) [persistent disk](https://cloud.google.com/compute/docs/disks) (PD) into your Pod. @@ -1146,7 +1158,7 @@ to the [volume plugin FAQ](https://github.com/kubernetes/community/blob/master/s (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads. -Please read the [CSI design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) for more information. +Please read the [CSI design proposal](https://git.k8s.io/design-proposals-archive/storage/container-storage-interface.md) for more information. {{< note >}} Support for CSI spec versions 0.2 and 0.3 are deprecated in Kubernetes @@ -1166,8 +1178,8 @@ CSI driver. A `csi` volume can be used in a Pod in three different ways: * through a reference to a [PersistentVolumeClaim](#persistentvolumeclaim) -* with a [generic ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volume) -* with a [CSI ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) +* with a [generic ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes) +* with a [CSI ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volumes) if the driver supports that (beta feature) The following fields are available to storage administrators to configure a CSI @@ -1234,12 +1246,26 @@ You can set up your You can directly configure CSI volumes within the Pod specification. Volumes specified in this way are ephemeral and do not persist across pod restarts. See [Ephemeral -Volumes](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) +Volumes](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volumes) for more information. For more information on how to develop a CSI driver, refer to the [kubernetes-csi documentation](https://kubernetes-csi.github.io/docs/) +#### Windows CSI proxy + +{{< feature-state for_k8s_version="v1.22" state="stable" >}} + +CSI node plugins need to perform various privileged +operations like scanning of disk devices and mounting of file systems. These operations +differ for each host operating system. For Linux worker nodes, containerized CSI node +node plugins are typically deployed as privileged containers. For Windows worker nodes, +privileged operations for containerized CSI node plugins is supported using +[csi-proxy](https://github.com/kubernetes-csi/csi-proxy), a community-managed, +stand-alone binary that needs to be pre-installed on each Windows node. + +For more details, refer to the deployment guide of the CSI plugin you wish to deploy. + #### Migrating to CSI drivers from in-tree plugins {{< feature-state for_k8s_version="v1.17" state="beta" >}} @@ -1256,6 +1282,14 @@ provisioning/delete, attach/detach, mount/unmount and resizing of volumes. In-tree plugins that support `CSIMigration` and have a corresponding CSI driver implemented are listed in [Types of Volumes](#volume-types). +The following in-tree plugins support persistent storage on Windows nodes: + +* [`awsElasticBlockStore`](#awselasticblockstore) +* [`azureDisk`](#azuredisk) +* [`azureFile`](#azurefile) +* [`gcePersistentDisk`](#gcepersistentdisk) +* [`vsphereVolume`](#vspherevolume) + ### flexVolume {{< feature-state for_k8s_version="v1.23" state="deprecated" >}} @@ -1267,6 +1301,12 @@ volume plugin path on each node and in some cases the control plane nodes as wel Pods interact with FlexVolume drivers through the `flexVolume` in-tree volume plugin. For more details, see the FlexVolume [README](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-storage/flexvolume.md#readme) document. +The following FlexVolume [plugins](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows), +deployed as PowerShell scripts on the host, support Windows nodes: + +* [SMB](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~smb.cmd) +* [iSCSI](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~iscsi.cmd) + {{< note >}} FlexVolume is deprecated. Using an out-of-tree CSI driver is the recommended way to integrate external storage with Kubernetes. diff --git a/content/en/docs/concepts/storage/windows-storage.md b/content/en/docs/concepts/storage/windows-storage.md new file mode 100644 index 0000000000..b8f40177ca --- /dev/null +++ b/content/en/docs/concepts/storage/windows-storage.md @@ -0,0 +1,71 @@ +--- +reviewers: +- jingxu97 +- mauriciopoppe +- jayunit100 +- jsturtevant +- marosset +- aravindhp +title: Windows Storage +content_type: concept +--- + + + +This page provides an storage overview specific to the Windows operating system. + + + +## Persistent storage {#storage} + +Windows has a layered filesystem driver to mount container layers and create a copy +filesystem based on NTFS. All file paths in the container are resolved only within +the context of that container. + +* With Docker, volume mounts can only target a directory in the container, and not + an individual file. This limitation does not apply to containerd. +* Volume mounts cannot project files or directories back to the host filesystem. +* Read-only filesystems are not supported because write access is always required + for the Windows registry and SAM database. However, read-only volumes are supported. +* Volume user-masks and permissions are not available. Because the SAM is not shared + between the host & container, there's no mapping between them. All permissions are + resolved within the context of the container. + +As a result, the following storage functionality is not supported on Windows nodes: + +* Volume subpath mounts: only the entire volume can be mounted in a Windows container +* Subpath volume mounting for Secrets +* Host mount projection +* Read-only root filesystem (mapped volumes still support `readOnly`) +* Block device mapping +* Memory as the storage medium (for example, `emptyDir.medium` set to `Memory`) +* File system features like uid/gid; per-user Linux filesystem permissions +* Setting [secret permissions with DefaultMode](/docs/concepts/configuration/secret/#secret-files-permissions) (due to UID/GID dependency) +* NFS based storage/volume support +* Expanding the mounted volume (resizefs) + +Kubernetes {{< glossary_tooltip text="volumes" term_id="volume" >}} enable complex +applications, with data persistence and Pod volume sharing requirements, to be deployed +on Kubernetes. Management of persistent volumes associated with a specific storage +back-end or protocol includes actions such as provisioning/de-provisioning/resizing +of volumes, attaching/detaching a volume to/from a Kubernetes node and +mounting/dismounting a volume to/from individual containers in a pod that needs to +persist data. + +Volume management components are shipped as Kubernetes volume +[plugin](/docs/concepts/storage/volumes/#types-of-volumes). +The following broad classes of Kubernetes volume plugins are supported on Windows: + +* [`FlexVolume plugins`](/docs/concepts/storage/volumes/#flexVolume) + * Please note that FlexVolumes have been deprecated as of 1.23 +* [`CSI Plugins`](/docs/concepts/storage/volumes/#csi) + +##### In-tree volume plugins + +The following in-tree plugins support persistent storage on Windows nodes: + +* [`awsElasticBlockStore`](/docs/concepts/storage/volumes/#awselasticblockstore) +* [`azureDisk`](/docs/concepts/storage/volumes/#azuredisk) +* [`azureFile`](/docs/concepts/storage/volumes/#azurefile) +* [`gcePersistentDisk`](/docs/concepts/storage/volumes/#gcepersistentdisk) +* [`vsphereVolume`](/docs/concepts/storage/volumes/#vspherevolume) diff --git a/content/en/docs/setup/production-environment/windows/_index.md b/content/en/docs/concepts/windows/_index.md similarity index 100% rename from content/en/docs/setup/production-environment/windows/_index.md rename to content/en/docs/concepts/windows/_index.md diff --git a/content/en/docs/concepts/windows/intro.md b/content/en/docs/concepts/windows/intro.md new file mode 100644 index 0000000000..91e9412759 --- /dev/null +++ b/content/en/docs/concepts/windows/intro.md @@ -0,0 +1,387 @@ +--- +reviewers: +- jayunit100 +- jsturtevant +- marosset +- perithompson +title: Windows containers in Kubernetes +content_type: concept +weight: 65 +--- + + + +Windows applications constitute a large portion of the services and applications that +run in many organizations. [Windows containers](https://aka.ms/windowscontainers) +provide a way to encapsulate processes and package dependencies, making it easier +to use DevOps practices and follow cloud native patterns for Windows applications. + +Organizations with investments in Windows-based applications and Linux-based +applications don't have to look for separate orchestrators to manage their workloads, +leading to increased operational efficiencies across their deployments, regardless +of operating system. + + + +## Windows nodes in Kubernetes + +To enable the orchestration of Windows containers in Kubernetes, include Windows nodes +in your existing Linux cluster. Scheduling Windows containers in +{{< glossary_tooltip text="Pods" term_id="pod" >}} on Kubernetes is similar to +scheduling Linux-based containers. + +In order to run Windows containers, your Kubernetes cluster must include +multiple operating systems. +While you can only run the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} on Linux, +you can deploy worker nodes running either Windows or Linux. + +Windows {{< glossary_tooltip text="nodes" term_id="node" >}} are +[supported](#windows-os-version-support) provided that the operating system is +Windows Server 2019. + +This document uses the term *Windows containers* to mean Windows containers with +process isolation. Kubernetes does not support running Windows containers with +[Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container). + +## Compatibility and limitations {#limitations} + +Some node features are only available if you use a specific +[container runtime](#container-runtime); others are not available on Windows nodes, +including: + +* HugePages: not supported for Windows containers +* Privileged containers: not supported for Windows containers. + [HostProcess Containers](/docs/tasks/configure-pod-container/create-hostprocess-pod/) offer similar functionality. +* TerminationGracePeriod: requires containerD + +Not all features of shared namespaces are supported. See [API compatibility](#api) +for more details. + +See [Windows OS version compatibility](#windows-os-version-support) for details on +the Windows versions that Kubernetes is tested against. + +From an API and kubectl perspective, Windows containers behave in much the same +way as Linux-based containers. However, there are some notable differences in key +functionality which are outlined in this section. + +### Comparison with Linux {#compatibility-linux-similarities} + +Key Kubernetes elements work the same way in Windows as they do in Linux. This +section refers to several key workload abstractions and how they map to Windows. + +* [Pods](/docs/concepts/workloads/pods/) + + A Pod is the basic building block of Kubernetes–the smallest and simplest unit in + the Kubernetes object model that you create or deploy. You may not deploy Windows and + Linux containers in the same Pod. All containers in a Pod are scheduled onto a single + Node where each Node represents a specific platform and architecture. The following + Pod capabilities, properties and events are supported with Windows containers: + + * Single or multiple containers per Pod with process isolation and volume sharing + * Pod `status` fields + * Readiness, liveness, and startup probes + * postStart & preStop container lifecycle hooks + * ConfigMap, Secrets: as environment variables or volumes + * `emptyDir` volumes + * Named pipe host mounts + * Resource limits + * OS field: + + The `.spec.os.name` field should be set to `windows` to indicate that the current Pod uses Windows containers. + The `IdentifyPodOS` feature gate needs to be enabled for this field to be recognized. + + {{< note >}} + Starting from 1.24, the `IdentifyPodOS` feature gate is in Beta stage and defaults to be enabled. + {{< /note >}} + + If the `IdentifyPodOS` feature gate is enabled and you set the `.spec.os.name` field to `windows`, + you must not set the following fields in the `.spec` of that Pod: + + * `spec.hostPID` + * `spec.hostIPC` + * `spec.securityContext.seLinuxOptions` + * `spec.securityContext.seccompProfile` + * `spec.securityContext.fsGroup` + * `spec.securityContext.fsGroupChangePolicy` + * `spec.securityContext.sysctls` + * `spec.shareProcessNamespace` + * `spec.securityContext.runAsUser` + * `spec.securityContext.runAsGroup` + * `spec.securityContext.supplementalGroups` + * `spec.containers[*].securityContext.seLinuxOptions` + * `spec.containers[*].securityContext.seccompProfile` + * `spec.containers[*].securityContext.capabilities` + * `spec.containers[*].securityContext.readOnlyRootFilesystem` + * `spec.containers[*].securityContext.privileged` + * `spec.containers[*].securityContext.allowPrivilegeEscalation` + * `spec.containers[*].securityContext.procMount` + * `spec.containers[*].securityContext.runAsUser` + * `spec.containers[*].securityContext.runAsGroup` + + In the above list, wildcards (`*`) indicate all elements in a list. + For example, `spec.containers[*].securityContext` refers to the SecurityContext object + for all containers. If any of these fields is specified, the Pod will + not be admited by the API server. + +* [Workload resources](/docs/concepts/workloads/controllers/) including: + * ReplicaSet + * Deployment + * StatefulSet + * DaemonSet + * Job + * CronJob + * ReplicationController +* {{< glossary_tooltip text="Services" term_id="service" >}} + See [Load balancing and Services](#load-balancing-and-services) for more details. + +Pods, workload resources, and Services are critical elements to managing Windows +workloads on Kubernetes. However, on their own they are not enough to enable +the proper lifecycle management of Windows workloads in a dynamic cloud native +environment. + +* `kubectl exec` +* Pod and container metrics +* {{< glossary_tooltip text="Horizontal pod autoscaling" term_id="horizontal-pod-autoscaler" >}} +* {{< glossary_tooltip text="Resource quotas" term_id="resource-quota" >}} +* Scheduler preemption + +### Command line options for the kubelet {#kubelet-compatibility} + +Some kubelet command line options behave differently on Windows, as described below: + +* The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process + (see [CPU resource management](/docs/concepts/configuration/windows-resource-management/#resource-management-cpu)) +* The `--kube-reserved`, `--system-reserved` , and `--eviction-hard` flags update + [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +* Eviction by using `--enforce-node-allocable` is not implemented +* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented +* When running on a Windows node the kubelet does not have memory or CPU + restrictions. `--kube-reserved` and `--system-reserved` only subtract from `NodeAllocatable` + and do not guarantee resource provided for workloads. + See [Resource Management for Windows nodes](/docs/concepts/configuration/windows-resource-management/#resource-reservation) + for more information. +* The `MemoryPressure` Condition is not implemented +* The kubelet does not take OOM eviction actions + +### API compatibility {#api} + +There are subtle differences in the way the Kubernetes APIs work for Windows due to the OS +and container runtime. Some workload properties were designed for Linux, and fail to run on Windows. + +At a high level, these OS concepts are different: + +* Identity - Linux uses userID (UID) and groupID (GID) which + are represented as integer types. User and group names + are not canonical - they are just an alias in `/etc/groups` + or `/etc/passwd` back to UID+GID. Windows uses a larger binary + [security identifier](https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/security-identifiers) (SID) + which is stored in the Windows Security Access Manager (SAM) database. This + database is not shared between the host and containers, or between containers. +* File permissions - Windows uses an access control list based on (SIDs), whereas + POSIX systems such as Linux use a bitmask based on object permissions and UID+GID, + plus _optional_ access control lists. +* File paths - the convention on Windows is to use `\` instead of `/`. The Go IO + libraries typically accept both and just make it work, but when you're setting a + path or command line that's interpreted inside a container, `\` may be needed. +* Signals - Windows interactive apps handle termination differently, and can + implement one or more of these: + * A UI thread handles well-defined messages including `WM_CLOSE`. + * Console apps handle Ctrl-C or Ctrl-break using a Control Handler. + * Services register a Service Control Handler function that can accept + `SERVICE_CONTROL_STOP` control codes. + +Container exit codes follow the same convention where 0 is success, and nonzero is failure. +The specific error codes may differ across Windows and Linux. However, exit codes +passed from the Kubernetes components (kubelet, kube-proxy) are unchanged. + +#### Field compatibility for container specifications {#compatibility-v1-pod-spec-containers} + +The following list documents differences between how Pod container specifications +work between Windows and Linux: + +* Huge pages are not implemented in the Windows container + runtime, and are not available. They require [asserting a user + privilege](https://docs.microsoft.com/en-us/windows/desktop/Memory/large-page-support) + that's not configurable for containers. +* `requests.cpu` and `requests.memory` - requests are subtracted + from node available resources, so they can be used to avoid overprovisioning a + node. However, they cannot be used to guarantee resources in an overprovisioned + node. They should be applied to all containers as a best practice if the operator + wants to avoid overprovisioning entirely. +* `securityContext.allowPrivilegeEscalation` - + not possible on Windows; none of the capabilities are hooked up +* `securityContext.capabilities` - + POSIX capabilities are not implemented on Windows +* `securityContext.privileged` - + Windows doesn't support privileged containers +* `securityContext.procMount` - + Windows doesn't have a `/proc` filesystem +* `securityContext.readOnlyRootFilesystem` - + not possible on Windows; write access is required for registry & system + processes to run inside the container +* `securityContext.runAsGroup` - + not possible on Windows as there is no GID support +* `securityContext.runAsNonRoot` - + this setting will prevent containers from running as `ContainerAdministrator` + which is the closest equivalent to a root user on Windows. +* `securityContext.runAsUser` - + use [`runAsUserName`](/docs/tasks/configure-pod-container/configure-runasusername) + instead +* `securityContext.seLinuxOptions` - + not possible on Windows as SELinux is Linux-specific +* `terminationMessagePath` - + this has some limitations in that Windows doesn't support mapping single files. The + default value is `/dev/termination-log`, which does work because it does not + exist on Windows by default. + +#### Field compatibility for Pod specifications {#compatibility-v1-pod} + +The following list documents differences between how Pod specifications work between Windows and Linux: + +* `hostIPC` and `hostpid` - host namespace sharing is not possible on Windows +* `hostNetwork` - There is no Windows OS support to share the host network +* `dnsPolicy` - setting the Pod `dnsPolicy` to `ClusterFirstWithHostNet` is + not supported on Windows because host networking is not provided. Pods always + run with a container network. +* `podSecurityContext` (see below) +* `shareProcessNamespace` - this is a beta feature, and depends on Linux namespaces + which are not implemented on Windows. Windows cannot share process namespaces or + the container's root filesystem. Only the network can be shared. +* `terminationGracePeriodSeconds` - this is not fully implemented in Docker on Windows, + see the [GitHub issue](https://github.com/moby/moby/issues/25982). + The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT, + then Windows waits 5 seconds by default, and finally shuts down + all processes using the normal Windows shutdown behavior. The 5 + second default is actually in the Windows registry + [inside the container](https://github.com/moby/moby/issues/25982#issuecomment-426441183), + so it can be overridden when the container is built. +* `volumeDevices` - this is a beta feature, and is not implemented on Windows. + Windows cannot attach raw block devices to pods. +* `volumes` + * If you define an `emptyDir` volume, you cannot set its volume source to `memory`. +* You cannot enable `mountPropagation` for volume mounts as this is not + supported on Windows. + +#### Field compatibility for Pod security context {#compatibility-v1-pod-spec-containers-securitycontext} + +None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context) fields work on Windows. + +## Node problem detector + +The node problem detector (see +[Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/)) +has preliminary support for Windows. +For more information, visit the project's [GitHub page](https://github.com/kubernetes/node-problem-detector#windows). + +## Pause container + +In a Kubernetes Pod, an infrastructure or “pause” container is first created +to host the container. In Linux, the cgroups and namespaces that make up a pod +need a process to maintain their continued existence; the pause process provides +this. Containers that belong to the same pod, including infrastructure and worker +containers, share a common network endpoint (same IPv4 and / or IPv6 address, same +network port spaces). Kubernetes uses pause containers to allow for worker containers +crashing or restarting without losing any of the networking configuration. + +Kubernetes maintains a multi-architecture image that includes support for Windows. +For Kubernetes v{{< skew currentVersion >}} the recommended pause image is `k8s.gcr.io/pause:3.6`. +The [source code](https://github.com/kubernetes/kubernetes/tree/master/build/pause) +is available on GitHub. + +Microsoft maintains a different multi-architecture image, with Linux and Windows +amd64 support, that you can find as `mcr.microsoft.com/oss/kubernetes/pause:3.6`. +This image is built from the same source as the Kubernetes maintained image but +all of the Windows binaries are [authenticode signed](https://docs.microsoft.com/en-us/windows-hardware/drivers/install/authenticode) by Microsoft. +The Kubernetes project recommends using the Microsoft maintained image if you are +deploying to a production or production-like environment that requires signed +binaries. + +## Container runtimes {#container-runtime} + +You need to install a +{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} +into each node in the cluster so that Pods can run there. + +The following container runtimes work with Windows: + +{{% thirdparty-content %}} + +### ContainerD + +{{< feature-state for_k8s_version="v1.20" state="stable" >}} + +You can use {{< glossary_tooltip term_id="containerd" text="ContainerD" >}} 1.4.0+ +as the container runtime for Kubernetes nodes that run Windows. + +Learn how to [install ContainerD on a Windows node](/docs/setup/production-environment/container-runtimes/#install-containerd). + +{{< note >}} +There is a [known limitation](/docs/tasks/configure-pod-container/configure-gmsa/#gmsa-limitations) +when using GMSA with containerd to access Windows network shares, which requires a +kernel patch. +{{< /note >}} + +### Mirantis Container Runtime {#mcr} + +[Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) +is available as a container runtime for all Windows Server 2019 and later versions. + +See [Install MCR on Windows Servers](https://docs.mirantis.com/mcr/20.10/install/mcr-windows.html) for more information. + +## Windows OS version compatibility {#windows-os-version-support} + +On Windows nodes, strict compatibility rules apply where the host OS version must +match the container base image OS version. Only Windows containers with a container +operating system of Windows Server 2019 are fully supported. + +For Kubernetes v{{< skew currentVersion >}}, operating system compatibility for Windows nodes (and Pods) +is as follows: + +Windows Server LTSC release +: Windows Server 2019 +: Windows Server 2022 + +Windows Server SAC release +: Windows Server version 20H2 + +The Kubernetes [version-skew policy](/docs/setup/release/version-skew-policy/) also applies. + +## Getting help and troubleshooting {#troubleshooting} + +Your main source of help for troubleshooting your Kubernetes cluster should start +with the [Troubleshooting](/docs/tasks/debug/) +page. + +Some additional, Windows-specific troubleshooting help is included +in this section. Logs are an important element of troubleshooting +issues in Kubernetes. Make sure to include them any time you seek +troubleshooting assistance from other contributors. Follow the +instructions in the +SIG Windows [contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs). + +### Reporting issues and feature requests + +If you have what looks like a bug, or you would like to +make a feature request, please follow the [SIG Windows contributing guide](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#reporting-issues-and-feature-requests) to create a new issue. +You should first search the list of issues in case it was +reported previously and comment with your experience on the issue and add additional +logs. SIG Windows channel on the Kubernetes Slack is also a great avenue to get some initial support and +troubleshooting ideas prior to creating a ticket. + +## Deployment tools + +The kubeadm tool helps you to deploy a Kubernetes cluster, providing the control +plane to manage the cluster it, and nodes to run your workloads. +[Adding Windows nodes](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/) +explains how to deploy Windows nodes to your cluster using kubeadm. + +The Kubernetes [cluster API](https://cluster-api.sigs.k8s.io/) project also provides means to automate deployment of Windows nodes. + +## Windows distribution channels + +For a detailed explanation of Windows distribution channels see the +[Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). + +Information on the different Windows Server servicing channels +including their support models can be found at +[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started/servicing-channels-comparison). diff --git a/content/en/docs/setup/production-environment/windows/user-guide-windows-containers.md b/content/en/docs/concepts/windows/user-guide.md similarity index 79% rename from content/en/docs/setup/production-environment/windows/user-guide-windows-containers.md rename to content/en/docs/concepts/windows/user-guide.md index 7177a4aa05..450a1bb5e0 100644 --- a/content/en/docs/setup/production-environment/windows/user-guide-windows-containers.md +++ b/content/en/docs/concepts/windows/user-guide.md @@ -3,7 +3,6 @@ reviewers: - jayunit100 - jsturtevant - marosset -- perithompson title: Guide for scheduling Windows containers in Kubernetes content_type: concept weight: 75 @@ -11,31 +10,29 @@ weight: 75 -Windows applications constitute a large portion of the services and applications that run in many organizations. -This guide walks you through the steps to configure and deploy a Windows container in Kubernetes. - - +Windows applications constitute a large portion of the services and applications that run in many organizations. +This guide walks you through the steps to configure and deploy Windows containers in Kubernetes. ## Objectives * Configure an example deployment to run Windows containers on the Windows node -* (Optional) Configure an Active Directory Identity for your Pod using Group Managed Service Accounts (GMSA) +* Highlight Windows specific funcationality in Kubernetes ## Before you begin -* Create a Kubernetes cluster that includes a +* Create a Kubernetes cluster that includes a control plane and a [worker node running Windows Server](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/) -* It is important to note that creating and deploying services and workloads on Kubernetes -behaves in much the same way for Linux and Windows containers. -[Kubectl commands](/docs/reference/kubectl/) to interface with the cluster are identical. +* It is important to note that creating and deploying services and workloads on Kubernetes +behaves in much the same way for Linux and Windows containers. +[Kubectl commands](/docs/reference/kubectl/) to interface with the cluster are identical. The example in the section below is provided to jumpstart your experience with Windows containers. ## Getting Started: Deploying a Windows container -To deploy a Windows container on Kubernetes, you must first create an example application. -The example YAML file below creates a simple webserver application. +The example YAML file below deploys a simple webserver application running inside a Windows container. + Create a service spec named `win-webserver.yaml` with the contents below: ```yaml @@ -83,8 +80,8 @@ spec: ``` {{< note >}} -Port mapping is also supported, but for simplicity in this example -the container port 80 is exposed directly to the service. +Port mapping is also supported, but for simplicity this example exposes +port 80 of the container directly to the Service. {{< /note >}} 1. Check that all nodes are healthy: @@ -104,20 +101,19 @@ the container port 80 is exposed directly to the service. 1. Check that the deployment succeeded. To verify: - * Two containers per pod on the Windows node, use `docker ps` - * Two pods listed from the Linux control plane node, use `kubectl get pods` - * Node-to-pod communication across the network, `curl` port 80 of your pod IPs from the Linux control plane node + * Two pods listed from the Linux control plane node, use `kubectl get pods` + * Node-to-pod communication across the network, `curl` port 80 of your pod IPs from the Linux control plane node to check for a web server response - * Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) + * Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) using docker exec or kubectl exec - * Service-to-pod communication, `curl` the virtual service IP (seen under `kubectl get services`) + * Service-to-pod communication, `curl` the virtual service IP (seen under `kubectl get services`) from the Linux control plane node and from individual pods * Service discovery, `curl` the service name with the Kubernetes [default DNS suffix](/docs/concepts/services-networking/dns-pod-service/#services) * Inbound connectivity, `curl` the NodePort from the Linux control plane node or machines outside of the cluster * Outbound connectivity, `curl` external IPs from inside the pod using kubectl exec {{< note >}} -Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. +Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. Only Windows pods are able to access service IPs. {{< /note >}} @@ -125,43 +121,43 @@ Only Windows pods are able to access service IPs. ### Capturing logs from workloads -Logs are an important element of observability; they enable users to gain insights -into the operational aspect of workloads and are a key ingredient to troubleshooting issues. -Because Windows containers and workloads inside Windows containers behave differently from Linux containers, -users had a hard time collecting logs, limiting operational visibility. -Windows workloads for example are usually configured to log to ETW (Event Tracing for Windows) -or push entries to the application event log. -[LogMonitor](https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor), an open source tool by Microsoft, -is the recommended way to monitor configured log sources inside a Windows container. -LogMonitor supports monitoring event logs, ETW providers, and custom application logs, +Logs are an important element of observability; they enable users to gain insights +into the operational aspect of workloads and are a key ingredient to troubleshooting issues. +Because Windows containers and workloads inside Windows containers behave differently from Linux containers, +users had a hard time collecting logs, limiting operational visibility. +Windows workloads for example are usually configured to log to ETW (Event Tracing for Windows) +or push entries to the application event log. +[LogMonitor](https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor), an open source tool by Microsoft, +is the recommended way to monitor configured log sources inside a Windows container. +LogMonitor supports monitoring event logs, ETW providers, and custom application logs, piping them to STDOUT for consumption by `kubectl logs `. -Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files +Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files to all your containers and add the necessary entrypoints for LogMonitor to push your logs to STDOUT. -## Using configurable Container usernames +## Configuring container user -Starting with Kubernetes v1.16, Windows containers can be configured to run their entrypoints and processes -with different usernames than the image defaults. -The way this is achieved is a bit different from the way it is done for Linux containers. +### Using configurable Container usernames + +Windows containers can be configured to run their entrypoints and processes +with different usernames than the image defaults. Learn more about it [here](/docs/tasks/configure-pod-container/configure-runasusername/). -## Managing Workload Identity with Group Managed Service Accounts +### Managing Workload Identity with Group Managed Service Accounts -Starting with Kubernetes v1.14, Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). -Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, -simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. -Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. +Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). +Group Managed Service Accounts are a specific type of Active Directory account that provide automatic password management, +simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. +Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. Learn more about configuring and using GMSA for Windows containers [here](/docs/tasks/configure-pod-container/configure-gmsa/). ## Taints and Tolerations -Users today need to use some combination of taints and node selectors in order to -keep Linux and Windows workloads on their respective OS-specific nodes. -This likely imposes a burden only on Windows users. The recommended approach is outlined below, +Users need to use some combination of taints and node selectors in order to +schedule Linux and Windows workloads to their respective OS-specific nodes. +The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads. - If the `IdentifyPodOS` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled, you can (and should) set `.spec.os.name` for a Pod to indicate the operating system that the containers in that Pod are designed for. For Pods that run Linux containers, set @@ -184,26 +180,26 @@ so taints and tolerations and node selectors are still required ### Ensuring OS-specific workloads land on the appropriate container host -Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations. +Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations. All Kubernetes nodes today have the following default labels: * kubernetes.io/os = [windows|linux] * kubernetes.io/arch = [amd64|arm64|...] -If a Pod specification does not specify a nodeSelector like `"kubernetes.io/os": windows`, -it is possible the Pod can be scheduled on any host, Windows or Linux. -This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. +If a Pod specification does not specify a nodeSelector like `"kubernetes.io/os": windows`, +it is possible the Pod can be scheduled on any host, Windows or Linux. +This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice is to use a nodeSelector. -However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, -as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. -In those situations, you may be hesitant to make the configuration change to add nodeSelectors. -The alternative is to use Taints. Because the kubelet can set Taints during registration, +However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, +as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. +In those situations, you may be hesitant to make the configuration change to add nodeSelectors. +The alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only. For example: `--register-with-taints='os=windows:NoSchedule'` -By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). +By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector and the appropriate matching toleration to choose Windows. @@ -223,26 +219,24 @@ tolerations: The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows Server versions in the same cluster, then you should set additional node labels and nodeSelectors. -Kubernetes 1.17 automatically adds a new label `node.kubernetes.io/windows-build` to simplify this. +Kubernetes 1.17 automatically adds a new label `node.kubernetes.io/windows-build` to simplify this. If you're running an older version, then it's recommended to add this label manually to Windows nodes. -This label reflects the Windows major, minor, and build number that need to match for compatibility. +This label reflects the Windows major, minor, and build number that need to match for compatibility. Here are values used today for each Windows Server version. | Product Name | Build Number(s) | |--------------------------------------|------------------------| | Windows Server 2019 | 10.0.17763 | -| Windows Server version 1809 | 10.0.17763 | -| Windows Server version 1903 | 10.0.18362 | - +| Windows Server, Version 20H2 | 10.0.19042 | +| Windows Server 2022 | 10.0.20348 | ### Simplifying with RuntimeClass -[RuntimeClass] can be used to simplify the process of using taints and tolerations. +[RuntimeClass] can be used to simplify the process of using taints and tolerations. A cluster administrator can create a `RuntimeClass` object which is used to encapsulate these taints and tolerations. - -1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector` +1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector` for the Windows OS, architecture, and version. ```yaml @@ -313,7 +307,4 @@ spec: app: iis-2019 ``` - - - [RuntimeClass]: https://kubernetes.io/docs/concepts/containers/runtime-class/ diff --git a/content/en/docs/concepts/workloads/controllers/job.md b/content/en/docs/concepts/workloads/controllers/job.md index 74a232831a..9d917505de 100644 --- a/content/en/docs/concepts/workloads/controllers/job.md +++ b/content/en/docs/concepts/workloads/controllers/job.md @@ -51,13 +51,8 @@ job.batch/pi created Check on the status of the Job with `kubectl`: -```shell -kubectl describe jobs/pi -``` - -The output is similar to this: - -``` +{{< tabs name="Check status of Job" >}} +{{< tab name="kubectl describe job pi" codelang="bash" >}} Name: pi Namespace: default Selector: controller-uid=c9948307-e56d-4b5d-8302-ae2d7b7da67c @@ -91,7 +86,62 @@ Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 14m job-controller Created pod: pi-5rwd7 -``` +{{< /tab >}} +{{< tab name="kubectl get job pi -o yaml" codelang="bash" >}} +apiVersion: batch/v1 +kind: Job +metadata: + annotations: + kubectl.kubernetes.io/last-applied-configuration: | + {"apiVersion":"batch/v1","kind":"Job","metadata":{"annotations":{},"name":"pi","namespace":"default"},"spec":{"backoffLimit":4,"template":{"spec":{"containers":[{"command":["perl","-Mbignum=bpi","-wle","print bpi(2000)"],"image":"perl","name":"pi"}],"restartPolicy":"Never"}}}} + creationTimestamp: "2022-06-15T08:40:15Z" + generation: 1 + labels: + controller-uid: 863452e6-270d-420e-9b94-53a54146c223 + job-name: pi + name: pi + namespace: default + resourceVersion: "987" + uid: 863452e6-270d-420e-9b94-53a54146c223 +spec: + backoffLimit: 4 + completionMode: NonIndexed + completions: 1 + parallelism: 1 + selector: + matchLabels: + controller-uid: 863452e6-270d-420e-9b94-53a54146c223 + suspend: false + template: + metadata: + creationTimestamp: null + labels: + controller-uid: 863452e6-270d-420e-9b94-53a54146c223 + job-name: pi + spec: + containers: + - command: + - perl + - -Mbignum=bpi + - -wle + - print bpi(2000) + image: perl + imagePullPolicy: Always + name: pi + resources: {} + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + dnsPolicy: ClusterFirst + restartPolicy: Never + schedulerName: default-scheduler + securityContext: {} + terminationGracePeriodSeconds: 30 +status: + active: 1 + ready: 1 + startTime: "2022-06-15T08:40:15Z" +{{< /tab >}} +{{< /tabs >}} To view completed Pods of a Job, use `kubectl get pods`. @@ -119,7 +169,7 @@ kubectl logs $pods The output is similar to this: -```shell +``` 3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901 ``` @@ -253,9 +303,19 @@ due to a logical error in configuration etc. To do so, set `.spec.backoffLimit` to specify the number of retries before considering a Job as failed. The back-off limit is set by default to 6. Failed Pods associated with the Job are recreated by the Job controller with an -exponential back-off delay (10s, 20s, 40s ...) capped at six minutes. The -back-off count is reset when a Job's Pod is deleted or successful without any -other Pods for the Job failing around that time. +exponential back-off delay (10s, 20s, 40s ...) capped at six minutes. + +The number of retries is calculated in two ways: +- The number of Pods with `.status.phase = "Failed"`. +- When using `restartPolicy = "OnFailure"`, the number of retries in all the + containers of Pods with `.status.phase` equal to `Pending` or `Running`. + +If either of the calculations reaches the `.spec.backoffLimit`, the Job is +considered failed. + +When the [`JobTrackingWithFinalizers`](#job-tracking-with-finalizers) feature is +disabled, the number of failed Pods is only based on Pods that are still present +in the API. {{< note >}} If your job has `restartPolicy = "OnFailure"`, keep in mind that your Pod running the Job @@ -405,7 +465,7 @@ The pattern names are also links to examples and more detailed description. | ----------------------------------------- |:-----------------:|:---------------------------:|:-------------------:| | [Queue with Pod Per Work Item] | ✓ | | sometimes | | [Queue with Variable Pod Count] | ✓ | ✓ | | -| [Indexed Job with Static Work Assignment] | ✓ | | ✓ | +| [Indexed Job with Static Work Assignment] | ✓ | | ✓ | | [Job Template Expansion] | | | ✓ | When you specify completions with `.spec.completions`, each Pod created by the Job controller @@ -631,7 +691,6 @@ In order to use this behavior, you must enable the `JobTrackingWithFinalizers` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/) and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/). -It is enabled by default. When enabled, the control plane tracks new Jobs using the behavior described below. Jobs created before the feature was enabled are unaffected. As a user, diff --git a/content/en/docs/concepts/workloads/controllers/replicaset.md b/content/en/docs/concepts/workloads/controllers/replicaset.md index 8024e6246a..470a5e5024 100644 --- a/content/en/docs/concepts/workloads/controllers/replicaset.md +++ b/content/en/docs/concepts/workloads/controllers/replicaset.md @@ -78,7 +78,7 @@ kubectl describe rs/frontend And you will see output similar to: -```shell +``` Name: frontend Namespace: default Selector: tier=frontend @@ -130,7 +130,7 @@ kubectl get pods frontend-b2zdv -o yaml The output will look similar to this, with the frontend ReplicaSet's info set in the metadata's ownerReferences field: -```shell +```yaml apiVersion: v1 kind: Pod metadata: @@ -181,7 +181,7 @@ kubectl get pods The output shows that the new Pods are either already terminated, or in the process of being terminated: -```shell +``` NAME READY STATUS RESTARTS AGE frontend-b2zdv 1/1 Running 0 10m frontend-vcmts 1/1 Running 0 10m @@ -210,7 +210,7 @@ kubectl get pods ``` Will reveal in its output: -```shell +``` NAME READY STATUS RESTARTS AGE frontend-hmmj2 1/1 Running 0 9s pod1 1/1 Running 0 36s @@ -387,7 +387,7 @@ As such, it is recommended to use Deployments when you want ReplicaSets. ### Bare Pods -Unlike the case where a user directly created Pods, a ReplicaSet replaces Pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your application requires only a single Pod. Think of it similarly to a process supervisor, only it supervises multiple Pods across multiple nodes instead of individual processes on a single node. A ReplicaSet delegates local container restarts to some agent on the node (for example, Kubelet or Docker). +Unlike the case where a user directly created Pods, a ReplicaSet replaces Pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your application requires only a single Pod. Think of it similarly to a process supervisor, only it supervises multiple Pods across multiple nodes instead of individual processes on a single node. A ReplicaSet delegates local container restarts to some agent on the node such as Kubelet. ### Job diff --git a/content/en/docs/concepts/workloads/pods/downward-api.md b/content/en/docs/concepts/workloads/pods/downward-api.md new file mode 100644 index 0000000000..fcee383c45 --- /dev/null +++ b/content/en/docs/concepts/workloads/pods/downward-api.md @@ -0,0 +1,131 @@ +--- +title: Downward API +content_type: concept +description: > + There are two ways to expose Pod and container fields to a running container: + environment variables, and as files that are populated by a special volume type. + Together, these two ways of exposing Pod and container fields are called the downward API. +--- + + + +It is sometimes useful for a container to have information about itself, without +being overly coupled to Kubernetes. The _downward API_ allows containers to consume +information about themselves or the cluster without using the Kubernetes client +or API server. + +An example is an existing application that assumes a particular well-known +environment variable holds a unique identifier. One possibility is to wrap the +application, but that is tedious and error-prone, and it violates the goal of low +coupling. A better option would be to use the Pod's name as an identifier, and +inject the Pod's name into the well-known environment variable. + +In Kubernetes, there are two ways to expose Pod and container fields to a running container: + +* as [environment variables](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api) +* as [files in a `downwardAPI` volume](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) + +Together, these two ways of exposing Pod and container fields are called the +_downward API_. + + + +## Available fields + +Only some Kubernetes API fields are available through the downward API. This +section lists which fields you can make available. + +You can pass information from available Pod-level fields using `fieldRef`. +At the API level, the `spec` for a Pod always defines at least one +[Container](/docs/reference/kubernetes-api/workload-resources/pod-v1/#Container). +You can pass information from available Container-level fields using +`resourceFieldRef`. + +### Information available via `fieldRef` {#downwardapi-fieldRef} + +For most Pod-level fields, you can provide them to a container either as +an environment variable or using a `downwardAPI` volume. The fields available +via either mechanism are: + +`metadata.name` +: the pod's name + +`metadata.namespace` +: the pod's {{< glossary_tooltip text="namespace" term_id="namespace" >}} + +`metadata.uid` +: the pod's unique ID + +`metadata.annotations['']` +: the value of the pod's {{< glossary_tooltip text="annotation" term_id="annotation" >}} named `` (for example, `metadata.annotations['myannotation']`) + +`metadata.labels['']` +: the text value of the pod's {{< glossary_tooltip text="label" term_id="label" >}} named `` (for example, `metadata.labels['mylabel']`) + +`spec.serviceAccountName` +: the name of the pod's {{< glossary_tooltip text="service account" term_id="service-account" >}} + +`spec.nodeName` +: the name of the {{< glossary_tooltip term_id="node" text="node">}} where the Pod is executing + +`status.hostIP` +: the primary IP address of the node to which the Pod is assigned + +`status.podIP` +: the pod's primary IP address (usually, its IPv4 address) + +In addition, the following information is available through +a `downwardAPI` volume `fieldRef`, but **not as environment variables**: + +`metadata.labels` +: all of the pod's labels, formatted as `label-key="escaped-label-value"` with one label per line + +`metadata.annotations` +: all of the pod's annotations, formatted as `annotation-key="escaped-annotation-value"` with one annotation per line + +### Information available via `resourceFieldRef` {#downwardapi-resourceFieldRef} + +These container-level fields allow you to provide information about +[requests and limits](/docs/concepts/configuration/manage-resources-containers/#requests-and-limits) +for resources such as CPU and memory. + + +`resource: limits.cpu` +: A container's CPU limit + +`resource: requests.cpu` +: A container's CPU request + +`resource: limits.memory` +: A container's memory limit + +`resource: requests.memory` +: A container's memory request + +`resource: limits.hugepages-*` +: A container's hugepages limit (provided that the `DownwardAPIHugePages` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled) + +`resource: requests.hugepages-*` +: A container's hugepages request (provided that the `DownwardAPIHugePages` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled) + +`resource: limits.ephemeral-storage` +: A container's ephemeral-storage limit + +`resource: requests.ephemeral-storage` +: A container's ephemeral-storage request + +#### Fallback information for resource limits + +If CPU and memory limits are not specified for a container, and you use the +downward API to try to expose that information, then the +kubelet defaults to exposing the maximum allocatable value for CPU and memory +based on the [node allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) +calculation. + +## {{% heading "whatsnext" %}} + +You can read about [`downwardAPI` volumes](/docs/concepts/storage/volumes/#downwardapi). + +You can try using the downward API to expose container- or Pod-level information: +* as [environment variables](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api) +* as [files in `downwardAPI` volume](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) diff --git a/content/en/docs/concepts/workloads/pods/init-containers.md b/content/en/docs/concepts/workloads/pods/init-containers.md index 371ef58dac..61b90d17d0 100644 --- a/content/en/docs/concepts/workloads/pods/init-containers.md +++ b/content/en/docs/concepts/workloads/pods/init-containers.md @@ -334,4 +334,3 @@ Kubernetes, consult the documentation for the version you are using. * Read about [creating a Pod that has an init container](/docs/tasks/configure-pod-container/configure-pod-initialization/#create-a-pod-that-has-an-init-container) * Learn how to [debug init containers](/docs/tasks/debug/debug-application/debug-init-containers/) - diff --git a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md b/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md index 34ca33d6b1..fe50cf7e2d 100644 --- a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md +++ b/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md @@ -64,6 +64,7 @@ metadata: spec: topologySpreadConstraints: - maxSkew: + minDomains: topologyKey: whenUnsatisfiable: labelSelector: @@ -143,7 +144,7 @@ If we want an incoming Pod to be evenly spread with existing Pods across zones, `topologyKey: zone` implies the even distribution will only be applied to the nodes which have label pair "zone:<any value>" present. `whenUnsatisfiable: DoNotSchedule` tells the scheduler to let it stay pending if the incoming Pod can't satisfy the constraint. -If the scheduler placed this incoming Pod into "zoneA", the Pods distribution would become [3, 1], hence the actual skew is 2 (3 - 1) - which violates `maxSkew: 1`. In this example, the incoming Pod can only be placed onto "zoneB": +If the scheduler placed this incoming Pod into "zoneA", the Pods distribution would become [3, 1], hence the actual skew is 2 (3 - 1) - which violates `maxSkew: 1`. In this example, the incoming Pod can only be placed into "zoneB": {{}} graph BT @@ -188,7 +189,7 @@ graph BT You can tweak the Pod spec to meet various kinds of requirements: -- Change `maxSkew` to a bigger value like "2" so that the incoming Pod can be placed onto "zoneA" as well. +- Change `maxSkew` to a bigger value like "2" so that the incoming Pod can be placed into "zoneA" as well. - Change `topologyKey` to "node" so as to distribute the Pods evenly across nodes instead of zones. In the above example, if `maxSkew` remains "1", the incoming Pod can only be placed onto "node4". - Change `whenUnsatisfiable: DoNotSchedule` to `whenUnsatisfiable: ScheduleAnyway` to ensure the incoming Pod to be always schedulable (suppose other scheduling APIs are satisfied). However, it's preferred to be placed onto the topology domain which has fewer matching Pods. (Be aware that this preferability is jointly normalized with other internal scheduling priorities like resource usage ratio, etc.) @@ -219,7 +220,7 @@ You can use 2 TopologySpreadConstraints to control the Pods spreading on both zo {{< codenew file="pods/topology-spread-constraints/two-constraints.yaml" >}} -In this case, to match the first constraint, the incoming Pod can only be placed onto "zoneB"; while in terms of the second constraint, the incoming Pod can only be placed onto "node4". Then the results of 2 constraints are ANDed, so the only viable option is to place on "node4". +In this case, to match the first constraint, the incoming Pod can only be placed into "zoneB"; while in terms of the second constraint, the incoming Pod can only be placed onto "node4". Then the results of 2 constraints are ANDed, so the only viable option is to place on "node4". Multiple constraints can lead to conflicts. Suppose you have a 3-node cluster across 2 zones: @@ -242,7 +243,7 @@ graph BT class zoneA,zoneB cluster; {{< /mermaid >}} -If you apply "two-constraints.yaml" to this cluster, you will notice "mypod" stays in `Pending` state. This is because: to satisfy the first constraint, "mypod" can only be put to "zoneB"; while in terms of the second constraint, "mypod" can only put to "node2". Then a joint result of "zoneB" and "node2" returns nothing. +If you apply "two-constraints.yaml" to this cluster, you will notice "mypod" stays in `Pending` state. This is because: to satisfy the first constraint, "mypod" can only placed into "zoneB"; while in terms of the second constraint, "mypod" can only be placed onto "node2". Then a joint result of "zoneB" and "node2" returns nothing. To overcome this situation, you can either increase the `maxSkew` or modify one of the constraints to use `whenUnsatisfiable: ScheduleAnyway`. @@ -286,7 +287,7 @@ class n5 k8s; class zoneC cluster; {{< /mermaid >}} -and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed onto "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected. +and you know that "zoneC" must be excluded. In this case, you can compose the yaml as below, so that "mypod" will be placed into "zoneB" instead of "zoneC". Similarly `spec.nodeSelector` is also respected. {{< codenew file="pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml" >}} @@ -301,9 +302,9 @@ There are some implicit conventions worth noting here: - The scheduler will bypass the nodes without `topologySpreadConstraints[*].topologyKey` present. This implies that: 1. the Pods located on those nodes do not impact `maxSkew` calculation - in the above example, suppose "node1" does not have label "zone", then the 2 Pods will be disregarded, hence the incoming Pod will be scheduled into "zoneA". - 2. the incoming Pod has no chances to be scheduled onto this kind of nodes - in the above example, suppose a "node5" carrying label `{zone-typo: zoneC}` joins the cluster, it will be bypassed due to the absence of label key "zone". + 2. the incoming Pod has no chances to be scheduled onto such nodes - in the above example, suppose a "node5" carrying label `{zone-typo: zoneC}` joins the cluster, it will be bypassed due to the absence of label key "zone". -- Be aware of what will happen if the incomingPod's `topologySpreadConstraints[*].labelSelector` doesn't match its own labels. In the above example, if we remove the incoming Pod's labels, it can still be placed onto "zoneB" since the constraints are still satisfied. However, after the placement, the degree of imbalance of the cluster remains unchanged - it's still zoneA having 2 Pods which hold label {foo:bar}, and zoneB having 1 Pod which holds label {foo:bar}. So if this is not what you expect, we recommend the workload's `topologySpreadConstraints[*].labelSelector` to match its own labels. +- Be aware of what will happen if the incoming Pod's `topologySpreadConstraints[*].labelSelector` doesn't match its own labels. In the above example, if we remove the incoming Pod's labels, it can still be placed into "zoneB" since the constraints are still satisfied. However, after the placement, the degree of imbalance of the cluster remains unchanged - it's still zoneA having 2 Pods which hold label {foo:bar}, and zoneB having 1 Pod which holds label {foo:bar}. So if this is not what you expect, we recommend the workload's `topologySpreadConstraints[*].labelSelector` to match its own labels. ### Cluster-level default constraints diff --git a/content/en/docs/contribute/_index.md b/content/en/docs/contribute/_index.md index 61a4e0a118..8627e8dd93 100644 --- a/content/en/docs/contribute/_index.md +++ b/content/en/docs/contribute/_index.md @@ -18,8 +18,15 @@ card: {{< note >}} To learn more about contributing to Kubernetes in general, see the [contributor documentation](https://www.kubernetes.dev/docs/). + +You can also read the +{{< glossary_tooltip text="CNCF" term_id="cncf" >}} +[page](https://contribute.cncf.io/contributors/projects/#kubernetes) +about contributing to Kubernetes. {{< /note >}} +--- + This website is maintained by [Kubernetes SIG Docs](/docs/contribute/#get-involved-with-sig-docs). Kubernetes documentation contributors: diff --git a/content/en/docs/contribute/advanced.md b/content/en/docs/contribute/advanced.md index 9c37906f3d..34e0107035 100644 --- a/content/en/docs/contribute/advanced.md +++ b/content/en/docs/contribute/advanced.md @@ -136,7 +136,7 @@ The role of co-chair is one of service: co-chairs build contributor capacity, ha Responsibilities include: - Keep SIG Docs focused on maximizing developer happiness through excellent documentation -- Exemplify the [community code of conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) and hold SIG members accountable to it +- Exemplify the [community code of conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md) and hold SIG members accountable to it - Learn and set best practices for the SIG by updating contribution guidelines - Schedule and run SIG meetings: weekly status updates, quarterly retro/planning sessions, and others as needed - Schedule and run doc sprints at KubeCon events and other conferences @@ -147,7 +147,7 @@ Responsibilities include: To schedule and run effective meetings, these guidelines show what to do, how to do it, and why. -**Uphold the [community code of conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md)**: +**Uphold the [community code of conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md)**: - Hold respectful, inclusive discussions with respectful, inclusive language. diff --git a/content/en/docs/contribute/localization.md b/content/en/docs/contribute/localization.md index 7630bdc7d9..7bfcbb6375 100644 --- a/content/en/docs/contribute/localization.md +++ b/content/en/docs/contribute/localization.md @@ -126,12 +126,14 @@ Add a configuration block for the new language to `config.toml`, under the exist title = "Kubernetes" description = "Produktionsreife Container-Verwaltung" languageName = "Deutsch (German)" -languageNameLatinScript = "German" +languageNameLatinScript = "Deutsch" contentDir = "content/de" weight = 8 ``` -The value for `languageName` will be listed in language selection bar. Assign "language name in native script (language name in latin script)" to `languageName`, for example, `languageName = "한국어 (Korean)"`. `languageNameLatinScript` can be used to access the language name in latin script and use it in the theme. Assign "language name in latin script" to `languageNameLatinScript`, for example, `languageNameLatinScript ="Korean"`. +The value for `languageName` will be listed in language selection bar. Assign "language name in native script and language (English language name in latin script)" to `languageName`. For example, `languageName = "한국어 (Korean)"` or `languageName = "Deutsch (German)"`. + +`languageNameLatinScript` can be used to access the language name in latin script and use it in the theme. Assign "language name in latin script" to `languageNameLatinScript`. For example, `languageNameLatinScript ="Korean"` or `languageNameLatinScript = "Deutsch"`. When assigning a `weight` parameter for your block, find the language block with the highest weight and add 1 to that value. @@ -157,7 +159,7 @@ For example, for German the strings live in `data/i18n/de/de.toml`, and ### Localize the community code of conduct -Open a PR against the [`cncf/foundation`](https://github.com/cncf/foundation/tree/master/code-of-conduct-languages) repository to add the code of conduct in your language. +Open a PR against the [`cncf/foundation`](https://github.com/cncf/foundation/tree/main/code-of-conduct-languages) repository to add the code of conduct in your language. ### Setting up the OWNERS files @@ -365,4 +367,3 @@ For more information about working from forks or directly from the repository, s ## Upstream contributions SIG Docs welcomes upstream contributions and corrections to the English source. - diff --git a/content/en/docs/contribute/new-content/blogs-case-studies.md b/content/en/docs/contribute/new-content/blogs-case-studies.md index 83b950105c..3034f66ce8 100644 --- a/content/en/docs/contribute/new-content/blogs-case-studies.md +++ b/content/en/docs/contribute/new-content/blogs-case-studies.md @@ -16,16 +16,17 @@ Case studies require extensive review before they're approved. ## The Kubernetes Blog -The Kubernetes blog is used by the project to communicate new features, community reports, and any news that might be relevant to the Kubernetes community. -This includes end users and developers. -Most of the blog's content is about things happening in the core project, but we encourage you to submit about things happening elsewhere in the ecosystem too! +The Kubernetes blog is used by the project to communicate new features, community reports, and any +news that might be relevant to the Kubernetes community. This includes end users and developers. +Most of the blog's content is about things happening in the core project, but we encourage you to +submit about things happening elsewhere in the ecosystem too! Anyone can write a blog post and submit it for review. ### Submit a Post -Blog posts should not be commercial in nature and should consist of original content that applies broadly to the Kubernetes community. -Appropriate blog content includes: +Blog posts should not be commercial in nature and should consist of original content that applies +broadly to the Kubernetes community. Appropriate blog content includes: - New Kubernetes capabilities - Kubernetes projects updates @@ -43,75 +44,138 @@ Unsuitable content includes: To submit a blog post, follow these steps: -1. [Sign the CLA](https://kubernetes.io/docs/contribute/start/#sign-the-cla) if you have not yet done so. -1. Have a look at the Markdown format for existing blog posts in the [website repository](https://github.com/kubernetes/website/tree/master/content/en/blog/_posts). +1. [Sign the CLA](https://github.com/kubernetes/community/blob/master/CLA.md) + if you have not yet done so. + +1. Have a look at the Markdown format for existing blog posts in the + [website repository](https://github.com/kubernetes/website/tree/master/content/en/blog/_posts). + 1. Write out your blog post in a text editor of your choice. -1. On the same link from step 2, click the Create new file button. Paste your content into the editor. Name the file to match the proposed title of the blog post, but don’t put the date in the file name. The blog reviewers will work with you on the final file name and the date the blog will be published. + +1. On the same link from step 2, click the Create new file button. Paste your content into the editor. + Name the file to match the proposed title of the blog post, but don’t put the date in the file name. + The blog reviewers will work with you on the final file name and the date the blog will be published. + 1. When you save the file, GitHub will walk you through the pull request process. -1. A blog post reviewer will review your submission and work with you on feedback and final details. When the blog post is approved, the blog will be scheduled for publication. + +1. A blog post reviewer will review your submission and work with you on feedback and final details. + When the blog post is approved, the blog will be scheduled for publication. ### Guidelines and expectations - Blog posts should not be vendor pitches. - - Articles must contain content that applies broadly to the Kubernetes community. For example, a submission should focus on upstream Kubernetes as opposed to vendor-specific configurations. Check the [Documentation style guide](/docs/contribute/style/content-guide/#what-s-allowed) for what is typically allowed on Kubernetes properties. - - Links should primarily be to the official Kubernetes documentation. When using external references, links should be diverse - For example a submission shouldn't contain only links back to a single company's blog. - - Sometimes this is a delicate balance. The [blog team](https://kubernetes.slack.com/messages/sig-docs-blog/) is there to give guidance on whether a post is appropriate for the Kubernetes blog, so don't hesitate to reach out. + + - Articles must contain content that applies broadly to the Kubernetes community. For example, a + submission should focus on upstream Kubernetes as opposed to vendor-specific configurations. + Check the [Documentation style guide](/docs/contribute/style/content-guide/#what-s-allowed) for + what is typically allowed on Kubernetes properties. + - Links should primarily be to the official Kubernetes documentation. When using external + references, links should be diverse - For example a submission shouldn't contain only links + back to a single company's blog. + - Sometimes this is a delicate balance. The [blog team](https://kubernetes.slack.com/messages/sig-docs-blog/) + is there to give guidance on whether a post is appropriate for the Kubernetes blog, so don't + hesitate to reach out. + - Blog posts are not published on specific dates. - - Articles are reviewed by community volunteers. We'll try our best to accommodate specific timing, but we make no guarantees. - - Many core parts of the Kubernetes projects submit blog posts during release windows, delaying publication times. Consider submitting during a quieter period of the release cycle. - - If you are looking for greater coordination on post release dates, coordinating with [CNCF marketing](https://www.cncf.io/about/contact/) is a more appropriate choice than submitting a blog post. - - Sometimes reviews can get backed up. If you feel your review isn't getting the attention it needs, you can reach out to the blog team via [this slack channel](https://kubernetes.slack.com/messages/sig-docs-blog/) to ask in real time. + + - Articles are reviewed by community volunteers. We'll try our best to accommodate specific + timing, but we make no guarantees. + - Many core parts of the Kubernetes projects submit blog posts during release windows, delaying + publication times. Consider submitting during a quieter period of the release cycle. + - If you are looking for greater coordination on post release dates, coordinating with + [CNCF marketing](https://www.cncf.io/about/contact/) is a more appropriate choice than submitting a blog post. + - Sometimes reviews can get backed up. If you feel your review isn't getting the attention it needs, + you can reach out to the blog team via [this slack channel](https://kubernetes.slack.com/messages/sig-docs-blog/) + to ask in real time. + - Blog posts should be relevant to Kubernetes users. - - Topics related to participation in or results of Kubernetes SIGs activities are always on topic (see the work in the [Upstream Marketing Team](https://github.com/kubernetes/community/blob/master/communication/marketing-team/blog-guidelines.md#upstream-marketing-blog-guidelines) for support on these posts). - - The components of Kubernetes are purposely modular, so tools that use existing integration points like CNI and CSI are on topic. - - Posts about other CNCF projects may or may not be on topic. We recommend asking the blog team before submitting a draft. - - Many CNCF projects have their own blog. These are often a better choice for posts. There are times of major feature or milestone for a CNCF project that users would be interested in reading on the Kubernetes blog. - - Blog posts about contributing to the Kubernetes project should be in the [Kubernetes Contributors site](https://kubernetes.dev) + + - Topics related to participation in or results of Kubernetes SIGs activities are always on + topic (see the work in the [Upstream Marketing Team](https://github.com/kubernetes/community/blob/master/communication/marketing-team/blog-guidelines.md#upstream-marketing-blog-guidelines) + for support on these posts). + - The components of Kubernetes are purposely modular, so tools that use existing integration + points like CNI and CSI are on topic. + - Posts about other CNCF projects may or may not be on topic. We recommend asking the blog team + before submitting a draft. + - Many CNCF projects have their own blog. These are often a better choice for posts. There are + times of major feature or milestone for a CNCF project that users would be interested in + reading on the Kubernetes blog. + - Blog posts about contributing to the Kubernetes project should be in the + [Kubernetes Contributors site](https://kubernetes.dev) + - Blog posts should be original content + - The official blog is not for repurposing existing content from a third party as new content. - - The [license](https://github.com/kubernetes/website/blob/main/LICENSE) for the blog allows commercial use of the content for commercial purposes, but not the other way around. + - The [license](https://github.com/kubernetes/website/blob/main/LICENSE) for the blog allows + commercial use of the content for commercial purposes, but not the other way around. + - Blog posts should aim to be future proof - - Given the development velocity of the project, we want evergreen content that won't require updates to stay accurate for the reader. - - It can be a better choice to add a tutorial or update official documentation than to write a high level overview as a blog post. - - Consider concentrating the long technical content as a call to action of the blog post, and focus on the problem space or why readers should care. + + - Given the development velocity of the project, we want evergreen content that won't require + updates to stay accurate for the reader. + - It can be a better choice to add a tutorial or update official documentation than to write a + high level overview as a blog post. + - Consider concentrating the long technical content as a call to action of the blog post, and + focus on the problem space or why readers should care. ### Technical Considerations for submitting a blog post -Submissions need to be in Markdown format to be used by the [Hugo](https://gohugo.io/) generator for the blog. There are [many resources available](https://gohugo.io/documentation/) on how to use this technology stack. +Submissions need to be in Markdown format to be used by the [Hugo](https://gohugo.io/) generator +for the blog. There are [many resources available](https://gohugo.io/documentation/) on how to use +this technology stack. -We recognize that this requirement makes the process more difficult for less-familiar folks to submit, and we're constantly looking at solutions to lower this bar. If you have ideas on how to lower the barrier, please volunteer to help out. +We recognize that this requirement makes the process more difficult for less-familiar folks to +submit, and we're constantly looking at solutions to lower this bar. If you have ideas on how to +lower the barrier, please volunteer to help out. -The SIG Docs [blog subproject](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject) manages the review process for blog posts. For more information, see [Submit a post](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject#submit-a-post). +The SIG Docs [blog subproject](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject) +manages the review process for blog posts. For more information, see +[Submit a post](https://github.com/kubernetes/community/tree/master/sig-docs/blog-subproject#submit-a-post). To submit a blog post follow these directions: -- [Open a pull request](/docs/contribute/new-content/open-a-pr/#fork-the-repo) with a new blog post. New blog posts go under the [`content/en/blog/_posts`](https://github.com/kubernetes/website/tree/main/content/en/blog/_posts) directory. +- [Open a pull request](/docs/contribute/new-content/open-a-pr/#fork-the-repo) with a new blog post. + New blog posts go under the [`content/en/blog/_posts`](https://github.com/kubernetes/website/tree/main/content/en/blog/_posts) + directory. -- Ensure that your blog post follows the correct naming conventions and the following frontmatter (metadata) information: +- Ensure that your blog post follows the correct naming conventions and the following frontmatter + (metadata) information: - - The Markdown file name must follow the format `YYYY-MM-DD-Your-Title-Here.md`. For example, `2020-02-07-Deploying-External-OpenStack-Cloud-Provider-With-Kubeadm.md`. - - Do **not** include dots in the filename. A name like `2020-01-01-whats-new-in-1.19.md` causes failures during a build. + - The Markdown file name must follow the format `YYYY-MM-DD-Your-Title-Here.md`. For example, + `2020-02-07-Deploying-External-OpenStack-Cloud-Provider-With-Kubeadm.md`. + - Do **not** include dots in the filename. A name like `2020-01-01-whats-new-in-1.19.md` causes + failures during a build. - The front matter must include the following: - ```yaml - --- - layout: blog - title: "Your Title Here" - date: YYYY-MM-DD - slug: text-for-URL-link-here-no-spaces - --- - ``` - - The first or initial commit message should be a short summary of the work being done and should stand alone as a description of the blog post. Please note that subsequent edits to your blog will be squashed into this main commit, so it should be as useful as possible. + ```yaml + --- + layout: blog + title: "Your Title Here" + date: YYYY-MM-DD + slug: text-for-URL-link-here-no-spaces + --- + ``` + + - The first or initial commit message should be a short summary of the work being done and + should stand alone as a description of the blog post. Please note that subsequent edits to + your blog will be squashed into this main commit, so it should be as useful as possible. + - Examples of a good commit message: - - _Add blog post on the foo kubernetes feature_ - - _blog: foobar announcement_ + - _Add blog post on the foo kubernetes feature_ + - _blog: foobar announcement_ - Examples of bad commit message: - _Add blog post_ - _._ - _initial commit_ - _draft post_ - - The blog team will then review your PR and give you comments on things you might need to fix. After that the bot will merge your PR and your blog post will be published. - - If the content of the blog post contains only content that is not expected to require updates to stay accurate for the reader, it can be marked as evergreen and exempted from the automatic warning about outdated content added to blog posts older than one year. + + - The blog team will then review your PR and give you comments on things you might need to fix. + After that the bot will merge your PR and your blog post will be published. + + - If the content of the blog post contains only content that is not expected to require updates + to stay accurate for the reader, it can be marked as evergreen and exempted from the automatic + warning about outdated content added to blog posts older than one year. + - To mark a blog post as evergreen, add this to the front matter: ```yaml @@ -121,13 +185,15 @@ To submit a blog post follow these directions: - **Tutorials** that only apply to specific releases or versions and not all future versions - References to pre-GA APIs or features - ## Submit a case study -Case studies highlight how organizations are using Kubernetes to solve -real-world problems. The Kubernetes marketing team and members of the {{< glossary_tooltip text="CNCF" term_id="cncf" >}} collaborate with you on all case studies. +Case studies highlight how organizations are using Kubernetes to solve real-world problems. The +Kubernetes marketing team and members of the {{< glossary_tooltip text="CNCF" term_id="cncf" >}} +collaborate with you on all case studies. Have a look at the source for the [existing case studies](https://github.com/kubernetes/website/tree/main/content/en/case-studies). -Refer to the [case study guidelines](https://github.com/cncf/foundation/blob/master/case-study-guidelines.md) and submit your request as outlined in the guidelines. +Refer to the [case study guidelines](https://github.com/cncf/foundation/blob/master/case-study-guidelines.md) +and submit your request as outlined in the guidelines. + diff --git a/content/en/docs/contribute/new-content/open-a-pr.md b/content/en/docs/contribute/new-content/open-a-pr.md index 548dbac5d0..666dfdce96 100644 --- a/content/en/docs/contribute/new-content/open-a-pr.md +++ b/content/en/docs/contribute/new-content/open-a-pr.md @@ -28,7 +28,7 @@ If your changes are large, read [Work from a local fork](#fork-the-repo) to lear ## Changes using GitHub If you're less experienced with git workflows, here's an easier method of -opening a pull request. The figure below outlines the steps and the details follow. +opening a pull request. Figure 1 outlines the steps and the details follow. @@ -61,7 +61,7 @@ class tasks,tasks2 white class id1 k8s {{}} -***Figure - Steps for opening a PR using GitHub*** +Figure 1. Steps for opening a PR using GitHub. 1. On the page where you see the issue, select the pencil icon at the top right. You can also scroll to the bottom of the page and select **Edit this page**. @@ -122,7 +122,7 @@ work from a local fork. Make sure you have [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed on your computer. You can also use a git UI application. -The figure below shows the steps to follow when you work from a local fork. The details for each step follow. +Figure 2 shows the steps to follow when you work from a local fork. The details for each step follow. @@ -151,7 +151,8 @@ class 1,2,3,3a,4,5,6 grey class S,T spacewhite class changes,changes2 white {{}} -***Figure - Working from a local fork to make your changes*** + +Figure 2. Working from a local fork to make your changes. ### Fork the kubernetes/website repository @@ -291,34 +292,24 @@ You can either build the website's container image or run Hugo locally. Building The commands below use Docker as default container engine. Set the `CONTAINER_ENGINE` environment variable to override this behaviour. {{< /note >}} -1. Build the image locally: +1. Build the container image locally + _You only need this step if you are testing a change to the Hugo tool itself_ + ```bash + # Run this in a terminal (if required) + make container-image + ``` - ```bash - # Use docker (default) - make container-image +1. Start Hugo in a container: - ### OR ### + ```bash + # Run this in a terminal + make container-serve + ``` - # Use podman - CONTAINER_ENGINE=podman make container-image - ``` - -2. After building the `kubernetes-hugo` image locally, build and serve the site: - - ```bash - # Use docker (default) - make container-serve - - ### OR ### - - # Use podman - CONTAINER_ENGINE=podman make container-serve - ``` - -3. In a web browser, navigate to `https://localhost:1313`. Hugo watches the +1. In a web browser, navigate to `https://localhost:1313`. Hugo watches the changes and rebuilds the site as needed. -4. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`, +1. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`, or close the terminal window. {{% /tab %}} @@ -353,7 +344,7 @@ Alternately, install and use the `hugo` command on your computer: ### Open a pull request from your fork to kubernetes/website {#open-a-pr} -The figure below shows the steps to open a PR from your fork to the K8s/website. The details follow. +Figure 3 shows the steps to open a PR from your fork to the K8s/website. The details follow. @@ -379,7 +370,8 @@ classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:b class 1,2,3,4,5,6,7,8 grey class first,second white {{}} -***Figure - Steps to open a PR from your fork to the K8s/website*** + +Figure 3. Steps to open a PR from your fork to the K8s/website. 1. In a web browser, go to the [`kubernetes/website`](https://github.com/kubernetes/website/) repository. 2. Select **New Pull Request**. diff --git a/content/en/docs/contribute/participate/pr-wranglers.md b/content/en/docs/contribute/participate/pr-wranglers.md index 865af35805..ee553c64d6 100644 --- a/content/en/docs/contribute/participate/pr-wranglers.md +++ b/content/en/docs/contribute/participate/pr-wranglers.md @@ -84,7 +84,7 @@ To close a pull request, leave a `/close` comment on the PR. {{< note >}} -The [`fejta-bot`](https://github.com/fejta-bot) bot marks issues as stale after 90 days of inactivity. After 30 more days it marks issues as rotten and closes them. PR wranglers should close issues after 14-30 days of inactivity. +The [`k8s-triage-robot`](https://github.com/k8s-triage-robot) bot marks issues as stale after 90 days of inactivity. After 30 more days it marks issues as rotten and closes them. PR wranglers should close issues after 14-30 days of inactivity. {{< /note >}} @@ -100,4 +100,4 @@ In late 2021, SIG Docs introduced the PR Wrangler Shadow Program. The program wa - Others can reach out on the [#sig-docs Slack channel](https://kubernetes.slack.com/messages/sig-docs) for requesting to shadow an assigned PR Wrangler for a specific week. Feel free to reach out to Brad Topol (`@bradtopol`) or one of the [SIG Docs co-chairs/leads](https://github.com/kubernetes/community/tree/master/sig-docs#leadership). -- Once you've signed up to shadow a PR Wrangler, introduce yourself to the PR Wrangler on the [Kubernetes Slack](slack.k8s.io). \ No newline at end of file +- Once you've signed up to shadow a PR Wrangler, introduce yourself to the PR Wrangler on the [Kubernetes Slack](https://slack.k8s.io). diff --git a/content/en/docs/contribute/review/reviewing-prs.md b/content/en/docs/contribute/review/reviewing-prs.md index 3e71e9c434..5a5bbce533 100644 --- a/content/en/docs/contribute/review/reviewing-prs.md +++ b/content/en/docs/contribute/review/reviewing-prs.md @@ -27,7 +27,7 @@ Before reviewing, it's a good idea to: Before you start a review: -- Read the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md) and ensure that you abide by it at all times. +- Read the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md) and ensure that you abide by it at all times. - Be polite, considerate, and helpful. - Comment on positive aspects of PRs as well as changes. - Be empathetic and mindful of how your review may be received. @@ -95,7 +95,7 @@ Figure 1. Review process steps. 1. Click on the `+` symbol beside the line you want to comment on. 2. Fill in any comments you have about the line and click either **Add single comment** (if you have only one comment to make) or **Start a review** (if you have multiple comments to make). 3. When finished, click **Review changes** at the top of the page. Here, you can add - add a summary of your review (and leave some positive comments for the contributor!), + a summary of your review (and leave some positive comments for the contributor!), approve the PR, comment or request changes as needed. New contributors should always choose **Comment**. diff --git a/content/en/docs/home/supported-doc-versions.md b/content/en/docs/home/supported-doc-versions.md index b955f95f56..fd3559a4d3 100644 --- a/content/en/docs/home/supported-doc-versions.md +++ b/content/en/docs/home/supported-doc-versions.md @@ -10,3 +10,8 @@ card: This website contains documentation for the current version of Kubernetes and the four previous versions of Kubernetes. + +The availability of documentation for a Kubernetes version is separate from whether +that release is currently supported. +Read [Support period](/releases/patch-releases/#support-period) to learn about +which versions of Kubernetes are officially supported, and for how long. \ No newline at end of file diff --git a/content/en/docs/images/ingress.svg b/content/en/docs/images/ingress.svg new file mode 100644 index 0000000000..450a0aae9b --- /dev/null +++ b/content/en/docs/images/ingress.svg @@ -0,0 +1 @@ +
    cluster
    Ingress-managed
    load balancer
    routing rule
    Ingress
    Pod
    Service
    Pod
    client
     \ No newline at end of file diff --git a/content/en/docs/images/ingressFanOut.svg b/content/en/docs/images/ingressFanOut.svg new file mode 100644 index 0000000000..a6bf202635 --- /dev/null +++ b/content/en/docs/images/ingressFanOut.svg @@ -0,0 +1 @@ +
    cluster
    Ingress-managed
    load balancer
    /foo
    /bar
    Ingress, 178.91.123.132
    Pod
    Service service1:4200
    Pod
    Pod
    Service service2:8080
    Pod
    client
     \ No newline at end of file diff --git a/content/en/docs/images/ingressNameBased.svg b/content/en/docs/images/ingressNameBased.svg new file mode 100644 index 0000000000..7e1d7be98c --- /dev/null +++ b/content/en/docs/images/ingressNameBased.svg @@ -0,0 +1 @@ +
    cluster
    Ingress-managed
    load balancer
    Host: foo.bar.com
    Host: bar.foo.com
    Ingress, 178.91.123.132
    Pod
    Service service1:80
    Pod
    Pod
    Service service2:80
    Pod
    client
     \ No newline at end of file diff --git a/content/en/docs/images/tutor-service-nodePort-fig01.svg b/content/en/docs/images/tutor-service-nodePort-fig01.svg new file mode 100644 index 0000000000..bb4d866f85 --- /dev/null +++ b/content/en/docs/images/tutor-service-nodePort-fig01.svg @@ -0,0 +1 @@ +
    SNAT
    SNAT
    client
    Node 2
    Node 1
    Endpoint
     \ No newline at end of file diff --git a/content/en/docs/images/tutor-service-nodePort-fig02.svg b/content/en/docs/images/tutor-service-nodePort-fig02.svg new file mode 100644 index 0000000000..1a891575e5 --- /dev/null +++ b/content/en/docs/images/tutor-service-nodePort-fig02.svg @@ -0,0 +1 @@ +
    client
    Node 1
    Node 2
    endpoint
     \ No newline at end of file diff --git a/content/en/docs/reference/_index.md b/content/en/docs/reference/_index.md index c41d20bdbb..0ebd59c406 100644 --- a/content/en/docs/reference/_index.md +++ b/content/en/docs/reference/_index.md @@ -77,9 +77,11 @@ operator to use or manage a cluster. * [kube-apiserver configuration (v1alpha1)](/docs/reference/config-api/apiserver-config.v1alpha1/) * [kube-apiserver configuration (v1)](/docs/reference/config-api/apiserver-config.v1/) * [kube-apiserver encryption (v1)](/docs/reference/config-api/apiserver-encryption.v1/) +* [kube-apiserver event rate limit (v1alpha1)](/docs/reference/config-api/apiserver-eventratelimit.v1/) * [kubelet configuration (v1alpha1)](/docs/reference/config-api/kubelet-config.v1alpha1/) and [kubelet configuration (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/) * [kubelet credential providers (v1alpha1)](/docs/reference/config-api/kubelet-credentialprovider.v1alpha1/) +* [kubelet credential providers (v1beta1)](/docs/reference/config-api/kubelet-credentialprovider.v1beta1/) * [kube-scheduler configuration (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) and [kube-scheduler configuration (v1beta3)](/docs/reference/config-api/kube-scheduler-config.v1beta3/) * [kube-proxy configuration (v1alpha1)](/docs/reference/config-api/kube-proxy-config.v1alpha1/) @@ -87,6 +89,7 @@ operator to use or manage a cluster. * [Client authentication API (v1beta1)](/docs/reference/config-api/client-authentication.v1beta1/) and [Client authentication API (v1)](/docs/reference/config-api/client-authentication.v1/) * [WebhookAdmission configuration (v1)](/docs/reference/config-api/apiserver-webhookadmission.v1/) +* [ImagePolicy API (v1alpha1)](/docs/reference/config-api/imagepolicy.v1alpha1/) ## Config API for kubeadm @@ -96,6 +99,6 @@ operator to use or manage a cluster. ## Design Docs An archive of the design docs for Kubernetes functionality. Good starting points are -[Kubernetes Architecture](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md) and -[Kubernetes Design Overview](https://git.k8s.io/community/contributors/design-proposals). +[Kubernetes Architecture](https://git.k8s.io/design-proposals-archive/architecture/architecture.md) and +[Kubernetes Design Overview](https://git.k8s.io/design-proposals-archive). diff --git a/content/en/docs/reference/access-authn-authz/_index.md b/content/en/docs/reference/access-authn-authz/_index.md index 86d06488a8..3677f79c57 100644 --- a/content/en/docs/reference/access-authn-authz/_index.md +++ b/content/en/docs/reference/access-authn-authz/_index.md @@ -24,3 +24,5 @@ Reference documentation: - Service accounts - [Developer guide](/docs/tasks/configure-pod-container/configure-service-account/) - [Administration](/docs/reference/access-authn-authz/service-accounts-admin/) +- [Kubelet Authentication & Authorization](/docs/reference/access-authn-authz/kubelet-authn-authz/) + - including kubelet [TLS bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) diff --git a/content/en/docs/reference/access-authn-authz/admission-controllers.md b/content/en/docs/reference/access-authn-authz/admission-controllers.md index abc3ee9681..f03b04f8e3 100644 --- a/content/en/docs/reference/access-authn-authz/admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/admission-controllers.md @@ -94,7 +94,7 @@ kube-apiserver -h | grep enable-admission-plugins In the current version, the default ones are: ```shell -CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook +CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook ``` ## What does each admission controller do? @@ -139,7 +139,7 @@ requests with the `spec.signerName` requested on the CertificateSigningRequest r See [Certificate Signing Requests](/docs/reference/access-authn-authz/certificate-signing-requests/) for more information on the permissions required to perform different actions on CertificateSigningRequest resources. -### CertificateSubjectRestrictions {#certificatesubjectrestrictions} +### CertificateSubjectRestriction {#certificatesubjectrestriction} This admission controller observes creation of CertificateSigningRequest resources that have a `spec.signerName` of `kubernetes.io/kube-apiserver-client`. It rejects any request that specifies a 'group' (or 'organization attribute') @@ -232,12 +232,10 @@ of it. This admission controller mitigates the problem where the API server gets flooded by event requests. The cluster admin can specify event rate limits by: - * Enabling the `EventRateLimit` admission controller; - * Referencing an `EventRateLimit` configuration file from the file provided to the API - server's command line flag `--admission-control-config-file`: +* Enabling the `EventRateLimit` admission controller; +* Referencing an `EventRateLimit` configuration file from the file provided to the API + server's command line flag `--admission-control-config-file`: -{{< tabs name="eventratelimit_example" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -246,19 +244,6 @@ plugins: path: eventconfig.yaml ... ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: EventRateLimit - path: eventconfig.yaml -... -``` -{{% /tab %}} -{{< /tabs >}} There are four types of limits that can be specified in the configuration: @@ -283,7 +268,7 @@ limits: burst: 50 ``` -See the [EventRateLimit proposal](https://git.k8s.io/community/contributors/design-proposals/api-machinery/admission_control_event_rate_limit.md) +See the [EventRateLimit Config API (v1alpha1)](/docs/reference/config-api/apiserver-eventratelimit.v1alpha1/) for more details. ### ExtendedResourceToleration {#extendedresourcetoleration} @@ -319,8 +304,6 @@ imagePolicy: Reference the ImagePolicyWebhook configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: -{{< tabs name="imagepolicywebhook_example1" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -329,24 +312,9 @@ plugins: path: imagepolicyconfig.yaml ... ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: ImagePolicyWebhook - path: imagepolicyconfig.yaml -... -``` -{{% /tab %}} -{{< /tabs >}} Alternatively, you can embed the configuration directly in the file: -{{< tabs name="imagepolicywebhook_example2" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -360,31 +328,14 @@ plugins: retryBackoff: 500 defaultAllow: true ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: ImagePolicyWebhook - configuration: - imagePolicy: - kubeConfigFile: - allowTTL: 50 - denyTTL: 50 - retryBackoff: 500 - defaultAllow: true -``` -{{% /tab %}} -{{< /tabs >}} The ImagePolicyWebhook config file must reference a [kubeconfig](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) formatted file which sets up the connection to the backend. It is required that the backend communicate over TLS. -The kubeconfig file's cluster field must point to the remote service, and the user field must contain the returned authorizer. +The kubeconfig file's `cluster` field must point to the remote service, and the `user` field +must contain the returned authorizer. ```yaml # clusters refers to the remote service. @@ -405,11 +356,21 @@ users: For additional HTTP configuration, refer to the [kubeconfig](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) documentation. -#### Request Payloads +#### Request payloads -When faced with an admission decision, the API Server POSTs a JSON serialized `imagepolicy.k8s.io/v1alpha1` `ImageReview` object describing the action. This object contains fields describing the containers being admitted, as well as any pod annotations that match `*.image-policy.k8s.io/*`. +When faced with an admission decision, the API Server POSTs a JSON serialized +`imagepolicy.k8s.io/v1alpha1` `ImageReview` object describing the action. +This object contains fields describing the containers being admitted, as well as +any pod annotations that match `*.image-policy.k8s.io/*`. -Note that webhook API objects are subject to the same versioning compatibility rules as other Kubernetes API objects. Implementers should be aware of looser compatibility promises for alpha objects and check the "apiVersion" field of the request to ensure correct deserialization. Additionally, the API Server must enable the imagepolicy.k8s.io/v1alpha1 API extensions group (`--runtime-config=imagepolicy.k8s.io/v1alpha1=true`). +{{ note }} +The webhook API objects are subject to the same versioning compatibility rules +as other Kubernetes API objects. Implementers should be aware of looser compatibility +promises for alpha objects and check the `apiVersion` field of the request to +ensure correct deserialization. +Additionally, the API Server must enable the `imagepolicy.k8s.io/v1alpha1` API extensions +group (`--runtime-config=imagepolicy.k8s.io/v1alpha1=true`). +{{ /note }} An example request body: @@ -434,7 +395,9 @@ An example request body: } ``` -The remote service is expected to fill the `ImageReviewStatus` field of the request and respond to either allow or disallow access. The response body's "spec" field is ignored and may be omitted. A permissive response would return: +The remote service is expected to fill the `ImageReviewStatus` field of the request and +respond to either allow or disallow access. The response body's `spec` field is ignored and +may be omitted. A permissive response would return: ```json { @@ -459,19 +422,23 @@ To disallow access, the service would return: } ``` -For further documentation refer to the `imagepolicy.v1alpha1` API objects and `plugin/pkg/admission/imagepolicy/admission.go`. +For further documentation refer to the +[`imagepolicy.v1alpha1` API](/docs/reference/config-api/imagepolicy.v1alpha1/). #### Extending with Annotations -All annotations on a Pod that match `*.image-policy.k8s.io/*` are sent to the webhook. Sending annotations allows users who are aware of the image policy backend to send extra information to it, and for different backends implementations to accept different information. +All annotations on a Pod that match `*.image-policy.k8s.io/*` are sent to the webhook. +Sending annotations allows users who are aware of the image policy backend to +send extra information to it, and for different backends implementations to +accept different information. Examples of information you might put here are: - * request to "break glass" to override a policy, in case of emergency. - * a ticket number from a ticket system that documents the break-glass request - * provide a hint to the policy server as to the imageID of the image being provided, to save it a lookup +* request to "break glass" to override a policy, in case of emergency. +* a ticket number from a ticket system that documents the break-glass request +* provide a hint to the policy server as to the imageID of the image being provided, to save it a lookup -In any case, the annotations are provided by the user and are not validated by Kubernetes in any way. In the future, if an annotation is determined to be widely useful, it may be promoted to a named field of `ImageReviewSpec`. +In any case, the annotations are provided by the user and are not validated by Kubernetes in any way. ### LimitPodHardAntiAffinityTopology {#limitpodhardantiaffinitytopology} @@ -480,14 +447,16 @@ This admission controller denies any pod that defines `AntiAffinity` topology ke ### LimitRanger {#limitranger} -This admission controller will observe the incoming request and ensure that it does not violate any of the constraints -enumerated in the `LimitRange` object in a `Namespace`. If you are using `LimitRange` objects in -your Kubernetes deployment, you MUST use this admission controller to enforce those constraints. LimitRanger can also -be used to apply default resource requests to Pods that don't specify any; currently, the default LimitRanger -applies a 0.1 CPU requirement to all Pods in the `default` namespace. +This admission controller will observe the incoming request and ensure that it does not violate +any of the constraints enumerated in the `LimitRange` object in a `Namespace`. If you are using +`LimitRange` objects in your Kubernetes deployment, you MUST use this admission controller to +enforce those constraints. LimitRanger can also be used to apply default resource requests to Pods +that don't specify any; currently, the default LimitRanger applies a 0.1 CPU requirement to all +Pods in the `default` namespace. -See the [limitRange design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) -and the [example of Limit Range](/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/) for more details. +See the [LimitRange API reference](/docs/reference/kubernetes-api/policy-resources/limit-range-v1/) +and the [example of LimitRange](/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/) +for more details. ### MutatingAdmissionWebhook {#mutatingadmissionwebhook} @@ -502,21 +471,20 @@ webhooks or validating admission controllers will permit the request to finish. If you disable the MutatingAdmissionWebhook, you must also disable the `MutatingWebhookConfiguration` object in the `admissionregistration.k8s.io/v1` -group/version via the `--runtime-config` flag (both are on by default in -versions >= 1.9). +group/version via the `--runtime-config` flag, both are on by default. #### Use caution when authoring and installing mutating webhooks - * Users may be confused when the objects they try to create are different from - what they get back. - * Built in control loops may break when the objects they try to create are - different when read back. - * Setting originally unset fields is less likely to cause problems than - overwriting fields set in the original request. Avoid doing the latter. - * Future changes to control loops for built-in resources or third-party resources - may break webhooks that work well today. Even when the webhook installation API - is finalized, not all possible webhook behaviors will be guaranteed to be supported - indefinitely. +* Users may be confused when the objects they try to create are different from + what they get back. +* Built in control loops may break when the objects they try to create are + different when read back. + * Setting originally unset fields is less likely to cause problems than + overwriting fields set in the original request. Avoid doing the latter. +* Future changes to control loops for built-in resources or third-party resources + may break webhooks that work well today. Even when the webhook installation API + is finalized, not all possible webhook behaviors will be guaranteed to be supported + indefinitely. ### NamespaceAutoProvision {#namespaceautoprovision} @@ -533,26 +501,28 @@ If the namespace referenced from a request doesn't exist, the request is rejecte ### NamespaceLifecycle {#namespacelifecycle} -This admission controller enforces that a `Namespace` that is undergoing termination cannot have new objects created in it, -and ensures that requests in a non-existent `Namespace` are rejected. This admission controller also prevents deletion of -three system reserved namespaces `default`, `kube-system`, `kube-public`. +This admission controller enforces that a `Namespace` that is undergoing termination cannot have +new objects created in it, and ensures that requests in a non-existent `Namespace` are rejected. +This admission controller also prevents deletion of three system reserved namespaces `default`, +`kube-system`, `kube-public`. -A `Namespace` deletion kicks off a sequence of operations that remove all objects (pods, services, etc.) in that -namespace. In order to enforce integrity of that process, we strongly recommend running this admission controller. +A `Namespace` deletion kicks off a sequence of operations that remove all objects (pods, services, +etc.) in that namespace. In order to enforce integrity of that process, we strongly recommend +running this admission controller. ### NodeRestriction {#noderestriction} This admission controller limits the `Node` and `Pod` objects a kubelet can modify. In order to be limited by this admission controller, kubelets must use credentials in the `system:nodes` group, with a username in the form `system:node:`. Such kubelets will only be allowed to modify their own `Node` API object, and only modify `Pod` API objects that are bound to their node. -In Kubernetes 1.11+, kubelets are not allowed to update or remove taints from their `Node` API object. +kubelets are not allowed to update or remove taints from their `Node` API object. -In Kubernetes 1.13+, the `NodeRestriction` admission plugin prevents kubelets from deleting their `Node` API object, +The `NodeRestriction` admission plugin prevents kubelets from deleting their `Node` API object, and enforces kubelet modification of labels under the `kubernetes.io/` or `k8s.io/` prefixes as follows: * **Prevents** kubelets from adding/removing/updating labels with a `node-restriction.kubernetes.io/` prefix. -This label prefix is reserved for administrators to label their `Node` objects for workload isolation purposes, -and kubelets will not be allowed to modify labels with that prefix. + This label prefix is reserved for administrators to label their `Node` objects for workload isolation purposes, + and kubelets will not be allowed to modify labels with that prefix. * **Allows** kubelets to add/remove/update these labels and label prefixes: * `kubernetes.io/hostname` * `kubernetes.io/arch` @@ -566,9 +536,11 @@ and kubelets will not be allowed to modify labels with that prefix. * `kubelet.kubernetes.io/`-prefixed labels * `node.kubernetes.io/`-prefixed labels -Use of any other labels under the `kubernetes.io` or `k8s.io` prefixes by kubelets is reserved, and may be disallowed or allowed by the `NodeRestriction` admission plugin in the future. +Use of any other labels under the `kubernetes.io` or `k8s.io` prefixes by kubelets is reserved, +and may be disallowed or allowed by the `NodeRestriction` admission plugin in the future. -Future versions may add additional restrictions to ensure kubelets have the minimal set of permissions required to operate correctly. +Future versions may add additional restrictions to ensure kubelets have the minimal set of +permissions required to operate correctly. ### OwnerReferencesPermissionEnforcement {#ownerreferencespermissionenforcement} @@ -582,7 +554,8 @@ subresource of the referenced *owner* can change it. {{< feature-state for_k8s_version="v1.24" state="stable" >}} -This admission controller implements additional validations for checking incoming `PersistentVolumeClaim` resize requests. +This admission controller implements additional validations for checking incoming +`PersistentVolumeClaim` resize requests. Enabling the `PersistentVolumeClaimResize` admission controller is recommended. This admission controller prevents resizing of all claims by default unless a claim's `StorageClass` @@ -624,9 +597,10 @@ Starting from 1.11, this admission controller is disabled by default. {{< feature-state for_k8s_version="v1.5" state="alpha" >}} -This admission controller defaults and limits what node selectors may be used within a namespace by reading a namespace annotation and a global configuration. +This admission controller defaults and limits what node selectors may be used within a namespace +by reading a namespace annotation and a global configuration. -#### Configuration File Format +#### Configuration file format `PodNodeSelector` uses a configuration file to set options for the behavior of the backend. Note that the configuration file format will move to a versioned file in a future release. @@ -639,10 +613,9 @@ podNodeSelectorPluginConfig: namespace2: name-of-node-selector ``` -Reference the `PodNodeSelector` configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: +Reference the `PodNodeSelector` configuration file from the file provided to the API server's +command line flag `--admission-control-config-file`: -{{< tabs name="podnodeselector_example1" >}} -{{% tab name="apiserver.config.k8s.io/v1" %}} ```yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration @@ -651,23 +624,11 @@ plugins: path: podnodeselector.yaml ... ``` -{{% /tab %}} -{{% tab name="apiserver.k8s.io/v1alpha1" %}} -```yaml -# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1 -apiVersion: apiserver.k8s.io/v1alpha1 -kind: AdmissionConfiguration -plugins: -- name: PodNodeSelector - path: podnodeselector.yaml -... -``` -{{% /tab %}} -{{< /tabs >}} #### Configuration Annotation Format -`PodNodeSelector` uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign node selectors to namespaces. +`PodNodeSelector` uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign +node selectors to namespaces. ```yaml apiVersion: v1 @@ -682,13 +643,14 @@ metadata: This admission controller has the following behavior: -1. If the `Namespace` has an annotation with a key `scheduler.alpha.kubernetes.io/node-selector`, use its value as the -node selector. -2. If the namespace lacks such an annotation, use the `clusterDefaultNodeSelector` defined in the `PodNodeSelector` -plugin configuration file as the node selector. -3. Evaluate the pod's node selector against the namespace node selector for conflicts. Conflicts result in rejection. -4. Evaluate the pod's node selector against the namespace-specific allowed selector defined the plugin configuration file. -Conflicts result in rejection. +1. If the `Namespace` has an annotation with a key `scheduler.alpha.kubernetes.io/node-selector`, + use its value as the node selector. +2. If the namespace lacks such an annotation, use the `clusterDefaultNodeSelector` defined in the + `PodNodeSelector` plugin configuration file as the node selector. +3. Evaluate the pod's node selector against the namespace node selector for conflicts. Conflicts + result in rejection. +4. Evaluate the pod's node selector against the namespace-specific allowed selector defined the + plugin configuration file. Conflicts result in rejection. {{< note >}} PodNodeSelector allows forcing pods to run on specifically labeled nodes. Also see the PodTolerationRestriction @@ -721,7 +683,8 @@ for more information. {{< feature-state for_k8s_version="v1.7" state="alpha" >}} -The PodTolerationRestriction admission controller verifies any conflict between tolerations of a pod and the tolerations of its namespace. +The PodTolerationRestriction admission controller verifies any conflict between tolerations of a +pod and the tolerations of its namespace. It rejects the pod request if there is a conflict. It then merges the tolerations annotated on the namespace into the tolerations of the pod. The resulting tolerations are checked against a list of allowed tolerations annotated to the namespace. @@ -748,16 +711,18 @@ metadata: ### Priority {#priority} -The priority admission controller uses the `priorityClassName` field and populates the integer value of the priority. +The priority admission controller uses the `priorityClassName` field and populates the integer +value of the priority. If the priority class is not found, the Pod is rejected. ### ResourceQuota {#resourcequota} -This admission controller will observe the incoming request and ensure that it does not violate any of the constraints -enumerated in the `ResourceQuota` object in a `Namespace`. If you are using `ResourceQuota` -objects in your Kubernetes deployment, you MUST use this admission controller to enforce quota constraints. +This admission controller will observe the incoming request and ensure that it does not violate +any of the constraints enumerated in the `ResourceQuota` object in a `Namespace`. If you are +using `ResourceQuota` objects in your Kubernetes deployment, you MUST use this admission +controller to enforce quota constraints. -See the [resourceQuota design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_resource_quota.md) +See the [ResourceQuota API reference](/docs/reference/kubernetes-api/policy-resources/resource-quota-v1/) and the [example of Resource Quota](/docs/concepts/policy/resource-quotas/) for more details. ### RuntimeClass {#runtimeclass} @@ -793,14 +758,15 @@ pod privileges. This admission controller implements automation for [serviceAccounts](/docs/tasks/configure-pod-container/configure-service-account/). -We strongly recommend using this admission controller if you intend to make use of Kubernetes `ServiceAccount` objects. +We strongly recommend using this admission controller if you intend to make use of Kubernetes +`ServiceAccount` objects. ### StorageObjectInUseProtection The `StorageObjectInUseProtection` plugin adds the `kubernetes.io/pvc-protection` or `kubernetes.io/pv-protection` finalizers to newly created Persistent Volume Claims (PVCs) or Persistent Volumes (PV). In case a user deletes a PVC or PV the PVC or PV is not removed until the finalizer is removed -from the PVC or PV by PVC or PV Protection Controller. +from the PVC or PV by PVC or PV Protection Controller. Refer to the [Storage Object in Use Protection](/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection) for more detailed information. @@ -809,7 +775,10 @@ for more detailed information. {{< feature-state for_k8s_version="v1.17" state="stable" >}} -This admission controller {{< glossary_tooltip text="taints" term_id="taint" >}} newly created Nodes as `NotReady` and `NoSchedule`. That tainting avoids a race condition that could cause Pods to be scheduled on new Nodes before their taints were updated to accurately reflect their reported conditions. +This admission controller {{< glossary_tooltip text="taints" term_id="taint" >}} newly created +Nodes as `NotReady` and `NoSchedule`. That tainting avoids a race condition that could cause Pods +to be scheduled on new Nodes before their taints were updated to accurately reflect their reported +conditions. ### ValidatingAdmissionWebhook {#validatingadmissionwebhook} diff --git a/content/en/docs/reference/access-authn-authz/authentication.md b/content/en/docs/reference/access-authn-authz/authentication.md index b33b239199..1641cb8e54 100644 --- a/content/en/docs/reference/access-authn-authz/authentication.md +++ b/content/en/docs/reference/access-authn-authz/authentication.md @@ -856,6 +856,14 @@ rules: resourceNames: ["06f6ce97-e2c5-4ab8-7ba5-7654dd08d52b"] ``` +{{< note >}} +Impersonating a user or group allows you to perform any action as if you were that user or group; +for that reason, impersonation is not namespace scoped. +If you want to allow impersonation using Kubernetes RBAC, +this requires using a `ClusterRole` and a `ClusterRoleBinding`, +not a `Role` and `RoleBinding`. +{{< /note >}} + ## client-go credential plugins {{< feature-state for_k8s_version="v1.22" state="stable" >}} diff --git a/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md b/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md index 7e743be63d..74367d50c9 100644 --- a/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md +++ b/content/en/docs/reference/access-authn-authz/bootstrap-tokens.md @@ -15,7 +15,7 @@ creating new clusters or joining new nodes to an existing cluster. It was built to support [kubeadm](/docs/reference/setup-tools/kubeadm/), but can be used in other contexts for users that wish to start clusters without `kubeadm`. It is also built to work, via RBAC policy, with the -[Kubelet TLS Bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) system. +[Kubelet TLS Bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) system. @@ -70,7 +70,7 @@ controller on the controller manager. Each valid token is backed by a secret in the `kube-system` namespace. You can find the full design doc -[here](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/bootstrap-discovery.md). +[here](https://git.k8s.io/design-proposals-archive/cluster-lifecycle/bootstrap-discovery.md). Here is what the secret looks like. diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet-authentication-authorization.md b/content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md similarity index 100% rename from content/en/docs/reference/command-line-tools-reference/kubelet-authentication-authorization.md rename to content/en/docs/reference/access-authn-authz/kubelet-authn-authz.md diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md b/content/en/docs/reference/access-authn-authz/kubelet-tls-bootstrapping.md similarity index 100% rename from content/en/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping.md rename to content/en/docs/reference/access-authn-authz/kubelet-tls-bootstrapping.md diff --git a/content/en/docs/reference/access-authn-authz/node.md b/content/en/docs/reference/access-authn-authz/node.md index 6e7c538eb0..bc9863219f 100644 --- a/content/en/docs/reference/access-authn-authz/node.md +++ b/content/en/docs/reference/access-authn-authz/node.md @@ -43,7 +43,7 @@ have the minimal set of permissions required to operate correctly. In order to be authorized by the Node authorizer, kubelets must use a credential that identifies them as being in the `system:nodes` group, with a username of `system:node:`. This group and user name format match the identity created for each kubelet as part of -[kubelet TLS bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/). +[kubelet TLS bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/). The value of `` **must** match precisely the name of the node as registered by the kubelet. By default, this is the host name as provided by `hostname`, or overridden via the [kubelet option](/docs/reference/command-line-tools-reference/kubelet/) `--hostname-override`. However, when using the `--cloud-provider` kubelet option, the specific hostname may be determined by the cloud provider, ignoring the local `hostname` and the `--hostname-override` option. For specifics about how the kubelet determines the hostname, see the [kubelet options reference](/docs/reference/command-line-tools-reference/kubelet/). diff --git a/content/en/docs/reference/access-authn-authz/psp-to-pod-security-standards.md b/content/en/docs/reference/access-authn-authz/psp-to-pod-security-standards.md index 468579f982..6c820a6e99 100644 --- a/content/en/docs/reference/access-authn-authz/psp-to-pod-security-standards.md +++ b/content/en/docs/reference/access-authn-authz/psp-to-pod-security-standards.md @@ -17,7 +17,7 @@ For each applicable parameter, the allowed values for the [Baseline](/docs/concepts/security/pod-security-standards/#baseline) and [Restricted](/docs/concepts/security/pod-security-standards/#restricted) profiles are listed. Anything outside the allowed values for those profiles would fall under the -[Privileged](/docs/concepts/security/pod-security-standards/#priveleged) profile. "No opinion" +[Privileged](/docs/concepts/security/pod-security-standards/#privileged) profile. "No opinion" means all values are allowed under all Pod Security Standards. For a step-by-step migration guide, see diff --git a/content/en/docs/reference/access-authn-authz/rbac.md b/content/en/docs/reference/access-authn-authz/rbac.md index 57a074a29a..d085251e43 100644 --- a/content/en/docs/reference/access-authn-authz/rbac.md +++ b/content/en/docs/reference/access-authn-authz/rbac.md @@ -798,7 +798,7 @@ This is commonly used by add-on API servers for unified authentication and autho system:node-bootstrapper None Allows access to the resources required to perform -kubelet TLS bootstrapping. +kubelet TLS bootstrapping. system:node-problem-detector diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 8eb23b3a47..9d1e67b3c0 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -761,7 +761,7 @@ Each feature gate is designed for enabling/disabling a specific feature: Requires Portworx CSI driver to be installed and configured in the cluster. - `CSINodeInfo`: Enable all logic related to the CSINodeInfo API object in `csi.storage.k8s.io`. - `CSIPersistentVolume`: Enable discovering and mounting volumes provisioned through a - [CSI (Container Storage Interface)](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) + [CSI (Container Storage Interface)](https://git.k8s.io/design-proposals-archive/storage/container-storage-interface.md) compatible volume plugin. - `CSIServiceAccountToken`: Enable CSI drivers to receive the pods' service account token that they mount volumes for. See @@ -1086,10 +1086,10 @@ Each feature gate is designed for enabling/disabling a specific feature: [Bound Service Account Tokens](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md) for more details. - `RotateKubeletClientCertificate`: Enable the rotation of the client TLS certificate on the kubelet. - See [kubelet configuration](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#kubelet-configuration) + See [kubelet configuration](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#kubelet-configuration) for more details. - `RotateKubeletServerCertificate`: Enable the rotation of the server TLS certificate on the kubelet. - See [kubelet configuration](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#kubelet-configuration) + See [kubelet configuration](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#kubelet-configuration) for more details. - `RunAsGroup`: Enable control over the primary group ID set on the init processes of containers. diff --git a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md index 774c1efd0b..ad2dc545b7 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md @@ -81,13 +81,6 @@ kube-apiserver [flags]

    Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.

    - ---apiserver-count int     Default: 1 - - -

    The number of apiservers running in the cluster, must be a positive number. (In use when --endpoint-reconciler-type=master-count is enabled.)

    - - --audit-log-batch-buffer-size int     Default: 10000 @@ -526,7 +519,7 @@ kube-apiserver [flags] --endpoint-reconciler-type string     Default: "lease" -

    Use an endpoint reconciler (master-count, lease, none)

    +

    Use an endpoint reconciler (master-count, lease, none) master-count is deprecated, and will be removed in a future version.

    @@ -606,13 +599,6 @@ kube-apiserver [flags]

    Amount of time to retain events.

    - ---experimental-logging-sanitization - - -

    [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
    Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

    - - --external-hostname string @@ -624,7 +610,7 @@ kube-apiserver [flags] --feature-gates <comma-separated 'key=True|False' pairs> -

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (ALPHA - default=false)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureDisk=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=false)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationOpenStack=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIStorageCapacity=true|false (BETA - default=true)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    CSRDuration=true|false (BETA - default=true)
    ControllerManagerLeaderMigration=true|false (BETA - default=true)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DefaultPodTopologySpread=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EfficientWatchResumption=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandCSIVolumes=true|false (BETA - default=true)
    ExpandInUsePersistentVolumes=true|false (BETA - default=true)
    ExpandPersistentVolumes=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (ALPHA - default=false)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (ALPHA - default=false)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (ALPHA - default=false)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    IndexedJob=true|false (BETA - default=true)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (ALPHA - default=false)
    JobTrackingWithFinalizers=true|false (BETA - default=true)
    KubeletCredentialProviders=true|false (ALPHA - default=false)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (ALPHA - default=false)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NodeSwap=true|false (ALPHA - default=false)
    NonPreemptingPriority=true|false (BETA - default=true)
    OpenAPIEnums=true|false (ALPHA - default=false)
    OpenAPIV3=true|false (ALPHA - default=false)
    PodAffinityNamespaceSelector=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodOverhead=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    PreferNominatedNode=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RemoveSelfLink=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    ServiceLBNodePortControl=true|false (BETA - default=true)
    ServiceLoadBalancerClass=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    SuspendJob=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=false)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)
    csiMigrationRBD=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (BETA - default=true)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=true)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationRBD=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    ContextualLogging=true|false (ALPHA - default=false)
    CronJobTimeZone=true|false (ALPHA - default=false)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (BETA - default=true)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (BETA - default=true)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (BETA - default=true)
    JobTrackingWithFinalizers=true|false (BETA - default=false)
    KubeletCredentialProviders=true|false (BETA - default=true)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (BETA - default=true)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NetworkPolicyStatus=true|false (ALPHA - default=false)
    NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
    NodeSwap=true|false (ALPHA - default=false)
    OpenAPIEnums=true|false (BETA - default=true)
    OpenAPIV3=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceIPStaticSubrange=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=true)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md index 6f1f3489a0..f0d0ee4e98 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md @@ -442,13 +442,6 @@ kube-controller-manager [flags]

    The length of endpoint slice updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated

    - ---experimental-logging-sanitization - - -

    [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
    Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

    - - --external-cloud-volume-plugin string @@ -460,7 +453,7 @@ kube-controller-manager [flags] --feature-gates <comma-separated 'key=True|False' pairs> -

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (ALPHA - default=false)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureDisk=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=false)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationOpenStack=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIStorageCapacity=true|false (BETA - default=true)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    CSRDuration=true|false (BETA - default=true)
    ControllerManagerLeaderMigration=true|false (BETA - default=true)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DefaultPodTopologySpread=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EfficientWatchResumption=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandCSIVolumes=true|false (BETA - default=true)
    ExpandInUsePersistentVolumes=true|false (BETA - default=true)
    ExpandPersistentVolumes=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (ALPHA - default=false)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (ALPHA - default=false)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (ALPHA - default=false)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    IndexedJob=true|false (BETA - default=true)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (ALPHA - default=false)
    JobTrackingWithFinalizers=true|false (BETA - default=true)
    KubeletCredentialProviders=true|false (ALPHA - default=false)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (ALPHA - default=false)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NodeSwap=true|false (ALPHA - default=false)
    NonPreemptingPriority=true|false (BETA - default=true)
    OpenAPIEnums=true|false (ALPHA - default=false)
    OpenAPIV3=true|false (ALPHA - default=false)
    PodAffinityNamespaceSelector=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodOverhead=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    PreferNominatedNode=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RemoveSelfLink=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    ServiceLBNodePortControl=true|false (BETA - default=true)
    ServiceLoadBalancerClass=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    SuspendJob=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=false)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)
    csiMigrationRBD=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (BETA - default=true)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=true)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationRBD=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    ContextualLogging=true|false (ALPHA - default=false)
    CronJobTimeZone=true|false (ALPHA - default=false)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (BETA - default=true)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (BETA - default=true)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (BETA - default=true)
    JobTrackingWithFinalizers=true|false (BETA - default=false)
    KubeletCredentialProviders=true|false (BETA - default=true)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (BETA - default=true)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NetworkPolicyStatus=true|false (ALPHA - default=false)
    NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
    NodeSwap=true|false (ALPHA - default=false)
    OpenAPIEnums=true|false (BETA - default=true)
    OpenAPIV3=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceIPStaticSubrange=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=true)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)

    @@ -579,7 +572,7 @@ kube-controller-manager [flags] --leader-elect-resource-lock string     Default: "leases" -

    The type of resource object that is used for locking during leader election. Supported options are 'endpoints', 'configmaps', 'leases', 'endpointsleases' and 'configmapsleases'.

    +

    The type of resource object that is used for locking during leader election. Supported options are 'leases', 'endpointsleases' and 'configmapsleases'.

    diff --git a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md index abe749b801..34ced450e0 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md @@ -43,17 +43,24 @@ kube-proxy [flags] ---azure-container-registry-config string +--add_dir_header -

    Path to the file containing Azure container registry configuration information.

    +

    If true, adds the file directory to the header of the log messages

    + + + +--alsologtostderr + + +

    log to standard error as well as files

    --bind-address string     Default: 0.0.0.0 -

    The IP address for the proxy server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces)

    +

    The IP address for the proxy server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces). This parameter is ignored if a config file is specified by --config.

    @@ -63,13 +70,6 @@ kube-proxy [flags]

    If true kube-proxy will treat failure to bind to a port as fatal and exit

    - ---boot-id-file string     Default: "/proc/sys/kernel/random/boot_id" - - -

    Comma-separated list of files to check for boot-id. Use the first one that exists.

    - - --boot_id_file string     Default: "/proc/sys/kernel/random/boot_id" @@ -84,25 +84,11 @@ kube-proxy [flags]

    If true cleanup iptables and ipvs rules and exit.

    - ---cloud-provider-gce-l7lb-src-cidrs cidrs     Default: 130.211.0.0/22,35.191.0.0/16 - - -

    CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks

    - - - ---cloud-provider-gce-lb-src-cidrs cidrs     Default: 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 - - -

    CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks

    - - --cluster-cidr string -

    The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead

    +

    The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead. For dual-stack clusters, a comma-separated list is accepted with at least one CIDR per IP family (IPv4 and IPv6). This parameter is ignored if a config file is specified by --config.

    @@ -147,39 +133,25 @@ kube-proxy [flags]

    Idle timeout for established TCP connections (0 to leave as-is)

    - ---default-not-ready-toleration-seconds int     Default: 300 - - -

    Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration.

    - - - ---default-unreachable-toleration-seconds int     Default: 300 - - -

    Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.

    - - --detect-local-mode LocalMode -

    Mode to use to detect local traffic

    +

    Mode to use to detect local traffic. This parameter is ignored if a config file is specified by --config.

    --feature-gates <comma-separated 'key=True|False' pairs> -

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (ALPHA - default=false)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureDisk=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=false)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationOpenStack=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIStorageCapacity=true|false (BETA - default=true)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    CSRDuration=true|false (BETA - default=true)
    ControllerManagerLeaderMigration=true|false (BETA - default=true)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DefaultPodTopologySpread=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EfficientWatchResumption=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandCSIVolumes=true|false (BETA - default=true)
    ExpandInUsePersistentVolumes=true|false (BETA - default=true)
    ExpandPersistentVolumes=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (ALPHA - default=false)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (ALPHA - default=false)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (ALPHA - default=false)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    IndexedJob=true|false (BETA - default=true)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (ALPHA - default=false)
    JobTrackingWithFinalizers=true|false (BETA - default=true)
    KubeletCredentialProviders=true|false (ALPHA - default=false)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (ALPHA - default=false)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NodeSwap=true|false (ALPHA - default=false)
    NonPreemptingPriority=true|false (BETA - default=true)
    OpenAPIEnums=true|false (ALPHA - default=false)
    OpenAPIV3=true|false (ALPHA - default=false)
    PodAffinityNamespaceSelector=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodOverhead=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    PreferNominatedNode=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RemoveSelfLink=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    ServiceLBNodePortControl=true|false (BETA - default=true)
    ServiceLoadBalancerClass=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    SuspendJob=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=false)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)
    csiMigrationRBD=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (BETA - default=true)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=true)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationRBD=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    CronJobTimeZone=true|false (ALPHA - default=false)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (BETA - default=true)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (BETA - default=true)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (BETA - default=true)
    JobTrackingWithFinalizers=true|false (BETA - default=false)
    KubeletCredentialProviders=true|false (BETA - default=true)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (BETA - default=true)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NetworkPolicyStatus=true|false (ALPHA - default=false)
    NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
    NodeSwap=true|false (ALPHA - default=false)
    OpenAPIEnums=true|false (BETA - default=true)
    OpenAPIV3=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceIPStaticSubrange=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=true)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)This parameter is ignored if a config file is specified by --config.

    --healthz-bind-address ipport     Default: 0.0.0.0:10256 -

    The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '[::]:10256' for all IPv6 interfaces). Set empty to disable.

    +

    The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '[::]:10256' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.

    @@ -302,10 +274,38 @@ kube-proxy [flags] ---machine-id-file string     Default: "/etc/machine-id,/var/lib/dbus/machine-id" +--log_backtrace_at <a string in the form 'file:N'>     Default: :0 -

    Comma-separated list of files to check for machine-id. Use the first one that exists.

    +

    when logging hits line file:N, emit a stack trace

    + + + +--log_dir string + + +

    If non-empty, write log files in this directory

    + + + +--log_file string + + +

    If non-empty, use this log file

    + + + +--log_file_max_size uint     Default: 1800 + + +

    Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.

    + + + +--logtostderr     Default: true + + +

    log to standard error instead of files

    @@ -333,35 +333,56 @@ kube-proxy [flags] --metrics-bind-address ipport     Default: 127.0.0.1:10249 -

    The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable.

    +

    The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.

    --nodeport-addresses strings -

    A string slice of values which specify the addresses to use for NodePorts. Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32). The default empty string slice ([]) means to use all local addresses.

    +

    A string slice of values which specify the addresses to use for NodePorts. Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32). The default empty string slice ([]) means to use all local addresses. This parameter is ignored if a config file is specified by --config.

    + + + +--one_output + + +

    If true, only write logs to their native severity level (vs also writing to each lower severity level)

    --oom-score-adj int32     Default: -999 -

    The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]

    +

    The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]. This parameter is ignored if a config file is specified by --config.

    + + + +--pod-bridge-interface string + + +

    A bridge interface name in the cluster. Kube-proxy considers traffic as local if originating from an interface which matches the value. This argument should be set if DetectLocalMode is set to BridgeInterface.

    + + + +--pod-interface-name-prefix string + + +

    An interface prefix in the cluster. Kube-proxy considers traffic as local if originating from interfaces that match the given prefix. This argument should be set if DetectLocalMode is set to InterfaceNamePrefix.

    --profiling -

    If true enables profiling via web interface on /debug/pprof handler.

    +

    If true enables profiling via web interface on /debug/pprof handler. This parameter is ignored if a config file is specified by --config.

    --proxy-mode ProxyMode -

    Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' or 'kernelspace' (windows). If blank, use the best-available proxy (currently iptables). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.

    +

    Which proxy mode to use: 'iptables' (Linux-only), 'ipvs' (Linux-only), 'kernelspace' (Windows-only), or 'userspace' (Linux/Windows, deprecated). The default value is 'iptables' on Linux and 'userspace' on Windows.This parameter is ignored if a config file is specified by --config.

    @@ -375,7 +396,28 @@ kube-proxy [flags] --show-hidden-metrics-for-version string -

    The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.

    +

    The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is <major>.<minor>, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.This parameter is ignored if a config file is specified by --config.

    + + + +--skip_headers + + +

    If true, avoid header prefixes in the log messages

    + + + +--skip_log_headers + + +

    If true, avoid headers when opening log files

    + + + +--stderrthreshold int     Default: 2 + + +

    logs at or above this threshold go to stderr

    @@ -385,6 +427,13 @@ kube-proxy [flags]

    How long an idle UDP connection will be kept open (e.g. '250ms', '2s'). Must be greater than 0. Only applicable for proxy-mode=userspace

    + +-v, --v int + + +

    number for the log level verbosity

    + + --version version[=true] @@ -392,6 +441,13 @@ kube-proxy [flags]

    Print version information and quit

    + +--vmodule <comma-separated 'pattern=N' settings> + + +

    comma-separated list of pattern=N settings for file-filtered logging

    + + --write-config-to string diff --git a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md index 6f0115d6ac..ed0980e299 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md @@ -155,18 +155,11 @@ kube-scheduler [flags]

    This flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.

    - ---experimental-logging-sanitization - - -

    [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
    Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.

    - - --feature-gates <comma-separated 'key=True|False' pairs> -

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (ALPHA - default=false)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureDisk=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=false)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationOpenStack=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIStorageCapacity=true|false (BETA - default=true)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    CSRDuration=true|false (BETA - default=true)
    ControllerManagerLeaderMigration=true|false (BETA - default=true)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DefaultPodTopologySpread=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EfficientWatchResumption=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandCSIVolumes=true|false (BETA - default=true)
    ExpandInUsePersistentVolumes=true|false (BETA - default=true)
    ExpandPersistentVolumes=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (ALPHA - default=false)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (ALPHA - default=false)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (ALPHA - default=false)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    IndexedJob=true|false (BETA - default=true)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (ALPHA - default=false)
    JobTrackingWithFinalizers=true|false (BETA - default=true)
    KubeletCredentialProviders=true|false (ALPHA - default=false)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (ALPHA - default=false)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NodeSwap=true|false (ALPHA - default=false)
    NonPreemptingPriority=true|false (BETA - default=true)
    OpenAPIEnums=true|false (ALPHA - default=false)
    OpenAPIV3=true|false (ALPHA - default=false)
    PodAffinityNamespaceSelector=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodOverhead=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    PreferNominatedNode=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RemoveSelfLink=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    ServiceLBNodePortControl=true|false (BETA - default=true)
    ServiceLoadBalancerClass=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    SuspendJob=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=false)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)
    csiMigrationRBD=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    APIListChunking=true|false (BETA - default=true)
    APIPriorityAndFairness=true|false (BETA - default=true)
    APIResponseCompression=true|false (BETA - default=true)
    APIServerIdentity=true|false (ALPHA - default=false)
    APIServerTracing=true|false (ALPHA - default=false)
    AllAlpha=true|false (ALPHA - default=false)
    AllBeta=true|false (BETA - default=false)
    AnyVolumeDataSource=true|false (BETA - default=true)
    AppArmor=true|false (BETA - default=true)
    CPUManager=true|false (BETA - default=true)
    CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
    CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
    CPUManagerPolicyOptions=true|false (BETA - default=true)
    CSIInlineVolume=true|false (BETA - default=true)
    CSIMigration=true|false (BETA - default=true)
    CSIMigrationAWS=true|false (BETA - default=true)
    CSIMigrationAzureFile=true|false (BETA - default=true)
    CSIMigrationGCE=true|false (BETA - default=true)
    CSIMigrationPortworx=true|false (ALPHA - default=false)
    CSIMigrationRBD=true|false (ALPHA - default=false)
    CSIMigrationvSphere=true|false (BETA - default=false)
    CSIVolumeHealth=true|false (ALPHA - default=false)
    ContextualLogging=true|false (ALPHA - default=false)
    CronJobTimeZone=true|false (ALPHA - default=false)
    CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
    CustomResourceValidationExpressions=true|false (ALPHA - default=false)
    DaemonSetUpdateSurge=true|false (BETA - default=true)
    DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
    DevicePlugins=true|false (BETA - default=true)
    DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
    DisableCloudProviders=true|false (ALPHA - default=false)
    DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
    DownwardAPIHugePages=true|false (BETA - default=true)
    EndpointSliceTerminatingCondition=true|false (BETA - default=true)
    EphemeralContainers=true|false (BETA - default=true)
    ExpandedDNSConfig=true|false (ALPHA - default=false)
    ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
    GRPCContainerProbe=true|false (BETA - default=true)
    GracefulNodeShutdown=true|false (BETA - default=true)
    GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
    HPAContainerMetrics=true|false (ALPHA - default=false)
    HPAScaleToZero=true|false (ALPHA - default=false)
    HonorPVReclaimPolicy=true|false (ALPHA - default=false)
    IdentifyPodOS=true|false (BETA - default=true)
    InTreePluginAWSUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
    InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
    InTreePluginGCEUnregister=true|false (ALPHA - default=false)
    InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
    InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
    InTreePluginRBDUnregister=true|false (ALPHA - default=false)
    InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
    JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
    JobReadyPods=true|false (BETA - default=true)
    JobTrackingWithFinalizers=true|false (BETA - default=false)
    KubeletCredentialProviders=true|false (BETA - default=true)
    KubeletInUserNamespace=true|false (ALPHA - default=false)
    KubeletPodResources=true|false (BETA - default=true)
    KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
    LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
    LocalStorageCapacityIsolation=true|false (BETA - default=true)
    LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
    LogarithmicScaleDown=true|false (BETA - default=true)
    MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
    MemoryManager=true|false (BETA - default=true)
    MemoryQoS=true|false (ALPHA - default=false)
    MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
    MixedProtocolLBService=true|false (BETA - default=true)
    NetworkPolicyEndPort=true|false (BETA - default=true)
    NetworkPolicyStatus=true|false (ALPHA - default=false)
    NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
    NodeSwap=true|false (ALPHA - default=false)
    OpenAPIEnums=true|false (BETA - default=true)
    OpenAPIV3=true|false (BETA - default=true)
    PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
    PodDeletionCost=true|false (BETA - default=true)
    PodSecurity=true|false (BETA - default=true)
    ProbeTerminationGracePeriod=true|false (BETA - default=false)
    ProcMountType=true|false (ALPHA - default=false)
    ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
    QOSReserved=true|false (ALPHA - default=false)
    ReadWriteOncePod=true|false (ALPHA - default=false)
    RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
    RemainingItemCount=true|false (BETA - default=true)
    RotateKubeletServerCertificate=true|false (BETA - default=true)
    SeccompDefault=true|false (ALPHA - default=false)
    ServerSideFieldValidation=true|false (ALPHA - default=false)
    ServiceIPStaticSubrange=true|false (ALPHA - default=false)
    ServiceInternalTrafficPolicy=true|false (BETA - default=true)
    SizeMemoryBackedVolumes=true|false (BETA - default=true)
    StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
    StatefulSetMinReadySeconds=true|false (BETA - default=true)
    StorageVersionAPI=true|false (ALPHA - default=false)
    StorageVersionHash=true|false (BETA - default=true)
    TopologyAwareHints=true|false (BETA - default=true)
    TopologyManager=true|false (BETA - default=true)
    VolumeCapacityPriority=true|false (ALPHA - default=false)
    WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)

    @@ -236,7 +229,7 @@ kube-scheduler [flags] --leader-elect-resource-lock string     Default: "leases" -

    The type of resource object that is used for locking during leader election. Supported options are 'endpoints', 'configmaps', 'leases', 'endpointsleases' and 'configmapsleases'.

    +

    The type of resource object that is used for locking during leader election. Supported options are 'leases', 'endpointsleases' and 'configmapsleases'.

    @@ -309,6 +302,13 @@ kube-scheduler [flags]

    If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]

    + +--pod-max-in-unschedulable-pods-duration duration     Default: 5m0s + + +

    DEPRECATED: the maximum time a pod can stay in unschedulablePods. If a pod stays in unschedulablePods for longer than this value, the pod will be moved from unschedulablePods to backoffQ or activeQ. This flag is deprecated and will be removed in 1.26

    + + --profiling     Default: true diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet.md b/content/en/docs/reference/command-line-tools-reference/kubelet.md index 555414a6eb..b10e1b7057 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet.md @@ -44,70 +44,70 @@ kubelet [flags] --add-dir-header -If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see here.) --address string     Default: 0.0.0.0 -The IP address for the Kubelet to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The IP address for the Kubelet to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --allowed-unsafe-sysctls strings -Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --alsologtostderr -Log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +Log to standard error as well as files (DEPRECATED: will be removed in a future release, see here.) --anonymous-auth     Default: true -Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authentication-token-webhook -Use the TokenReview API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Use the TokenReview API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authentication-token-webhook-cache-ttl duration     Default: 2m0s -The duration to cache responses from the webhook token authenticator. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The duration to cache responses from the webhook token authenticator. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authorization-mode string     Default: AlwaysAllow -Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authorization-webhook-cache-authorized-ttl duration     Default: 5m0s -The duration to cache 'authorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The duration to cache 'authorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --authorization-webhook-cache-unauthorized-ttl duration     Default: 30s -The duration to cache 'unauthorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The duration to cache 'unauthorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -135,28 +135,28 @@ kubelet [flags] --cgroup-driver string     Default: cgroupfs -Driver that the kubelet uses to manipulate cgroups on the host. Possible values: cgroupfs, systemd. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)/td> +Driver that the kubelet uses to manipulate cgroups on the host. Possible values: cgroupfs, systemd. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cgroup-root string     Default: '' -Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cgroups-per-qos     Default: true -Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --client-ca-file string -If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -177,14 +177,14 @@ kubelet [flags] --cluster-dns strings -Comma-separated list of DNS server IP address. This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst".
    Note: all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Comma-separated list of DNS server IP address. This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst".
    Note: all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cluster-domain string -Domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -219,14 +219,14 @@ kubelet [flags] --container-log-max-files int32     Default: 5 -<Warning: Beta feature> Set the maximum number of container log files that can be present for a container. The number must be >= 2. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Beta feature> Set the maximum number of container log files that can be present for a container. The number must be >= 2. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --container-log-max-size string     Default: 10Mi -<Warning: Beta feature> Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Beta feature> Set the maximum size (e.g. 10Mi) of container log file before it is rotated. This flag can only be used with --container-runtime=remote. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -248,42 +248,42 @@ kubelet [flags] --contention-profiling -Enable lock contention profiling, if profiling is enabled (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable lock contention profiling, if profiling is enabled (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-cfs-quota     Default: true -Enable CPU CFS quota enforcement for containers that specify CPU limits (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable CPU CFS quota enforcement for containers that specify CPU limits (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-cfs-quota-period duration     Default: 100ms -Sets CPU CFS quota period value, cpu.cfs_period_us, defaults to Linux Kernel default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Sets CPU CFS quota period value, cpu.cfs_period_us, defaults to Linux Kernel default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-manager-policy string     Default: none -CPU Manager policy to use. Possible values: none, static. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +CPU Manager policy to use. Possible values: none, static. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-manager-policy-options mapStringString -Comma-separated list of options to fine-tune the behavior of the selected CPU Manager policy. If not supplied, keep the default behaviour. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Comma-separated list of options to fine-tune the behavior of the selected CPU Manager policy. If not supplied, keep the default behaviour. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --cpu-manager-reconcile-period duration     Default: 10s -<Warning: Alpha feature> CPU Manager reconciliation period. Examples: 10s, or 1m. If not supplied, defaults to node status update frequency. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Alpha feature> CPU Manager reconciliation period. Examples: 10s, or 1m. If not supplied, defaults to node status update frequency. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -304,84 +304,84 @@ kubelet [flags] --enable-controller-attach-detach     Default: true -Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --enable-debugging-handlers     Default: true -Enables server endpoints for log collection and local running of containers and commands. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enables server endpoints for log collection and local running of containers and commands. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --enable-server     Default: true -Enable the Kubelet's server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Enable the Kubelet's server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --enforce-node-allocatable strings     Default: pods -A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptable options are none, pods, system-reserved, and kube-reserved. If the latter two options are specified, --system-reserved-cgroup and --kube-reserved-cgroup must also be set, respectively. If none is specified, no additional options should be set. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptable options are none, pods, system-reserved, and kube-reserved. If the latter two options are specified, --system-reserved-cgroup and --kube-reserved-cgroup must also be set, respectively. If none is specified, no additional options should be set. See here for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --event-burst int32     Default: 10 -Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding --event-qps. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding --event-qps. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --event-qps int32     Default: 5 -QPS to limit event creations. The number must be >= 0. If 0 will use default QPS (5). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +QPS to limit event creations. The number must be >= 0. If 0 will use default QPS (5). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-hard mapStringString     Default: imagefs.available<15%,memory.available<100Mi,nodefs.available<10% -A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction. On a Linux node, the default value also includes nodefs.inodesFree<5%. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction. On a Linux node, the default value also includes nodefs.inodesFree<5%. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-max-pod-grace-period int32 - Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) + Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-minimum-reclaim mapStringString -A set of minimum reclaims (e.g. imagefs.available=2Gi) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of minimum reclaims (e.g. imagefs.available=2Gi) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-pressure-transition-period duration     Default: 5m0s -Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-soft mapStringString -A set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --eviction-soft-grace-period mapStringString -A set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -395,7 +395,7 @@ kubelet [flags] --experimental-allocatable-ignore-eviction     Default: false -When set to true, hard eviction thresholds will be ignored while calculating node allocatable. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (DEPRECATED: will be removed in 1.24 or later) +When set to true, hard eviction thresholds will be ignored while calculating node allocatable. See here for more details. (DEPRECATED: will be removed in 1.24 or later) @@ -409,14 +409,14 @@ kubelet [flags] --experimental-kernel-memcg-notification -Use kernelMemcgNotification configuration, this flag will be removed in 1.24 or later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Use kernelMemcgNotification configuration, this flag will be removed in 1.24 or later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --experimental-log-sanitization bool -[Experimental] When enabled, prevents logging of fields tagged as sensitive (passwords, keys, tokens). Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +[Experimental] When enabled, prevents logging of fields tagged as sensitive (passwords, keys, tokens). Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -430,7 +430,7 @@ kubelet [flags] --fail-swap-on     Default: true -Makes the Kubelet fail to start if swap is enabled on the node. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Makes the Kubelet fail to start if swap is enabled on the node. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -548,35 +548,35 @@ WinDSR=true|false (ALPHA - default=false)
    WinOverlay=true|false (BETA - default=true)
    WindowsHostProcessContainers=true|false (BETA - default=true)
    csiMigrationRBD=true|false (ALPHA - default=false)
    -(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --file-check-frequency duration     Default: 20s -Duration between checking config files for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Duration between checking config files for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --hairpin-mode string     Default: promiscuous-bridge -How should the kubelet setup hairpin NAT. This allows endpoints of a Service to load balance back to themselves if they should try to access their own Service. Valid values are promiscuous-bridge, hairpin-veth and none. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +How should the kubelet setup hairpin NAT. This allows endpoints of a Service to load balance back to themselves if they should try to access their own Service. Valid values are promiscuous-bridge, hairpin-veth and none. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --healthz-bind-address string     Default: 127.0.0.1 -The IP address for the healthz server to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The IP address for the healthz server to serve on (set to 0.0.0.0 or :: for listening in all interfaces and IP families). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --healthz-port int32     Default: 10248 -The port of the localhost healthz endpoint (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The port of the localhost healthz endpoint (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -597,7 +597,7 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --http-check-frequency duration     Default: 20s -Duration between checking HTTP for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Duration between checking HTTP for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -618,14 +618,14 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --image-gc-high-threshold int32     Default: 85 -The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --image-gc-low-threshold int32     Default: 80 -The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -646,14 +646,14 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --iptables-drop-bit int32     Default: 15 -The bit of the fwmark space to mark packets for dropping. Must be within the range [0, 31]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The bit of the fwmark space to mark packets for dropping. Must be within the range [0, 31]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --iptables-masquerade-bit int32     Default: 14 -The bit of the fwmark space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in kube-proxy. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The bit of the fwmark space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in kube-proxy. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -667,42 +667,42 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --kernel-memcg-notification -If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-api-burst int32     Default: 10 -Burst to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Burst to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-api-content-type string     Default: application/vnd.kubernetes.protobuf -Content type of requests sent to apiserver. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Content type of requests sent to apiserver. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-api-qps int32     Default: 5 -QPS to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default QPS (5). Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +QPS to use while talking with kubernetes API server. The number must be >= 0. If 0 will use default QPS (5). Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-reserved mapStringString     Default: <None> -A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local ephemeral-storage for root file system are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local ephemeral-storage for root file system are supported. See here for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --kube-reserved-cgroup string     Default: '' -Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via --kube-reserved flag. Ex. /kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via --kube-reserved flag. Ex. /kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -716,7 +716,7 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --kubelet-cgroups string -Optional absolute name of cgroups to create and run the Kubelet in. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Optional absolute name of cgroups to create and run the Kubelet in. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -730,28 +730,28 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --log-backtrace-at <A string of format 'file:line'>     Default: ":0" -When logging hits line :, emit a stack trace. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +When logging hits line :, emit a stack trace. (DEPRECATED: will be removed in a future release, see here.) --log-dir string -If non-empty, write log files in this directory. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +If non-empty, write log files in this directory. (DEPRECATED: will be removed in a future release, see here.) --log-file string -If non-empty, use this log file. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +If non-empty, use this log file. (DEPRECATED: will be removed in a future release, see here.) --log-file-max-size uint     Default: 1800 -Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see here.) @@ -765,49 +765,49 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --log-json-info-buffer-size string     Default: '0' -[Experimental] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +[Experimental] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --log-json-split-stream -[Experimental] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +[Experimental] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --logging-format string     Default: text -Sets the log format. Permitted formats: text, json.
    Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --skip_headers, --skip_log_headers, --stderrthreshold, --log-flush-frequency.
    Non-default choices are currently alpha and subject to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Sets the log format. Permitted formats: text, json.
    Non-default formats don't honor these flags: --add-dir-header, --alsologtostderr, --log-backtrace-at, --log-dir, --log-file, --log-file-max-size, --logtostderr, --skip_headers, --skip_log_headers, --stderrthreshold, --log-flush-frequency.
    Non-default choices are currently alpha and subject to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --logtostderr     Default: true -log to standard error instead of files. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +log to standard error instead of files. (DEPRECATED: will be removed in a future release, see here.) --make-iptables-util-chains     Default: true -If true, kubelet will ensure iptables utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If true, kubelet will ensure iptables utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --manifest-url string -URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --manifest-url-header string -Comma-separated list of HTTP headers to use when accessing the URL provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Comma-separated list of HTTP headers to use when accessing the URL provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -821,14 +821,14 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --max-open-files int     Default: 1000000 -Number of files that can be opened by Kubelet process. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Number of files that can be opened by Kubelet process. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --max-pods int32     Default: 110 -Number of Pods that can run on this Kubelet. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Number of Pods that can run on this Kubelet. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -849,7 +849,7 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --memory-manager-policy string     Default: None -Memory Manager policy to use. Possible values: 'None', 'Static'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Memory Manager policy to use. Possible values: 'None', 'Static'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -863,7 +863,7 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --minimum-image-ttl-duration duration     Default: 2m0s -Minimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Minimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -898,14 +898,14 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --node-status-max-images int32     Default: 50 -The maximum number of images to report in node.status.images. If -1 is specified, no cap will be applied. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The maximum number of images to report in node.status.images. If -1 is specified, no cap will be applied. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --node-status-update-frequency duration     Default: 10s -Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in Node controller. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in Node controller. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -919,21 +919,21 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --one-output -If true, only write logs to their native severity level (vs also writing to each lower severity level). (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +If true, only write logs to their native severity level (vs also writing to each lower severity level). (DEPRECATED: will be removed in a future release, see here.) --oom-score-adj int32     Default: -999 -The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --pod-cidr string -The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -947,56 +947,56 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --pod-manifest-path string -Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --pod-max-pids int     Default: -1 -Set the maximum number of processes per pod. If -1, the kubelet defaults to the node allocatable PID capacity. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Set the maximum number of processes per pod. If -1, the kubelet defaults to the node allocatable PID capacity. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --pods-per-core int32 -Number of Pods per core that can run on this kubelet. The total number of pods on this kubelet cannot exceed --max-pods, so --max-pods will be used if this calculation results in a larger number of pods allowed on the kubelet. A value of 0 disables this limit. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Number of Pods per core that can run on this kubelet. The total number of pods on this kubelet cannot exceed --max-pods, so --max-pods will be used if this calculation results in a larger number of pods allowed on the kubelet. A value of 0 disables this limit. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --port int32     Default: 10250 -The port for the kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The port for the kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the kubelet's --config flag. See kubelet-config-file for more information.) --protect-kernel-defaults - Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) + Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --provider-id string -Unique identifier for identifying the node in a machine database, i.e cloud provider. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Unique identifier for identifying the node in a machine database, i.e cloud provider. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --qos-reserved mapStringString -<Warning: Alpha feature> A set of <resource name>=<percentage> (e.g. memory=50%) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. Requires the QOSReserved feature gate to be enabled. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Alpha feature> A set of <resource name>=<percentage> (e.g. memory=50%) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. Requires the QOSReserved feature gate to be enabled. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --read-only-port int32     Default: 10255 -The read-only port for the kubelet to serve on with no authentication/authorization (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The read-only port for the kubelet to serve on with no authentication/authorization (set to 0 to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1010,7 +1010,7 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --register-node     Default: true -Register the node with the API server. If --kubeconfig is not provided, this flag is irrelevant, as the Kubelet won't have an API server to register with. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Register the node with the API server. If --kubeconfig is not provided, this flag is irrelevant, as the Kubelet won't have an API server to register with. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1024,42 +1024,42 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --register-with-taints mapStringString -Register the node with the given list of taints (comma separated <key>=<value>:<effect>). No-op if --register-node is false. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Register the node with the given list of taints (comma separated <key>=<value>:<effect>). No-op if --register-node is false. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --registry-burst int32     Default: 10 -Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding --registry-qps. Only used if --registry-qps is greater than 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding --registry-qps. Only used if --registry-qps is greater than 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --registry-qps int32     Default: 5 -If > 0, limit registry pull QPS to this value. If 0, unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If > 0, limit registry pull QPS to this value. If 0, unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --reserved-cpus string -A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in --system-reserved and --kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in --system-reserved and --kube-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --reserved-memory string -A comma-separated list of memory reservations for NUMA nodes. (e.g. --reserved-memory 0:memory=1Gi,hugepages-1M=2Gi --reserved-memory 1:memory=2Gi). The total sum for each memory type should be equal to the sum of --kube-reserved, --system-reserved and --eviction-threshold. See https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/#reserved-memory-flag for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A comma-separated list of memory reservations for NUMA nodes. (e.g. --reserved-memory 0:memory=1Gi,hugepages-1M=2Gi --reserved-memory 1:memory=2Gi). The total sum for each memory type should be equal to the sum of --kube-reserved, --system-reserved and --eviction-threshold. See here for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --resolv-conf string     Default: /etc/resolv.conf -Resolver configuration file used as the basis for the container DNS resolution configuration. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Resolver configuration file used as the basis for the container DNS resolution configuration. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1073,21 +1073,21 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --rotate-certificates -<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --rotate-server-certificates -Auto-request and rotate the kubelet serving certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. Requires the RotateKubeletServerCertificate feature gate to be enabled, and approval of the submitted CertificateSigningRequest objects. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Auto-request and rotate the kubelet serving certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. Requires the RotateKubeletServerCertificate feature gate to be enabled, and approval of the submitted CertificateSigningRequest objects. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --runonce -If true, exit after spawning pods from local manifests or remote urls. Exclusive with --enable-server (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +If true, exit after spawning pods from local manifests or remote urls. Exclusive with --enable-server (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1101,7 +1101,7 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --runtime-request-timeout duration     Default: 2m0s -Timeout of all runtime requests except long running request - pull, logs, exec and attach. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Timeout of all runtime requests except long running request - pull, logs, exec and attach. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1115,70 +1115,70 @@ csiMigrationRBD=true|false (ALPHA - default=false)
    --serialize-image-pulls     Default: true -Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an aufs storage backend. Issue #10959 has more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an aufs storage backend. Issue #10959 has more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --skip-headers -If true, avoid header prefixes in the log messages. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +If true, avoid header prefixes in the log messages. (DEPRECATED: will be removed in a future release, see here.) --skip-log-headers -If true, avoid headers when opening log files. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +If true, avoid headers when opening log files. (DEPRECATED: will be removed in a future release, see here.) --stderrthreshold int     Default: 2 -logs at or above this threshold go to stderr. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +logs at or above this threshold go to stderr. (DEPRECATED: will be removed in a future release, see here.) --streaming-connection-idle-timeout duration     Default: 4h0m0s -Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: 5m. Note: All connections to the kubelet server have a maximum duration of 4 hours. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: 5m. Note: All connections to the kubelet server have a maximum duration of 4 hours. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --sync-frequency duration     Default: 1m0s -Max period between synchronizing running containers and config. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Max period between synchronizing running containers and config. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --system-cgroups string -Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under '/'. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under '/'. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --system-reserved mapStringString     Default: <none> -A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +A set of <resource name>=<resource quantity> (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100') pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See here for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --system-reserved-cgroup string     Default: '' -Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via --system-reserved flag. Ex. /system-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via --system-reserved flag. Ex. /system-reserved. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --tls-cert-file string -File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1190,21 +1190,21 @@ Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384
    Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA. -(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --tls-min-version string -Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --tls-private-key-file string -File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1212,14 +1212,14 @@ TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_E --topology-manager-policy string     Default: 'none' -Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --topology-manager-scope string     Default: container -Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) @@ -1247,14 +1247,14 @@ TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_E --volume-plugin-dir string     Default: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ -The full path of the directory in which to search for additional third party volume plugins. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +The full path of the directory in which to search for additional third party volume plugins. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) --volume-stats-agg-period duration     Default: 1m0s -Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) +Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See kubelet-config-file for more information.) diff --git a/content/en/docs/reference/config-api/apiserver-audit.v1.md b/content/en/docs/reference/config-api/apiserver-audit.v1.md index 3cca579663..e48824c649 100644 --- a/content/en/docs/reference/config-api/apiserver-audit.v1.md +++ b/content/en/docs/reference/config-api/apiserver-audit.v1.md @@ -89,7 +89,15 @@ For non-resource requests, this is the lower-cased HTTP method.

    []string -

    Source IPs, from where the request originated and intermediate proxies.

    +

    Source IPs, from where the request originated and intermediate proxies. +The source IPs are listed from (in order):

    +
      +
    1. X-Forwarded-For request header IPs
      2. +
    2. X-Real-Ip header, if not present in the X-Forwarded-For list
      3. +
    3. The remote address for the connection, if it doesn't match the last +IP in the list up to here (X-Forwarded-For or X-Real-Ip). +Note: All but the last IP can be arbitrarily set by the client.
      4. +
    userAgent
    diff --git a/content/en/docs/reference/config-api/apiserver-eventratelimit.v1alpha1.md b/content/en/docs/reference/config-api/apiserver-eventratelimit.v1alpha1.md new file mode 100644 index 0000000000..2189c4910d --- /dev/null +++ b/content/en/docs/reference/config-api/apiserver-eventratelimit.v1alpha1.md @@ -0,0 +1,121 @@ +--- +title: Event Rate Limit Configuration (v1alpha1) +content_type: tool-reference +package: eventratelimit.admission.k8s.io/v1alpha1 +auto_generated: true +--- + + +## Resource Types + + +- [Configuration](#eventratelimit-admission-k8s-io-v1alpha1-Configuration) + + + +## `Configuration` {#eventratelimit-admission-k8s-io-v1alpha1-Configuration} + + + +

    Configuration provides configuration for the EventRateLimit admission +controller.

    + + + + + + + + + + + + + + +
    FieldDescription
    apiVersion
    string    		eventratelimit.admission.k8s.io/v1alpha1
    kind
    string    		Configuration
    limits [Required]
    +[]Limit +    		 +

    limits are the limits to place on event queries received. +Limits can be placed on events received server-wide, per namespace, +per user, and per source+object. +At least one limit is required.

    +
    + +## `Limit` {#eventratelimit-admission-k8s-io-v1alpha1-Limit} + + +**Appears in:** + +- [Configuration](#eventratelimit-admission-k8s-io-v1alpha1-Configuration) + + +

    Limit is the configuration for a particular limit type

    + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    type [Required]
    +LimitType +    		 +

    type is the type of limit to which this configuration applies

    +
    qps [Required]
    +int32 +    		 +

    qps is the number of event queries per second that are allowed for this +type of limit. The qps and burst fields are used together to determine if +a particular event query is accepted. The qps determines how many queries +are accepted once the burst amount of queries has been exhausted.

    +
    burst [Required]
    +int32 +    		 +

    burst is the burst number of event queries that are allowed for this type +of limit. The qps and burst fields are used together to determine if a +particular event query is accepted. The burst determines the maximum size +of the allowance granted for a particular bucket. For example, if the burst +is 10 and the qps is 3, then the admission control will accept 10 queries +before blocking any queries. Every second, 3 more queries will be allowed. +If some of that allowance is not used, then it will roll over to the next +second, until the maximum allowance of 10 is reached.

    +
    cacheSize
    +int32 +    		 +

    cacheSize is the size of the LRU cache for this type of limit. If a bucket +is evicted from the cache, then the allowance for that bucket is reset. If +more queries are later received for an evicted bucket, then that bucket +will re-enter the cache with a clean slate, giving that bucket a full +allowance of burst queries.

    +

    The default cache size is 4096.

    +

    If limitType is 'server', then cacheSize is ignored.

    +
    + +## `LimitType` {#eventratelimit-admission-k8s-io-v1alpha1-LimitType} + +(Alias of `string`) + +**Appears in:** + +- [Limit](#eventratelimit-admission-k8s-io-v1alpha1-Limit) + + +

    LimitType is the type of the limit (e.g., per-namespace)

    + + + + \ No newline at end of file diff --git a/content/en/docs/reference/config-api/imagepolicy.v1alpha1.md b/content/en/docs/reference/config-api/imagepolicy.v1alpha1.md new file mode 100644 index 0000000000..f420623559 --- /dev/null +++ b/content/en/docs/reference/config-api/imagepolicy.v1alpha1.md @@ -0,0 +1,168 @@ +--- +title: Image Policy API (v1alpha1) +content_type: tool-reference +package: imagepolicy.k8s.io/v1alpha1 +auto_generated: true +--- + + +## Resource Types + + +- [ImageReview](#imagepolicy-k8s-io-v1alpha1-ImageReview) + + + +## `ImageReview` {#imagepolicy-k8s-io-v1alpha1-ImageReview} + + + +

    ImageReview checks if the set of images in a pod are allowed.

    + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    apiVersion
    string    		imagepolicy.k8s.io/v1alpha1
    kind
    string    		ImageReview
    metadata
    +meta/v1.ObjectMeta +    		 +

    Standard object's metadata. +More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

    +Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec [Required]
    +ImageReviewSpec +    		 +

    Spec holds information about the pod being evaluated

    +
    status
    +ImageReviewStatus +    		 +

    Status is filled in by the backend and indicates whether the pod should be allowed.

    +
    + +## `ImageReviewContainerSpec` {#imagepolicy-k8s-io-v1alpha1-ImageReviewContainerSpec} + + +**Appears in:** + +- [ImageReviewSpec](#imagepolicy-k8s-io-v1alpha1-ImageReviewSpec) + + +

    ImageReviewContainerSpec is a description of a container within the pod creation request.

    + + + + + + + + + + + +
    FieldDescription
    image
    +string +    		 +

    This can be in the form image:tag or image@SHA:012345679abcdef.

    +
    + +## `ImageReviewSpec` {#imagepolicy-k8s-io-v1alpha1-ImageReviewSpec} + + +**Appears in:** + +- [ImageReview](#imagepolicy-k8s-io-v1alpha1-ImageReview) + + +

    ImageReviewSpec is a description of the pod creation request.

    + + + + + + + + + + + + + + + + + +
    FieldDescription
    containers
    +[]ImageReviewContainerSpec +    		 +

    Containers is a list of a subset of the information in each container of the Pod being created.

    +
    annotations
    +map[string]string +    		 +

    Annotations is a list of key-value pairs extracted from the Pod's annotations. +It only includes keys which match the pattern *.image-policy.k8s.io/*. +It is up to each webhook backend to determine how to interpret these annotations, if at all.

    +
    namespace
    +string +    		 +

    Namespace is the namespace the pod is being created in.

    +
    + +## `ImageReviewStatus` {#imagepolicy-k8s-io-v1alpha1-ImageReviewStatus} + + +**Appears in:** + +- [ImageReview](#imagepolicy-k8s-io-v1alpha1-ImageReview) + + +

    ImageReviewStatus is the result of the review for the pod creation request.

    + + + + + + + + + + + + + + + + + +
    FieldDescription
    allowed [Required]
    +bool +    		 +

    Allowed indicates that all images were allowed to be run.

    +
    reason
    +string +    		 +

    Reason should be empty unless Allowed is false in which case it +may contain a short description of what is wrong. Kubernetes +may truncate excessively long errors when displaying to the user.

    +
    auditAnnotations
    +map[string]string +    		 +

    AuditAnnotations will be added to the attributes object of the +admission controller request using 'AddAnnotation'. The keys should +be prefix-less (i.e., the admission controller will add an +appropriate prefix).

    +
    + \ No newline at end of file diff --git a/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md b/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md index b3d1fc98ff..8c794f2755 100644 --- a/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md +++ b/content/en/docs/reference/config-api/kube-proxy-config.v1alpha1.md @@ -193,6 +193,50 @@ An empty string slice is meant to select all network interfaces.

    DetectLocalMode determines mode to use for detecting local traffic, defaults to LocalModeClusterCIDR

    +detectLocal [Required]
    +DetectLocalConfiguration + + +

    DetectLocal contains optional configuration settings related to DetectLocalMode.

    + + + + + +## `DetectLocalConfiguration` {#kubeproxy-config-k8s-io-v1alpha1-DetectLocalConfiguration} + + +**Appears in:** + +- [KubeProxyConfiguration](#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration) + + +

    DetectLocalConfiguration contains optional settings related to DetectLocalMode option

    + + + + + + + + + + + + +
    FieldDescription
    bridgeInterface [Required]
    +string +    		 +

    BridgeInterface is a string argument which represents a single bridge interface name. +Kube-proxy considers traffic as local if originating from this given bridge. +This argument should be set if DetectLocalMode is set to LocalModeBridgeInterface.

    +
    interfaceNamePrefix [Required]
    +string +    		 +

    InterfaceNamePrefix is a string argument which represents a single interface prefix name. +Kube-proxy considers traffic as local if originating from one or more interfaces which match +the given prefix. This argument should be set if DetectLocalMode is set to LocalModeInterfaceNamePrefix.

    +
    @@ -424,6 +468,22 @@ NAT when loadbalancing

    with DSR

    +rootHnsEndpointName [Required]
    +string + + +

    RootHnsEndpointName is the name of hnsendpoint that is attached to +l2bridge for root network namespace

    + + +forwardHealthCheckVip [Required]
    +bool + + +

    ForwardHealthCheckVip forwards service VIP for health check port on +Windows

    + + @@ -747,8 +807,9 @@ default value of format is text

    time.Duration -

    Maximum number of seconds between log flushes. Ignored if the -selected logging backend writes log messages without buffering.

    +

    Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

    verbosity [Required]
    @@ -769,14 +830,6 @@ are always logged.

    Only supported for "text" log format.

    -sanitization [Required]
    -bool - - -

    [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)

    - - options [Required]
    FormatOptions diff --git a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md index 8c5f981659..9025c78dcb 100644 --- a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md +++ b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta2.md @@ -285,8 +285,9 @@ default value of format is text

    time.Duration -

    Maximum number of seconds between log flushes. Ignored if the -selected logging backend writes log messages without buffering.

    +

    Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

    verbosity [Required]
    @@ -307,14 +308,6 @@ are always logged.

    Only supported for "text" log format.

    -sanitization [Required]
    -bool - - -

    [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)

    - - options [Required]
    FormatOptions @@ -663,8 +656,7 @@ of "System" or "List".

    Nodes and Zones.
  • "List": Use constraints defined in .defaultConstraints.
    • -

    Defaults to "List" if feature gate DefaultPodTopologySpread is disabled -and to "System" if enabled.

    +

    Defaults to "System".

    diff --git a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta3.md b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta3.md index 6c8635415e..aeddbe89be 100644 --- a/content/en/docs/reference/config-api/kube-scheduler-config.v1beta3.md +++ b/content/en/docs/reference/config-api/kube-scheduler-config.v1beta3.md @@ -20,6 +20,329 @@ auto_generated: true +## `ClientConnectionConfiguration` {#ClientConnectionConfiguration} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + + +

    ClientConnectionConfiguration contains details for constructing a client.

    + + + + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    kubeconfig [Required]
    +string +    		 +

    kubeconfig is the path to a KubeConfig file.

    +
    acceptContentTypes [Required]
    +string +    		 +

    acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the +default value of 'application/json'. This field will control all connections to the server used by a particular +client.

    +
    contentType [Required]
    +string +    		 +

    contentType is the content type used when sending data to the server from this client.

    +
    qps [Required]
    +float32 +    		 +

    qps controls the number of queries per second allowed for this connection.

    +
    burst [Required]
    +int32 +    		 +

    burst allows extra queries to accumulate when a client is exceeding its rate.

    +
    + +## `DebuggingConfiguration` {#DebuggingConfiguration} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) + + +

    DebuggingConfiguration holds configuration for Debugging related features.

    + + + + + + + + + + + + + + +
    FieldDescription
    enableProfiling [Required]
    +bool +    		 +

    enableProfiling enables profiling via web interface host:port/debug/pprof/

    +
    enableContentionProfiling [Required]
    +bool +    		 +

    enableContentionProfiling enables lock contention profiling, if +enableProfiling is true.

    +
    + +## `FormatOptions` {#FormatOptions} + + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

    FormatOptions contains options for the different logging formats.

    + + + + + + + + + + + +
    FieldDescription
    json [Required]
    +JSONOptions +    		 +

    [Experimental] JSON contains options for logging format "json".

    +
    + +## `JSONOptions` {#JSONOptions} + + +**Appears in:** + +- [FormatOptions](#FormatOptions) + + +

    JSONOptions contains options for logging format "json".

    + + + + + + + + + + + + + + +
    FieldDescription
    splitStream [Required]
    +bool +    		 +

    [Experimental] SplitStream redirects error messages to stderr while +info messages go to stdout, with buffering. The default is to write +both to stdout, without buffering.

    +
    infoBufferSize [Required]
    +k8s.io/apimachinery/pkg/api/resource.QuantityValue +    		 +

    [Experimental] InfoBufferSize sets the size of the info stream when +using split streams. The default is zero, which disables buffering.

    +
    + +## `LeaderElectionConfiguration` {#LeaderElectionConfiguration} + + +**Appears in:** + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) + +- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) + + +

    LeaderElectionConfiguration defines the configuration of leader election +clients for components that can run with leader election enabled.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    leaderElect [Required]
    +bool +    		 +

    leaderElect enables a leader election client to gain leadership +before executing the main loop. Enable this when running replicated +components for high availability.

    +
    leaseDuration [Required]
    +meta/v1.Duration +    		 +

    leaseDuration is the duration that non-leader candidates will wait +after observing a leadership renewal until attempting to acquire +leadership of a led but unrenewed leader slot. This is effectively the +maximum duration that a leader can be stopped before it is replaced +by another candidate. This is only applicable if leader election is +enabled.

    +
    renewDeadline [Required]
    +meta/v1.Duration +    		 +

    renewDeadline is the interval between attempts by the acting master to +renew a leadership slot before it stops leading. This must be less +than or equal to the lease duration. This is only applicable if leader +election is enabled.

    +
    retryPeriod [Required]
    +meta/v1.Duration +    		 +

    retryPeriod is the duration the clients should wait between attempting +acquisition and renewal of a leadership. This is only applicable if +leader election is enabled.

    +
    resourceLock [Required]
    +string +    		 +

    resourceLock indicates the resource object type that will be used to lock +during leader election cycles.

    +
    resourceName [Required]
    +string +    		 +

    resourceName indicates the name of resource object that will be used to lock +during leader election cycles.

    +
    resourceNamespace [Required]
    +string +    		 +

    resourceName indicates the namespace of resource object that will be used to lock +during leader election cycles.

    +
    + +## `LoggingConfiguration` {#LoggingConfiguration} + + +**Appears in:** + +- [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) + + +

    LoggingConfiguration contains logging options +Refer Logs Options for more information.

    + + + + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    format [Required]
    +string +    		 +

    Format Flag specifies the structure of log messages. +default value of format is text

    +
    flushFrequency [Required]
    +time.Duration +    		 +

    Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

    +
    verbosity [Required]
    +uint32 +    		 +

    Verbosity is the threshold that determines which log messages are +logged. Default is zero which logs only the most important +messages. Higher values enable additional messages. Error messages +are always logged.

    +
    vmodule [Required]
    +VModuleConfiguration +    		 +

    VModule overrides the verbosity threshold for individual files. +Only supported for "text" log format.

    +
    options [Required]
    +FormatOptions +    		 +

    [Experimental] Options holds additional parameters that are specific +to the different logging formats. Only the options for the selected +format get used, but all of them get validated.

    +
    + +## `VModuleConfiguration` {#VModuleConfiguration} + +(Alias of `[]k8s.io/component-base/config/v1alpha1.VModuleItem`) + +**Appears in:** + +- [LoggingConfiguration](#LoggingConfiguration) + + +

    VModuleConfiguration is a collection of individual file names or patterns +and the corresponding verbosity threshold.

    + + + + + + ## `DefaultPreemptionArgs` {#kubescheduler-config-k8s-io-v1beta3-DefaultPreemptionArgs} @@ -323,8 +646,7 @@ of "System" or "List".

    Nodes and Zones.
  • "List": Use constraints defined in .defaultConstraints.
    • -

    Defaults to "List" if feature gate DefaultPodTopologySpread is disabled -and to "System" if enabled.

    +

    Defaults to "System".

    @@ -1061,333 +1383,3 @@ Weight defaults to 1 if not specified or explicitly set to 0.

    - - - -## `ClientConnectionConfiguration` {#ClientConnectionConfiguration} - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) - - -

    ClientConnectionConfiguration contains details for constructing a client.

    - - - - - - - - - - - - - - - - - - - - - - - -
    FieldDescription
    kubeconfig [Required]
    -string -    		 -

    kubeconfig is the path to a KubeConfig file.

    -
    acceptContentTypes [Required]
    -string -    		 -

    acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the -default value of 'application/json'. This field will control all connections to the server used by a particular -client.

    -
    contentType [Required]
    -string -    		 -

    contentType is the content type used when sending data to the server from this client.

    -
    qps [Required]
    -float32 -    		 -

    qps controls the number of queries per second allowed for this connection.

    -
    burst [Required]
    -int32 -    		 -

    burst allows extra queries to accumulate when a client is exceeding its rate.

    -
    - -## `DebuggingConfiguration` {#DebuggingConfiguration} - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) - - -

    DebuggingConfiguration holds configuration for Debugging related features.

    - - - - - - - - - - - - - - -
    FieldDescription
    enableProfiling [Required]
    -bool -    		 -

    enableProfiling enables profiling via web interface host:port/debug/pprof/

    -
    enableContentionProfiling [Required]
    -bool -    		 -

    enableContentionProfiling enables lock contention profiling, if -enableProfiling is true.

    -
    - -## `FormatOptions` {#FormatOptions} - - -**Appears in:** - -- [LoggingConfiguration](#LoggingConfiguration) - - -

    FormatOptions contains options for the different logging formats.

    - - - - - - - - - - - -
    FieldDescription
    json [Required]
    -JSONOptions -    		 -

    [Experimental] JSON contains options for logging format "json".

    -
    - -## `JSONOptions` {#JSONOptions} - - -**Appears in:** - -- [FormatOptions](#FormatOptions) - - -

    JSONOptions contains options for logging format "json".

    - - - - - - - - - - - - - - -
    FieldDescription
    splitStream [Required]
    -bool -    		 -

    [Experimental] SplitStream redirects error messages to stderr while -info messages go to stdout, with buffering. The default is to write -both to stdout, without buffering.

    -
    infoBufferSize [Required]
    -k8s.io/apimachinery/pkg/api/resource.QuantityValue -    		 -

    [Experimental] InfoBufferSize sets the size of the info stream when -using split streams. The default is zero, which disables buffering.

    -
    - -## `LeaderElectionConfiguration` {#LeaderElectionConfiguration} - - -**Appears in:** - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta2-KubeSchedulerConfiguration) - -- [KubeSchedulerConfiguration](#kubescheduler-config-k8s-io-v1beta3-KubeSchedulerConfiguration) - - -

    LeaderElectionConfiguration defines the configuration of leader election -clients for components that can run with leader election enabled.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    FieldDescription
    leaderElect [Required]
    -bool -    		 -

    leaderElect enables a leader election client to gain leadership -before executing the main loop. Enable this when running replicated -components for high availability.

    -
    leaseDuration [Required]
    -meta/v1.Duration -    		 -

    leaseDuration is the duration that non-leader candidates will wait -after observing a leadership renewal until attempting to acquire -leadership of a led but unrenewed leader slot. This is effectively the -maximum duration that a leader can be stopped before it is replaced -by another candidate. This is only applicable if leader election is -enabled.

    -
    renewDeadline [Required]
    -meta/v1.Duration -    		 -

    renewDeadline is the interval between attempts by the acting master to -renew a leadership slot before it stops leading. This must be less -than or equal to the lease duration. This is only applicable if leader -election is enabled.

    -
    retryPeriod [Required]
    -meta/v1.Duration -    		 -

    retryPeriod is the duration the clients should wait between attempting -acquisition and renewal of a leadership. This is only applicable if -leader election is enabled.

    -
    resourceLock [Required]
    -string -    		 -

    resourceLock indicates the resource object type that will be used to lock -during leader election cycles.

    -
    resourceName [Required]
    -string -    		 -

    resourceName indicates the name of resource object that will be used to lock -during leader election cycles.

    -
    resourceNamespace [Required]
    -string -    		 -

    resourceName indicates the namespace of resource object that will be used to lock -during leader election cycles.

    -
    - -## `LoggingConfiguration` {#LoggingConfiguration} - - -**Appears in:** - -- [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - - -

    LoggingConfiguration contains logging options -Refer Logs Options for more information.

    - - - - - - - - - - - - - - - - - - - - - - - - - - -
    FieldDescription
    format [Required]
    -string -    		 -

    Format Flag specifies the structure of log messages. -default value of format is text

    -
    flushFrequency [Required]
    -time.Duration -    		 -

    Maximum number of seconds between log flushes. Ignored if the -selected logging backend writes log messages without buffering.

    -
    verbosity [Required]
    -uint32 -    		 -

    Verbosity is the threshold that determines which log messages are -logged. Default is zero which logs only the most important -messages. Higher values enable additional messages. Error messages -are always logged.

    -
    vmodule [Required]
    -VModuleConfiguration -    		 -

    VModule overrides the verbosity threshold for individual files. -Only supported for "text" log format.

    -
    sanitization [Required]
    -bool -    		 -

    [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)

    -
    options [Required]
    -FormatOptions -    		 -

    [Experimental] Options holds additional parameters that are specific -to the different logging formats. Only the options for the selected -format get used, but all of them get validated.

    -
    - -## `VModuleConfiguration` {#VModuleConfiguration} - -(Alias of `[]k8s.io/component-base/config/v1alpha1.VModuleItem`) - -**Appears in:** - -- [LoggingConfiguration](#LoggingConfiguration) - - -

    VModuleConfiguration is a collection of individual file names or patterns -and the corresponding verbosity threshold.

    - - - diff --git a/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md b/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md index 8874cf6a36..377ac021b6 100644 --- a/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md +++ b/content/en/docs/reference/config-api/kubeadm-config.v1beta2.md @@ -143,7 +143,7 @@ configuration types to be used during a kubeadm init run.

    criSocket: "/var/run/dockershim.sock" taints: - key: "kubeadmNode" - value: "master" + value: "someValue" effect: "NoSchedule" kubeletExtraArgs: v: 4 @@ -348,7 +348,7 @@ could be used for assigning a stable DNS to the control plane. string -

    mageRepository sets the container registry to pull images from. +

    imageRepository sets the container registry to pull images from. If empty, k8s.gcr.io will be used by default; in case of kubernetes version is a CI build (kubernetes version starts with ci/) gcr.io/k8s-staging-ci-images is used as a default for control plane components and for kube-proxy, while @@ -876,7 +876,9 @@ cluster information.

    tlsBootstrapToken is a token used for TLS bootstrapping. -If bootstrapToken is set, this field is defaulted to .bootstrapToken.token, but can be overridden. If file` is set, this field must be set in case the KubeConfigFile does not +If bootstrapToken is set, this field is defaulted to .bootstrapToken.token, +but can be overridden. +If file is set, this field must be set in case the KubeConfigFile does not contain any other authentication information.

    @@ -1080,7 +1082,7 @@ originated from the Kubernetes/Kubernetes release process

    string -

    mageRepository sets the container registry to pull images from. +

    imageRepository sets the container registry to pull images from. If not set, the imageRepository defined in ClusterConfiguration will be used.

    @@ -1267,7 +1269,7 @@ Defaults to the hostname of the node if not provided.

    string -

    `criSocket is used to retrieve container runtime information. This information will +

    criSocket is used to retrieve container runtime information. This information will be annotated to the Node API object, for later re-use.

    @@ -1276,9 +1278,9 @@ be annotated to the Node API object, for later re-use.

    taints specifies the taints the Node API object should be registered with. -If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted to -'node-role.kubernetes.io/master=""'. If you don't want to taint your control-plane node, -set this field to an empty list, i.e. taints: [] in the YAML file. This field is +If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted with +a control-plane taint for control-plane nodes. If you don't want to taint your control-plane +node, set this field to an empty list, i.e. taints: [], in the YAML file. This field is solely used for Node registration.

    diff --git a/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md b/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md index 730973fd82..75fc7c1ecf 100644 --- a/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md +++ b/content/en/docs/reference/config-api/kubeadm-config.v1beta3.md @@ -152,7 +152,7 @@ configuration types to be used during a kubeadm init run.

     criSocket: "/var/run/dockershim.sock" taints: - key: "kubeadmNode" - value: "master" + value: "someValue" effect: "NoSchedule" kubeletExtraArgs: v: 4 @@ -264,109 +264,6 @@ node only (e.g. the node ip).

    -## `BootstrapToken` {#BootstrapToken} - - -**Appears in:** - -- [InitConfiguration](#kubeadm-k8s-io-v1beta3-InitConfiguration) - - -

    BootstrapToken describes one bootstrap token, stored as a Secret in the cluster

    - - - - - - - - - - - - - - - - - - - - - - - - - - -
    FieldDescription
    token [Required]
    -BootstrapTokenString -    		 -

    token is used for establishing bidirectional trust between nodes and control-planes. -Used for joining nodes in the cluster.

    -
    description
    -string -    		 -

    description sets a human-friendly message why this token exists and what it's used -for, so other administrators can know its purpose.

    -
    ttl
    -meta/v1.Duration -    		 -

    ttl defines the time to live for this token. Defaults to 24h. -expires and ttl are mutually exclusive.

    -
    expires
    -meta/v1.Time -    		 -

    expires specifies the timestamp when this token expires. Defaults to being set -dynamically at runtime based on the ttl. expires and ttl are mutually exclusive.

    -
    usages
    -[]string -    		 -

    usages describes the ways in which this token can be used. Can by default be used -for establishing bidirectional trust, but that can be changed here.

    -
    groups
    -[]string -    		 -

    groups specifies the extra groups that this token will authenticate as when/if -used for authentication

    -
    - -## `BootstrapTokenString` {#BootstrapTokenString} - - -**Appears in:** - -- [BootstrapToken](#BootstrapToken) - - -

    BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used -for both validation of the practically of the API server from a joining node's point -of view and as an authentication method for the node in the bootstrap phase of -"kubeadm join". This token is and should be short-lived.

    - - - - - - - - - - - - - - -
    FieldDescription
    - [Required]
    -string -    		 - No description provided.
    - [Required]
    -string -    		 - No description provided.
    - - - ## `ClusterConfiguration` {#kubeadm-k8s-io-v1beta3-ClusterConfiguration} @@ -1263,9 +1160,9 @@ This information will be annotated to the Node API object, for later re-use

    tains specifies the taints the Node API object should be registered with. -If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted to -taints: ["node-role.kubernetes.io/master:""]. -If you don't want to taint your control-plane node, set this field to an empty slice, +If this field is unset, i.e. nil, in the kubeadm init process it will be defaulted +with a control-plane taint for control-plane nodes. +If you don't want to taint your control-plane node, set this field to an empty list, i.e. taints: [] in the YAML file. This field is solely used for Node registration.

    @@ -1340,3 +1237,106 @@ first alpha-numerically.

    + + + +## `BootstrapToken` {#BootstrapToken} + + +**Appears in:** + +- [InitConfiguration](#kubeadm-k8s-io-v1beta3-InitConfiguration) + + +

    BootstrapToken describes one bootstrap token, stored as a Secret in the cluster

    + + + + + + + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    token [Required]
    +BootstrapTokenString +    		 +

    token is used for establishing bidirectional trust between nodes and control-planes. +Used for joining nodes in the cluster.

    +
    description
    +string +    		 +

    description sets a human-friendly message why this token exists and what it's used +for, so other administrators can know its purpose.

    +
    ttl
    +meta/v1.Duration +    		 +

    ttl defines the time to live for this token. Defaults to 24h. +expires and ttl are mutually exclusive.

    +
    expires
    +meta/v1.Time +    		 +

    expires specifies the timestamp when this token expires. Defaults to being set +dynamically at runtime based on the ttl. expires and ttl are mutually exclusive.

    +
    usages
    +[]string +    		 +

    usages describes the ways in which this token can be used. Can by default be used +for establishing bidirectional trust, but that can be changed here.

    +
    groups
    +[]string +    		 +

    groups specifies the extra groups that this token will authenticate as when/if +used for authentication

    +
    + +## `BootstrapTokenString` {#BootstrapTokenString} + + +**Appears in:** + +- [BootstrapToken](#BootstrapToken) + + +

    BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used +for both validation of the practically of the API server from a joining node's point +of view and as an authentication method for the node in the bootstrap phase of +"kubeadm join". This token is and should be short-lived.

    + + + + + + + + + + + + + + +
    FieldDescription
    - [Required]
    +string +    		 + No description provided.
    - [Required]
    +string +    		 + No description provided.
    diff --git a/content/en/docs/reference/config-api/kubelet-config.v1alpha1.md b/content/en/docs/reference/config-api/kubelet-config.v1alpha1.md index ccdec28e8d..2fe765dabe 100644 --- a/content/en/docs/reference/config-api/kubelet-config.v1alpha1.md +++ b/content/en/docs/reference/config-api/kubelet-config.v1alpha1.md @@ -13,166 +13,6 @@ auto_generated: true -## `CredentialProviderConfig` {#kubelet-config-k8s-io-v1alpha1-CredentialProviderConfig} - - - -

    CredentialProviderConfig is the configuration containing information about -each exec credential provider. Kubelet reads this configuration from disk and enables -each provider as specified by the CredentialProvider type.

    - - - - - - - - - - - - - - -
    FieldDescription
    apiVersion
    string    		kubelet.config.k8s.io/v1alpha1
    kind
    string    		CredentialProviderConfig
    providers [Required]
    -[]CredentialProvider -    		 -

    providers is a list of credential provider plugins that will be enabled by the kubelet. -Multiple providers may match against a single image, in which case credentials -from all providers will be returned to the kubelet. If multiple providers are called -for a single image, the results are combined. If providers return overlapping -auth keys, the value from the provider earlier in this list is used.

    -
    - -## `CredentialProvider` {#kubelet-config-k8s-io-v1alpha1-CredentialProvider} - - -**Appears in:** - -- [CredentialProviderConfig](#kubelet-config-k8s-io-v1alpha1-CredentialProviderConfig) - - -

    CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only -invoked when an image being pulled matches the images handled by the plugin (see matchImages).

    - - - - - - - - - - - - - - - - - - - - - - - - - - -
    FieldDescription
    name [Required]
    -string -    		 -

    name is the required name of the credential provider. It must match the name of the -provider executable as seen by the kubelet. The executable must be in the kubelet's -bin directory (set by the --image-credential-provider-bin-dir flag).

    -
    matchImages [Required]
    -[]string -    		 -

    matchImages is a required list of strings used to match against images in order to -determine if this provider should be invoked. If one of the strings matches the -requested image from the kubelet, the plugin will be invoked and given a chance -to provide credentials. Images are expected to contain the registry domain -and URL path.

    -

    Each entry in matchImages is a pattern which can optionally contain a port and a path. -Globs can be used in the domain, but not in the port or the path. Globs are supported -as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. -Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match -a single subdomain segment, so *.io does not match *.k8s.io.

    -

    A match exists between an image and a matchImage when all of the below are true:

    -
      -
    • Both contain the same number of domain parts and each part matches.
      • -
    • The URL path of an imageMatch must be a prefix of the target image URL path.
      • -
    • If the imageMatch contains a port, then the port must match in the image as well.
      • -
    -

    Example values of matchImages:

    -
      -
    • 123456789.dkr.ecr.us-east-1.amazonaws.com
      • -
    • *.azurecr.io
      • -
    • gcr.io
      • -
    • ..registry.io
      • -
    • registry.io:8080/path
      • -
    -
    defaultCacheDuration [Required]
    -meta/v1.Duration -    		 -

    defaultCacheDuration is the default duration the plugin will cache credentials in-memory -if a cache duration is not provided in the plugin response. This field is required.

    -
    apiVersion [Required]
    -string -    		 -

    Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse -MUST use the same encoding version as the input. Current supported values are:

    -
      -
    • credentialprovider.kubelet.k8s.io/v1alpha1
      • -
    -
    args
    -[]string -    		 -

    Arguments to pass to the command when executing it.

    -
    env
    -[]ExecEnvVar -    		 -

    Env defines additional environment variables to expose to the process. These -are unioned with the host's environment, as well as variables client-go uses -to pass argument to the plugin.

    -
    - -## `ExecEnvVar` {#kubelet-config-k8s-io-v1alpha1-ExecEnvVar} - - -**Appears in:** - -- [CredentialProvider](#kubelet-config-k8s-io-v1alpha1-CredentialProvider) - - -

    ExecEnvVar is used for setting environment variables when executing an exec-based -credential plugin.

    - - - - - - - - - - - - - - -
    FieldDescription
    name [Required]
    -string -    		 - No description provided.
    value [Required]
    -string -    		 - No description provided.
    - - - - ## `FormatOptions` {#FormatOptions} @@ -247,3 +87,163 @@ and the corresponding verbosity threshold.

    + + + +## `CredentialProviderConfig` {#kubelet-config-k8s-io-v1alpha1-CredentialProviderConfig} + + + +

    CredentialProviderConfig is the configuration containing information about +each exec credential provider. Kubelet reads this configuration from disk and enables +each provider as specified by the CredentialProvider type.

    + + + + + + + + + + + + + + +
    FieldDescription
    apiVersion
    string    		kubelet.config.k8s.io/v1alpha1
    kind
    string    		CredentialProviderConfig
    providers [Required]
    +[]CredentialProvider +    		 +

    providers is a list of credential provider plugins that will be enabled by the kubelet. +Multiple providers may match against a single image, in which case credentials +from all providers will be returned to the kubelet. If multiple providers are called +for a single image, the results are combined. If providers return overlapping +auth keys, the value from the provider earlier in this list is used.

    +
    + +## `CredentialProvider` {#kubelet-config-k8s-io-v1alpha1-CredentialProvider} + + +**Appears in:** + +- [CredentialProviderConfig](#kubelet-config-k8s-io-v1alpha1-CredentialProviderConfig) + + +

    CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only +invoked when an image being pulled matches the images handled by the plugin (see matchImages).

    + + + + + + + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    name [Required]
    +string +    		 +

    name is the required name of the credential provider. It must match the name of the +provider executable as seen by the kubelet. The executable must be in the kubelet's +bin directory (set by the --image-credential-provider-bin-dir flag).

    +
    matchImages [Required]
    +[]string +    		 +

    matchImages is a required list of strings used to match against images in order to +determine if this provider should be invoked. If one of the strings matches the +requested image from the kubelet, the plugin will be invoked and given a chance +to provide credentials. Images are expected to contain the registry domain +and URL path.

    +

    Each entry in matchImages is a pattern which can optionally contain a port and a path. +Globs can be used in the domain, but not in the port or the path. Globs are supported +as subdomains like *.k8s.io or k8s.*.io, and top-level-domains such as k8s.*. +Matching partial subdomains like app*.k8s.io is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

    +

    A match exists between an image and a matchImage when all of the below are true:

    +
      +
    • Both contain the same number of domain parts and each part matches.
      • +
    • The URL path of an imageMatch must be a prefix of the target image URL path.
      • +
    • If the imageMatch contains a port, then the port must match in the image as well.
      • +
    +

    Example values of matchImages:

    +
      +
    • 123456789.dkr.ecr.us-east-1.amazonaws.com
      • +
    • *.azurecr.io
      • +
    • gcr.io
      • +
    • *.*.registry.io
      • +
    • registry.io:8080/path
      • +
    +
    defaultCacheDuration [Required]
    +meta/v1.Duration +    		 +

    defaultCacheDuration is the default duration the plugin will cache credentials in-memory +if a cache duration is not provided in the plugin response. This field is required.

    +
    apiVersion [Required]
    +string +    		 +

    Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse +MUST use the same encoding version as the input. Current supported values are:

    +
      +
    • credentialprovider.kubelet.k8s.io/v1alpha1
      • +
    +
    args
    +[]string +    		 +

    Arguments to pass to the command when executing it.

    +
    env
    +[]ExecEnvVar +    		 +

    Env defines additional environment variables to expose to the process. These +are unioned with the host's environment, as well as variables client-go uses +to pass argument to the plugin.

    +
    + +## `ExecEnvVar` {#kubelet-config-k8s-io-v1alpha1-ExecEnvVar} + + +**Appears in:** + +- [CredentialProvider](#kubelet-config-k8s-io-v1alpha1-CredentialProvider) + + +

    ExecEnvVar is used for setting environment variables when executing an exec-based +credential plugin.

    + + + + + + + + + + + + + + +
    FieldDescription
    name [Required]
    +string +    		 + No description provided.
    value [Required]
    +string +    		 + No description provided.
    + diff --git a/content/en/docs/reference/config-api/kubelet-config.v1beta1.md b/content/en/docs/reference/config-api/kubelet-config.v1beta1.md index 9fc2bbfde0..42755470cb 100644 --- a/content/en/docs/reference/config-api/kubelet-config.v1beta1.md +++ b/content/en/docs/reference/config-api/kubelet-config.v1beta1.md @@ -9,11 +9,43 @@ auto_generated: true ## Resource Types +- [CredentialProviderConfig](#kubelet-config-k8s-io-v1beta1-CredentialProviderConfig) - [KubeletConfiguration](#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) - [SerializedNodeConfigSource](#kubelet-config-k8s-io-v1beta1-SerializedNodeConfigSource) +## `CredentialProviderConfig` {#kubelet-config-k8s-io-v1beta1-CredentialProviderConfig} + + + +

    CredentialProviderConfig is the configuration containing information about +each exec credential provider. Kubelet reads this configuration from disk and enables +each provider as specified by the CredentialProvider type.

    + + + + + + + + + + + + + + +
    FieldDescription
    apiVersion
    string    		kubelet.config.k8s.io/v1beta1
    kind
    string    		CredentialProviderConfig
    providers [Required]
    +[]CredentialProvider +    		 +

    providers is a list of credential provider plugins that will be enabled by the kubelet. +Multiple providers may match against a single image, in which case credentials +from all providers will be returned to the kubelet. If multiple providers are called +for a single image, the results are combined. If providers return overlapping +auth keys, the value from the provider earlier in this list is used.

    +
    + ## `KubeletConfiguration` {#kubelet-config-k8s-io-v1beta1-KubeletConfiguration} @@ -35,9 +67,6 @@ auto_generated: true

    enableServer enables Kubelet's secured server. Note: Kubelet's insecure port is controlled by the readOnlyPort option. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: true

    @@ -47,10 +76,6 @@ Default: true

    staticPodPath is the path to the directory containing local (static) pods to run, or the path to a single static pod file. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -the set of static pods specified at the new path may be different than the -ones the Kubelet initially started with, and this may disrupt your node. Default: ""

    @@ -60,11 +85,6 @@ Default: ""

    syncFrequency is the max period between synchronizing running containers and config. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening this duration may have a negative performance impact, especially -as the number of Pods on the node increases. Alternatively, increasing this -duration will result in longer refresh times for ConfigMaps and Secrets. Default: "1m"

    @@ -74,10 +94,6 @@ Default: "1m"

    fileCheckFrequency is the duration between checking config files for new data. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the duration will cause the Kubelet to reload local Static Pod -configurations more frequently, which may have a negative performance impact. Default: "20s"

    @@ -86,10 +102,6 @@ Default: "20s"

    httpCheckFrequency is the duration between checking http for new data. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the duration will cause the Kubelet to poll staticPodURL more -frequently, which may have a negative performance impact. Default: "20s"

    @@ -98,10 +110,6 @@ Default: "20s"

    staticPodURL is the URL for accessing static pods to run. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -the set of static pods specified at the new URL may be different than the -ones the Kubelet initially started with, and this may disrupt your node. Default: ""

    @@ -110,9 +118,6 @@ Default: ""

    staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt the ability to read the latest set of static pods from StaticPodURL. Default: nil

    @@ -122,9 +127,6 @@ Default: nil

    address is the IP address for the Kubelet to serve on (set to 0.0.0.0 for all interfaces). -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: "0.0.0.0"

    @@ -134,9 +136,6 @@ Default: "0.0.0.0"

    port is the port for the Kubelet to serve on. The port number must be between 1 and 65535, inclusive. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: 10250

    @@ -148,9 +147,6 @@ Default: 10250

    no authentication/authorization. The port number must be between 1 and 65535, inclusive. Setting this field to 0 disables the read-only service. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: 0 (disabled)

    @@ -163,9 +159,6 @@ if any, concatenated after server cert). If tlsCertFile and tlsPrivateKeyFile are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to the Kubelet's --cert-dir flag. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: ""

    @@ -174,9 +167,6 @@ Default: ""

    tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: ""

    @@ -186,9 +176,6 @@ Default: ""

    tlsCipherSuites is the list of allowed cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: nil

    @@ -198,9 +185,6 @@ Default: nil

    tlsMinVersion is the minimum TLS version supported. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: ""

    @@ -211,10 +195,6 @@ Default: ""

    rotateCertificates enables client certificate rotation. The Kubelet will request a new certificate from the certificates.k8s.io API. This requires an approver to approve the certificate signing requests. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it may disrupt the Kubelet's ability to authenticate with the API server -after the current certificate expires. Default: false

    @@ -227,11 +207,6 @@ signing a serving certificate, the Kubelet will request a certificate from the 'certificates.k8s.io' API. This requires an approver to approve the certificate signing requests (CSR). The RotateKubeletServerCertificate feature must be enabled when setting this field. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it will stop the renewal of Kubelet server certificates, which can -disrupt components that interact with the Kubelet server in the long term, -due to certificate expiration. Default: false

    @@ -240,9 +215,6 @@ Default: false

    authentication specifies how requests to the Kubelet's server are authenticated. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Defaults: anonymous: enabled: false @@ -256,9 +228,6 @@ cacheTTL: "2m"

    authorization specifies how requests to the Kubelet's server are authorized. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Defaults: mode: Webhook webhook: @@ -273,10 +242,6 @@ cacheUnauthorizedTTL: "30s"

    registryPullQPS is the limit of registry pulls per second. The value must not be a negative number. Setting it to 0 means no limit. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced -by image pulls. Default: 5

    @@ -288,10 +253,6 @@ Default: 5

    pulls to burst to this number, while still not exceeding registryPullQPS. The value must not be a negative number. Only used if registryPullQPS is greater than 0. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced -by image pulls. Default: 10

    @@ -301,10 +262,6 @@ Default: 10

    eventRecordQPS is the maximum event creations per second. If 0, there is no limit enforced. The value cannot be a negative number. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced by -event creations. Default: 5

    @@ -316,10 +273,6 @@ Default: 5

    allows event creations to burst to this number, while still not exceeding eventRecordQPS. This field canot be a negative number and it is only used when eventRecordQPS > 0. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic produced by -event creations. Default: 10

    @@ -330,9 +283,6 @@ Default: 10

    enableDebuggingHandlers enables server endpoints for log access and local running of containers and commands, including the exec, attach, logs, and portforward features. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it may disrupt components that interact with the Kubelet server. Default: true

    @@ -341,9 +291,6 @@ Default: true

    enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -enabling it may carry a performance impact. Default: false

    @@ -353,9 +300,6 @@ Default: false

    healthzPort is the port of the localhost healthz endpoint (set to 0 to disable). A valid number is between 1 and 65535. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that monitor Kubelet health. Default: 10248

    @@ -364,9 +308,6 @@ Default: 10248

    healthzBindAddress is the IP address for the healthz server to serve on. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that monitor Kubelet health. Default: "127.0.0.1"

    @@ -376,9 +317,6 @@ Default: "127.0.0.1"

    oomScoreAdj is The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the stability of nodes under memory pressure. Default: -999

    @@ -389,8 +327,6 @@ Default: -999

    clusterDomain is the DNS domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains. -Dynamic Kubelet Config (deprecated): Dynamically updating this field is not recommended, -as it should be kept in sync with the rest of the cluster. Default: ""

    @@ -401,10 +337,6 @@ Default: ""

    clusterDNS is a list of IP addresses for the cluster DNS server. If set, kubelet will configure all containers to use this for DNS resolution instead of the host's DNS servers. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes will only take effect on Pods created after the update. Draining -the node is recommended before changing this field. Default: nil

    @@ -414,10 +346,6 @@ Default: nil

    streamingConnectionIdleTimeout is the maximum time a streaming connection can be idle before the connection is automatically closed. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact components that rely on infrequent updates over streaming -connections to the Kubelet server. Default: "4h"

    @@ -430,12 +358,6 @@ status. If node lease feature is not enabled, it is also the frequency that kubelet posts node status to master. Note: When node lease feature is not enabled, be cautious when changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact node scalability, and also that the node controller's -nodeMonitorGracePeriod must be set to N*NodeStatusUpdateFrequency, -where N is the number of retries before the node controller marks -the node unhealthy. Default: "10s"

    @@ -464,10 +386,6 @@ If the lease expires, the node can be considered unhealthy. The lease is currently renewed every 10s, per KEP-0009. In the future, the lease renewal interval may be set based on the lease duration. The field value must be greater than 0. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -decreasing the duration may reduce tolerance for issues that temporarily prevent -the Kubelet from renewing the lease (e.g. a short-lived network issue). Default: 40

    @@ -476,9 +394,7 @@ Default: 40

    imageMinimumGCAge is the minimum age for an unused image before it is -garbage collected. If DynamicKubeletConfig (deprecated; default off) -is on, when dynamically updating this field, consider that it may trigger or -delay garbage collection, and may change the image overhead on the node. +garbage collected. Default: "2m"

    @@ -491,10 +407,6 @@ image garbage collection is always run. The percent is calculated by dividing this field value by 100, so this field must be between 0 and 100, inclusive. When specified, the value must be greater than imageGCLowThresholdPercent. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay garbage collection, and may change the image overhead -on the node. Default: 85

    @@ -507,10 +419,6 @@ image garbage collection is never run. Lowest disk usage to garbage collect to. The percent is calculated by dividing this field value by 100, so the field value must be between 0 and 100, inclusive. When specified, the value must be less than imageGCHighThresholdPercent. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay garbage collection, and may change the image overhead -on the node. Default: 80

    @@ -520,9 +428,6 @@ Default: 80

    volumeStatsAggPeriod is the frequency for calculating and caching volume disk usage for all pods. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the period may carry a performance impact. Default: "1m"

    @@ -531,8 +436,6 @@ Default: "1m"

    kubeletCgroups is the absolute name of cgroups to isolate the kubelet in -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: ""

    @@ -544,8 +447,6 @@ Default: ""

    all non-kernel processes that are not already in a container. Empty for no container. Rolling back the flag requires a reboot. The cgroupRoot must be specified if this field is not empty. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: ""

    @@ -554,10 +455,7 @@ Default: ""

    cgroupRoot is the root cgroup to use for pods. This is handled by the -container runtime on a best effort basis. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. -Default: ""

    +container runtime on a best effort basis.

    cgroupsPerQOS
    @@ -567,8 +465,6 @@ Default: ""

    cgroupsPerQOS enable QoS based CGroup hierarchy: top level CGroups for QoS classes and all Burstable and BestEffort Pods are brought up under their specific top level QoS CGroup. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: true

    @@ -578,8 +474,6 @@ Default: true

    cgroupDriver is the driver kubelet uses to manipulate CGroups on the host (cgroupfs or systemd). -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: "cgroupfs"

    @@ -589,8 +483,6 @@ Default: "cgroupfs"

    cpuManagerPolicy is the name of the policy to use. Requires the CPUManager feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: "None"

    @@ -601,8 +493,6 @@ Default: "None"

    cpuManagerPolicyOptions is a set of key=value which allows to set extra options to fine tune the behaviour of the cpu manager policies. Requires both the "CPUManager" and "CPUManagerPolicyOptions" feature gates to be enabled. -Dynamic Kubelet Config (beta): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: nil

    @@ -612,9 +502,6 @@ Default: nil

    cpuManagerReconcilePeriod is the reconciliation period for the CPU Manager. Requires the CPUManager feature gate to be enabled. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -shortening the period may carry a performance impact. Default: "10s"

    @@ -624,8 +511,6 @@ Default: "10s"

    memoryManagerPolicy is the name of the policy to use by memory manager. Requires the MemoryManager feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: "none"

    @@ -645,8 +530,6 @@ resources; of CPU and device resources.

    Policies other than "none" require the TopologyManager feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: "none"

    @@ -673,8 +556,6 @@ the minimum percentage of a resource reserved for exclusive use by the guaranteed QoS tier. Currently supported resources: "memory" Requires the QOSReserved feature gate to be enabled. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: nil

    @@ -684,9 +565,6 @@ Default: nil

    runtimeRequestTimeout is the timeout for all runtime requests except long running requests - pull, logs, exec and attach. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may disrupt components that interact with the Kubelet server. Default: "2m"

    @@ -705,9 +583,6 @@ themselves if they should try to access their own Service. Values:

    Generally, one must set --hairpin-mode=hairpin-veth to achieve hairpin NAT, because promiscuous-bridge assumes the existence of a container bridge named cbr0. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may require a node reboot, depending on the network plugin. Default: "promiscuous-bridge"

    @@ -717,12 +592,6 @@ Default: "promiscuous-bridge"

    maxPods is the maximum number of Pods that can run on this Kubelet. The value must be a non-negative integer. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes may cause Pods to fail admission on Kubelet restart, and may change -the value reported in Node.Status.Capacity[v1.ResourcePods], thus affecting -future scheduling decisions. Increasing this value may also decrease performance, -as more Pods can be packed into a single node. Default: 110

    @@ -732,8 +601,6 @@ Default: 110

    podCIDR is the CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the control plane. -Dynamic Kubelet Config (deprecated): This field should always be set to the empty default. -It should only set for standalone Kubelets, which cannot use Dynamic Kubelet Config. Default: ""

    @@ -742,9 +609,6 @@ Default: ""

    podPidsLimit is the maximum number of PIDs in any pod. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it may prevent container processes from forking after the change. Default: -1

    @@ -754,10 +618,6 @@ Default: -1

    resolvConf is the resolver configuration file used as the basis for the container DNS resolution configuration. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes will only take effect on Pods created after the update. Draining -the node is recommended before changing this field. If set to the empty string, will override the default and effectively disable DNS lookups. Default: "/etc/resolv.conf"

    @@ -777,9 +637,6 @@ Default: false

    cpuCFSQuota enables CPU CFS quota enforcement for containers that specify CPU limits. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it may reduce node stability. Default: true

    @@ -790,10 +647,6 @@ Default: true

    cpuCFSQuotaPeriod is the CPU CFS quota period value, cpu.cfs_period_us. The value must be between 1 us and 1 second, inclusive. Requires the CustomCPUCFSQuotaPeriod feature gate to be enabled. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -limits set for containers will result in different cpu.cfs_quota settings. This -will trigger container restarts on the node being reconfigured. Default: "100ms"

    @@ -804,9 +657,6 @@ Default: "100ms"

    nodeStatusMaxImages caps the number of images reported in Node.status.images. The value must be greater than -2. Note: If -1 is specified, no cap will be applied. If 0 is specified, no image is returned. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -different values can be reported on node status. Default: 50

    @@ -816,9 +666,6 @@ Default: 50

    maxOpenFiles is Number of files that can be opened by Kubelet process. The value must be a non-negative number. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the ability of the Kubelet to interact with the node's filesystem. Default: 1000000

    @@ -827,11 +674,6 @@ Default: 1000000

    contentType is contentType of requests sent to apiserver. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the ability for the Kubelet to communicate with the API server. -If the Kubelet loses contact with the API server due to a change to this field, -the change cannot be reverted via dynamic Kubelet config. Default: "application/vnd.kubernetes.protobuf"

    @@ -840,10 +682,6 @@ Default: "application/vnd.kubernetes.protobuf"

    kubeAPIQPS is the QPS to use while talking with kubernetes apiserver. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic the Kubelet -sends to the API server. Default: 5

    @@ -853,10 +691,6 @@ Default: 5

    kubeAPIBurst is the burst to allow while talking with kubernetes API server. This field cannot be a negative number. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact scalability by changing the amount of traffic the Kubelet -sends to the API server. Default: 10

    @@ -868,9 +702,6 @@ Default: 10

    at a time. We recommend not changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the performance of image pulls. Default: true

    @@ -881,9 +712,6 @@ Default: true

    evictionHard is a map of signal names to quantities that defines hard eviction thresholds. For example: {"memory.available": "300Mi"}. To explicitly disable, pass a 0% or 100% threshold on an arbitrary resource. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay Pod evictions. Default: memory.available: "100Mi" nodefs.available: "10%" @@ -897,10 +725,6 @@ imagefs.available: "15%"

    evictionSoft is a map of signal names to quantities that defines soft eviction thresholds. For example: {"memory.available": "300Mi"}. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay Pod evictions, and may change the allocatable reported -by the node. Default: nil

    @@ -910,9 +734,6 @@ Default: nil

    evictionSoftGracePeriod is a map of signal names to quantities that defines grace periods for each soft eviction signal. For example: {"memory.available": "30s"}. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger or delay Pod evictions. Default: nil

    @@ -922,9 +743,6 @@ Default: nil

    evictionPressureTransitionPeriod is the duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it may decrease the stability of the node when the node is overcommitted. Default: "5m"

    @@ -938,10 +756,6 @@ effectively caps the Pod's terminationGracePeriodSeconds value during soft evict Note: Due to issue #64530, the behavior has a bug where this value currently just overrides the grace period during soft eviction, which can increase the grace period from what is set on the Pod. This bug will be fixed in a future release. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it decreases the amount of time Pods will have to gracefully clean -up before being killed during a soft eviction. Default: 0

    @@ -953,9 +767,6 @@ Default: 0

    which describe the minimum amount of a given resource the kubelet will reclaim when performing a pod eviction while that resource is under pressure. For example: {"imagefs.available": "2Gi"}. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may change how well eviction can manage resource pressure. Default: nil

    @@ -966,12 +777,6 @@ Default: nil

    podsPerCore is the maximum number of pods per core. Cannot exceed maxPods. The value must be a non-negative integer. If 0, there is no limit on the number of Pods. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changes may cause Pods to fail admission on Kubelet restart, and may change -the value reported in Node.status.capacity.pods, thus affecting -future scheduling decisions. Increasing this value may also decrease performance, -as more Pods can be packed into a single node. Default: 0

    @@ -982,13 +787,8 @@ Default: 0

    enableControllerAttachDetach enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -changing which component is responsible for volume management on a live node -may result in volumes refusing to detach if the node is not drained prior to -the update, and if Pods are scheduled to the node before the -volumes.kubernetes.io/controller-managed-attach-detach annotation is updated by the -Kubelet. In general, it is safest to leave this value set the same as local config. +Note: attaching/detaching CSI volumes is not supported by the kubelet, +so this option needs to be true for that use case. Default: true

    @@ -999,10 +799,6 @@ Default: true

    protectKernelDefaults, if true, causes the Kubelet to error if kernel flags are not as it expects. Otherwise the Kubelet will attempt to modify kernel flags to match its expectation. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -enabling it may cause the Kubelet to crash-loop if the Kernel is not configured as -Kubelet expects. Default: false

    @@ -1014,9 +810,6 @@ Default: false

    are present on host. These rules will serve as utility rules for various components, e.g. kube-proxy. The rules will be created based on iptablesMasqueradeBit and iptablesDropBit. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -disabling it will prevent the Kubelet from healing locally misconfigured iptables rules. Default: true

    @@ -1028,10 +821,6 @@ Default: true

    Values must be within the range [0, 31]. Must be different from other mark bits. Warning: Please match the value of the corresponding parameter in kube-proxy. TODO: clean up IPTablesMasqueradeBit in kube-proxy. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it needs to be coordinated with other components, like kube-proxy, and the update -will only be effective if MakeIPTablesUtilChains is enabled. Default: 14

    @@ -1041,10 +830,6 @@ Default: 14

    iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets. Values must be within the range [0, 31]. Must be different from other mark bits. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it needs to be coordinated with other components, like kube-proxy, and the update -will only be effective if MakeIPTablesUtilChains is enabled. Default: 15

    @@ -1055,12 +840,6 @@ Default: 15

    featureGates is a map of feature names to bools that enable or disable experimental features. This field modifies piecemeal the built-in default values from "k8s.io/kubernetes/pkg/features/kube_features.go". -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider the -documentation for the features you are enabling or disabling. While we -encourage feature developers to make it possible to dynamically enable -and disable features, some changes may require node reboots, and some -features may require careful coordination to retroactively disable. Default: nil

    @@ -1069,9 +848,6 @@ Default: nil

    failSwapOn tells the Kubelet to fail to start if swap is enabled on the node. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -setting it to true will cause the Kubelet to crash-loop if swap is enabled. Default: true

    @@ -1088,9 +864,6 @@ Default: true

    containerLogMaxSize is a quantity defining the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki". -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may trigger log rotation. Default: "10Mi"

    @@ -1100,9 +873,6 @@ Default: "10Mi"

    containerLogMaxFiles specifies the maximum number of container log files that can be present for a container. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -lowering it may cause log files to be deleted. Default: 5

    @@ -1128,11 +898,6 @@ managers are running. Valid values include:

    pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may not be possible to increase the reserved resources, because this -requires resizing cgroups. Always look for a NodeAllocatableEnforced event -after updating this field to ensure that the update was successful. Default: nil

    @@ -1145,11 +910,6 @@ that describe resources reserved for kubernetes system components. Currently cpu, memory and local storage for root file system are supported. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more details. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may not be possible to increase the reserved resources, because this -requires resizing cgroups. Always look for a NodeAllocatableEnforced event -after updating this field to ensure that the update was successful. Default: nil

    @@ -1183,10 +943,8 @@ Default: ""

    systemReservedCgroup helps the kubelet identify absolute name of top level CGroup used to enforce systemReserved compute resource reservation for OS system daemons. -Refer to Node Allocatable +Refer to Node Allocatable doc for more information. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: ""

    @@ -1196,10 +954,8 @@ Default: ""

    kubeReservedCgroup helps the kubelet identify absolute name of top level CGroup used to enforce KubeReserved compute resource reservation for Kubernetes node system daemons. -Refer to Node Allocatable +Refer to Node Allocatable doc for more information. -Dynamic Kubelet Config (deprecated): This field should not be updated without a full node -reboot. It is safest to keep this value the same as the local config. Default: ""

    @@ -1214,15 +970,8 @@ If none is specified, no other options may be specified. When system-reserved is in the list, systemReservedCgroup must be specified. When kube-reserved is in the list, kubeReservedCgroup must be specified. This field is supported only when cgroupsPerQOS is set to true. -Refer to Node Allocatable +Refer to Node Allocatable for more information. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -removing enforcements may reduce the stability of the node. Alternatively, adding -enforcements may reduce the stability of components which were using more than -the reserved amount of resources; for example, enforcing kube-reserved may cause -Kubelets to OOM if it uses more than the reserved resources, and enforcing system-reserved -may cause system daemons to OOM if they use more than the reserved resources. Default: ["pods"]

    @@ -1242,9 +991,6 @@ Default: []

    volumePluginDir is the full path of the directory in which to search for additional third party volume plugins. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that changing -the volumePluginDir may disrupt workloads relying on third party volume plugins. Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"

    @@ -1254,9 +1000,6 @@ Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"

    providerID, if set, sets the unique ID of the instance that an external provider (i.e. cloudprovider) can use to identify a specific node. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the ability of the Kubelet to interact with cloud providers. Default: ""

    @@ -1267,9 +1010,6 @@ Default: ""

    kernelMemcgNotification, if set, instructs the the kubelet to integrate with the kernel memcg notification for determining if memory eviction thresholds are exceeded rather than polling. -If DynamicKubeletConfig (deprecated; default off) is on, when -dynamically updating this field, consider that -it may impact the way Kubelet interacts with the kernel. Default: false

    @@ -1457,6 +1197,132 @@ It exists in the kubeletconfig API group because it is classified as a versioned +## `CredentialProvider` {#kubelet-config-k8s-io-v1beta1-CredentialProvider} + + +**Appears in:** + +- [CredentialProviderConfig](#kubelet-config-k8s-io-v1beta1-CredentialProviderConfig) + + +

    CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only +invoked when an image being pulled matches the images handled by the plugin (see matchImages).

    + + + + + + + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    name [Required]
    +string +    		 +

    name is the required name of the credential provider. It must match the name of the +provider executable as seen by the kubelet. The executable must be in the kubelet's +bin directory (set by the --image-credential-provider-bin-dir flag).

    +
    matchImages [Required]
    +[]string +    		 +

    matchImages is a required list of strings used to match against images in order to +determine if this provider should be invoked. If one of the strings matches the +requested image from the kubelet, the plugin will be invoked and given a chance +to provide credentials. Images are expected to contain the registry domain +and URL path.

    +

    Each entry in matchImages is a pattern which can optionally contain a port and a path. +Globs can be used in the domain, but not in the port or the path. Globs are supported +as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. +Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

    +

    A match exists between an image and a matchImage when all of the below are true:

    +
      +
    • Both contain the same number of domain parts and each part matches.
      • +
    • The URL path of an imageMatch must be a prefix of the target image URL path.
      • +
    • If the imageMatch contains a port, then the port must match in the image as well.
      • +
    +

    Example values of matchImages:

    +
      +
    • 123456789.dkr.ecr.us-east-1.amazonaws.com
      • +
    • *.azurecr.io
      • +
    • gcr.io
      • +
    • ..registry.io
      • +
    • registry.io:8080/path
      • +
    +
    defaultCacheDuration [Required]
    +meta/v1.Duration +    		 +

    defaultCacheDuration is the default duration the plugin will cache credentials in-memory +if a cache duration is not provided in the plugin response. This field is required.

    +
    apiVersion [Required]
    +string +    		 +

    Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse +MUST use the same encoding version as the input. Current supported values are:

    +
      +
    • credentialprovider.kubelet.k8s.io/v1beta1
      • +
    +
    args
    +[]string +    		 +

    Arguments to pass to the command when executing it.

    +
    env
    +[]ExecEnvVar +    		 +

    Env defines additional environment variables to expose to the process. These +are unioned with the host's environment, as well as variables client-go uses +to pass argument to the plugin.

    +
    + +## `ExecEnvVar` {#kubelet-config-k8s-io-v1beta1-ExecEnvVar} + + +**Appears in:** + +- [CredentialProvider](#kubelet-config-k8s-io-v1beta1-CredentialProvider) + + +

    ExecEnvVar is used for setting environment variables when executing an exec-based +credential plugin.

    + + + + + + + + + + + + + + +
    FieldDescription
    name [Required]
    +string +    		 + No description provided.
    value [Required]
    +string +    		 + No description provided.
    + ## `KubeletAnonymousAuthentication` {#kubelet-config-k8s-io-v1beta1-KubeletAnonymousAuthentication} @@ -1859,8 +1725,9 @@ default value of format is text

    time.Duration -

    Maximum number of seconds between log flushes. Ignored if the -selected logging backend writes log messages without buffering.

    +

    Maximum number of nanoseconds (i.e. 1s = 1000000000) between log +flushes. Ignored if the selected logging backend writes log +messages without buffering.

    verbosity [Required]
    @@ -1881,14 +1748,6 @@ are always logged.

    Only supported for "text" log format.

    -sanitization [Required]
    -bool - - -

    [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens). -Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)

    - - options [Required]
    FormatOptions diff --git a/content/en/docs/reference/config-api/kubelet-credentialprovider.v1alpha1.md b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1alpha1.md index 070fe96d09..029d5ac62c 100644 --- a/content/en/docs/reference/config-api/kubelet-credentialprovider.v1alpha1.md +++ b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1alpha1.md @@ -14,16 +14,14 @@ auto_generated: true - ## `CredentialProviderRequest` {#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderRequest} - - -CredentialProviderRequest includes the image that the kubelet requires authentication for. +

    CredentialProviderRequest includes the image that the kubelet requires authentication for. Kubelet will pass this request object to the plugin via stdin. In general, plugins should -prefer responding with the same apiVersion they were sent. +prefer responding with the same apiVersion they were sent.

    + @@ -32,33 +30,27 @@ prefer responding with the same apiVersion they were sent. - - +to extract any information required to fetch credentials.

    + - -
    FieldDescription
    apiVersion
    string    		credentialprovider.kubelet.k8s.io/v1alpha1
    kind
    string    		CredentialProviderRequest
    image [Required]
    string     		- image is the container image that is being pulled as part of the +

    image is the container image that is being pulled as part of the credential provider plugin request. Plugins may optionally parse the image -to extract any information required to fetch credentials.
    - - ## `CredentialProviderResponse` {#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderResponse} - - -CredentialProviderResponse holds credentials that the kubelet should use for the specified +

    CredentialProviderResponse holds credentials that the kubelet should use for the specified image provided in the original request. Kubelet will read the response from the plugin via stdout. -This response should be set to the same apiVersion as CredentialProviderRequest. +This response should be set to the same apiVersion as CredentialProviderRequest.

    + @@ -67,119 +59,106 @@ This response should be set to the same apiVersion as CredentialProviderRequest. - - +Global. If an invalid value is specified, the response will NOT be used by the kubelet.

    + - - +CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.

    + - - +as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. +Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

    +

    The kubelet will match images against the key when all of the below are true:

    +
      +
    • Both contain the same number of domain parts and each part matches.
      • +
    • The URL path of an imageMatch must be a prefix of the target image URL path.
      • +
    • If the imageMatch contains a port, then the port must match in the image as well.
      • +
    +

    When multiple keys are returned, the kubelet will traverse all keys in reverse order so that:

    +
      +
    • longer keys come before shorter keys with the same prefix
      • +
    • non-wildcard keys come before wildcard keys with the same prefix.
      • +
    +

    For any given match, the kubelet will attempt an image pull with the provided credentials, +stopping after the first successfully authenticated pull.

    +

    Example keys:

    +
      +
    • 123456789.dkr.ecr.us-east-1.amazonaws.com
      • +
    • *.azurecr.io
      • +
    • gcr.io
      • +
    • ..registry.io
      • +
    • registry.io:8080/path
      • +
    + - -
    FieldDescription
    apiVersion
    string    		credentialprovider.kubelet.k8s.io/v1alpha1
    kind
    string    		CredentialProviderResponse
    cacheKeyType [Required]
    PluginCacheKeyType     		- cacheKeyType indiciates the type of caching key to use based on the image provided +

    cacheKeyType indiciates the type of caching key to use based on the image provided in the request. There are three valid values for the cache key type: Image, Registry, and -Global. If an invalid value is specified, the response will NOT be used by the kubelet.
    cacheDuration
    -meta/v1.Duration +meta/v1.Duration     		- cacheDuration indicates the duration the provided credentials should be cached for. +

    cacheDuration indicates the duration the provided credentials should be cached for. The kubelet will use this field to set the in-memory cache duration for credentials in the AuthConfig. If null, the kubelet will use defaultCacheDuration provided in -CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.
    auth
    map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1alpha1.AuthConfig     		- auth is a map containing authentication information passed into the kubelet. +

    auth is a map containing authentication information passed into the kubelet. Each key is a match image string (more on this below). The corresponding authConfig value should be valid for all images that match against this key. A plugin should set -this field to null if no valid credentials can be returned for the requested image. - -Each key in the map is a pattern which can optionally contain a port and a path. +this field to null if no valid credentials can be returned for the requested image.

    +

    Each key in the map is a pattern which can optionally contain a port and a path. Globs can be used in the domain, but not in the port or the path. Globs are supported -as subdomains like '∗.k8s.io' or 'k8s.∗.io', and top-level-domains such as 'k8s.∗'. -Matching partial subdomains like 'app∗.k8s.io' is also supported. Each glob can only match -a single subdomain segment, so ∗.io does not match ∗.k8s.io. - -The kubelet will match images against the key when all of the below are true: -- Both contain the same number of domain parts and each part matches. -- The URL path of an imageMatch must be a prefix of the target image URL path. -- If the imageMatch contains a port, then the port must match in the image as well. - -When multiple keys are returned, the kubelet will traverse all keys in reverse order so that: -- longer keys come before shorter keys with the same prefix -- non-wildcard keys come before wildcard keys with the same prefix. - -For any given match, the kubelet will attempt an image pull with the provided credentials, -stopping after the first successfully authenticated pull. - -Example keys: - - 123456789.dkr.ecr.us-east-1.amazonaws.com - - ∗.azurecr.io - - gcr.io - - ∗.∗.registry.io - - registry.io:8080/path
    - - ## `AuthConfig` {#credentialprovider-kubelet-k8s-io-v1alpha1-AuthConfig} - - **Appears in:** - [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderResponse) -AuthConfig contains authentication information for a container registry. +

    AuthConfig contains authentication information for a container registry. Only username/password based authentication is supported today, but more authentication -mechanisms may be added in the future. +mechanisms may be added in the future.

    + - +

    username is the username used for authenticating to the container registry +An empty username is valid.

    + - - +

    password is the password used for authenticating to the container registry +An empty password is valid.

    + - -
    FieldDescription
    username [Required]
    string     		- username is the username used for authenticating to the container registry -An empty username is valid.
    password [Required]
    string     		- password is the password used for authenticating to the container registry -An empty password is valid.
    - - ## `PluginCacheKeyType` {#credentialprovider-kubelet-k8s-io-v1alpha1-PluginCacheKeyType} (Alias of `string`) - **Appears in:** - [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1alpha1-CredentialProviderResponse) @@ -187,6 +166,4 @@ An empty password is valid. - - diff --git a/content/en/docs/reference/config-api/kubelet-credentialprovider.v1beta1.md b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1beta1.md new file mode 100644 index 0000000000..a849970d4b --- /dev/null +++ b/content/en/docs/reference/config-api/kubelet-credentialprovider.v1beta1.md @@ -0,0 +1,169 @@ +--- +title: Kubelet CredentialProvider (v1beta1) +content_type: tool-reference +package: credentialprovider.kubelet.k8s.io/v1beta1 +auto_generated: true +--- + + +## Resource Types + + +- [CredentialProviderRequest](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderRequest) +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse) + + + +## `CredentialProviderRequest` {#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderRequest} + + + +

    CredentialProviderRequest includes the image that the kubelet requires authentication for. +Kubelet will pass this request object to the plugin via stdin. In general, plugins should +prefer responding with the same apiVersion they were sent.

    + + + + + + + + + + + + + + +
    FieldDescription
    apiVersion
    string    		credentialprovider.kubelet.k8s.io/v1beta1
    kind
    string    		CredentialProviderRequest
    image [Required]
    +string +    		 +

    image is the container image that is being pulled as part of the +credential provider plugin request. Plugins may optionally parse the image +to extract any information required to fetch credentials.

    +
    + +## `CredentialProviderResponse` {#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse} + + + +

    CredentialProviderResponse holds credentials that the kubelet should use for the specified +image provided in the original request. Kubelet will read the response from the plugin via stdout. +This response should be set to the same apiVersion as CredentialProviderRequest.

    + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    apiVersion
    string    		credentialprovider.kubelet.k8s.io/v1beta1
    kind
    string    		CredentialProviderResponse
    cacheKeyType [Required]
    +PluginCacheKeyType +    		 +

    cacheKeyType indiciates the type of caching key to use based on the image provided +in the request. There are three valid values for the cache key type: Image, Registry, and +Global. If an invalid value is specified, the response will NOT be used by the kubelet.

    +
    cacheDuration
    +meta/v1.Duration +    		 +

    cacheDuration indicates the duration the provided credentials should be cached for. +The kubelet will use this field to set the in-memory cache duration for credentials +in the AuthConfig. If null, the kubelet will use defaultCacheDuration provided in +CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.

    +
    auth
    +map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1beta1.AuthConfig +    		 +

    auth is a map containing authentication information passed into the kubelet. +Each key is a match image string (more on this below). The corresponding authConfig value +should be valid for all images that match against this key. A plugin should set +this field to null if no valid credentials can be returned for the requested image.

    +

    Each key in the map is a pattern which can optionally contain a port and a path. +Globs can be used in the domain, but not in the port or the path. Globs are supported +as subdomains like '.k8s.io' or 'k8s..io', and top-level-domains such as 'k8s.'. +Matching partial subdomains like 'app.k8s.io' is also supported. Each glob can only match +a single subdomain segment, so *.io does not match *.k8s.io.

    +

    The kubelet will match images against the key when all of the below are true:

    +
      +
    • Both contain the same number of domain parts and each part matches.
      • +
    • The URL path of an imageMatch must be a prefix of the target image URL path.
      • +
    • If the imageMatch contains a port, then the port must match in the image as well.
      • +
    +

    When multiple keys are returned, the kubelet will traverse all keys in reverse order so that:

    +
      +
    • longer keys come before shorter keys with the same prefix
      • +
    • non-wildcard keys come before wildcard keys with the same prefix.
      • +
    +

    For any given match, the kubelet will attempt an image pull with the provided credentials, +stopping after the first successfully authenticated pull.

    +

    Example keys:

    +
      +
    • 123456789.dkr.ecr.us-east-1.amazonaws.com
      • +
    • *.azurecr.io
      • +
    • gcr.io
      • +
    • ..registry.io
      • +
    • registry.io:8080/path
      • +
    +
    + +## `AuthConfig` {#credentialprovider-kubelet-k8s-io-v1beta1-AuthConfig} + + +**Appears in:** + +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse) + + +

    AuthConfig contains authentication information for a container registry. +Only username/password based authentication is supported today, but more authentication +mechanisms may be added in the future.

    + + + + + + + + + + + + + + +
    FieldDescription
    username [Required]
    +string +    		 +

    username is the username used for authenticating to the container registry +An empty username is valid.

    +
    password [Required]
    +string +    		 +

    password is the password used for authenticating to the container registry +An empty password is valid.

    +
    + +## `PluginCacheKeyType` {#credentialprovider-kubelet-k8s-io-v1beta1-PluginCacheKeyType} + +(Alias of `string`) + +**Appears in:** + +- [CredentialProviderResponse](#credentialprovider-kubelet-k8s-io-v1beta1-CredentialProviderResponse) + + + + + diff --git a/content/en/docs/reference/glossary/downward-api.md b/content/en/docs/reference/glossary/downward-api.md new file mode 100644 index 0000000000..a3d9a46336 --- /dev/null +++ b/content/en/docs/reference/glossary/downward-api.md @@ -0,0 +1,28 @@ +--- +title: Downward API +id: downward-api +date: 2022-03-21 +short_description: > + A mechanism to expose Pod and container field values to code running in a container. +aka: +full_link: /docs/concepts/workloads/pods/downward-api/ +tags: +- architecture +--- +Kubernetes' mechanism to expose Pod and container field values to code running in a container. + +It is sometimes useful for a container to have information about itself, without +needing to make changes to the container code that directly couple it to Kubernetes. + +The Kubernetes downward API allows containers to consume information about themselves +or their context in a Kubernetes cluster. Applications in containers can have +access to that information, without the application needing to act as a client of +the Kubernetes API. + +There are two ways to expose Pod and container fields to a running container: + +- using [environment variables](/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) +- using [a `downwardAPI` volume](/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) + +Together, these two ways of exposing Pod and container fields are called the _downward API_. + diff --git a/content/en/docs/reference/kubectl/cheatsheet.md b/content/en/docs/reference/kubectl/cheatsheet.md index e1711ea542..3f96fd6e4c 100644 --- a/content/en/docs/reference/kubectl/cheatsheet.md +++ b/content/en/docs/reference/kubectl/cheatsheet.md @@ -30,7 +30,7 @@ You can also use a shorthand alias for `kubectl` that also works with completion ```bash alias k=kubectl -complete -F __start_kubectl k +complete -o default -F __start_kubectl k ``` ### ZSH @@ -280,7 +280,7 @@ kubectl patch deployment valid-deployment --type json -p='[{"op": "remove", " # Add a new element to a positional array kubectl patch sa default --type='json' -p='[{"op": "add", "path": "/secrets/1", "value": {"name": "whatever" } }]' -# Update a deployment's replicas count by patching it's scale subresource +# Update a deployment's replica count by patching its scale subresource kubectl patch deployment nginx-deployment --subresource='scale' --type='merge' -p '{"spec":{"replicas":2}}' ``` diff --git a/content/en/docs/reference/kubernetes-api/workload-resources/pod-v1.md b/content/en/docs/reference/kubernetes-api/workload-resources/pod-v1.md index dbcc8f889e..d26dfdbaf9 100644 --- a/content/en/docs/reference/kubernetes-api/workload-resources/pod-v1.md +++ b/content/en/docs/reference/kubernetes-api/workload-resources/pod-v1.md @@ -1879,7 +1879,7 @@ PodStatus represents information about the status of a pod. Status may trail the - **qosClass** (string) - The Quality of Service (QOS) classification assigned to the pod based on resource requirements See PodQOSClass type for available QOS classes More info: https://git.k8s.io/community/contributors/design-proposals/node/resource-qos.md + The Quality of Service (QOS) classification assigned to the pod based on resource requirements See PodQOSClass type for available QOS classes More info: https://git.k8s.io/design-proposals-archive/node/resource-qos.md diff --git a/content/en/docs/reference/labels-annotations-taints/_index.md b/content/en/docs/reference/labels-annotations-taints/_index.md index c6baa12557..927c45c613 100644 --- a/content/en/docs/reference/labels-annotations-taints/_index.md +++ b/content/en/docs/reference/labels-annotations-taints/_index.md @@ -17,7 +17,7 @@ This document serves both as a reference to the values and as a coordination poi ### app.kubernetes.io/component -Example: `app.kubernetes.io/component=database` +Example: `app.kubernetes.io/component: "database"` Used on: All Objects @@ -27,7 +27,7 @@ One of the [recommended labels](/docs/concepts/overview/working-with-objects/com ### app.kubernetes.io/created-by -Example: `app.kubernetes.io/created-by=controller-manager` +Example: `app.kubernetes.io/created-by: "controller-manager"` Used on: All Objects @@ -37,7 +37,7 @@ One of the [recommended labels](/docs/concepts/overview/working-with-objects/com ### app.kubernetes.io/instance -Example: `app.kubernetes.io/instance=mysql-abcxzy` +Example: `app.kubernetes.io/instance: "mysql-abcxzy"` Used on: All Objects @@ -47,7 +47,7 @@ One of the [recommended labels](/docs/concepts/overview/working-with-objects/com ### app.kubernetes.io/managed-by -Example: `app.kubernetes.io/managed-by=helm` +Example: `app.kubernetes.io/managed-by: "helm"` Used on: All Objects @@ -57,7 +57,7 @@ One of the [recommended labels](/docs/concepts/overview/working-with-objects/com ### app.kubernetes.io/name -Example: `app.kubernetes.io/name=mysql` +Example: `app.kubernetes.io/name: "mysql"` Used on: All Objects @@ -67,7 +67,7 @@ One of the [recommended labels](/docs/concepts/overview/working-with-objects/com ### app.kubernetes.io/part-of -Example: `app.kubernetes.io/part-of=wordpress` +Example: `app.kubernetes.io/part-of: "wordpress"` Used on: All Objects @@ -77,7 +77,7 @@ One of the [recommended labels](/docs/concepts/overview/working-with-objects/com ### app.kubernetes.io/version -Example: `app.kubernetes.io/version="5.7.21"` +Example: `app.kubernetes.io/version: "5.7.21"` Used on: All Objects @@ -87,7 +87,7 @@ One of the [recommended labels](/docs/concepts/overview/working-with-objects/com ### kubernetes.io/arch -Example: `kubernetes.io/arch=amd64` +Example: `kubernetes.io/arch: "amd64"` Used on: Node @@ -95,7 +95,7 @@ The Kubelet populates this with `runtime.GOARCH` as defined by Go. This can be h ### kubernetes.io/os -Example: `kubernetes.io/os=linux` +Example: `kubernetes.io/os: "linux"` Used on: Node @@ -103,7 +103,7 @@ The Kubelet populates this with `runtime.GOOS` as defined by Go. This can be han ### kubernetes.io/metadata.name -Example: `kubernetes.io/metadata.name=mynamespace` +Example: `kubernetes.io/metadata.name: "mynamespace"` Used on: Namespaces @@ -124,7 +124,7 @@ This label has been deprecated. Please use `kubernetes.io/os` instead. ### kubernetes.io/hostname {#kubernetesiohostname} -Example: `kubernetes.io/hostname=ip-172-20-114-199.ec2.internal` +Example: `kubernetes.io/hostname: "ip-172-20-114-199.ec2.internal"` Used on: Node @@ -135,7 +135,7 @@ This label is also used as part of the topology hierarchy. See [topology.kubern ### kubernetes.io/change-cause {#change-cause} -Example: `kubernetes.io/change-cause=kubectl edit --record deployment foo` +Example: `kubernetes.io/change-cause: "kubectl edit --record deployment foo"` Used on: All Objects @@ -161,7 +161,7 @@ The value for this annotation must be **true** to take effect. This annotation i ### controller.kubernetes.io/pod-deletion-cost {#pod-deletion-cost} -Example: `controller.kubernetes.io/pod-deletion-cost=10` +Example: `controller.kubernetes.io/pod-deletion-cost: "10"` Used on: Pod @@ -212,7 +212,7 @@ For example, `10M` means 10 megabits per second. ### node.kubernetes.io/instance-type {#nodekubernetesioinstance-type} -Example: `node.kubernetes.io/instance-type=m3.medium` +Example: `node.kubernetes.io/instance-type: "m3.medium"` Used on: Node @@ -237,7 +237,7 @@ See [topology.kubernetes.io/zone](#topologykubernetesiozone). Example: -`statefulset.kubernetes.io/pod-name=mystatefulset-7` +`statefulset.kubernetes.io/pod-name: "mystatefulset-7"` When a StatefulSet controller creates a Pod for the StatefulSet, the control plane sets this label on that Pod. The value of the label is the name of the Pod being created. @@ -249,7 +249,7 @@ StatefulSet topic for more details. Example: -`topology.kubernetes.io/region=us-east-1` +`topology.kubernetes.io/region: "us-east-1"` See [topology.kubernetes.io/zone](#topologykubernetesiozone). @@ -257,7 +257,7 @@ See [topology.kubernetes.io/zone](#topologykubernetesiozone). Example: -`topology.kubernetes.io/zone=us-east-1c` +`topology.kubernetes.io/zone: "us-east-1c"` Used on: Node, PersistentVolume @@ -286,7 +286,7 @@ adding the labels manually (or adding support for `PersistentVolumeLabel`). With ### volume.beta.kubernetes.io/storage-provisioner (deprecated) -Example: `volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath` +Example: `volume.beta.kubernetes.io/storage-provisioner: "k8s.io/minikube-hostpath"` Used on: PersistentVolumeClaim @@ -310,7 +310,7 @@ This annotation will be added to dynamic provisioning required PVC. ### node.kubernetes.io/windows-build {#nodekubernetesiowindows-build} -Example: `node.kubernetes.io/windows-build=10.0.17763` +Example: `node.kubernetes.io/windows-build: "10.0.17763"` Used on: Node @@ -320,7 +320,7 @@ The label's value is in the format "MajorVersion.MinorVersion.BuildNumber". ### service.kubernetes.io/headless {#servicekubernetesioheadless} -Example: `service.kubernetes.io/headless=""` +Example: `service.kubernetes.io/headless: ""` Used on: Service @@ -328,15 +328,33 @@ The control plane adds this label to an Endpoints object when the owning Service ### kubernetes.io/service-name {#kubernetesioservice-name} -Example: `kubernetes.io/service-name="nginx"` +Example: `kubernetes.io/service-name: "nginx"` Used on: Service Kubernetes uses this label to differentiate multiple Services. Used currently for `ELB`(Elastic Load Balancer) only. +### kubernetes.io/service-account.name + +Example: `kubernetes.io/service-account.name: "sa-name"` + +Used on: Secret + +This annotation records the {{< glossary_tooltip term_id="name" text="name">}} of the +ServiceAccount that the token (stored in the Secret of type `kubernetes.io/service-account-token`) represents. + +### kubernetes.io/service-account.uid + +Example: `kubernetes.io/service-account.uid: da68f9c6-9d26-11e7-b84e-002dc52800da` + +Used on: Secret + +This annotation records the {{< glossary_tooltip term_id="uid" text="unique ID" >}} of the +ServiceAccount that the token (stored in the Secret of type `kubernetes.io/service-account-token`) represents. + ### endpointslice.kubernetes.io/managed-by {#endpointslicekubernetesiomanaged-by} -Example: `endpointslice.kubernetes.io/managed-by="controller"` +Example: `endpointslice.kubernetes.io/managed-by: "controller"` Used on: EndpointSlices @@ -344,7 +362,7 @@ The label is used to indicate the controller or entity that manages an EndpointS ### endpointslice.kubernetes.io/skip-mirror {#endpointslicekubernetesioskip-mirror} -Example: `endpointslice.kubernetes.io/skip-mirror="true"` +Example: `endpointslice.kubernetes.io/skip-mirror: "true"` Used on: Endpoints @@ -352,7 +370,7 @@ The label can be set to `"true"` on an Endpoints resource to indicate that the E ### service.kubernetes.io/service-proxy-name {#servicekubernetesioservice-proxy-name} -Example: `service.kubernetes.io/service-proxy-name="foo-bar"` +Example: `service.kubernetes.io/service-proxy-name: "foo-bar"` Used on: Service @@ -364,7 +382,7 @@ Example: `experimental.windows.kubernetes.io/isolation-type: "hyperv"` Used on: Pod -The annotation is used to run Windows containers with Hyper-V isolation. To use Hyper-V isolation feature and create a Hyper-V isolated container, the kubelet should be started with feature gates HyperVContainer=true and the Pod should include the annotation experimental.windows.kubernetes.io/isolation-type=hyperv. +The annotation is used to run Windows containers with Hyper-V isolation. To use Hyper-V isolation feature and create a Hyper-V isolated container, the kubelet should be started with feature gates HyperVContainer=true and the Pod should include the annotation `experimental.windows.kubernetes.io/isolation-type: hyperv`. {{< note >}} You can only set this annotation on Pods that have a single container. @@ -387,7 +405,7 @@ Starting in v1.18, this annotation is deprecated in favor of `spec.ingressClassN ### storageclass.kubernetes.io/is-default-class -Example: `storageclass.kubernetes.io/is-default-class=true` +Example: `storageclass.kubernetes.io/is-default-class: "true"` Used on: StorageClass @@ -449,43 +467,43 @@ Use [Taints and Tolerations](/docs/concepts/scheduling-eviction/taint-and-tolera ### node.kubernetes.io/not-ready -Example: `node.kubernetes.io/not-ready:NoExecute` +Example: `node.kubernetes.io/not-ready: "NoExecute"` The node controller detects whether a node is ready by monitoring its health and adds or removes this taint accordingly. ### node.kubernetes.io/unreachable -Example: `node.kubernetes.io/unreachable:NoExecute` +Example: `node.kubernetes.io/unreachable: "NoExecute"` The node controller adds the taint to a node corresponding to the [NodeCondition](/docs/concepts/architecture/nodes/#condition) `Ready` being `Unknown`. ### node.kubernetes.io/unschedulable -Example: `node.kubernetes.io/unschedulable:NoSchedule` +Example: `node.kubernetes.io/unschedulable: "NoSchedule"` The taint will be added to a node when initializing the node to avoid race condition. ### node.kubernetes.io/memory-pressure -Example: `node.kubernetes.io/memory-pressure:NoSchedule` +Example: `node.kubernetes.io/memory-pressure: "NoSchedule"` The kubelet detects memory pressure based on `memory.available` and `allocatableMemory.available` observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed. ### node.kubernetes.io/disk-pressure -Example: `node.kubernetes.io/disk-pressure:NoSchedule` +Example: `node.kubernetes.io/disk-pressure :"NoSchedule"` The kubelet detects disk pressure based on `imagefs.available`, `imagefs.inodesFree`, `nodefs.available` and `nodefs.inodesFree`(Linux only) observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed. ### node.kubernetes.io/network-unavailable -Example: `node.kubernetes.io/network-unavailable:NoSchedule` +Example: `node.kubernetes.io/network-unavailable: "NoSchedule"` This is initially set by the kubelet when the cloud provider used indicates a requirement for additional network configuration. Only when the route on the cloud is configured properly will the taint be removed by the cloud provider. ### node.kubernetes.io/pid-pressure -Example: `node.kubernetes.io/pid-pressure:NoSchedule` +Example: `node.kubernetes.io/pid-pressure: "NoSchedule"` The kubelet checks D-value of the size of `/proc/sys/kernel/pid_max` and the PIDs consumed by Kubernetes on a node to get the number of available PIDs that referred to as the `pid.available` metric. The metric is then compared to the corresponding threshold that can be set on the kubelet to determine if the node condition and taint should be added/removed. @@ -506,19 +524,19 @@ for further details about when and how to use this taint. ### node.cloudprovider.kubernetes.io/uninitialized -Example: `node.cloudprovider.kubernetes.io/uninitialized:NoSchedule` +Example: `node.cloudprovider.kubernetes.io/uninitialized: "NoSchedule"` Sets this taint on a node to mark it as unusable, when kubelet is started with the "external" cloud provider, until a controller from the cloud-controller-manager initializes this node, and then removes the taint. ### node.cloudprovider.kubernetes.io/shutdown -Example: `node.cloudprovider.kubernetes.io/shutdown:NoSchedule` +Example: `node.cloudprovider.kubernetes.io/shutdown: "NoSchedule"` If a Node is in a cloud provider specified shutdown state, the Node gets tainted accordingly with `node.cloudprovider.kubernetes.io/shutdown` and the taint effect of `NoSchedule`. ### pod-security.kubernetes.io/enforce -Example: `pod-security.kubernetes.io/enforce: baseline` +Example: `pod-security.kubernetes.io/enforce: "baseline"` Used on: Namespace @@ -532,7 +550,7 @@ for more information. ### pod-security.kubernetes.io/enforce-version -Example: `pod-security.kubernetes.io/enforce-version: {{< skew latestVersion >}}` +Example: `pod-security.kubernetes.io/enforce-version: "{{< skew currentVersion >}}"` Used on: Namespace @@ -545,7 +563,7 @@ for more information. ### pod-security.kubernetes.io/audit -Example: `pod-security.kubernetes.io/audit: baseline` +Example: `pod-security.kubernetes.io/audit: "baseline"` Used on: Namespace @@ -559,7 +577,7 @@ for more information. ### pod-security.kubernetes.io/audit-version -Example: `pod-security.kubernetes.io/audit-version: {{< skew latestVersion >}}` +Example: `pod-security.kubernetes.io/audit-version: "{{< skew currentVersion >}}"` Used on: Namespace @@ -572,7 +590,7 @@ for more information. ### pod-security.kubernetes.io/warn -Example: `pod-security.kubernetes.io/warn: baseline` +Example: `pod-security.kubernetes.io/warn: "baseline"` Used on: Namespace @@ -588,7 +606,7 @@ for more information. ### pod-security.kubernetes.io/warn-version -Example: `pod-security.kubernetes.io/warn-version: {{< skew latestVersion >}}` +Example: `pod-security.kubernetes.io/warn-version: "{{< skew currentVersion >}}"` Used on: Namespace diff --git a/content/en/docs/reference/ports-and-protocols.md b/content/en/docs/reference/ports-and-protocols.md index 91d6cba8e7..8ca5bc0774 100644 --- a/content/en/docs/reference/ports-and-protocols.md +++ b/content/en/docs/reference/ports-and-protocols.md @@ -7,7 +7,7 @@ weight: 50 When running Kubernetes in an environment with strict network boundaries, such as on-premises datacenter with physical network firewalls or Virtual Networks in Public Cloud, it is useful to be aware of the ports and protocols -used by Kubernetes components +used by Kubernetes components. ## Control plane diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_generate-csr.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_generate-csr.md index 1abc7d9bac..9cd82c4316 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_generate-csr.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_generate-csr.md @@ -92,6 +92,3 @@ kubeadm certs generate-csr [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_apiserver.md index 7dc59c45d4..b770895500 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_apiserver.md @@ -87,6 +87,3 @@ kubeadm certs renew apiserver [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-healthcheck-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-healthcheck-client.md index 84d75bfd36..0fde99368c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-healthcheck-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-healthcheck-client.md @@ -87,6 +87,3 @@ kubeadm certs renew etcd-healthcheck-client [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-peer.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-peer.md index 60acaae1db..214b353b00 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-peer.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-peer.md @@ -87,6 +87,3 @@ kubeadm certs renew etcd-peer [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-server.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-server.md index 969157fe3e..cd8b73908d 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-server.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_certs_renew_etcd-server.md @@ -87,6 +87,3 @@ kubeadm certs renew etcd-server [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md index 0f85b4fbc2..7dd3a4f820 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md @@ -67,6 +67,3 @@ kubeadm config images [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md index 0700538f56..d46abeab03 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md @@ -55,7 +55,7 @@ kubeadm config images list [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    @@ -116,6 +116,3 @@ kubeadm config images list [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md index 7423ad1934..775881483b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md @@ -48,7 +48,7 @@ kubeadm config images pull [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md index 1c634871eb..ea7c83e14b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md @@ -79,6 +79,3 @@ kubeadm config print join-defaults [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md index f71e83cf1b..645d11c80b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md @@ -138,7 +138,7 @@ kubeadm init [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md index 64777661d0..1ff70e5cb8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md @@ -60,6 +60,3 @@ kubeadm init phase addon [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md index 198aee2657..292abcf5fb 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md @@ -62,7 +62,7 @@ kubeadm init phase addon all [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md index 8b07e647da..8eddf5bb5d 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md @@ -41,7 +41,7 @@ kubeadm init phase addon coredns [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md index 547601e364..a47b8390be 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md @@ -85,6 +85,3 @@ kubeadm init phase certs etcd-ca [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md index a3df321d88..d6c0630075 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md @@ -69,6 +69,3 @@ kubeadm init phase certs sa [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md index c5b6ce6dba..1dfaa1a7cb 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md @@ -101,7 +101,7 @@ kubeadm init phase control-plane all [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md index 4f589f3dbd..9d5f9583b2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md @@ -83,7 +83,7 @@ kubeadm init phase control-plane apiserver [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md index f1ebdbcf12..1eb52e8287 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md @@ -116,6 +116,3 @@ kubeadm init phase kubeconfig all [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-finalize_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-finalize_all.md index 70e4c634b0..55a98fae96 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-finalize_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-finalize_all.md @@ -81,6 +81,3 @@ kubeadm init phase kubelet-finalize all [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md index 11d2407499..a79f41f610 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md @@ -88,6 +88,3 @@ kubeadm init phase kubelet-start [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md index 345621f703..61e5a0c946 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md @@ -81,6 +81,3 @@ kubeadm init phase preflight [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md index 515060a76c..e242f4eedd 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md @@ -95,6 +95,3 @@ kubeadm init phase upload-certs [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md index 3c087368a7..ba4e91fa89 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md @@ -74,6 +74,3 @@ kubeadm init phase upload-config all [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md index 13e561f486..930b2d94a2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md @@ -83,6 +83,3 @@ kubeadm init phase upload-config kubeadm [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md index ba27f728cb..df7caadce6 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md @@ -15,7 +15,7 @@ Upload the kubelet component config to a ConfigMap ### Synopsis -Upload kubelet configuration extracted from the kubeadm InitConfiguration object to a ConfigMap of the form kubelet-config-1.X in the cluster, where X is the minor version of the current (API Server) Kubernetes version. +Upload the kubelet configuration extracted from the kubeadm InitConfiguration object to a kubelet-config ConfigMap in the cluster ``` kubeadm init phase upload-config kubelet [flags] diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md index 07768a16c6..f7ea8ea39a 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md @@ -67,6 +67,3 @@ kubeadm join phase control-plane-join [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md index 7a3517652d..496213ce90 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md @@ -95,6 +95,3 @@ kubeadm join phase control-plane-join all [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md index c06ddaae40..d127f67fd8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md @@ -9,7 +9,6 @@ guide. You can file document formatting bugs against the [reference-docs](https://github.com/kubernetes-sigs/reference-docs/) project. --> - Add a new local etcd member ### Synopsis diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md index 6952dbca80..3dc12615a9 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md @@ -67,6 +67,3 @@ kubeadm join phase control-plane-prepare [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md index 661edf597d..1d5351f3af 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md @@ -151,6 +151,3 @@ kubeadm join phase control-plane-prepare all [api-server-endpoint] [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md index 6475115940..81355a775e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md @@ -110,8 +110,6 @@ kubeadm join phase control-plane-prepare certs [api-server-endpoint] [flags] - - ### Options inherited from parent commands @@ -130,6 +128,3 @@ kubeadm join phase control-plane-prepare certs [api-server-endpoint] [flags]
    - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md index 5896b25337..cafb58658e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md @@ -123,6 +123,3 @@ kubeadm join phase kubelet-start [api-server-endpoint] [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md index e5cffa35f8..ae869d7f47 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md @@ -52,6 +52,13 @@ kubeadm reset [flags]

    Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.

    + +--dry-run + + +

    Don't apply any changes; just output what would be done.

    + + -f, --force @@ -110,6 +117,3 @@ kubeadm reset [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md index ceabd2045e..4120c0a97d 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md @@ -74,6 +74,3 @@ kubeadm reset phase cleanup-node [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md index d2c1060ff4..3fd91a98a6 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md @@ -67,6 +67,3 @@ kubeadm reset phase remove-etcd-member [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md index 5384fc4d6c..025ab1efac 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md @@ -91,6 +91,3 @@ kubeadm token [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md index a2a217033c..4687fcbba1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md @@ -9,7 +9,6 @@ guide. You can file document formatting bugs against the [reference-docs](https://github.com/kubernetes-sigs/reference-docs/) project. --> - Create bootstrap tokens on the server ### Synopsis diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md index 2040bd3f94..30b7678798 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md @@ -79,6 +79,3 @@ kubeadm token delete [token-value] ... - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md index 9fd720662e..629fed4ff4 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md @@ -76,7 +76,7 @@ kubeadm upgrade apply [version] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md index eb5e3c4cac..718257f8af 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md @@ -102,6 +102,3 @@ kubeadm upgrade diff [version] [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md index a8a3138c88..12fbe5b8f3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md @@ -117,6 +117,3 @@ kubeadm upgrade node [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md index 6b86c95054..ce5b6f8429 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md @@ -56,6 +56,3 @@ Use this command to invoke single phase of the node workflow - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md index d2b03974c2..e03f59f4f3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md @@ -15,7 +15,7 @@ Upgrade the kubelet configuration for this node ### Synopsis -Download the kubelet configuration from a ConfigMap of the form "kubelet-config-1.X" in the cluster, where X is the minor version of the kubelet. kubeadm uses the KuberneteVersion field in the kubeadm-config ConfigMap to determine what the _desired_ kubelet version is. +Download the kubelet configuration from the kubelet-config ConfigMap stored in the cluster ``` kubeadm upgrade node phase kubelet-config [flags] diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_preflight.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_preflight.md index d82a193898..ff44d0d7da 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_preflight.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_preflight.md @@ -67,6 +67,3 @@ kubeadm upgrade node phase preflight [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md index d592249f76..f7913a8452 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md @@ -55,7 +55,7 @@ kubeadm upgrade plan [version] [flags] --feature-gates string -

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (ALPHA - default=false)

    +

    A set of key=value pairs that describe feature gates for various features. Options are:
    PublicKeysECDSA=true|false (ALPHA - default=false)
    RootlessControlPlane=true|false (ALPHA - default=false)
    UnversionedKubeletConfigMap=true|false (BETA - default=true)

    diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md index b86c725977..38cc27bee9 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md @@ -67,6 +67,3 @@ kubeadm version [flags] - - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md b/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md index b98fdcd66e..bc752e4d0b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md +++ b/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md @@ -219,7 +219,7 @@ Other API server flags that are set unconditionally are: - `--insecure-port=0` to avoid insecure connections to the api server - `--enable-bootstrap-token-auth=true` to enable the `BootstrapTokenAuthenticator` authentication module. - See [TLS Bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) for more details + See [TLS Bootstrapping](/docs/reference/access-authn-authn/kubelet-tls-bootstrapping/) for more details - `--allow-privileged` to `true` (required e.g. by kube proxy) - `--requestheader-client-ca-file` to `front-proxy-ca.crt` - `--enable-admission-plugins` to: @@ -266,7 +266,7 @@ The static Pod manifest for the controller manager is affected by following para Other flags that are set unconditionally are: - `--controllers` enabling all the default controllers plus `BootstrapSigner` and `TokenCleaner` controllers for TLS bootstrap. - See [TLS Bootstrapping](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) for more details + See [TLS Bootstrapping](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) for more details - `--use-service-account-credentials` to `true` - Flags for using certificates generated in previous steps: - `--root-ca-file` to `ca.crt` @@ -327,7 +327,7 @@ individually with the [`kubeadm init phase mark-control-plane`](/docs/reference/ ### Configure TLS-Bootstrapping for node joining Kubeadm uses [Authenticating with Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) for joining new nodes to an -existing cluster; for more details see also [design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/bootstrap-discovery.md). +existing cluster; for more details see also [design proposal](https://git.k8s.io/design-proposals-archive/cluster-lifecycle/bootstrap-discovery.md). `kubeadm init` ensures that everything is properly configured for this process, and this includes following steps as well as setting API server and controller flags as already described in previous paragraphs. @@ -418,7 +418,7 @@ Similarly to `kubeadm init`, also `kubeadm join` internal workflow consists of a This is split into discovery (having the Node trust the Kubernetes Master) and TLS bootstrap (having the Kubernetes Master trust the Node). -see [Authenticating with Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) or the corresponding [design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/bootstrap-discovery.md). +see [Authenticating with Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) or the corresponding [design proposal](https://git.k8s.io/design-proposals-archive/cluster-lifecycle/bootstrap-discovery.md). ### Preflight checks diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md index fc87e796c2..fdb117c5d5 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -52,7 +52,7 @@ following steps: 1. Makes all the necessary configurations for allowing node joining with the [Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) and - [TLS Bootstrap](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/) + [TLS Bootstrap](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) mechanism: - Write a ConfigMap for making available all the information required @@ -242,6 +242,15 @@ where `config.yaml` contains the custom `imageRepository`, and/or `imageTag` for etcd and CoreDNS. * Pass the same `config.yaml` to `kubeadm init`. +#### Custom sandbox (pause) images {#custom-pause-image} + +To set a custom image for these you need to configure this in your +{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} +to use the image. +Consult the documentation for your container runtime to find out how to change this setting; +for selected container runtimes, you can also find advice within the +[Container Runtimes]((/docs/setup/production-environment/container-runtimes/) topic. + ### Uploading control-plane certificates to the cluster By adding the flag `--upload-certs` to `kubeadm init` you can temporary upload diff --git a/content/en/docs/reference/using-api/_index.md b/content/en/docs/reference/using-api/_index.md index 5e335fb191..6592deb3c7 100644 --- a/content/en/docs/reference/using-api/_index.md +++ b/content/en/docs/reference/using-api/_index.md @@ -39,7 +39,7 @@ The JSON and Protobuf serialization schemas follow the same guidelines for schema changes. The following descriptions cover both formats. The API versioning and software versioning are indirectly related. -The [API and release versioning proposal](https://git.k8s.io/community/contributors/design-proposals/release/versioning.md) +The [API and release versioning proposal](https://git.k8s.io/design-proposals-archive/release/versioning.md) describes the relationship between API versioning and software versioning. Different API versions indicate different levels of stability and support. You @@ -83,7 +83,7 @@ Here's a summary of each level: ## API groups -[API groups](https://git.k8s.io/community/contributors/design-proposals/api-machinery/api-group.md) +[API groups](https://git.k8s.io/design-proposals-archive/api-machinery/api-group.md) make it easier to extend the Kubernetes API. The API group is specified in a REST path and in the `apiVersion` field of a serialized object. @@ -124,4 +124,4 @@ Kubernetes stores its serialized state in terms of the API resources by writing - Learn more about [API conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#api-conventions) - Read the design documentation for - [aggregator](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/aggregated-api-servers.md) + [aggregator](https://git.k8s.io/design-proposals-archive/api-machinery/aggregated-api-servers.md) diff --git a/content/en/docs/reference/using-api/api-concepts.md b/content/en/docs/reference/using-api/api-concepts.md index 2e4fb85df2..1a722acffd 100644 --- a/content/en/docs/reference/using-api/api-concepts.md +++ b/content/en/docs/reference/using-api/api-concepts.md @@ -15,8 +15,8 @@ primary resources via the standard HTTP verbs (POST, PUT, PATCH, DELETE, GET). For some resources, the API includes additional subresources that allow -fine grained authorization (such as a separating viewing details for a Pod from -retrieving its logs), and can accept and serve those resources in different +fine grained authorization (such as separate views for Pod details and +log retrievals), and can accept and serve those resources in different representations for convenience or efficiency. Kubernetes supports efficient change notifications on resources via *watches*. diff --git a/content/en/docs/reference/using-api/deprecation-guide.md b/content/en/docs/reference/using-api/deprecation-guide.md index ed53601841..d448344504 100644 --- a/content/en/docs/reference/using-api/deprecation-guide.md +++ b/content/en/docs/reference/using-api/deprecation-guide.md @@ -110,8 +110,10 @@ The **policy/v1beta1** API version of PodDisruptionBudget will no longer be serv PodSecurityPolicy in the **policy/v1beta1** API version will no longer be served in v1.25, and the PodSecurityPolicy admission controller will be removed. -PodSecurityPolicy replacements are still under discussion, but current use can be migrated to -[3rd-party admission webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers/) now. +Migrate to [Pod Security Admission](/docs/concepts/security/pod-security-admission/) +or a [3rd party admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/). +For a migration guide, see [Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller](/docs/tasks/configure-pod-container/migrate-from-psp/). +For more information on the deprecation, see [PodSecurityPolicy Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). #### RuntimeClass {#runtimeclass-v125} @@ -174,7 +176,7 @@ The **authentication.k8s.io/v1beta1** API version of TokenReview is no longer se #### SubjectAccessReview resources {#subjectaccessreview-resources-v122} -The **authorization.k8s.io/v1beta1** API version of LocalSubjectAccessReview, SelfSubjectAccessReview, and SubjectAccessReview is no longer served as of v1.22. +The **authorization.k8s.io/v1beta1** API version of LocalSubjectAccessReview, SelfSubjectAccessReview, SubjectAccessReview, and SelfSubjectRulesReview is no longer served as of v1.22. * Migrate manifests and API clients to use the **authorization.k8s.io/v1** API version, available since v1.6. * Notable changes: @@ -336,14 +338,14 @@ to locate use of deprecated APIs. * Update custom integrations and controllers to call the non-deprecated APIs * Change YAML files to reference the non-deprecated APIs - You can use the `kubectl-convert` command (`kubectl convert` prior to v1.20) - to automatically convert an existing object: + You can use the `kubectl-convert` command (`kubectl convert` prior to v1.20) + to automatically convert an existing object: - `kubectl-convert -f --output-version /`. + `kubectl-convert -f --output-version /`. - For example, to convert an older Deployment to `apps/v1`, you can run: + For example, to convert an older Deployment to `apps/v1`, you can run: - `kubectl-convert -f ./my-deployment.yaml --output-version apps/v1` + `kubectl-convert -f ./my-deployment.yaml --output-version apps/v1` - Note that this may use non-ideal default values. To learn more about a specific - resource, check the Kubernetes [API reference](/docs/reference/kubernetes-api/). + Note that this may use non-ideal default values. To learn more about a specific + resource, check the Kubernetes [API reference](/docs/reference/kubernetes-api/). diff --git a/content/en/docs/reference/using-api/server-side-apply.md b/content/en/docs/reference/using-api/server-side-apply.md index 6b932278dc..e9f951a76a 100644 --- a/content/en/docs/reference/using-api/server-side-apply.md +++ b/content/en/docs/reference/using-api/server-side-apply.md @@ -125,7 +125,8 @@ this occurs, the applier has 3 options to resolve the conflicts: * **Overwrite value, become sole manager:** If overwriting the value was intentional (or if the applier is an automated process like a controller) the - applier should set the `force` query parameter to true and make the request + applier should set the `force` query parameter to true (in kubectl, it can be done by + using the `--force-conflicts` flag with the apply command) and make the request again. This forces the operation to succeed, changes the value of the field, and removes the field from all other managers' entries in managedFields. diff --git a/content/en/docs/setup/_index.md b/content/en/docs/setup/_index.md index bb73375553..adc7304033 100644 --- a/content/en/docs/setup/_index.md +++ b/content/en/docs/setup/_index.md @@ -27,6 +27,13 @@ control, available resources, and expertise required to operate and manage a clu You can [download Kubernetes](/releases/download/) to deploy a Kubernetes cluster on a local machine, into the cloud, or for your own datacenter. +Several [Kubernetes components](/docs/concepts/overview/components/) such as `kube-apiserver` or `kube-proxy` can also be +deployed as [container images](/releases/download/#container-images) within the cluster. + +It is **recommended** to run Kubernetes components as container images wherever +that is possible, and to have Kubernetes manage those components. +Components that run containers - notably, the kubelet - can't be included in this category. + If you don't want to manage a Kubernetes cluster yourself, you could pick a managed service, including [certified platforms](/docs/setup/production-environment/turnkey-solutions/). There are also other standardized and custom solutions across a wide range of cloud and diff --git a/content/en/docs/setup/best-practices/certificates.md b/content/en/docs/setup/best-practices/certificates.md index 6d6d576c39..23e4ac8df7 100644 --- a/content/en/docs/setup/best-practices/certificates.md +++ b/content/en/docs/setup/best-practices/certificates.md @@ -22,7 +22,7 @@ This page explains the certificates that your cluster requires. Kubernetes requires PKI for the following operations: * Client certificates for the kubelet to authenticate to the API server -* Kubelet [server certificates](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates) +* Kubelet [server certificates](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#client-and-serving-certificates) for the API server to talk to the kubelets * Server certificate for the API server endpoint * Client certificates for administrators of the cluster to authenticate to the API server diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index 1f658fa262..deaee23022 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -46,6 +46,41 @@ check the documentation for that version. +## Install and configure prerequisites + +The following steps apply common settings for Kubernetes nodes on Linux. + +You can skip a particular setting if you're certain you don't need it. + +For more information, see [Network Plugin Requirements](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#network-plugin-requirements) or the documentation for your specific container runtime. + +### Forwarding IPv4 and letting iptables see bridged traffic + +Verify that the `br_netfilter` module is loaded by running `lsmod | grep br_netfilter`. + +To load it explicitly, run `sudo modprobe br_netfilter`. + +In order for a Linux node's iptables to correctly view bridged traffic, verify that `net.bridge.bridge-nf-call-iptables` is set to 1 in your `sysctl` config. For example: + +```bash +cat <}} +{{% tab name="Linux" %}} +You can find this file under the path `/etc/containerd/config.toml`. +{{% /tab %}} +{{< tab name="Windows" >}} +You can find this file under the path `C:\Program Files\containerd\config.toml`. +{{< /tab >}} +{{< /tabs >}} On Linux the default CRI socket for containerd is `/run/containerd/containerd.sock`. On Windows the default CRI endpoint is `npipe://./pipe/containerd-containerd`. @@ -185,6 +197,14 @@ To use the `systemd` cgroup driver in `/etc/containerd/config.toml` with `runc`, [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true ``` +{{< note >}} +If you installed containerd from a package (for example, RPM or `.deb`), you may find +that the CRI integration plugin is disabled by default. + +You need CRI support enabled to use containerd with Kubernetes. Make sure that `cri` +is not included in the`disabled_plugins` list within `/etc/containerd/config.toml`; +if you made changes to that file, also restart `containerd`. +{{< /note >}} If you apply this change, make sure to restart containerd: @@ -193,7 +213,19 @@ sudo systemctl restart containerd ``` When using kubeadm, manually configure the -[cgroup driver for kubelet](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-control-plane-node). +[cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver). + +#### Overriding the sandbox (pause) image {#override-pause-image-containerd} + +In your [containerd config](https://github.com/containerd/cri/blob/master/docs/config.md) you can overwrite the +sandbox image by setting the following config: + +```toml +[plugins."io.containerd.grpc.v1.cri"] + sandbox_image = "k8s.gcr.io/pause:3.2" +``` + +You might need to restart `containerd` as well once you've updated the config file: `systemctl restart containerd`. ### CRI-O @@ -221,6 +253,19 @@ in sync. For CRI-O, the CRI socket is `/var/run/crio/crio.sock` by default. +#### Overriding the sandbox (pause) image {#override-pause-image-cri-o} + +In your [CRI-O config](https://github.com/cri-o/cri-o/blob/main/docs/crio.conf.5.md) you can set the following +config value: + +```toml +[crio.image] +pause_image="registry.k8s.io/pause:3.6" +``` + +This config option supports live configuration reload to apply this change: `systemctl reload crio` or by sending +`SIGHUP` to the `crio` process. + ### Docker Engine {#docker} {{< note >}} @@ -237,6 +282,12 @@ Docker Engine with Kubernetes. For `cri-dockerd`, the CRI socket is `/run/cri-dockerd.sock` by default. +#### Overriding the sandbox (pause) image {#override-pause-image-cri-dockerd} + +The `cri-dockerd` adapter accepts a command line argument for +specifying which container image to use as the Pod infrastructure container (“pause image”). +The command line argument to use is `--pod-infra-container-image`. + ### Mirantis Container Runtime {#mcr} [Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) is a commercially @@ -251,6 +302,12 @@ visit [MCR Deployment Guide](https://docs.mirantis.com/mcr/20.10/install.html). Check the systemd unit named `cri-docker.socket` to find out the path to the CRI socket. +#### Overriding the sandbox (pause) image {#override-pause-image-cri-dockerd-mcr} + +The `cri-dockerd` adapter accepts a command line argument for +specifying which container image to use as the Pod infrastructure container (“pause image”). +The command line argument to use is `--pod-infra-container-image`. + ## {{% heading "whatsnext" %}} As well as a container runtime, your cluster will need a working diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md index 9a7e4056dc..7f4f28a2cc 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md @@ -539,8 +539,8 @@ field when using `--config`. This option will control the versions of kube-apiserver, kube-controller-manager, kube-scheduler and kube-proxy. Example: -* kubeadm is at {{< skew latestVersion >}} -* `kubernetesVersion` must be at {{< skew latestVersion >}} or {{< skew prevMinorVersion >}} +* kubeadm is at {{< skew currentVersion >}} +* `kubernetesVersion` must be at {{< skew currentVersion >}} or {{< skew currentVersionAddMinor -1 >}} ### kubeadm's skew against the kubelet @@ -548,8 +548,8 @@ Similarly to the Kubernetes version, kubeadm can be used with a kubelet version version as kubeadm or one version older. Example: -* kubeadm is at {{< skew latestVersion >}} -* kubelet on the host must be at {{< skew latestVersion >}} or {{< skew prevMinorVersion >}} +* kubeadm is at {{< skew currentVersion >}} +* kubelet on the host must be at {{< skew currentVersion >}} or {{< skew currentVersionAddMinor -1 >}} ### kubeadm's skew against kubeadm @@ -562,17 +562,17 @@ the same node with `kubeadm upgrade`. Similar rules apply to the rest of the kub with the exception of `kubeadm upgrade`. Example for `kubeadm join`: -* kubeadm version {{< skew latestVersion >}} was used to create a cluster with `kubeadm init` -* Joining nodes must use a kubeadm binary that is at version {{< skew latestVersion >}} +* kubeadm version {{< skew currentVersion >}} was used to create a cluster with `kubeadm init` +* Joining nodes must use a kubeadm binary that is at version {{< skew currentVersion >}} Nodes that are being upgraded must use a version of kubeadm that is the same MINOR version or one MINOR version newer than the version of kubeadm used for managing the node. Example for `kubeadm upgrade`: -* kubeadm version {{< skew prevMinorVersion >}} was used to create or upgrade the node -* The version of kubeadm used for upgrading the node must be at {{< skew prevMinorVersion >}} -or {{< skew latestVersion >}} +* kubeadm version {{< skew currentVersionAddMinor -1 >}} was used to create or upgrade the node +* The version of kubeadm used for upgrading the node must be at {{< skew currentVersionAddMinor -1 >}} +or {{< skew currentVersion >}} To learn more about the version skew between the different Kubernetes component see the [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/). @@ -598,7 +598,7 @@ Workarounds: kubeadm deb/rpm packages and binaries are built for amd64, arm (32-bit), arm64, ppc64le, and s390x following the [multi-platform -proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multi-platform.md). +proposal](https://git.k8s.io/design-proposals-archive/multi-platform.md). Multiplatform container images for the control plane and addons are also supported since v1.12. diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md index 9281d5cb41..010f0aa94b 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md @@ -45,26 +45,6 @@ may [fail](https://github.com/kubernetes/kubeadm/issues/31). If you have more than one network adapter, and your Kubernetes components are not reachable on the default route, we recommend you add IP route(s) so Kubernetes cluster addresses go via the appropriate adapter. -## Letting iptables see bridged traffic - -Make sure that the `br_netfilter` module is loaded. This can be done by running `lsmod | grep br_netfilter`. To load it explicitly call `sudo modprobe br_netfilter`. - -As a requirement for your Linux Node's iptables to correctly see bridged traffic, you should ensure `net.bridge.bridge-nf-call-iptables` is set to 1 in your `sysctl` config, e.g. - -```bash -cat <}} -kubeadm contains all the necessary crytographic machinery to generate +kubeadm contains all the necessary cryptographic machinery to generate the certificates described below; no other cryptographic tooling is required for this example. {{< /note >}} diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md index 147ccd000e..de8ddce39e 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md @@ -114,7 +114,7 @@ sudo kubeadm reset A possible solution is to restart the container runtime and then re-run `kubeadm reset`. You can also use `crictl` to debug the state of the container runtime. See -[Debugging Kubernetes nodes with crictl](/docs/tasks/debug-application-cluster/crictl/). +[Debugging Kubernetes nodes with crictl](/docs/tasks/debug/debug-cluster/crictl/). ## Pods in `RunContainerError`, `CrashLoopBackOff` or `Error` state diff --git a/content/en/docs/setup/production-environment/tools/kubespray.md b/content/en/docs/setup/production-environment/tools/kubespray.md index c4a8c8f59e..fd594b92f8 100644 --- a/content/en/docs/setup/production-environment/tools/kubespray.md +++ b/content/en/docs/setup/production-environment/tools/kubespray.md @@ -39,7 +39,7 @@ Provision servers with the following [requirements](https://github.com/kubernete * The target servers are configured to allow **IPv4 forwarding** * **Your ssh key must be copied** to all the servers in your inventory * **Firewalls are not managed by kubespray**. You'll need to implement appropriate rules as needed. You should disable your firewall in order to avoid any issues during deployment -* If kubespray is ran from a non-root user account, correct privilege escalation method should be configured in the target servers and the `ansible_become` flag or command parameters `--become` or `-b` should be specified +* If kubespray is run from a non-root user account, correct privilege escalation method should be configured in the target servers and the `ansible_become` flag or command parameters `--become` or `-b` should be specified Kubespray provides the following utilities to help provision your environment: diff --git a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md deleted file mode 100644 index 9c6ab896d7..0000000000 --- a/content/en/docs/setup/production-environment/windows/intro-windows-in-kubernetes.md +++ /dev/null @@ -1,917 +0,0 @@ ---- -reviewers: -- jayunit100 -- jsturtevant -- marosset -- perithompson -title: Windows containers in Kubernetes -content_type: concept -weight: 65 ---- - - - -Windows applications constitute a large portion of the services and applications that -run in many organizations. [Windows containers](https://aka.ms/windowscontainers) -provide a way to encapsulate processes and package dependencies, making it easier -to use DevOps practices and follow cloud native patterns for Windows applications. - -Organizations with investments in Windows-based applications and Linux-based -applications don't have to look for separate orchestrators to manage their workloads, -leading to increased operational efficiencies across their deployments, regardless -of operating system. - - - -## Windows nodes in Kubernetes - -To enable the orchestration of Windows containers in Kubernetes, include Windows nodes -in your existing Linux cluster. Scheduling Windows containers in -{{< glossary_tooltip text="Pods" term_id="pod" >}} on Kubernetes is similar to -scheduling Linux-based containers. - -In order to run Windows containers, your Kubernetes cluster must include -multiple operating systems. -While you can only run the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} on Linux, you can deploy worker nodes running either Windows or Linux depending on your workload needs. - -Windows {{< glossary_tooltip text="nodes" term_id="node" >}} are -[supported](#windows-os-version-support) provided that the operating system is -Windows Server 2019. - -This document uses the term *Windows containers* to mean Windows containers with -process isolation. Kubernetes does not support running Windows containers with -[Hyper-V isolation](https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container). - -## Compatibility and limitations {#limitations} - -Some node features are only available if you use a specific -[container runtime](#container-runtime); others are not available on Windows nodes, -including: - -* HugePages: not supported for Windows containers -* Privileged containers: not supported for Windows containers -* TerminationGracePeriod: requires containerD - -Not all features of shared namespaces are supported. See [API compatibility](#api) -for more details. - -See [Windows OS version compatibility](#windows-os-version-support) for details on -the Windows versions that Kubernetes is tested against. - -From an API and kubectl perspective, Windows containers behave in much the same -way as Linux-based containers. However, there are some notable differences in key -functionality which are outlined in this section. - -### Comparison with Linux {#compatibility-linux-similarities} - -Key Kubernetes elements work the same way in Windows as they do in Linux. This -section refers to several key workload enablers and how they map to Windows. - -* [Pods](/docs/concepts/workloads/pods/) - - A Pod is the basic building block of Kubernetes–the smallest and simplest unit in - the Kubernetes object model that you create or deploy. You may not deploy Windows and - Linux containers in the same Pod. All containers in a Pod are scheduled onto a single - Node where each Node represents a specific platform and architecture. The following - Pod capabilities, properties and events are supported with Windows containers: - - * Single or multiple containers per Pod with process isolation and volume sharing - * Pod `status` fields - * Readiness and Liveness probes - * postStart & preStop container lifecycle events - * ConfigMap, Secrets: as environment variables or volumes - * `emptyDir` volumes - * Named pipe host mounts - * Resource limits - * OS field: - - The `.spec.os.name` field should be set to `windows` to indicate that the current Pod uses Windows containers. - The `IdentifyPodOS` feature gate needs to be enabled for this field to be recognized and used by control plane - components and kubelet. - - {{< note >}} - Starting from 1.24, the `IdentifyPodOS` feature gate is in Beta stage and defaults to be enabled. - {{< /note >}} - - If the `IdentifyPodOS` feature gate is enabled and you set the `.spec.os.name` field to `windows`, - you must not set the following fields in the `.spec` of that Pod: - - * `spec.hostPID` - * `spec.hostIPC` - * `spec.securityContext.seLinuxOptions` - * `spec.securityContext.seccompProfile` - * `spec.securityContext.fsGroup` - * `spec.securityContext.fsGroupChangePolicy` - * `spec.securityContext.sysctls` - * `spec.shareProcessNamespace` - * `spec.securityContext.runAsUser` - * `spec.securityContext.runAsGroup` - * `spec.securityContext.supplementalGroups` - * `spec.containers[*].securityContext.seLinuxOptions` - * `spec.containers[*].securityContext.seccompProfile` - * `spec.containers[*].securityContext.capabilities` - * `spec.containers[*].securityContext.readOnlyRootFilesystem` - * `spec.containers[*].securityContext.privileged` - * `spec.containers[*].securityContext.allowPrivilegeEscalation` - * `spec.containers[*].securityContext.procMount` - * `spec.containers[*].securityContext.runAsUser` - * `spec.containers[*].securityContext.runAsGroup` - - In the above list, wildcards (`*`) indicate all elements in a list. - For example, `spec.containers[*].securityContext` refers to the SecurityContext object - for all containers. If any of these fields is specified, the Pod will - not be admited by the API server. - -* [Workload resources](/docs/concepts/workloads/controllers/) including: - * ReplicaSet - * Deployments - * StatefulSets - * DaemonSet - * Job - * CronJob - * ReplicationController -* {{< glossary_tooltip text="Services" term_id="service" >}} - See [Load balancing and Services](#load-balancing-and-services) for more details. - -Pods, workload resources, and Services are critical elements to managing Windows -workloads on Kubernetes. However, on their own they are not enough to enable -the proper lifecycle management of Windows workloads in a dynamic cloud native -environment. Kubernetes also supports: - -* `kubectl exec` -* Pod and container metrics -* {{< glossary_tooltip text="Horizontal pod autoscaling" term_id="horizontal-pod-autoscaler" >}} -* {{< glossary_tooltip text="Resource quotas" term_id="resource-quota" >}} -* Scheduler preemption - - -### Networking on Windows nodes {#compatibility-networking} - -Networking for Windows containers is exposed through -[CNI plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/). -Windows containers function similarly to virtual machines in regards to -networking. Each container has a virtual network adapter (vNIC) which is connected -to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the -Host Compute Service (HCS) work together to create containers and attach container -vNICs to networks. HCS is responsible for the management of containers whereas HNS -is responsible for the management of networking resources such as: - -* Virtual networks (including creation of vSwitches) -* Endpoints / vNICs -* Namespaces -* Policies including packet encapsulations, load-balancing rules, ACLs, and NAT rules. - -#### Container networking {#networking} - -The Windows HNS and vSwitch implement namespacing and can -create virtual NICs as needed for a pod or container. However, many configurations such -as DNS, routes, and metrics are stored in the Windows registry database rather than as -files inside `/etc`, which is how Linux stores those configurations. The Windows registry for the container -is separate from that of the host, so concepts like mapping `/etc/resolv.conf` from -the host into a container don't have the same effect they would on Linux. These must -be configured using Windows APIs run in the context of that container. Therefore -CNI implementations need to call the HNS instead of relying on file mappings to pass -network details into the pod or container. - -The following networking functionality is _not_ supported on Windows nodes: - -* Host networking mode -* Local NodePort access from the node itself (works for other nodes or external clients) -* More than 64 backend pods (or unique destination addresses) for a single Service -* IPv6 communication between Windows pods connected to overlay networks -* Local Traffic Policy in non-DSR mode -* Outbound communication using the ICMP protocol via the `win-overlay`, `win-bridge`, or using the Azure-CNI plugin.\ - Specifically, the Windows data plane ([VFP](https://www.microsoft.com/en-us/research/project/azure-virtual-filtering-platform/)) doesn't support ICMP packet transpositions, and this means: - * ICMP packets directed to destinations within the same network (such as pod to pod communication via ping) work as expected and without any limitations; - * TCP/UDP packets work as expected and without any limitations; - * ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping) cannot be transposed and thus will not be routed back to their source; - * Since TCP/UDP packets can still be transposed, you can substitute `ping ` with `curl ` to get some debugging insight into connectivity with the outside world. - -Overlay networking support in kube-proxy is a beta feature. In addition, it requires -[KB4482887](https://support.microsoft.com/en-us/help/4482887/windows-10-update-kb4482887) -to be installed on Windows Server 2019. - -#### Network modes - -Windows supports five different networking drivers/modes: L2bridge, L2tunnel, -Overlay (beta), Transparent, and NAT. In a heterogeneous cluster with Windows and Linux -worker nodes, you need to select a networking solution that is compatible on both -Windows and Linux. The following out-of-tree plugins are supported on Windows, -with recommendations on when to use each CNI: - -| Network Driver | Description | Container Packet Modifications | Network Plugins | Network Plugin Characteristics | -| -------------- | ----------- | ------------------------------ | --------------- | ------------------------------ | -| L2bridge | Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress. | MAC is rewritten to host MAC, IP may be rewritten to host IP using HNS OutboundNAT policy. | [win-bridge](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-bridge), [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), Flannel host-gateway uses win-bridge | win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires user-defined routes (UDR) for inter-node connectivity. | -| L2Tunnel | This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied. | MAC rewritten, IP visible on the underlay network | [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) | Azure-CNI allows integration of containers with Azure vNET, and allows them to leverage the set of capabilities that [Azure Virtual Network provides](https://azure.microsoft.com/en-us/services/virtual-network/). For example, securely connect to Azure services or use Azure NSGs. See [azure-cni for some examples](https://docs.microsoft.com/en-us/azure/aks/concepts-network#azure-cni-advanced-networking) | -| Overlay (Overlay networking for Windows in Kubernetes is in *alpha* stage) | Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation. | Encapsulated with an outer header. | [win-overlay](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-overlay), Flannel VXLAN (uses win-overlay) | win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option requires [KB4489899](https://support.microsoft.com/help/4489899) on Windows Server 2019. | -| Transparent (special use case for [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes)) | Requires an external vSwitch. Containers are attached to an external vSwitch which enables intra-pod communication via logical networks (logical switches and routers). | Packet is encapsulated either via [GENEVE](https://datatracker.ietf.org/doc/draft-gross-geneve/) or [STT](https://datatracker.ietf.org/doc/draft-davie-stt/) tunneling to reach pods which are not on the same host.
    Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller.
    NAT is done for north-south communication. | [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes) | [Deploy via ansible](https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib). Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh. | -| NAT (*not used in Kubernetes*) | Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called [WinNAT](https://techcommunity.microsoft.com/t5/virtualization/windows-nat-winnat-capabilities-and-limitations/ba-p/382303) | MAC and IP is rewritten to host MAC/IP. | [nat](https://github.com/Microsoft/windows-container-networking/tree/master/plugins/nat) | Included here for completeness | - -As outlined above, the [Flannel](https://github.com/coreos/flannel) -CNI [meta plugin](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel) -is also [supported](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel#windows-support-experimental) on Windows via the -[VXLAN network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) (**alpha support** ; delegates to win-overlay) -and [host-gateway network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) (stable support; delegates to win-bridge). - -This plugin supports delegating to one of the reference CNI plugins (win-overlay, -win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for -automatic node subnet lease assignment and HNS network creation. This plugin reads -in its own configuration file (cni.conf), and aggregates it with the environment -variables from the FlannelD generated subnet.env file. It then delegates to one of -the reference CNI plugins for network plumbing, and sends the correct configuration -containing the node-assigned subnet to the IPAM plugin (for example: `host-local`). - -For Node, Pod, and Service objects, the following network flows are supported for -TCP/UDP traffic: - -* Pod → Pod (IP) -* Pod → Pod (Name) -* Pod → Service (Cluster IP) -* Pod → Service (PQDN, but only if there are no ".") -* Pod → Service (FQDN) -* Pod → external (IP) -* Pod → external (DNS) -* Node → Pod -* Pod → Node - -#### CNI plugin limitations - -* Windows reference network plugins win-bridge and win-overlay do not implement - [CNI spec](https://github.com/containernetworking/cni/blob/master/SPEC.md) v0.4.0, - due to a missing `CHECK` implementation. -* The Flannel VXLAN CNI plugin has the following limitations on Windows: - -1. Node-pod connectivity isn't possible by design. It's only possible for local pods with Flannel v0.12.0 (or higher). -2. Flannel is restricted to using VNI 4096 and UDP port 4789. See the official - [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) - backend docs for more details on these parameters. - -#### IP address management (IPAM) {#ipam} - -The following IPAM options are supported on Windows: - -* [host-local](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/host-local) -* HNS IPAM (Inbox platform IPAM, this is a fallback when no IPAM is set) -* [azure-vnet-ipam](https://github.com/Azure/azure-container-networking/blob/master/docs/ipam.md) (for azure-cni only) - -#### Load balancing and Services - -A Kubernetes {{< glossary_tooltip text="Service" term_id="service" >}} is an abstraction -that defines a logical set of Pods and a means to access them over a network. -In a cluster that includes Windows nodes, you can use the following types of Service: - - * `NodePort` - * `ClusterIP` - * `LoadBalancer` - * `ExternalName` - -{{< warning >}} -There are known issue with NodePort services on overlay networking, if the target destination node is running Windows Server 2022. -To avoid the issue entirely, you can configure the service with `externalTrafficPolicy: Local`. - -There are known issues with pod to pod connectivity on l2bridge network on Windows Server 2022 with KB5005619 or higher installed. -To workaround the issue and restore pod-pod connectivity, you can disable the WinDSR feature in kube-proxy. - -These issues require OS fixes. -Please follow https://github.com/microsoft/Windows-Containers/issues/204 for updates. -{{< /warning >}} - -Windows container networking differs in some important ways from Linux networking. -The [Microsoft documentation for Windows Container Networking](https://docs.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture) provides -additional details and background. - -On Windows, you can use the following settings to configure Services and load -balancing behavior: - -{{< table caption="Windows Service Settings" >}} -| Feature | Description | Supported Kubernetes version | Supported Windows OS build | How to enable | -| ------- | ----------- | ----------------------------- | -------------------------- | ------------- | -| Session affinity | Ensures that connections from a particular client are passed to the same Pod each time. | v1.20+ | [Windows Server vNext Insider Preview Build 19551](https://blogs.windows.com/windowsexperience/2020/01/28/announcing-windows-server-vnext-insider-preview-build-19551/) (or higher) | Set `service.spec.sessionAffinity` to "ClientIP" | -| Direct Server Return (DSR) | Load balancing mode where the IP address fixups and the LBNAT occurs at the container vSwitch port directly; service traffic arrives with the source IP set as the originating pod IP. | v1.20+ | Windows Server 2019 | Set the following flags in kube-proxy: `--feature-gates="WinDSR=true" --enable-dsr=true` | -| Preserve-Destination | Skips DNAT of service traffic, thereby preserving the virtual IP of the target service in packets reaching the backend Pod. Also disables node-node forwarding. | v1.20+ | Windows Server, version 1903 (or higher) | Set `"preserve-destination": "true"` in service annotations and enable DSR in kube-proxy. | -| IPv4/IPv6 dual-stack networking | Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster | v1.19+ | Windows Server, version 2019 | See [IPv4/IPv6 dual-stack](#ipv4ipv6-dual-stack) | -| Client IP preservation | Ensures that source IP of incoming ingress traffic gets preserved. Also disables node-node forwarding. | v1.20+ | Windows Server, version 2019 | Set `service.spec.externalTrafficPolicy` to "Local" and enable DSR in kube-proxy | -{{< /table >}} - -##### Session affinity - -Setting the maximum session sticky time for Windows services using -`service.spec.sessionAffinityConfig.clientIP.timeoutSeconds` is not supported. - -#### DNS {#dns-limitations} - -* ClusterFirstWithHostNet is not supported for DNS. Windows treats all names with a - `.` as a FQDN and skips FQDN resolution -* On Linux, you have a DNS suffix list, which is used when trying to resolve PQDNs. On - Windows, you can only have 1 DNS suffix, which is the DNS suffix associated with that - pod's namespace (mydns.svc.cluster.local for example). Windows can resolve FQDNs - and services or names resolvable with just that suffix. For example, a pod spawned - in the default namespace, will have the DNS suffix **default.svc.cluster.local**. - Inside a Windows pod, you can resolve both **kubernetes.default.svc.cluster.local** - and **kubernetes**, but not the in-betweens, like **kubernetes.default** or - **kubernetes.default.svc**. -* On Windows, there are multiple DNS resolvers that can be used. As these come with - slightly different behaviors, using the `Resolve-DNSName` utility for name query - resolutions is recommended. - -#### IPv6 networking - -Kubernetes on Windows does not support single-stack "IPv6-only" networking. However, -dual-stack IPv4/IPv6 networking for pods and nodes with single-family services -is supported. - -You can use IPv4/IPv6 dual-stack networking with `l2bridge` networks. See [configure IPv4/IPv6 dual stack](/docs/concepts/services-networking/dual-stack#configure-ipv4-ipv6-dual-stack) for more details. - -{{< note >}} -Overlay (VXLAN) networks on Windows do not support dual-stack networking. -{{< /note >}} - -### Persistent storage {#compatibility-storage} - -Windows has a layered filesystem driver to mount container layers and create a copy -filesystem based on NTFS. All file paths in the container are resolved only within -the context of that container. - -* With Docker, volume mounts can only target a directory in the container, and not - an individual file. This limitation does not exist with CRI-containerD runtime. -* Volume mounts cannot project files or directories back to the host filesystem. -* Read-only filesystems are not supported because write access is always required - for the Windows registry and SAM database. However, read-only volumes are supported. -* Volume user-masks and permissions are not available. Because the SAM is not shared - between the host & container, there's no mapping between them. All permissions are - resolved within the context of the container. - -As a result, the following storage functionality is not supported on Windows nodes: - -* Volume subpath mounts: only the entire volume can be mounted in a Windows container -* Subpath volume mounting for Secrets -* Host mount projection -* Read-only root filesystem (mapped volumes still support `readOnly`) -* Block device mapping -* Memory as the storage medium (for example, `emptyDir.medium` set to `Memory`) -* File system features like uid/gid; per-user Linux filesystem permissions -* DefaultMode (due to UID/GID dependency) -* NFS based storage/volume support -* Expanding the mounted volume (resizefs) - -Kubernetes {{< glossary_tooltip text="volumes" term_id="volume" >}} enable complex -applications, with data persistence and Pod volume sharing requirements, to be deployed -on Kubernetes. Management of persistent volumes associated with a specific storage -back-end or protocol includes actions such as provisioning/de-provisioning/resizing -of volumes, attaching/detaching a volume to/from a Kubernetes node and -mounting/dismounting a volume to/from individual containers in a pod that needs to -persist data. - -The code implementing these volume management actions for a specific storage back-end -or protocol is shipped in the form of a Kubernetes volume -[plugin](/docs/concepts/storage/volumes/#types-of-volumes). -The following broad classes of Kubernetes volume plugins are supported on Windows: - -##### In-tree volume plugins - -Code associated with in-tree volume plugins ship as part of the core Kubernetes code -base. Deployment of in-tree volume plugins do not require installation of additional -scripts or deployment of separate containerized plugin components. These plugins can -handle provisioning/de-provisioning and resizing of volumes in the storage backend, -attaching/detaching of volumes to/from a Kubernetes node and mounting/dismounting a -volume to/from individual containers in a pod. The following in-tree plugins support -persistent storage on Windows nodes: - -* [`awsElasticBlockStore`](/docs/concepts/storage/volumes/#awselasticblockstore) -* [`azureDisk`](/docs/concepts/storage/volumes/#azuredisk) -* [`azureFile`](/docs/concepts/storage/volumes/#azurefile) -* [`gcePersistentDisk`](/docs/concepts/storage/volumes/#gcepersistentdisk) -* [`vsphereVolume`](/docs/concepts/storage/volumes/#vspherevolume) - -#### FlexVolume plugins - -Code associated with [FlexVolume](/docs/concepts/storage/volumes/#flexVolume) -plugins ship as out-of-tree scripts or binaries that need to be deployed directly -on the host. FlexVolume plugins handle attaching/detaching of volumes to/from a -Kubernetes node and mounting/dismounting a volume to/from individual containers -in a pod. Provisioning/De-provisioning of persistent volumes associated -with FlexVolume plugins may be handled through an external provisioner that -is typically separate from the FlexVolume plugins. The following FlexVolume -[plugins](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows), -deployed as PowerShell scripts on the host, support Windows nodes: - -* [SMB](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~smb.cmd) -* [iSCSI](https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows/plugins/microsoft.com~iscsi.cmd) - -#### CSI plugins - -{{< feature-state for_k8s_version="v1.19" state="beta" >}} - -Code associated with {{< glossary_tooltip text="CSI" term_id="csi" >}} plugins ship -as out-of-tree scripts and binaries that are typically distributed as container -images and deployed using standard Kubernetes constructs like DaemonSets and -StatefulSets. -CSI plugins handle a wide range of volume management actions in Kubernetes: -provisioning/de-provisioning/resizing of volumes, attaching/detaching of volumes -to/from a Kubernetes node and mounting/dismounting a volume to/from individual -containers in a pod, backup/restore of persistent data using snapshots and cloning. -CSI plugins typically consist of node plugins (that run on each node as a DaemonSet) -and controller plugins. - -CSI node plugins (especially those associated with persistent volumes exposed as -either block devices or over a shared file-system) need to perform various privileged -operations like scanning of disk devices, mounting of file systems, etc. These -operations differ for each host operating system. For Linux worker nodes, containerized -CSI node plugins are typically deployed as privileged containers. For Windows worker -nodes, privileged operations for containerized CSI node plugins is supported using -[csi-proxy](https://github.com/kubernetes-csi/csi-proxy), a community-managed, -stand-alone binary that needs to be pre-installed on each Windows node. - -For more details, refer to the deployment guide of the CSI plugin you wish to deploy. - -### Command line options for the kubelet {#kubelet-compatibility} - -The behavior of some kubelet command line options behave differently on Windows, as described below: - -* The `--windows-priorityclass` lets you set the scheduling priority of the kubelet process (see [CPU resource management](/docs/concepts/configuration/windows-resource-management/#resource-management-cpu)) -* The `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable) -* Eviction by using `--enforce-node-allocable` is not implemented -* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented -* A kubelet running on a Windows node does not have memory - restrictions. `--kubelet-reserve` and `--system-reserve` do not set limits on - kubelet or processes running on the host. This means kubelet or a process on the host - could cause memory resource starvation outside the node-allocatable and scheduler. -* The `MemoryPressure` Condition is not implemented -* The kubelet does not take OOM eviction actions - -### API compatibility {#api} - -There are no differences in how most of the Kubernetes APIs work for Windows. The -subtleties around what's different come down to differences in the OS and container -runtime. In certain situations, some properties on workload resources were designed -under the assumption that they would be implemented on Linux, and fail to run on Windows. - -At a high level, these OS concepts are different: - -* Identity - Linux uses userID (UID) and groupID (GID) which - are represented as integer types. User and group names - are not canonical - they are just an alias in `/etc/groups` - or `/etc/passwd` back to UID+GID. Windows uses a larger binary - [security identifier](https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/security-identifiers) (SID) - which is stored in the Windows Security Access Manager (SAM) database. This - database is not shared between the host and containers, or between containers. -* File permissions - Windows uses an access control list based on (SIDs), whereas - POSIX systems such as Linux use a bitmask based on object permissions and UID+GID, - plus _optional_ access control lists. -* File paths - the convention on Windows is to use `\` instead of `/`. The Go IO - libraries typically accept both and just make it work, but when you're setting a - path or command line that's interpreted inside a container, `\` may be needed. -* Signals - Windows interactive apps handle termination differently, and can - implement one or more of these: - * A UI thread handles well-defined messages including `WM_CLOSE`. - * Console apps handle Ctrl-C or Ctrl-break using a Control Handler. - * Services register a Service Control Handler function that can accept - `SERVICE_CONTROL_STOP` control codes. - -Container exit codes follow the same convention where 0 is success, and nonzero is failure. -The specific error codes may differ across Windows and Linux. However, exit codes -passed from the Kubernetes components (kubelet, kube-proxy) are unchanged. - -##### Field compatibility for container specifications {#compatibility-v1-pod-spec-containers} - -The following list documents differences between how Pod container specifications -work between Windows and Linux: - -* Huge pages are not implemented in the Windows container - runtime, and are not available. They require [asserting a user - privilege](https://docs.microsoft.com/en-us/windows/desktop/Memory/large-page-support) - that's not configurable for containers. -* `requests.cpu` and `requests.memory` - requests are subtracted - from node available resources, so they can be used to avoid overprovisioning a - node. However, they cannot be used to guarantee resources in an overprovisioned - node. They should be applied to all containers as a best practice if the operator - wants to avoid overprovisioning entirely. -* `securityContext.allowPrivilegeEscalation` - - not possible on Windows; none of the capabilities are hooked up -* `securityContext.capabilities` - - POSIX capabilities are not implemented on Windows -* `securityContext.privileged` - - Windows doesn't support privileged containers -* `securityContext.procMount` - - Windows doesn't have a `/proc` filesystem -* `securityContext.readOnlyRootFilesystem` - - not possible on Windows; write access is required for registry & system - processes to run inside the container -* `securityContext.runAsGroup` - - not possible on Windows as there is no GID support -* `securityContext.runAsNonRoot` - - this setting will prevent containers from running as `ContainerAdministrator` - which is the closest equivalent to a root user on Windows. -* `securityContext.runAsUser` - - use [`runAsUserName`](/docs/tasks/configure-pod-container/configure-runasusername) - instead -* `securityContext.seLinuxOptions` - - not possible on Windows as SELinux is Linux-specific -* `terminationMessagePath` - - this has some limitations in that Windows doesn't support mapping single files. The - default value is `/dev/termination-log`, which does work because it does not - exist on Windows by default. - -##### Field compatibility for Pod specifications {#compatibility-v1-pod} - -The following list documents differences between how Pod specifications work between Windows and Linux: - -* `hostIPC` and `hostpid` - host namespace sharing is not possible on Windows -* `hostNetwork` - There is no Windows OS support to share the host network -* `dnsPolicy` - setting the Pod `dnsPolicy` to `ClusterFirstWithHostNet` is - not supported on Windows because host networking is not provided. Pods always - run with a container network. -* `podSecurityContext` (see below) -* `shareProcessNamespace` - this is a beta feature, and depends on Linux namespaces - which are not implemented on Windows. Windows cannot share process namespaces or - the container's root filesystem. Only the network can be shared. -* `terminationGracePeriodSeconds` - this is not fully implemented in Docker on Windows, - see the [GitHub issue](https://github.com/moby/moby/issues/25982). - The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT, - then Windows waits 5 seconds by default, and finally shuts down - all processes using the normal Windows shutdown behavior. The 5 - second default is actually in the Windows registry - [inside the container](https://github.com/moby/moby/issues/25982#issuecomment-426441183), - so it can be overridden when the container is built. -* `volumeDevices` - this is a beta feature, and is not implemented on Windows. - Windows cannot attach raw block devices to pods. -* `volumes` - * If you define an `emptyDir` volume, you cannot set its volume source to `memory`. -* You cannot enable `mountPropagation` for volume mounts as this is not - supported on Windows. - -##### Field compatibility for Pod security context {#compatibility-v1-pod-spec-containers-securitycontext} - -None of the Pod [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context) fields work on Windows. - -### Node problem detector - -The node problem detector (see -[Monitor Node Health](/docs/tasks/debug/debug-cluster/monitor-node-health/)) -is not compatible with Windows. - -### Pause container - -In a Kubernetes Pod, an infrastructure or “pause” container is first created -to host the container. In Linux, the cgroups and namespaces that make up a pod -need a process to maintain their continued existence; the pause process provides -this. Containers that belong to the same pod, including infrastructure and worker -containers, share a common network endpoint (same IPv4 and / or IPv6 address, same -network port spaces). Kubernetes uses pause containers to allow for worker containers -crashing or restarting without losing any of the networking configuration. - -Kubernetes maintains a multi-architecture image that includes support for Windows. -For Kubernetes v{{< skew currentVersion >}} the recommended pause image is `k8s.gcr.io/pause:3.6`. -The [source code](https://github.com/kubernetes/kubernetes/tree/master/build/pause) -is available on GitHub. - -Microsoft maintains a different multi-architecture image, with Linux and Windows -amd64 support, that you can find as `mcr.microsoft.com/oss/kubernetes/pause:3.6`. -This image is built from the same source as the Kubernetes maintained image but -all of the Windows binaries are [authenticode signed](https://docs.microsoft.com/en-us/windows-hardware/drivers/install/authenticode) by Microsoft. -The Kubernetes project recommends using the Microsoft maintained image if you are -deploying to a production or production-like environment that requires signed -binaries. - -### Container runtimes {#container-runtime} - -You need to install a -{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} -into each node in the cluster so that Pods can run there. - -The following container runtimes work with Windows: - -{{% thirdparty-content %}} - -#### cri-containerd - -{{< feature-state for_k8s_version="v1.20" state="stable" >}} - -You can use {{< glossary_tooltip term_id="containerd" text="ContainerD" >}} 1.4.0+ -as the container runtime for Kubernetes nodes that run Windows. - -Learn how to [install ContainerD on a Windows node](/docs/setup/production-environment/container-runtimes/#install-containerd). - -{{< note >}} -There is a [known limitation](/docs/tasks/configure-pod-container/configure-gmsa/#gmsa-limitations) -when using GMSA with containerd to access Windows network shares, which requires a -kernel patch. -{{< /note >}} - -#### Mirantis Container Runtime {#mcr} - -[Mirantis Container Runtime](https://docs.mirantis.com/mcr/20.10/overview.html) (MCR) is available as a container runtime for all Windows Server 2019 and later versions. - -See [Install MCR on Windows Servers](https://docs.mirantis.com/mcr/20.10/install/mcr-windows.html) for more information. - -## Windows OS version compatibility {#windows-os-version-support} - -On Windows nodes, strict compatibility rules apply where the host OS version must -match the container base image OS version. Only Windows containers with a container -operating system of Windows Server 2019 are fully supported. - -For Kubernetes v{{< skew currentVersion >}}, operating system compatibility for Windows nodes (and Pods) -is as follows: - -Windows Server LTSC release -: Windows Server 2019 -: Windows Server 2022 - -Windows Server SAC release -: Windows Server version 20H2 - -The Kubernetes [version-skew policy](/docs/setup/release/version-skew-policy/) also applies. - -## Getting help and troubleshooting {#troubleshooting} - -Your main source of help for troubleshooting your Kubernetes cluster should start -with the [Troubleshooting](/docs/tasks/debug/debug-cluster/) -page. - -Some additional, Windows-specific troubleshooting help is included -in this section. Logs are an important element of troubleshooting -issues in Kubernetes. Make sure to include them any time you seek -troubleshooting assistance from other contributors. Follow the -instructions in the -SIG Windows [contributing guide on gathering logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs). - -### Node-level troubleshooting {#troubleshooting-node} - -1. How do I know `start.ps1` completed successfully? - - You should see kubelet, kube-proxy, and (if you chose Flannel as your networking - solution) flanneld host-agent processes running on your node, with running logs - being displayed in separate PowerShell windows. In addition to this, your Windows - node should be listed as "Ready" in your Kubernetes cluster. - -1. Can I configure the Kubernetes node processes to run in the background as services? - - The kubelet and kube-proxy are already configured to run as native Windows Services, - offering resiliency by re-starting the services automatically in the event of - failure (for example a process crash). You have two options for configuring these - node components as services. - - 1. As native Windows Services - - You can run the kubelet and kube-proxy as native Windows Services using `sc.exe`. - - ```powershell - # Create the services for kubelet and kube-proxy in two separate commands - sc.exe create binPath= " --service " - - # Please note that if the arguments contain spaces, they must be escaped. - sc.exe create kubelet binPath= "C:\kubelet.exe --service --hostname-override 'minion' " - - # Start the services - Start-Service kubelet - Start-Service kube-proxy - - # Stop the service - Stop-Service kubelet (-Force) - Stop-Service kube-proxy (-Force) - - # Query the service status - Get-Service kubelet - Get-Service kube-proxy - ``` - - 1. Using `nssm.exe` - - You can also always use alternative service managers like - [nssm.exe](https://nssm.cc/) to run these processes (flanneld, - kubelet & kube-proxy) in the background for you. You can use this - [sample script](https://github.com/Microsoft/SDN/tree/master/Kubernetes/flannel/register-svc.ps1), - leveraging nssm.exe to register kubelet, kube-proxy, and flanneld.exe to run - as Windows services in the background. - - ```powershell - register-svc.ps1 -NetworkMode -ManagementIP -ClusterCIDR -KubeDnsServiceIP -LogDir - - # NetworkMode = The network mode l2bridge (flannel host-gw, also the default value) or overlay (flannel vxlan) chosen as a network solution - # ManagementIP = The IP address assigned to the Windows node. You can use ipconfig to find this - # ClusterCIDR = The cluster subnet range. (Default value 10.244.0.0/16) - # KubeDnsServiceIP = The Kubernetes DNS service IP (Default value 10.96.0.10) - # LogDir = The directory where kubelet and kube-proxy logs are redirected into their respective output files (Default value C:\k) - ``` - - If the above referenced script is not suitable, you can manually configure - `nssm.exe` using the following examples. - - ```powershell - # Register flanneld.exe - nssm install flanneld C:\flannel\flanneld.exe - nssm set flanneld AppParameters --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1 - nssm set flanneld AppEnvironmentExtra NODE_NAME= - nssm set flanneld AppDirectory C:\flannel - nssm start flanneld - - # Register kubelet.exe - # Microsoft releases the pause infrastructure container at mcr.microsoft.com/oss/kubernetes/pause:3.6 - nssm install kubelet C:\k\kubelet.exe - nssm set kubelet AppParameters --hostname-override= --v=6 --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:3.6 --resolv-conf="" --allow-privileged=true --enable-debugging-handlers --cluster-dns= --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false --log-dir= --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config - nssm set kubelet AppDirectory C:\k - nssm start kubelet - - # Register kube-proxy.exe (l2bridge / host-gw) - nssm install kube-proxy C:\k\kube-proxy.exe - nssm set kube-proxy AppDirectory c:\k - nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --hostname-override=--kubeconfig=c:\k\config --enable-dsr=false --log-dir= --logtostderr=false - nssm.exe set kube-proxy AppEnvironmentExtra KUBE_NETWORK=cbr0 - nssm set kube-proxy DependOnService kubelet - nssm start kube-proxy - - # Register kube-proxy.exe (overlay / vxlan) - nssm install kube-proxy C:\k\kube-proxy.exe - nssm set kube-proxy AppDirectory c:\k - nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --feature-gates="WinOverlay=true" --hostname-override= --kubeconfig=c:\k\config --network-name=vxlan0 --source-vip= --enable-dsr=false --log-dir= --logtostderr=false - nssm set kube-proxy DependOnService kubelet - nssm start kube-proxy - ``` - - For initial troubleshooting, you can use the following flags in [nssm.exe](https://nssm.cc/) to redirect stdout and stderr to a output file: - - ```powershell - nssm set AppStdout C:\k\mysvc.log - nssm set AppStderr C:\k\mysvc.log - ``` - - For additional details, see [NSSM - the Non-Sucking Service Manager](https://nssm.cc/usage). - -1. My Pods are stuck at "Container Creating" or restarting over and over - - Check that your pause image is compatible with your OS version. The - [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) - assume that both the OS and the containers are version 1803. If you have a later - version of Windows, such as an Insider build, you need to adjust the images - accordingly. See [Pause container](#pause-container) for more details. - -### Network troubleshooting {#troubleshooting-network} - -1. My Windows Pods do not have network connectivity - - If you are using virtual machines, ensure that MAC spoofing is **enabled** on all - the VM network adapter(s). - -1. My Windows Pods cannot ping external resources - - Windows Pods do not have outbound rules programmed for the ICMP protocol. However, - TCP/UDP is supported. When trying to demonstrate connectivity to resources - outside of the cluster, substitute `ping ` with corresponding - `curl ` commands. - - If you are still facing problems, most likely your network configuration in - [cni.conf](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf) - deserves some extra attention. You can always edit this static file. The - configuration update will apply to any new Kubernetes resources. - - One of the Kubernetes networking requirements - (see [Kubernetes model](/docs/concepts/cluster-administration/networking/)) is - for cluster communication to occur without - NAT internally. To honor this requirement, there is an - [ExceptionList](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf#L20) - for all the communication where you do not want outbound NAT to occur. However, - this also means that you need to exclude the external IP you are trying to query - from the `ExceptionList`. Only then will the traffic originating from your Windows - pods be SNAT'ed correctly to receive a response from the outside world. In this - regard, your `ExceptionList` in `cni.conf` should look as follows: - - ```conf - "ExceptionList": [ - "10.244.0.0/16", # Cluster subnet - "10.96.0.0/12", # Service subnet - "10.127.130.0/24" # Management (host) subnet - ] - ``` - -1. My Windows node cannot access `NodePort` type Services - - Local NodePort access from the node itself fails. This is a known - limitation. NodePort access works from other nodes or external clients. - -1. vNICs and HNS endpoints of containers are being deleted - - This issue can be caused when the `hostname-override` parameter is not passed to - [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/). To resolve - it, users need to pass the hostname to kube-proxy as follows: - - ```powershell - C:\k\kube-proxy.exe --hostname-override=$(hostname) - ``` - -1. With flannel, my nodes are having issues after rejoining a cluster - - Whenever a previously deleted node is being re-joined to the cluster, flannelD - tries to assign a new pod subnet to the node. Users should remove the old pod - subnet configuration files in the following paths: - - ```powershell - Remove-Item C:\k\SourceVip.json - Remove-Item C:\k\SourceVipRequest.json - ``` - -1. After launching `start.ps1`, flanneld is stuck in "Waiting for the Network to be created" - - There are numerous reports of this [issue](https://github.com/coreos/flannel/issues/1066); most likely it is a timing issue for when the management IP of the flannel network is set. A workaround is to relaunch `start.ps1` or relaunch it manually as follows: - - ```powershell - [Environment]::SetEnvironmentVariable("NODE_NAME", "") - C:\flannel\flanneld.exe --kubeconfig-file=c:\k\config --iface= --ip-masq=1 --kube-subnet-mgr=1 - ``` - -1. My Windows Pods cannot launch because of missing `/run/flannel/subnet.env` - - This indicates that Flannel didn't launch correctly. You can either try - to restart `flanneld.exe` or you can copy the files over manually from - `/run/flannel/subnet.env` on the Kubernetes master to `C:\run\flannel\subnet.env` - on the Windows worker node and modify the `FLANNEL_SUBNET` row to a different - number. For example, if node subnet 10.244.4.1/24 is desired: - - ```env - FLANNEL_NETWORK=10.244.0.0/16 - FLANNEL_SUBNET=10.244.4.1/24 - FLANNEL_MTU=1500 - FLANNEL_IPMASQ=true - ``` - -1. My Windows node cannot access my services using the service IP - - This is a known limitation of the networking stack on Windows. However, Windows Pods can access the Service IP. - -1. No network adapter is found when starting the kubelet - - The Windows networking stack needs a virtual adapter for Kubernetes networking to work. If the following commands return no results (in an admin shell), virtual network creation — a necessary prerequisite for the kubelet to work — has failed: - - ```powershell - Get-HnsNetwork | ? Name -ieq "cbr0" - Get-NetAdapter | ? Name -Like "vEthernet (Ethernet*" - ``` - - Often it is worthwhile to modify the [InterfaceName](https://github.com/microsoft/SDN/blob/master/Kubernetes/flannel/start.ps1#L7) parameter of the start.ps1 script, in cases where the host's network adapter isn't "Ethernet". Otherwise, consult the output of the `start-kubelet.ps1` script to see if there are errors during virtual network creation. - -1. DNS resolution is not properly working - - Check the DNS limitations for Windows in this [section](#dns-limitations). - -1. `kubectl port-forward` fails with "unable to do port forwarding: wincat not found" - - This was implemented in Kubernetes 1.15 by including `wincat.exe` in the pause infrastructure container `mcr.microsoft.com/oss/kubernetes/pause:3.6`. Be sure to use a supported version of Kubernetes. - If you would like to build your own pause infrastructure container be sure to include [wincat](https://github.com/kubernetes/kubernetes/tree/master/build/pause/windows/wincat). - -1. My Kubernetes installation is failing because my Windows Server node is behind a proxy - - If you are behind a proxy, the following PowerShell environment variables must be defined: - - ```PowerShell - [Environment]::SetEnvironmentVariable("HTTP_PROXY", "http://proxy.example.com:80/", [EnvironmentVariableTarget]::Machine) - [Environment]::SetEnvironmentVariable("HTTPS_PROXY", "http://proxy.example.com:443/", [EnvironmentVariableTarget]::Machine) - ``` - -### Further investigation - -If these steps don't resolve your problem, you can get help running Windows containers on Windows nodes in Kubernetes through: - -* StackOverflow [Windows Server Container](https://stackoverflow.com/questions/tagged/windows-server-container) topic -* Kubernetes Official Forum [discuss.kubernetes.io](https://discuss.kubernetes.io/) -* Kubernetes Slack [#SIG-Windows Channel](https://kubernetes.slack.com/messages/sig-windows) - -### Reporting issues and feature requests - -If you have what looks like a bug, or you would like to -make a feature request, please use the -[GitHub issue tracking system](https://github.com/kubernetes/kubernetes/issues). -You can open issues on -[GitHub](https://github.com/kubernetes/kubernetes/issues/new/choose) and assign -them to SIG-Windows. You should first search the list of issues in case it was -reported previously and comment with your experience on the issue and add additional -logs. SIG-Windows Slack is also a great avenue to get some initial support and -troubleshooting ideas prior to creating a ticket. - -If filing a bug, please include detailed information about how to reproduce the problem, such as: - -* Kubernetes version: output from `kubectl version` -* Environment details: Cloud provider, OS distro, networking choice and configuration, and Docker version -* Detailed steps to reproduce the problem -* [Relevant logs](https://github.com/kubernetes/community/blob/master/sig-windows/CONTRIBUTING.md#gathering-logs) - -It helps if you tag the issue as **sig/windows**, by commenting on the issue with `/sig windows`. This helps to bring -the issue to a SIG Windows member's attention - - -## {{% heading "whatsnext" %}} - -### Deployment tools - -The kubeadm tool helps you to deploy a Kubernetes cluster, providing the control -plane to manage the cluster it, and nodes to run your workloads. -[Adding Windows nodes](/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/) -explains how to deploy Windows nodes to your cluster using kubeadm. - -The Kubernetes [cluster API](https://cluster-api.sigs.k8s.io/) project also provides means to automate deployment of Windows nodes. - -### Windows distribution channels - -For a detailed explanation of Windows distribution channels see the [Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). - -Information on the different Windows Server servicing channels -including their support models can be found at -[Windows Server servicing channels](https://docs.microsoft.com/en-us/windows-server/get-started/servicing-channels-comparison). diff --git a/content/en/docs/tasks/access-application-cluster/access-cluster-services.md b/content/en/docs/tasks/access-application-cluster/access-cluster-services.md index 262071094c..456662692e 100644 --- a/content/en/docs/tasks/access-application-cluster/access-cluster-services.md +++ b/content/en/docs/tasks/access-application-cluster/access-cluster-services.md @@ -64,17 +64,17 @@ kubectl cluster-info The output is similar to this: ``` -Kubernetes master is running at https://104.197.5.247 -elasticsearch-logging is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy -kibana-logging is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/kibana-logging/proxy -kube-dns is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/kube-dns/proxy -grafana is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy -heapster is running at https://104.197.5.247/api/v1/namespaces/kube-system/services/monitoring-heapster/proxy +Kubernetes master is running at https://192.0.2.1 +elasticsearch-logging is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy +kibana-logging is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/kibana-logging/proxy +kube-dns is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/kube-dns/proxy +grafana is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy +heapster is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/monitoring-heapster/proxy ``` This shows the proxy-verb URL for accessing each service. For example, this cluster has cluster-level logging enabled (using Elasticsearch), which can be reached -at `https://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/` if suitable credentials are passed, or through a kubectl proxy at, for example: +at `https://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/` if suitable credentials are passed, or through a kubectl proxy at, for example: `http://localhost:8080/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/`. {{< note >}} @@ -104,13 +104,13 @@ The supported formats for the `` segment of the URL are: * To access the Elasticsearch service endpoint `_search?q=user:kimchy`, you would use: ``` - http://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_search?q=user:kimchy + http://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_search?q=user:kimchy ``` * To access the Elasticsearch cluster health information `_cluster/health?pretty=true`, you would use: ``` - https://104.197.5.247/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_cluster/health?pretty=true + https://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_cluster/health?pretty=true ``` The health information is similar to this: @@ -133,7 +133,7 @@ The supported formats for the `` segment of the URL are: * To access the *https* Elasticsearch service health information `_cluster/health?pretty=true`, you would use: ``` - https://104.197.5.247/api/v1/namespaces/kube-system/services/https:elasticsearch-logging/proxy/_cluster/health?pretty=true + https://192.0.2.1/api/v1/namespaces/kube-system/services/https:elasticsearch-logging:/proxy/_cluster/health?pretty=true ``` #### Using web browsers to access services running on the cluster diff --git a/content/en/docs/tasks/access-application-cluster/access-cluster.md b/content/en/docs/tasks/access-application-cluster/access-cluster.md index aae96d3e96..f20fe407e8 100644 --- a/content/en/docs/tasks/access-application-cluster/access-cluster.md +++ b/content/en/docs/tasks/access-application-cluster/access-cluster.md @@ -233,7 +233,7 @@ There are several different proxies you may encounter when using Kubernetes: - locates apiserver - adds authentication headers -1. The [apiserver proxy](#discovering-builtin-services): +1. The [apiserver proxy](/docs/tasks/access-application-cluster/access-cluster-services/#discovering-builtin-services): - is a bastion built into the apiserver - connects a user outside of the cluster to cluster IPs which otherwise might not be reachable diff --git a/content/en/docs/tasks/access-application-cluster/configure-access-multiple-clusters.md b/content/en/docs/tasks/access-application-cluster/configure-access-multiple-clusters.md index 8b79d7042f..0c3d05d62f 100644 --- a/content/en/docs/tasks/access-application-cluster/configure-access-multiple-clusters.md +++ b/content/en/docs/tasks/access-application-cluster/configure-access-multiple-clusters.md @@ -22,7 +22,8 @@ It does not mean that there is a file named `kubeconfig`. {{< warning >}} -Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. +Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig +file could result in malicious code execution or file exposure. If you must use an untrusted kubeconfig file, inspect it carefully first, much as you would a shell script. {{< /warning>}} @@ -50,7 +51,7 @@ to the scratch cluster requires authentication by username and password. Create a directory named `config-exercise`. In your `config-exercise` directory, create a file named `config-demo` with this content: -```shell +```yaml apiVersion: v1 kind: Config preferences: {} @@ -115,7 +116,7 @@ kubectl config --kubeconfig=config-demo view The output shows the two clusters, two users, and three contexts: -```shell +```yaml apiVersion: v1 clusters: - cluster: @@ -271,7 +272,7 @@ For example: ### Linux ```shell -export KUBECONFIG_SAVED=$KUBECONFIG +export KUBECONFIG_SAVED="$KUBECONFIG" ``` ### Windows PowerShell @@ -290,7 +291,7 @@ Temporarily append two paths to your `KUBECONFIG` environment variable. For exam ### Linux ```shell -export KUBECONFIG=$KUBECONFIG:config-demo:config-demo-2 +export KUBECONFIG="${KUBECONFIG}:config-demo:config-demo-2" ``` ### Windows PowerShell @@ -356,7 +357,7 @@ For example: ### Linux ```shell -export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config +export KUBECONFIG="${KUBECONFIG}:${HOME}/.kube/config" ``` ### Windows Powershell @@ -379,7 +380,7 @@ Return your `KUBECONFIG` environment variable to its original value. For example ### Linux ```shell -export KUBECONFIG=$KUBECONFIG_SAVED +export KUBECONFIG="$KUBECONFIG_SAVED" ``` ### Windows PowerShell diff --git a/content/en/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md b/content/en/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md index ba8f7b1244..3b2648f943 100644 --- a/content/en/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md +++ b/content/en/docs/tasks/access-application-cluster/port-forward-access-application-cluster.md @@ -11,180 +11,169 @@ This page shows how to use `kubectl port-forward` to connect to a MongoDB server running in a Kubernetes cluster. This type of connection can be useful for database debugging. - - - ## {{% heading "prerequisites" %}} - * {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} - * Install [MongoDB Shell](https://www.mongodb.com/try/download/shell). - - - ## Creating MongoDB deployment and service 1. Create a Deployment that runs MongoDB: - ```shell - kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-deployment.yaml - ``` + ```shell + kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-deployment.yaml + ``` - The output of a successful command verifies that the deployment was created: + The output of a successful command verifies that the deployment was created: - ``` - deployment.apps/mongo created - ``` + ``` + deployment.apps/mongo created + ``` - View the pod status to check that it is ready: + View the pod status to check that it is ready: - ```shell - kubectl get pods - ``` + ```shell + kubectl get pods + ``` - The output displays the pod created: + The output displays the pod created: - ``` - NAME READY STATUS RESTARTS AGE - mongo-75f59d57f4-4nd6q 1/1 Running 0 2m4s - ``` + ``` + NAME READY STATUS RESTARTS AGE + mongo-75f59d57f4-4nd6q 1/1 Running 0 2m4s + ``` - View the Deployment's status: + View the Deployment's status: - ```shell - kubectl get deployment - ``` + ```shell + kubectl get deployment + ``` - The output displays that the Deployment was created: + The output displays that the Deployment was created: - ``` - NAME READY UP-TO-DATE AVAILABLE AGE - mongo 1/1 1 1 2m21s - ``` + ``` + NAME READY UP-TO-DATE AVAILABLE AGE + mongo 1/1 1 1 2m21s + ``` - The Deployment automatically manages a ReplicaSet. - View the ReplicaSet status using: + The Deployment automatically manages a ReplicaSet. + View the ReplicaSet status using: - ```shell - kubectl get replicaset - ``` + ```shell + kubectl get replicaset + ``` - The output displays that the ReplicaSet was created: - - ``` - NAME DESIRED CURRENT READY AGE - mongo-75f59d57f4 1 1 1 3m12s - ``` + The output displays that the ReplicaSet was created: + ``` + NAME DESIRED CURRENT READY AGE + mongo-75f59d57f4 1 1 1 3m12s + ``` 2. Create a Service to expose MongoDB on the network: - ```shell - kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-service.yaml - ``` + ```shell + kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-service.yaml + ``` - The output of a successful command verifies that the Service was created: + The output of a successful command verifies that the Service was created: - ``` - service/mongo created - ``` + ``` + service/mongo created + ``` - Check the Service created: + Check the Service created: - ```shell - kubectl get service mongo - ``` + ```shell + kubectl get service mongo + ``` - The output displays the service created: + The output displays the service created: - ``` - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - mongo ClusterIP 10.96.41.183 27017/TCP 11s - ``` + ``` + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + mongo ClusterIP 10.96.41.183 27017/TCP 11s + ``` 3. Verify that the MongoDB server is running in the Pod, and listening on port 27017: - ```shell - # Change mongo-75f59d57f4-4nd6q to the name of the Pod - kubectl get pod mongo-75f59d57f4-4nd6q --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}' - ``` + ```shell + # Change mongo-75f59d57f4-4nd6q to the name of the Pod + kubectl get pod mongo-75f59d57f4-4nd6q --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}' + ``` - The output displays the port for MongoDB in that Pod: + The output displays the port for MongoDB in that Pod: - ``` - 27017 - ``` + ``` + 27017 + ``` - (this is the TCP port allocated to MongoDB on the internet). + 27017 is the TCP port allocated to MongoDB on the internet. ## Forward a local port to a port on the Pod -1. `kubectl port-forward` allows using resource name, such as a pod name, to select a matching pod to port forward to. +1. `kubectl port-forward` allows using resource name, such as a pod name, to select a matching pod to port forward to. - ```shell - # Change mongo-75f59d57f4-4nd6q to the name of the Pod - kubectl port-forward mongo-75f59d57f4-4nd6q 28015:27017 - ``` + ```shell + # Change mongo-75f59d57f4-4nd6q to the name of the Pod + kubectl port-forward mongo-75f59d57f4-4nd6q 28015:27017 + ``` - which is the same as + which is the same as - ```shell - kubectl port-forward pods/mongo-75f59d57f4-4nd6q 28015:27017 - ``` + ```shell + kubectl port-forward pods/mongo-75f59d57f4-4nd6q 28015:27017 + ``` - or + or - ```shell - kubectl port-forward deployment/mongo 28015:27017 - ``` + ```shell + kubectl port-forward deployment/mongo 28015:27017 + ``` - or + or - ```shell - kubectl port-forward replicaset/mongo-75f59d57f4 28015:27017 - ``` + ```shell + kubectl port-forward replicaset/mongo-75f59d57f4 28015:27017 + ``` - or + or - ```shell - kubectl port-forward service/mongo 28015:27017 - ``` + ```shell + kubectl port-forward service/mongo 28015:27017 + ``` - Any of the above commands works. The output is similar to this: + Any of the above commands works. The output is similar to this: - ``` - Forwarding from 127.0.0.1:28015 -> 27017 - Forwarding from [::1]:28015 -> 27017 - ``` + ``` + Forwarding from 127.0.0.1:28015 -> 27017 + Forwarding from [::1]:28015 -> 27017 + ``` -{{< note >}} + {{< note >}} + `kubectl port-forward` does not return. To continue with the exercises, you will need to open another terminal. + {{< /note >}} -`kubectl port-forward` does not return. To continue with the exercises, you will need to open another terminal. +2. Start the MongoDB command line interface: -{{< /note >}} + ```shell + mongosh --port 28015 + ``` -2. Start the MongoDB command line interface: +3. At the MongoDB command line prompt, enter the `ping` command: - ```shell - mongosh --port 28015 - ``` + ``` + db.runCommand( { ping: 1 } ) + ``` -3. At the MongoDB command line prompt, enter the `ping` command: + A successful ping request returns: - ``` - db.runCommand( { ping: 1 } ) - ``` - - A successful ping request returns: - - ``` - { ok: 1 } - ``` + ``` + { ok: 1 } + ``` ### Optionally let _kubectl_ choose the local port {#let-kubectl-choose-local-port} @@ -204,7 +193,6 @@ Forwarding from 127.0.0.1:63753 -> 27017 Forwarding from [::1]:63753 -> 27017 ``` - ## Discussion @@ -219,9 +207,7 @@ The support for UDP protocol is tracked in [issue 47862](https://github.com/kubernetes/kubernetes/issues/47862). {{< /note >}} - - - ## {{% heading "whatsnext" %}} Learn more about [kubectl port-forward](/docs/reference/generated/kubectl/kubectl-commands/#port-forward). + diff --git a/content/en/docs/tasks/administer-cluster/certificates.md b/content/en/docs/tasks/administer-cluster/certificates.md index 2338b0cdc7..44effe9340 100644 --- a/content/en/docs/tasks/administer-cluster/certificates.md +++ b/content/en/docs/tasks/administer-cluster/certificates.md @@ -1,5 +1,5 @@ --- -title: Certificates +title: Generate Certificates Manually content_type: task weight: 20 --- diff --git a/content/en/docs/tasks/administer-cluster/cluster-upgrade.md b/content/en/docs/tasks/administer-cluster/cluster-upgrade.md index 6e9dc302c4..e3f9595d26 100644 --- a/content/en/docs/tasks/administer-cluster/cluster-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/cluster-upgrade.md @@ -21,8 +21,8 @@ At a high level, the steps you perform are: ## {{% heading "prerequisites" %}} You must have an existing cluster. This page is about upgrading from Kubernetes -{{< skew prevMinorVersion >}} to Kubernetes {{< skew latestVersion >}}. If your cluster -is not currently running Kubernetes {{< skew prevMinorVersion >}} then please check +{{< skew currentVersionAddMinor -1 >}} to Kubernetes {{< skew currentVersion >}}. If your cluster +is not currently running Kubernetes {{< skew currentVersionAddMinor -1 >}} then please check the documentation for the version of Kubernetes that you plan to upgrade to. ## Upgrade approaches @@ -55,7 +55,7 @@ At this point you should [install the latest version of `kubectl`](/docs/tasks/tools/). For each node in your cluster, [drain](/docs/tasks/administer-cluster/safely-drain-node/) -that node and then either replace it with a new node that uses the {{< skew latestVersion >}} +that node and then either replace it with a new node that uses the {{< skew currentVersion >}} kubelet, or upgrade the kubelet on that node and bring the node back into service. ### Other deployments {#upgrade-other} diff --git a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md index bf5ddd8f5f..be77074dc1 100644 --- a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md +++ b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md @@ -150,7 +150,7 @@ access to clients with the certificate `k8sclient.cert`. Once etcd is configured correctly, only clients with valid certificates can access it. To give Kubernetes API servers the access, configure them with the -flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and +flags `--etcd-certfile=k8sclient.cert`, `--etcd-keyfile=k8sclient.key` and `--etcd-cafile=ca.cert`. {{< note >}} @@ -319,7 +319,7 @@ employed to recover the data of a failed cluster. Before starting the restore operation, a snapshot file must be present. It can either be a snapshot file from a previous backup operation, or from a remaining -[data directory]( https://etcd.io/docs/current/op-guide/configuration/#--data-dir). +[data directory](https://etcd.io/docs/current/op-guide/configuration/#--data-dir). Here is an example: ```shell diff --git a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md index 55030539cb..ff52a11f05 100644 --- a/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md +++ b/content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md @@ -252,6 +252,54 @@ linux/amd64, go1.10.3, 2e322f6 2018/09/07 15:29:04 [INFO] Reloading complete 172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s ``` +### Does CoreDNS have sufficient permissions? + +CoreDNS must be able to list {{< glossary_tooltip text="service" +term_id="service" >}} and {{< glossary_tooltip text="endpoint" +term_id="endpoint" >}} related resources to properly resolve service names. + +Sample error message: +``` +2022-03-18T07:12:15.699431183Z [INFO] 10.96.144.227:52299 - 3686 "A IN serverproxy.contoso.net.cluster.local. udp 52 false 512" SERVFAIL qr,aa,rd 145 0.000091221s +``` + +First, get the current ClusterRole of `system:coredns`: + +```shell +kubectl describe clusterrole system:coredns -n kube-system +``` + +Expected output: +``` +PolicyRule: + Resources Non-Resource URLs Resource Names Verbs + --------- ----------------- -------------- ----- + nodes [] [] [get] + endpoints [] [] [list watch] + namespaces [] [] [list watch] + pods [] [] [list watch] + services [] [] [list watch] + endpointslices.discovery.k8s.io [] [] [list watch] +``` + +If any permissions are missing, edit the ClusterRole to add them: + +```shell +kubectl edit clusterrole system:coredns -n kube-system +``` + +Example insertion of EndpointSlices permissions: +``` +... +- apiGroups: + - discovery.k8s.io + resources: + - endpointslices + verbs: + - list + - watch +... +``` ### Are you in the right namespace for the service? diff --git a/content/en/docs/tasks/administer-cluster/encrypt-data.md b/content/en/docs/tasks/administer-cluster/encrypt-data.md index c48f9ee2da..d510caff81 100644 --- a/content/en/docs/tasks/administer-cluster/encrypt-data.md +++ b/content/en/docs/tasks/administer-cluster/encrypt-data.md @@ -88,8 +88,8 @@ Name | Encryption | Strength | Speed | Key Length | Other Considerations `identity` | None | N/A | N/A | N/A | Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written. `secretbox` | XSalsa20 and Poly1305 | Strong | Faster | 32-byte | A newer standard and may not be considered acceptable in environments that require high levels of review. `aesgcm` | AES-GCM with random nonce | Must be rotated every 200k writes | Fastest | 16, 24, or 32-byte | Is not recommended for use except when an automated key rotation scheme is implemented. -`aescbc` | AES-CBC with PKCS#7 padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks. -`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with PKCS#7 padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/) +`aescbc` | AES-CBC with [PKCS#7](https://datatracker.ietf.org/doc/html/rfc2315) padding | Weak | Fast | 32-byte | Not recommended due to CBC's vulnerability to padding oracle attacks. +`kms` | Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with [PKCS#7](https://datatracker.ietf.org/doc/html/rfc2315) padding, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS) | Strongest | Fast | 32-bytes | The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user. [Configure the KMS provider](/docs/tasks/administer-cluster/kms-provider/) Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider is the first provider, the first key is used for encryption. diff --git a/content/en/docs/tasks/administer-cluster/kms-provider.md b/content/en/docs/tasks/administer-cluster/kms-provider.md index 15bc1290ff..d2ea73d761 100644 --- a/content/en/docs/tasks/administer-cluster/kms-provider.md +++ b/content/en/docs/tasks/administer-cluster/kms-provider.md @@ -19,35 +19,50 @@ This page shows how to configure a Key Management Service (KMS) provider and plu -The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd. The data is encrypted using a data encryption key (DEK); a new DEK is generated for each encryption. The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS. The KMS provider uses gRPC to communicate with a specific KMS -plugin. The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s) as the Kubernetes master(s), is responsible for all communication with the remote KMS. +The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd. +The data is encrypted using a data encryption key (DEK); a new DEK is generated for each encryption. +The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS. +The KMS provider uses gRPC to communicate with a specific KMS plugin. +The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s) +as the Kubernetes control plane, is responsible for all communication with the remote KMS. ## Configuring the KMS provider -To configure a KMS provider on the API server, include a provider of type ```kms``` in the providers array in the encryption configuration file and set the following properties: +To configure a KMS provider on the API server, include a provider of type `kms` in the +`providers` array in the encryption configuration file and set the following properties: * `name`: Display name of the KMS plugin. * `endpoint`: Listen address of the gRPC server (KMS plugin). The endpoint is a UNIX domain socket. * `cachesize`: Number of data encryption keys (DEKs) to be cached in the clear. When cached, DEKs can be used without another call to the KMS; whereas DEKs that are not cached require a call to the KMS to unwrap. -* `timeout`: How long should kube-apiserver wait for kms-plugin to respond before returning an error (default is 3 seconds). +* `timeout`: How long should `kube-apiserver` wait for kms-plugin to respond before + returning an error (default is 3 seconds). -See [Understanding the encryption at rest configuration.](/docs/tasks/administer-cluster/encrypt-data) +See [Understanding the encryption at rest configuration](/docs/tasks/administer-cluster/encrypt-data). ## Implementing a KMS plugin -To implement a KMS plugin, you can develop a new plugin gRPC server or enable a KMS plugin already provided by your cloud provider. You then integrate the plugin with the remote KMS and deploy it on the Kubernetes master. +To implement a KMS plugin, you can develop a new plugin gRPC server or enable a KMS plugin +already provided by your cloud provider. +You then integrate the plugin with the remote KMS and deploy it on the Kubernetes master. ### Enabling the KMS supported by your cloud provider + Refer to your cloud provider for instructions on enabling the cloud provider-specific KMS plugin. ### Developing a KMS plugin gRPC server -You can develop a KMS plugin gRPC server using a stub file available for Go. For other languages, you use a proto file to create a stub file that you can use to develop the gRPC server code. -* Using Go: Use the functions and data structures in the stub file: [service.pb.go](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/service.pb.go) to develop the gRPC server code +You can develop a KMS plugin gRPC server using a stub file available for Go. For other languages, +you use a proto file to create a stub file that you can use to develop the gRPC server code. -* Using languages other than Go: Use the protoc compiler with the proto file: [service.proto](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/service.proto) to generate a stub file for the specific language +* Using Go: Use the functions and data structures in the stub file: + [service.pb.go](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/service.pb.go) + to develop the gRPC server code + +* Using languages other than Go: Use the protoc compiler with the proto file: + [service.proto](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1/service.proto) + to generate a stub file for the specific language Then use the functions and data structures in the stub file to develop the server code. @@ -55,7 +70,7 @@ Then use the functions and data structures in the stub file to develop the serve * kms plugin version: `v1beta1` - In response to procedure call Version, a compatible KMS plugin should return v1beta1 as VersionResponse.version. + In response to procedure call Version, a compatible KMS plugin should return `v1beta1` as `VersionResponse.version`. * message version: `v1beta1` @@ -69,12 +84,15 @@ Then use the functions and data structures in the stub file to develop the serve The KMS plugin can communicate with the remote KMS using any protocol supported by the KMS. All configuration data, including authentication credentials the KMS plugin uses to communicate with the remote KMS, -are stored and managed by the KMS plugin independently. The KMS plugin can encode the ciphertext with additional metadata that may be required before sending it to the KMS for decryption. +are stored and managed by the KMS plugin independently. +The KMS plugin can encode the ciphertext with additional metadata that may be required before sending it to the KMS for decryption. ### Deploying the KMS plugin + Ensure that the KMS plugin runs on the same host(s) as the Kubernetes master(s). ## Encrypting your data with the KMS provider + To encrypt the data: 1. Create a new encryption configuration file using the appropriate properties for the `kms` provider: @@ -94,32 +112,43 @@ To encrypt the data: - identity: {} ``` -1. Set the `--encryption-provider-config` flag on the kube-apiserver to point to the location of the configuration file. +1. Set the `--encryption-provider-config` flag on the kube-apiserver to point to + the location of the configuration file. 1. Restart your API server. +For details about the `EncryptionConfiguration` format, please check the +[API server encryption API reference](/docs/reference/config-api/apiserver-encryption.v1/). + ## Verifying that the data is encrypted Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or updated secret should be encrypted when stored. To verify, you can use the `etcdctl` command line program to retrieve the contents of your secret. -1. Create a new secret called secret1 in the default namespace: - ``` +1. Create a new secret called `secret1` in the `default` namespace: + + ```shell kubectl create secret generic secret1 -n default --from-literal=mykey=mydata ``` -1. Using the etcdctl command line, read that secret out of etcd: - ``` + +1. Using the `etcdctl` command line, read that secret out of etcd: + + ```shell ETCDCTL_API=3 etcdctl get /kubernetes.io/secrets/default/secret1 [...] | hexdump -C ``` - where `[...]` must be the additional arguments for connecting to the etcd server. -1. Verify the stored secret is prefixed with `k8s:enc:kms:v1:`, which indicates that the `kms` provider has encrypted the resulting data. + where `[...]` contains the additional arguments for connecting to the etcd server. + +1. Verify the stored secret is prefixed with `k8s:enc:kms:v1:`, which indicates that + the `kms` provider has encrypted the resulting data. 1. Verify that the secret is correctly decrypted when retrieved via the API: - ``` + + ```shell kubectl describe secret secret1 -n default ``` - should match `mykey: mydata` + + The Secret should contain `mykey: mydata` ## Ensuring all secrets are encrypted @@ -129,7 +158,7 @@ The following command reads all secrets and then updates them to apply server si If an error occurs due to a conflicting write, retry the command. For larger clusters, you may wish to subdivide the secrets by namespace or script an update. -``` +```shell kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` @@ -156,12 +185,12 @@ To switch from a local encryption provider to the `kms` provider and re-encrypt secret: ``` -1. Restart all kube-apiserver processes. +1. Restart all `kube-apiserver` processes. 1. Run the following command to force all secrets to be re-encrypted using the `kms` provider. - ``` - kubectl get secrets --all-namespaces -o json| kubectl replace -f - + ```shell + kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` ## Disabling encryption at rest @@ -183,9 +212,12 @@ To disable encryption at rest: endpoint: unix:///tmp/socketfile.sock cachesize: 100 ``` -1. Restart all kube-apiserver processes. + +1. Restart all `kube-apiserver` processes. + 1. Run the following command to force all secrets to be decrypted. - ``` + + ```shell kubectl get secrets --all-namespaces -o json | kubectl replace -f - ``` diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md b/content/en/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md deleted file mode 100644 index 7009eb2d2a..0000000000 --- a/content/en/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes.md +++ /dev/null @@ -1,256 +0,0 @@ ---- -reviewers: -- jayunit100 -- jsturtevant -- marosset -- perithompson -title: Adding Windows nodes -min-kubernetes-server-version: 1.17 -content_type: tutorial -weight: 30 ---- - - - -{{< feature-state for_k8s_version="v1.18" state="beta" >}} - -You can use Kubernetes to run a mixture of Linux and Windows nodes, so you can mix Pods that run on Linux on with Pods that run on Windows. This page shows how to register Windows nodes to your cluster. - - -## {{% heading "prerequisites" %}} - {{< version-check >}} - -* Obtain a [Windows Server 2019 license](https://www.microsoft.com/en-us/cloud-platform/windows-server-pricing) -(or higher) in order to configure the Windows node that hosts Windows containers. -If you are using VXLAN/Overlay networking you must have also have [KB4489899](https://support.microsoft.com/help/4489899) installed. - -* A Linux-based Kubernetes kubeadm cluster in which you have access to the control plane (see [Creating a single control-plane cluster with kubeadm](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)). - - - - -## {{% heading "objectives" %}} - - -* Register a Windows node to the cluster -* Configure networking so Pods and Services on Linux and Windows can communicate with each other - - - - - - -## Getting Started: Adding a Windows Node to Your Cluster - -### Networking Configuration - -Once you have a Linux-based Kubernetes control-plane node you are ready to choose a networking solution. This guide illustrates using Flannel in VXLAN mode for simplicity. - -#### Configuring Flannel - -1. Prepare Kubernetes control plane for Flannel - - Some minor preparation is recommended on the Kubernetes control plane in our cluster. It is recommended to enable bridged IPv4 traffic to iptables chains when using Flannel. The following command must be run on all Linux nodes: - - ```bash - sudo sysctl net.bridge.bridge-nf-call-iptables=1 - ``` - -1. Download & configure Flannel for Linux - - Download the most recent Flannel manifest: - - ```bash - wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml - ``` - - Modify the `net-conf.json` section of the flannel manifest in order to set the VNI to 4096 and the Port to 4789. It should look as follows: - - ```json - net-conf.json: | - { - "Network": "10.244.0.0/16", - "Backend": { - "Type": "vxlan", - "VNI": 4096, - "Port": 4789 - } - } - ``` - - {{< note >}}The VNI must be set to 4096 and port 4789 for Flannel on Linux to interoperate with Flannel on Windows. See the [VXLAN documentation](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan). - for an explanation of these fields.{{< /note >}} - - {{< note >}}To use L2Bridge/Host-gateway mode instead change the value of `Type` to `"host-gw"` and omit `VNI` and `Port`.{{< /note >}} - -1. Apply the Flannel manifest and validate - - Let's apply the Flannel configuration: - - ```bash - kubectl apply -f kube-flannel.yml - ``` - - After a few minutes, you should see all the pods as running if the Flannel pod network was deployed. - - ```bash - kubectl get pods -n kube-system - ``` - - The output should include the Linux flannel DaemonSet as running: - - ``` - NAMESPACE NAME READY STATUS RESTARTS AGE - ... - kube-system kube-flannel-ds-54954 1/1 Running 0 1m - ``` - -1. Add Windows Flannel and kube-proxy DaemonSets - - Now you can add Windows-compatible versions of Flannel and kube-proxy. In order - to ensure that you get a compatible version of kube-proxy, you'll need to substitute - the tag of the image. The following example shows usage for Kubernetes {{< param "fullversion" >}}, - but you should adjust the version for your own deployment. - - ```bash - curl -L https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/kube-proxy.yml | sed 's/VERSION/{{< param "fullversion" >}}/g' | kubectl apply -f - - kubectl apply -f https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/flannel-overlay.yml - ``` - {{< note >}} - If you're using host-gateway use https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/flannel-host-gw.yml instead - {{< /note >}} - - {{< note >}} -If you're using a different interface rather than Ethernet (i.e. "Ethernet0 2") on the Windows nodes, you have to modify the line: - -```powershell -wins cli process run --path /k/flannel/setup.exe --args "--mode=overlay --interface=Ethernet" -``` - -in the `flannel-host-gw.yml` or `flannel-overlay.yml` file and specify your interface accordingly. - -```bash -# Example -curl -L https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/flannel-overlay.yml | sed 's/Ethernet/Ethernet0 2/g' | kubectl apply -f - -``` - {{< /note >}} - - - -### Joining a Windows worker node - -{{< note >}} -All code snippets in Windows sections are to be run in a PowerShell environment -with elevated permissions (Administrator) on the Windows worker node. -{{< /note >}} - -{{< tabs name="tab-windows-kubeadm-runtime-installation" >}} - -{{% tab name="CRI-containerD" %}} - -#### Install containerD - -```powershell -curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/Install-Containerd.ps1 -.\Install-Containerd.ps1 -``` - -{{< note >}} -To install a specific version of containerD specify the version with -ContainerDVersion. - -```powershell -# Example -.\Install-Containerd.ps1 -ContainerDVersion 1.4.1 -``` - -If you're using a different interface rather than Ethernet (i.e. "Ethernet0 2") on the Windows nodes, specify the name with `-netAdapterName`. - -```powershell -# Example -.\Install-Containerd.ps1 -netAdapterName "Ethernet0 2" -``` - -{{< /note >}} - -#### Install wins, kubelet, and kubeadm - -```PowerShell -curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/scripts/PrepareNode.ps1 -.\PrepareNode.ps1 -KubernetesVersion {{< param "fullversion" >}} -ContainerRuntime containerD -``` - -[Install `crictl` from the cri-tools package](https://github.com/kubernetes-sigs/cri-tools) -which is required so that kubeadm can talk to the CRI endpoint. - -#### Run `kubeadm` to join the node - -Use the command that was given to you when you ran `kubeadm init` on a control plane host. -If you no longer have this command, or the token has expired, you can run `kubeadm token create --print-join-command` -(on a control plane host) to generate a new token and join command. - -{{% /tab %}} - -{{% tab name="Docker Engine" %}} - -#### Install Docker Engine - -Install the `Containers` feature - -```powershell -Install-WindowsFeature -Name containers -``` - -Install Docker -Instructions to do so are available at [Install Docker Engine - Enterprise on Windows Servers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/set-up-environment?tabs=Windows-Server#install-docker). - -[Install cri-dockerd](https://github.com/Mirantis/cri-dockerd) which is required so that the kubelet -can communicate with Docker on a CRI compatible endpoint. - -{{< note >}} -Docker Engine does not implement the [CRI](/docs/concepts/architecture/cri/) -which is a requirement for a container runtime to work with Kubernetes. -For that reason, an additional service [cri-dockerd](https://github.com/Mirantis/cri-dockerd) -has to be installed. cri-dockerd is a project based on the legacy built-in -Docker Engine support that was [removed](/dockershim) from the kubelet in version 1.24. -{{< /note >}} - -Install `crictl` from the [cri-tools project](https://github.com/kubernetes-sigs/cri-tools) -which is required so that kubeadm can talk to the CRI endpoint. - -#### Install wins, kubelet, and kubeadm - -```PowerShell -curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/scripts/PrepareNode.ps1 -.\PrepareNode.ps1 -KubernetesVersion {{< param "fullversion" >}} -``` - -#### Run `kubeadm` to join the node - -Use the command that was given to you when you ran `kubeadm init` on a control plane host. -If you no longer have this command, or the token has expired, you can run `kubeadm token create --print-join-command` -(on a control plane host) to generate a new token and join command. - -{{% /tab %}} - -{{< /tabs >}} - -### Verifying your installation - -You should now be able to view the Windows node in your cluster by running: - -```bash -kubectl get nodes -o wide -``` - -If your new node is in the `NotReady` state it is likely because the flannel image is still downloading. -You can check the progress as before by checking on the flannel pods in the `kube-system` namespace: - -```shell -kubectl -n kube-system get pods -l app=flannel -``` - -Once the flannel Pod is running, your node should enter the `Ready` state and then be available to handle workloads. - -## {{% heading "whatsnext" %}} - -- [Upgrading Windows kubeadm nodes](/docs/tasks/administer-cluster/kubeadm/upgrading-windows-nodes) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md index d9df7fb38c..97fa2bc352 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md @@ -68,9 +68,7 @@ and passing it to the local node kubelet. ## Using the `cgroupfs` driver -As this guide explains using the `cgroupfs` driver with kubeadm is not recommended. - -To continue using `cgroupfs` and to prevent `kubeadm upgrade` from modifying the +To use `cgroupfs` and to prevent `kubeadm upgrade` from modifying the `KubeletConfiguration` cgroup driver on existing setups, you must be explicit about its value. This applies to a case where you do not wish future versions of kubeadm to apply the `systemd` driver by default. diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md index 696a69ba82..18032fc4b3 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -244,7 +244,7 @@ serverTLSBootstrap: true ``` If you have already created the cluster you must adapt it by doing the following: - - Find and edit the `kubelet-config-{{< skew latestVersion >}}` ConfigMap in the `kube-system` namespace. + - Find and edit the `kubelet-config-{{< skew currentVersion >}}` ConfigMap in the `kube-system` namespace. In that ConfigMap, the `kubelet` key has a [KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) document as its value. Edit the KubeletConfiguration document to set `serverTLSBootstrap: true`. @@ -276,7 +276,7 @@ By default, these serving certificate will expire after one year. Kubeadm sets t `KubeletConfiguration` field `rotateCertificates` to `true`, which means that close to expiration a new set of CSRs for the serving certificates will be created and must be approved to complete the rotation. To understand more see -[Certificate Rotation](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation). +[Certificate Rotation](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#certificate-rotation). If you are looking for a solution for automatic approval of these CSRs it is recommended that you contact your cloud provider and ask if they have a CSR signer that verifies diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index 8040e1185f..783b107927 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -29,7 +29,7 @@ The upgrade workflow at high level is the following: ## {{% heading "prerequisites" %}} -- Make sure you read the [release notes]({{< latest-release-notes >}}) carefully. +- Make sure you read the [release notes](https://git.k8s.io/kubernetes/CHANGELOG) carefully. - The cluster should use a static control plane and etcd pods or external etcd. - Make sure to back up any important components, such as app-level state stored in a database. `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice. @@ -79,83 +79,87 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc **For the first control plane node** -- Upgrade kubeadm: +- Upgrade kubeadm: -{{< tabs name="k8s_install_kubeadm_first_cp" >}} -{{% tab name="Ubuntu, Debian or HypriotOS" %}} - # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version - apt-mark unhold kubeadm && \ - apt-get update && apt-get install -y kubeadm={{< skew currentVersion >}}.x-00 && \ - apt-mark hold kubeadm -{{% /tab %}} -{{% tab name="CentOS, RHEL or Fedora" %}} - # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version - yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes -{{% /tab %}} -{{< /tabs >}} -
    + {{< tabs name="k8s_install_kubeadm_first_cp" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version + apt-mark unhold kubeadm && \ + apt-get update && apt-get install -y kubeadm={{< skew currentVersion >}}.x-00 && \ + apt-mark hold kubeadm + ``` + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version + yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes + ``` + {{% /tab %}} + {{< /tabs >}} +
    -- Verify that the download works and has the expected version: +- Verify that the download works and has the expected version: - ```shell - kubeadm version - ``` + ```shell + kubeadm version + ``` -- Verify the upgrade plan: +- Verify the upgrade plan: - ```shell - kubeadm upgrade plan - ``` + ```shell + kubeadm upgrade plan + ``` - This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to. - It also shows a table with the component config version states. + This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to. + It also shows a table with the component config version states. -{{< note >}} -`kubeadm upgrade` also automatically renews the certificates that it manages on this node. -To opt-out of certificate renewal the flag `--certificate-renewal=false` can be used. -For more information see the [certificate management guide](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs). -{{}} + {{< note >}} + `kubeadm upgrade` also automatically renews the certificates that it manages on this node. + To opt-out of certificate renewal the flag `--certificate-renewal=false` can be used. + For more information see the [certificate management guide](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs). + {{}} + + {{< note >}} + If `kubeadm upgrade plan` shows any component configs that require manual upgrade, users must provide + a config file with replacement configs to `kubeadm upgrade apply` via the `--config` command line flag. + Failing to do so will cause `kubeadm upgrade apply` to exit with an error and not perform an upgrade. + {{}} -{{< note >}} -If `kubeadm upgrade plan` shows any component configs that require manual upgrade, users must provide -a config file with replacement configs to `kubeadm upgrade apply` via the `--config` command line flag. -Failing to do so will cause `kubeadm upgrade apply` to exit with an error and not perform an upgrade. -{{}} +- Choose a version to upgrade to, and run the appropriate command. For example: -- Choose a version to upgrade to, and run the appropriate command. For example: + ```shell + # replace x with the patch version you picked for this upgrade + sudo kubeadm upgrade apply v{{< skew currentVersion >}}.x + ``` - ```shell - # replace x with the patch version you picked for this upgrade - sudo kubeadm upgrade apply v{{< skew currentVersion >}}.x - ``` + Once the command finishes you should see: - Once the command finishes you should see: + ``` + [upgrade/successful] SUCCESS! Your cluster was upgraded to "v{{< skew currentVersion >}}.x". Enjoy! - ``` - [upgrade/successful] SUCCESS! Your cluster was upgraded to "v{{< skew currentVersion >}}.x". Enjoy! + [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. + ``` - [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. - ``` +- Manually upgrade your CNI provider plugin. -- Manually upgrade your CNI provider plugin. + Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. + Check the [addons](/docs/concepts/cluster-administration/addons/) page to + find your CNI provider and see whether additional upgrade steps are required. - Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. - Check the [addons](/docs/concepts/cluster-administration/addons/) page to - find your CNI provider and see whether additional upgrade steps are required. - - This step is not required on additional control plane nodes if the CNI provider runs as a DaemonSet. + This step is not required on additional control plane nodes if the CNI provider runs as a DaemonSet. **For the other control plane nodes** Same as the first control plane node but use: -``` +```shell sudo kubeadm upgrade node ``` instead of: -``` +```shell sudo kubeadm upgrade apply ``` @@ -163,46 +167,50 @@ Also calling `kubeadm upgrade plan` and upgrading the CNI provider plugin is no ### Drain the node -- Prepare the node for maintenance by marking it unschedulable and evicting the workloads: +- Prepare the node for maintenance by marking it unschedulable and evicting the workloads: - ```shell - # replace with the name of your node you are draining - kubectl drain --ignore-daemonsets - ``` + ```shell + # replace with the name of your node you are draining + kubectl drain --ignore-daemonsets + ``` ### Upgrade kubelet and kubectl -- Upgrade the kubelet and kubectl: +- Upgrade the kubelet and kubectl: -{{< tabs name="k8s_install_kubelet" >}} -{{% tab name="Ubuntu, Debian or HypriotOS" %}} - # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version - apt-mark unhold kubelet kubectl && \ - apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \ - apt-mark hold kubelet kubectl -{{% /tab %}} -{{% tab name="CentOS, RHEL or Fedora" %}} - # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version - yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes -{{% /tab %}} -{{< /tabs >}} -
    + {{< tabs name="k8s_install_kubelet" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version + apt-mark unhold kubelet kubectl && \ + apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \ + apt-mark hold kubelet kubectl + ``` + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version + yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes + ``` + {{% /tab %}} + {{< /tabs >}} +
    -- Restart the kubelet: +- Restart the kubelet: - ```shell - sudo systemctl daemon-reload - sudo systemctl restart kubelet - ``` + ```shell + sudo systemctl daemon-reload + sudo systemctl restart kubelet + ``` ### Uncordon the node -- Bring the node back online by marking it schedulable: +- Bring the node back online by marking it schedulable: - ```shell - # replace with the name of your node - kubectl uncordon - ``` + ```shell + # replace with the name of your node + kubectl uncordon + ``` ## Upgrade worker nodes @@ -211,76 +219,83 @@ without compromising the minimum required capacity for running your workloads. ### Upgrade kubeadm -- Upgrade kubeadm: +- Upgrade kubeadm: -{{< tabs name="k8s_install_kubeadm_worker_nodes" >}} -{{% tab name="Ubuntu, Debian or HypriotOS" %}} - # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version - apt-mark unhold kubeadm && \ - apt-get update && apt-get install -y kubeadm={{< skew currentVersion >}}.x-00 && \ - apt-mark hold kubeadm -{{% /tab %}} -{{% tab name="CentOS, RHEL or Fedora" %}} - # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version - yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes -{{% /tab %}} -{{< /tabs >}} + {{< tabs name="k8s_install_kubeadm_worker_nodes" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version + apt-mark unhold kubeadm && \ + apt-get update && apt-get install -y kubeadm={{< skew currentVersion >}}.x-00 && \ + apt-mark hold kubeadm + ``` + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version + yum install -y kubeadm-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes + ``` + {{% /tab %}} + {{< /tabs >}} ### Call "kubeadm upgrade" -- For worker nodes this upgrades the local kubelet configuration: +- For worker nodes this upgrades the local kubelet configuration: - ```shell - sudo kubeadm upgrade node - ``` + ```shell + sudo kubeadm upgrade node + ``` ### Drain the node -- Prepare the node for maintenance by marking it unschedulable and evicting the workloads: +- Prepare the node for maintenance by marking it unschedulable and evicting the workloads: - ```shell - # replace with the name of your node you are draining - kubectl drain --ignore-daemonsets - ``` + ```shell + # replace with the name of your node you are draining + kubectl drain --ignore-daemonsets + ``` ### Upgrade kubelet and kubectl -- Upgrade the kubelet and kubectl: +- Upgrade the kubelet and kubectl: -{{< tabs name="k8s_kubelet_and_kubectl" >}} -{{% tab name="Ubuntu, Debian or HypriotOS" %}} - # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version - apt-mark unhold kubelet kubectl && \ - apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \ - apt-mark hold kubelet kubectl -{{% /tab %}} -{{% tab name="CentOS, RHEL or Fedora" %}} - # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version - yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes -{{% /tab %}} -{{< /tabs >}} -
    + {{< tabs name="k8s_kubelet_and_kubectl" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-00 with the latest patch version + apt-mark unhold kubelet kubectl && \ + apt-get update && apt-get install -y kubelet={{< skew currentVersion >}}.x-00 kubectl={{< skew currentVersion >}}.x-00 && \ + apt-mark hold kubelet kubectl + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + ```shell + # replace x in {{< skew currentVersion >}}.x-0 with the latest patch version + yum install -y kubelet-{{< skew currentVersion >}}.x-0 kubectl-{{< skew currentVersion >}}.x-0 --disableexcludes=kubernetes + ``` + {{% /tab %}} + {{< /tabs >}} +
    -- Restart the kubelet: +- Restart the kubelet: - ```shell - sudo systemctl daemon-reload - sudo systemctl restart kubelet - ``` + ```shell + sudo systemctl daemon-reload + sudo systemctl restart kubelet + ``` ### Uncordon the node -- Bring the node back online by marking it schedulable: +- Bring the node back online by marking it schedulable: - ```shell - # replace with the name of your node - kubectl uncordon - ``` + ```shell + # replace with the name of your node + kubectl uncordon + ``` ## Verify the status of the cluster -After the kubelet is upgraded on all nodes verify that all nodes are available again by running the following command -from anywhere kubectl can access the cluster: +After the kubelet is upgraded on all nodes verify that all nodes are available again by running +the following command from anywhere kubectl can access the cluster: ```shell kubectl get nodes @@ -296,6 +311,7 @@ This command is idempotent and eventually makes sure that the actual state is th To recover from a bad state, you can also run `kubeadm upgrade apply --force` without changing the version that your cluster is running. During upgrade kubeadm writes the following backup folders under `/etc/kubernetes/tmp`: + - `kubeadm-backup-etcd--