Merge remote-tracking branch 'upstream/master' into merged-master-dev-1.20
This commit is contained in:
Kristin Martin
@@ -0,0 +1,180 @@
|
||||
---
|
||||
layout: blog
|
||||
title: "Dockershim Deprecation FAQ"
|
||||
date: 2020-12-02
|
||||
slug: dockershim-faq
|
||||
aliases: [ '/dockershim' ]
|
||||
---
|
||||
|
||||
This document goes over some frequently asked questions regarding the Dockershim
|
||||
depreaction announced as a part of the Kubernetes v1.20 release. For more detail
|
||||
on the deprecation of Docker as a container runtime for Kubernetes kubelets, and
|
||||
what that means, check out the blog post
|
||||
[Don't Panic: Kubernetes and Docker](/blog/2020/12/02/dont-panic-kubernetes-and-docker/).
|
||||
|
||||
### Why is dockershim being deprecated?
|
||||
|
||||
Maintaining dockershim has become a heavy burden on the Kubernetes maintainers.
|
||||
The CRI standard was created to reduce this burden and allow smooth interoperability
|
||||
of different container runtimes. Docker itself doesn't currently implement CRI,
|
||||
thus the problem.
|
||||
|
||||
Dockershim was always intended to be a temporary solution (hence the name: shim).
|
||||
You can read more about the community discussion and planning in the
|
||||
[Dockershim Removal Kubernetes Enhancement Proposal][drkep].
|
||||
|
||||
Additionally, features that were largely incompatible with the dockershim, such
|
||||
as cgroups v2 and user namespaces are being implemented in these newer CRI
|
||||
runtimes. Removing support for the dockershim will allow further development in
|
||||
those areas.
|
||||
|
||||
[drkep]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1985-remove-dockershim
|
||||
|
||||
### Can I still use Docker in Kubernetes 1.20?
|
||||
|
||||
Yes, the only thing changing in 1.20 is a single warning log printed at [kubelet]
|
||||
startup if using Docker as the runtime.
|
||||
|
||||
[kubelet]: /docs/reference/command-line-tools-reference/kubelet/
|
||||
|
||||
|
||||
### When will dockershim be removed?
|
||||
|
||||
Given the impact of this change, we are using an extended deprecation timeline.
|
||||
It will not be removed before Kubernetes 1.22, meaning the earliest release without
|
||||
dockershim would be 1.23 in late 2021. We will be working closely with vendors
|
||||
and other ecosystem groups to ensure a smooth transition and will evaluate things
|
||||
as the situation evolves.
|
||||
|
||||
|
||||
### Will my existing Docker images still work?
|
||||
|
||||
Yes, the images produced from `docker build` will work with all CRI implementations.
|
||||
All your existing images will still work exactly the same.
|
||||
|
||||
|
||||
### What about private images?
|
||||
|
||||
Also yes. All CRI runtimes support the same pull secrets configuration used in
|
||||
Kubernetes, either via the PodSpec or ServiceAccount.
|
||||
|
||||
|
||||
### Are Docker and containers the same thing?
|
||||
|
||||
Docker popularized the Linux containers pattern and has been instrumental in
|
||||
developing the underlying technology, however containers in Linux have existed
|
||||
for a long time. The container ecosystem has grown to be much broader than just
|
||||
Docker. Standards like OCI and CRI have helped many tools grow and thrive in our
|
||||
ecosystem, some replacing aspects of Docker while others enhance existing
|
||||
functionality.
|
||||
|
||||
|
||||
### Are there examples of folks using other runtimes in production today?
|
||||
|
||||
All Kubernetes project produced artifacts (Kubernetes binaries) are validated
|
||||
with each release.
|
||||
|
||||
Additionally, the [kind] project has been using containerd for some time and has
|
||||
seen an improvement in stability for its use case. Kind and containerd are leveraged
|
||||
multiple times every day to validate any changes to the Kubernetes codebase. Other
|
||||
related projects follow a similar pattern as well, demonstrating the stability and
|
||||
usability of other container runtimes. As an example, OpenShift 4.x has been
|
||||
using the [CRI-O] runtime in production since June 2019.
|
||||
|
||||
For other examples and references you can look at the adopters of containerd and
|
||||
cri-o, two container runtimes under the Cloud Native Computing Foundation ([CNCF]).
|
||||
- [containerd](https://github.com/containerd/containerd/blob/master/ADOPTERS.md)
|
||||
- [CRI-O](https://github.com/cri-o/cri-o/blob/master/ADOPTERS.md)
|
||||
|
||||
[CRI-O]: https://cri-o.io/
|
||||
[kind]: https://kind.sigs.k8s.io/
|
||||
[CNCF]: https://cncf.io
|
||||
|
||||
|
||||
### People keep referencing OCI, what is that?
|
||||
|
||||
OCI stands for the [Open Container Initiative], which standardized many of the
|
||||
interfaces between container tools and technologies. They maintain a standard
|
||||
specification for packaging container images (OCI image-spec) and running containers
|
||||
(OCI runtime-spec). They also maintain an actual implementation of the runtime-spec
|
||||
in the form of [runc], which is the underlying default runtime for both
|
||||
[containerd] and [CRI-O]. The CRI builds on these low-level specifications to
|
||||
provide an end-to-end standard for managing containers.
|
||||
|
||||
[Open Container Initative]: https://opencontainers.org/about/overview/
|
||||
[runc]: https://github.com/opencontainers/runc
|
||||
[containerd]: https://containerd.io/
|
||||
|
||||
|
||||
### Which CRI implementation should I use?
|
||||
|
||||
That’s a complex question and it depends on a lot of factors. If Docker is
|
||||
working for you, moving to containerd should be a relatively easy swap and
|
||||
has have strictly better performance and less overhead. However we encourage you
|
||||
to explore all the options from the [CNCF landscape] in case another would be an
|
||||
even better fit for your environment.
|
||||
|
||||
[CNCF landscape]: https://landscape.cncf.io/category=container-runtime&format=card-mode&grouping=category
|
||||
|
||||
|
||||
### What should I look out for when changing CRI implementations?
|
||||
|
||||
While the underlying containerization code is the same between Docker and most
|
||||
CRIs (including containerd), there are a few differences around the edges. Some
|
||||
common things to consider when migrating are:
|
||||
|
||||
- Logging configuration
|
||||
- Runtime resource limitations
|
||||
- Node provisioning scripts that call docker or use docker via it's control socket
|
||||
- Kubectl plugins that require docker CLI or the control socket
|
||||
- Kubernetes tools that require direct access to Docker (e.g. kube-imagepuller)
|
||||
- Configuration of functionality like `registry-mirrors` and insecure registries
|
||||
- Other support scripts or daemons that expect docker to be available and are run
|
||||
outside of Kubernetes (e.g. monitoring or security agents)
|
||||
- GPUs or special hardware and how they integrate with your runtime and Kubernetes
|
||||
|
||||
If you use Kubernetes resource requests/limits or file-based log collection
|
||||
DaemonSets then they will continue to work the same, but if you’ve customized
|
||||
your dockerd configuration, you’ll need to adapt that for your new container
|
||||
runtime where possible.
|
||||
|
||||
Another thing to look out for is anything expecting to run for system maintenance
|
||||
or nested inside a container when building images will no longer work. For the
|
||||
former, you can use the [`crictl`][cr] tool as a drop-in replacement and for the
|
||||
latter you can use newer container build options like [img], [buildah], or
|
||||
[kaniko] that don’t require Docker.
|
||||
|
||||
[cr]: https://github.com/kubernetes-sigs/cri-tools
|
||||
[img]: https://github.com/genuinetools/img
|
||||
[buildah]: https://github.com/containers/buildah
|
||||
[kaniko]: https://github.com/GoogleContainerTools/kaniko
|
||||
|
||||
For containerd, you can start with their [documentation] to see what configuration
|
||||
options are available as you migrate things over.
|
||||
|
||||
[documentation]: https://github.com/containerd/cri/blob/master/docs/registry.md
|
||||
|
||||
For instructions on how to use containerd and CRI-O with Kubernetes, see the
|
||||
Kubernetes documentation on [Container Runtimes]
|
||||
|
||||
[Container Runtimes]: /docs/setup/production-environment/container-runtimes
|
||||
|
||||
|
||||
### What if I have more questions?
|
||||
|
||||
If you use a vendor-supported Kubernetes distribution, you can ask them about
|
||||
upgrade plans for their products. For end-user questions, please post them
|
||||
to our end user community forum: https://discuss.kubernetes.io/.
|
||||
|
||||
You can also check out the excellent blog post
|
||||
[Wait, Docker is deprecated in Kubernetes now?][dep] a more in-depth technical
|
||||
discussion of the changes.
|
||||
|
||||
[dep]: https://dev.to/inductor/wait-docker-is-deprecated-in-kubernetes-now-what-do-i-do-e4m
|
||||
|
||||
|
||||
### Can I have a hug?
|
||||
|
||||
Always and whenever you want! 🤗🤗
|
||||
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
layout: blog
|
||||
title: "Don't Panic: Kubernetes and Docker"
|
||||
date: 2020-12-02
|
||||
slug: dont-panic-kubernetes-and-docker
|
||||
---
|
||||
|
||||
**Authors:** Jorge Castro, Duffie Cooley, Kat Cosgrove, Justin Garrison, Noah Kantrowitz, Bob Killen, Rey Lejano, Dan “POP” Papandrea, Jeffrey Sica, Davanum “Dims” Srinivas
|
||||
|
||||
Kubernetes is [deprecating
|
||||
Docker](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#deprecation)
|
||||
as a container runtime after v1.20.
|
||||
|
||||
**You do not need to panic. It’s not as dramatic as it sounds.**
|
||||
|
||||
tl;dr Docker as an underlying runtime is being deprecated in favor of runtimes
|
||||
that use the [Container Runtime Interface(CRI)](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)
|
||||
created for Kubernetes. Docker-produced images will continue to work in your
|
||||
cluster with all runtimes, as they always have.
|
||||
|
||||
If you’re an end-user of Kubernetes, not a whole lot will be changing for you.
|
||||
This doesn’t mean the death of Docker, and it doesn’t mean you can’t, or
|
||||
shouldn’t, use Docker as a development tool anymore. Docker is still a useful
|
||||
tool for building containers, and the images that result from running `docker
|
||||
build` can still run in your Kubernetes cluster.
|
||||
|
||||
If you’re using a managed Kubernetes service like GKE, EKS, or AKS (which [defaults to containerd](https://github.com/Azure/AKS/releases/tag/2020-11-16)) you will need to
|
||||
make sure your worker nodes are using a supported container runtime before
|
||||
Docker support is removed in a future version of Kubernetes. If you have node
|
||||
customizations you may need to update them based on your environment and runtime
|
||||
requirements. Please work with your service provider to ensure proper upgrade
|
||||
testing and planning.
|
||||
|
||||
If you’re rolling your own clusters, you will also need to make changes to avoid
|
||||
your clusters breaking. At v1.20, you will get a deprecation warning for Docker.
|
||||
When Docker runtime support is removed in a future release (currently planned
|
||||
for the 1.22 release in late 2021) of Kubernetes it will no longer be supported
|
||||
and you will need to switch to one of the other compliant container runtimes,
|
||||
like containerd or CRI-O. Just make sure that the runtime you choose supports
|
||||
the docker daemon configurations you currently use (e.g. logging).
|
||||
|
||||
## So why the confusion and what is everyone freaking out about?
|
||||
|
||||
We’re talking about two different environments here, and that’s creating
|
||||
confusion. Inside of your Kubernetes cluster, there’s a thing called a container
|
||||
runtime that’s responsible for pulling and running your container images. Docker
|
||||
is a popular choice for that runtime (other common options include containerd
|
||||
and CRI-O), but Docker was not designed to be embedded inside Kubernetes, and
|
||||
that causes a problem.
|
||||
|
||||
You see, the thing we call “Docker” isn’t actually one thing -- it’s an entire
|
||||
tech stack, and one part of it is a thing called “containerd,” which is a
|
||||
high-level container runtime by itself. Docker is cool and useful because it has
|
||||
a lot of UX enhancements that make it really easy for humans to interact with
|
||||
while we’re doing development work, but those UX enhancements aren’t necessary
|
||||
for Kubernetes, because it isn’t a human.
|
||||
|
||||
As a result of this human-friendly abstraction layer, your Kubernetes cluster
|
||||
has to use another tool called Dockershim to get at what it really needs, which
|
||||
is containerd. That’s not great, because it gives us another thing that has to
|
||||
be maintained and can possibly break. What’s actually happening here is that
|
||||
Dockershim is being removed from Kubelet as early as v1.23 release, which
|
||||
removes support for Docker as a container runtime as a result. You might be
|
||||
thinking to yourself, but if containerd is included in the Docker stack, why
|
||||
does Kubernetes need the Dockershim?
|
||||
|
||||
Docker isn’t compliant with CRI, the [Container Runtime Interface](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/).
|
||||
If it were, we wouldn’t need the shim, and this wouldn’t be a thing. But it’s
|
||||
not the end of the world, and you don’t need to panic -- you just need to change
|
||||
your container runtime from Docker to another supported container runtime.
|
||||
|
||||
One thing to note: If you are relying on the underlying docker socket
|
||||
(/var/run/docker.sock) as part of a workflow within your cluster today, moving
|
||||
to a different runtime will break your ability to use it. This pattern is often
|
||||
called Docker in Docker. There are lots of options out there for this specific
|
||||
use case including things like
|
||||
[kaniko](https://github.com/GoogleContainerTools/kaniko),
|
||||
[img](https://github.com/genuinetools/img), and
|
||||
[buildah](https://github.com/containers/buildah).
|
||||
|
||||
## What does this change mean for developers, though? Do we still write Dockerfiles? Do we still build things with Docker?
|
||||
|
||||
This change addresses a different environment than most folks use to interact
|
||||
with Docker. The Docker installation you’re using in development is unrelated to
|
||||
the Docker runtime inside your Kubernetes cluster. It’s confusing, I know. As a
|
||||
developer, Docker is still useful to you in all the ways it was before this
|
||||
change was announced. The image that Docker produces isn’t really a
|
||||
Docker-specific image -- it’s an OCI ([Open Container Initiative](https://opencontainers.org/)) image.
|
||||
Any OCI-compliant image, regardless of the tool you use to build it, will look
|
||||
the same to Kubernetes. Both [containerd](https://containerd.io/) and
|
||||
[CRI-O](https://cri-o.io/) know how to pull those images and run them. This is
|
||||
why we have a standard for what containers should look like.
|
||||
|
||||
So, this change is coming. It’s going to cause issues for some, but it isn’t
|
||||
catastrophic, and generally it’s a good thing. Depending on how you interact
|
||||
with Kubernetes, this could mean nothing to you, or it could mean a bit of work.
|
||||
In the long run, it’s going to make things easier. If this is still confusing
|
||||
for you, that’s okay -- there’s a lot going on here, Kubernetes has a lot of
|
||||
moving parts, and nobody is an expert in 100% of it. We encourage any and all
|
||||
questions regardless of experience level or complexity! Our goal is to make sure
|
||||
everyone is educated as much as possible on the upcoming changes. `<3` We hope
|
||||
this has answered most of your questions and soothed some anxieties!
|
||||
|
||||
Looking for more answers? Check out our accompanying [Dockershim Deprecation FAQ](/blog/2020/12/02/dockershim-faq/).
|
||||
@@ -26,10 +26,9 @@ case_study_details:
|
||||
|
||||
<p>Speed of delivery increased. Some of the legacy VM-based deployments took 45 minutes; with Kubernetes, that time was "just a few seconds to a couple of minutes," says Engineering Manager Brian Balser. Adds Li: "Teams that used to deploy on weekly schedules or had to coordinate schedules with the infrastructure team now deploy their updates independently, and can do it daily when necessary." Adopting Cloud Native Computing Foundation technologies allows for a more unified approach to deployment across the engineering staff, and portability for the company.</p>
|
||||
|
||||
{{< case-studies/quote author="Deep Kapadia, Executive Director, Engineering at The New York Times">}}
|
||||
<iframe style="padding:1%:" width="380" height="215" src="https://www.youtube.com/embed/DqS_IPw-c6o" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
|
||||
<iframe style="padding:1%:" width="380" height="215" src="https://www.youtube.com/embed/Tm4VfJtOHt8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
|
||||
<br>
|
||||
{{< case-studies/quote author="Deep Kapadia, Executive Director, Engineering at The New York Times" >}}
|
||||
{{< youtube DqS_IPw-c6o youtube-quote-sm >}}
|
||||
{{< youtube Tm4VfJtOHt8 youtube-quote-sm >}}
|
||||
"I think once you get over the initial hump, things get a lot easier and actually a lot faster."
|
||||
{{< /case-studies/quote >}}
|
||||
|
||||
|
||||
@@ -158,6 +158,11 @@ tables to provide per-instance subnets to each host (which is limited to 50-100
|
||||
entries per VPC). In short, cni-ipvlan-vpc-k8s significantly reduces the
|
||||
network complexity required to deploy Kubernetes at scale within AWS.
|
||||
|
||||
### Coil
|
||||
|
||||
[Coil](https://github.com/cybozu-go/coil) is a CNI plugin designed for ease of integration, providing flexible egress networking.
|
||||
Coil operates with a low overhead compared to bare metal, and allows you to define arbitrary egress NAT gateways for external networks.
|
||||
|
||||
### Contiv
|
||||
|
||||
[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](https://contiv.io) is all open sourced.
|
||||
@@ -311,4 +316,4 @@ to run, and in both cases, the network provides one IP address per pod - as is s
|
||||
|
||||
The early design of the networking model and its rationale, and some future
|
||||
plans are described in more detail in the
|
||||
[networking design document](https://git.k8s.io/community/contributors/design-proposals/network/networking.md).
|
||||
[networking design document](https://git.k8s.io/community/contributors/design-proposals/network/networking.md).
|
||||
|
||||
@@ -104,7 +104,7 @@ The kubelet collects accelerator metrics through cAdvisor. To collect these metr
|
||||
|
||||
The responsibility for collecting accelerator metrics now belongs to the vendor rather than the kubelet. Vendors must provide a container that collects metrics and exposes them to the metrics service (for example, Prometheus).
|
||||
|
||||
The [`DisableAcceleratorUsageMetrics` feature gate](/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features:~:text= DisableAcceleratorUsageMetrics,-false) disables metrics collected by the kubelet.
|
||||
The [`DisableAcceleratorUsageMetrics` feature gate](/docs/reference/command-line-tools-reference/feature-gates/) disables metrics collected by the kubelet, with a [timeline for enabling this feature by default](https://github.com/kubernetes/enhancements/tree/411e51027db842355bd489691af897afc1a41a5e/keps/sig-node/1867-disable-accelerator-usage-metrics#graduation-criteria).
|
||||
|
||||
## Component metrics
|
||||
|
||||
|
||||
@@ -137,7 +137,7 @@ See the [ServiceAccount](/docs/tasks/configure-pod-container/configure-service-a
|
||||
documentation for more information on how service accounts work.
|
||||
You can also check the `automountServiceAccountToken` field and the
|
||||
`serviceAccountName` field of the
|
||||
[`Pod`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#secret-v1-core)
|
||||
[`Pod`](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core)
|
||||
for information on referencing service account from Pods.
|
||||
|
||||
### Docker config Secrets
|
||||
@@ -154,7 +154,7 @@ When using this Secret type, you have to ensure the Secret `data` field
|
||||
contains a `.dockercfg` key whose value is content of a `~/.dockercfg` file
|
||||
encoded in the base64 format.
|
||||
|
||||
The `kubernetes/dockerconfigjson` type is designed for storing a serialized
|
||||
The `kubernetes.io/dockerconfigjson` type is designed for storing a serialized
|
||||
JSON that follows the same format rules as the `~/.docker/config.json` file
|
||||
which is a new format for `~/.dockercfg`.
|
||||
When using this Secret type, the `data` field of the Secret object must
|
||||
@@ -248,7 +248,7 @@ configuration.
|
||||
|
||||
The builtin type `kubernetes.io/ssh-auth` is provided for storing data used in
|
||||
SSH authentication. When using this Secret type, you will have to specify a
|
||||
`ssh-privatekey` key-value pair in the `data` (or `stringData`) field.
|
||||
`ssh-privatekey` key-value pair in the `data` (or `stringData`) field
|
||||
as the SSH credential to use.
|
||||
|
||||
The following YAML is an example config for a SSH authentication Secret:
|
||||
@@ -349,22 +349,21 @@ data:
|
||||
usage-bootstrap-signing: dHJ1ZQ==
|
||||
```
|
||||
|
||||
A bootstrap type has the following keys specified under `data`:
|
||||
A bootstrap type Secret has the following keys specified under `data`:
|
||||
|
||||
- `token_id`: A random 6 character string as the token identifier. Required.
|
||||
- `token-secret`: A random 16 character string as the actual token secret. Required.
|
||||
- `description1`: A human-readable string that describes what the token is
|
||||
- `description`: A human-readable string that describes what the token is
|
||||
used for. Optional.
|
||||
- `expiration`: An absolute UTC time using RFC3339 specifying when the token
|
||||
should be expired. Optional.
|
||||
- `usage-bootstrap-<usage>`: A boolean flag indicating additional usage for
|
||||
the bootstrap token.
|
||||
- `auth-extra-groups`: A comma-separated list of group names that will be
|
||||
authenticated as in addition to system:bootstrappers group.
|
||||
authenticated as in addition to the `system:bootstrappers` group.
|
||||
|
||||
The above YAML may look confusing because the values are all in base64 encoded
|
||||
strings. In fact, you can create an identical Secret using the following YAML
|
||||
which results in an identical Secret object:
|
||||
strings. In fact, you can create an identical Secret using the following YAML:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
|
||||
@@ -84,7 +84,7 @@ Credentials can be provided in several ways:
|
||||
provider) can implement your mechanism for authenticating the node
|
||||
to the container registry.
|
||||
|
||||
These options are explaind in more detail below.
|
||||
These options are explained in more detail below.
|
||||
|
||||
### Configuring nodes to authenticate to a private registry
|
||||
|
||||
|
||||
@@ -77,19 +77,10 @@ about this format, see the [Kubernetes Protobuf serialization](https://github.co
|
||||
Interface Definition Language (IDL) files for each schema located in the Go
|
||||
packages that define the API objects.
|
||||
|
||||
## API changes
|
||||
## Persistence
|
||||
|
||||
Any system that is successful needs to grow and change as new use cases emerge or existing ones change.
|
||||
Therefore, Kubernetes has designed its features to allow the Kubernetes API to continuously change and grow.
|
||||
The Kubernetes project aims to _not_ break compatibility with existing clients, and to maintain that
|
||||
compatibility for a length of time so that other projects have an opportunity to adapt.
|
||||
|
||||
In general, new API resources and new resource fields can be added often and frequently.
|
||||
Elimination of resources or fields requires following the
|
||||
[API deprecation policy](/docs/reference/using-api/deprecation-policy/).
|
||||
|
||||
What constitutes a compatible change, and how to change the API, are detailed in
|
||||
[API changes](https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#readme).
|
||||
Kubernetes stores the serialized state of objects by writing them into
|
||||
{{< glossary_tooltip term_id="etcd" >}}.
|
||||
|
||||
## API groups and versioning
|
||||
|
||||
@@ -107,17 +98,44 @@ To make it easier to evolve and to extend its API, Kubernetes implements
|
||||
[enabled or disabled](/docs/reference/using-api/#enabling-or-disabling).
|
||||
|
||||
API resources are distinguished by their API group, resource type, namespace
|
||||
(for namespaced resources), and name. The API server may serve the same
|
||||
underlying data through multiple API version and handle the conversion between
|
||||
API versions transparently. All these different versions are actually
|
||||
representations of the same resource. For example, suppose there are two
|
||||
versions `v1` and `v1beta1` for the same resource. An object created by the
|
||||
`v1beta1` version can then be read, updated, and deleted by either the
|
||||
`v1beta1` or the `v1` versions.
|
||||
(for namespaced resources), and name. The API server handles the conversion between
|
||||
API versions transparently: all the different versions are actually representations
|
||||
of the same persisted data. The API server may serve the same underlying data
|
||||
through multiple API versions.
|
||||
|
||||
For example, suppose there are two API versions, `v1` and `v1beta1`, for the same
|
||||
resource. If you originally created an object using the `v1beta1` version of its
|
||||
API, you can later read, update, or delete that object
|
||||
using either the `v1beta1` or the `v1` API version.
|
||||
|
||||
### API changes
|
||||
|
||||
Any system that is successful needs to grow and change as new use cases emerge or existing ones change.
|
||||
Therefore, Kubernetes has designed the Kubernetes API to continuously change and grow.
|
||||
The Kubernetes project aims to _not_ break compatibility with existing clients, and to maintain that
|
||||
compatibility for a length of time so that other projects have an opportunity to adapt.
|
||||
|
||||
In general, new API resources and new resource fields can be added often and frequently.
|
||||
Elimination of resources or fields requires following the
|
||||
[API deprecation policy](/docs/reference/using-api/deprecation-policy/).
|
||||
|
||||
Kubernetes makes a strong commitment to maintain compatibility for official Kubernetes APIs
|
||||
once they reach general availability (GA), typically at API version `v1`. Additionally,
|
||||
Kubernetes keeps compatibility even for _beta_ API versions wherever feasible:
|
||||
if you adopt a beta API you can continue to interact with your cluster using that API,
|
||||
even after the feature goes stable.
|
||||
|
||||
{{< note >}}
|
||||
Although Kubernetes also aims to maintain compatibility for _alpha_ APIs versions, in some
|
||||
circumstances this is not possible. If you use any alpha API versions, check the release notes
|
||||
for Kubernetes when upgrading your cluster, in case the API did change.
|
||||
{{< /note >}}
|
||||
|
||||
Refer to [API versions reference](/docs/reference/using-api/#api-versioning)
|
||||
for more details on the API version level definitions.
|
||||
|
||||
|
||||
|
||||
## API Extension
|
||||
|
||||
The Kubernetes API can be extended in one of two ways:
|
||||
@@ -135,3 +153,5 @@ The Kubernetes API can be extended in one of two ways:
|
||||
how the cluster manages authentication and authorization for API access.
|
||||
- Learn about API endpoints, resource types and samples by reading
|
||||
[API Reference](/docs/reference/kubernetes-api/).
|
||||
- Learn about what constitutes a compatible change, and how to change the API, from
|
||||
[API changes](https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#readme).
|
||||
|
||||
@@ -140,10 +140,11 @@ partition
|
||||
!partition
|
||||
```
|
||||
|
||||
The first example selects all resources with key equal to `environment` and value equal to `production` or `qa`.
|
||||
The second example selects all resources with key equal to `tier` and values other than `frontend` and `backend`, and all resources with no labels with the `tier` key.
|
||||
The third example selects all resources including a label with key `partition`; no values are checked.
|
||||
The fourth example selects all resources without a label with key `partition`; no values are checked.
|
||||
* The first example selects all resources with key equal to `environment` and value equal to `production` or `qa`.
|
||||
* The second example selects all resources with key equal to `tier` and values other than `frontend` and `backend`, and all resources with no labels with the `tier` key.
|
||||
* The third example selects all resources including a label with key `partition`; no values are checked.
|
||||
* The fourth example selects all resources without a label with key `partition`; no values are checked.
|
||||
|
||||
Similarly the comma separator acts as an _AND_ operator. So filtering resources with a `partition` key (no matter the value) and with `environment` different than `qa` can be achieved using `partition,environment notin (qa)`.
|
||||
The _set-based_ label selector is a general form of equality since `environment=production` is equivalent to `environment in (production)`; similarly for `!=` and `notin`.
|
||||
|
||||
|
||||
@@ -32,15 +32,15 @@ You add a taint to a node using [kubectl taint](/docs/reference/generated/kubect
|
||||
For example,
|
||||
|
||||
```shell
|
||||
kubectl taint nodes node1 key=value:NoSchedule
|
||||
kubectl taint nodes node1 key1=value1:NoSchedule
|
||||
```
|
||||
|
||||
places a taint on node `node1`. The taint has key `key`, value `value`, and taint effect `NoSchedule`.
|
||||
places a taint on node `node1`. The taint has key `key1`, value `value1`, and taint effect `NoSchedule`.
|
||||
This means that no pod will be able to schedule onto `node1` unless it has a matching toleration.
|
||||
|
||||
To remove the taint added by the command above, you can run:
|
||||
```shell
|
||||
kubectl taint nodes node1 key=value:NoSchedule-
|
||||
kubectl taint nodes node1 key1=value1:NoSchedule-
|
||||
```
|
||||
|
||||
You specify a toleration for a pod in the PodSpec. Both of the following tolerations "match" the
|
||||
@@ -49,15 +49,15 @@ to schedule onto `node1`:
|
||||
|
||||
```yaml
|
||||
tolerations:
|
||||
- key: "key"
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "value"
|
||||
value: "value1"
|
||||
effect: "NoSchedule"
|
||||
```
|
||||
|
||||
```yaml
|
||||
tolerations:
|
||||
- key: "key"
|
||||
- key: "key1"
|
||||
operator: "Exists"
|
||||
effect: "NoSchedule"
|
||||
```
|
||||
@@ -80,7 +80,7 @@ There are two special cases:
|
||||
An empty `key` with operator `Exists` matches all keys, values and effects which means this
|
||||
will tolerate everything.
|
||||
|
||||
An empty `effect` matches all effects with key `key`.
|
||||
An empty `effect` matches all effects with key `key1`.
|
||||
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
@@ -13,9 +13,8 @@ Unlike other types of controllers which run as part of the `kube-controller-mana
|
||||
are not started automatically with a cluster. Use this page to choose the ingress controller implementation
|
||||
that best fits your cluster.
|
||||
|
||||
Kubernetes as a project currently supports and maintains [GCE](https://git.k8s.io/ingress-gce/README.md) and
|
||||
[nginx](https://git.k8s.io/ingress-nginx/README.md) controllers.
|
||||
|
||||
Kubernetes as a project supports and maintains [AWS](https://github.com/kubernetes-sigs/aws-load-balancer-controller#readme), [GCE](https://git.k8s.io/ingress-gce/README.md#readme), and
|
||||
[nginx](https://git.k8s.io/ingress-nginx/README.md#readme) ingress controllers.
|
||||
|
||||
|
||||
<!-- body -->
|
||||
@@ -24,31 +23,31 @@ Kubernetes as a project currently supports and maintains [GCE](https://git.k8s.i
|
||||
|
||||
{{% thirdparty-content %}}
|
||||
|
||||
* [AKS Application Gateway Ingress Controller](https://github.com/Azure/application-gateway-kubernetes-ingress) is an ingress controller that enables ingress to [AKS clusters](https://docs.microsoft.com/azure/aks/kubernetes-walkthrough-portal) using the [Azure Application Gateway](https://docs.microsoft.com/azure/application-gateway/overview).
|
||||
* [Ambassador](https://www.getambassador.io/) API Gateway is an [Envoy](https://www.envoyproxy.io) based ingress
|
||||
controller with [community](https://www.getambassador.io/docs) or
|
||||
[commercial](https://www.getambassador.io/pro/) support from [Datawire](https://www.datawire.io/).
|
||||
* [AppsCode Inc.](https://appscode.com) offers support and maintenance for the most widely used [HAProxy](https://www.haproxy.org/) based ingress controller [Voyager](https://appscode.com/products/voyager).
|
||||
* [AWS ALB Ingress Controller](https://github.com/kubernetes-sigs/aws-alb-ingress-controller) enables ingress using the [AWS Application Load Balancer](https://aws.amazon.com/elasticloadbalancing/).
|
||||
* [Contour](https://projectcontour.io/) is an [Envoy](https://www.envoyproxy.io/) based ingress controller
|
||||
provided and supported by VMware.
|
||||
* Citrix provides an [Ingress Controller](https://github.com/citrix/citrix-k8s-ingress-controller) for its hardware (MPX), virtualized (VPX) and [free containerized (CPX) ADC](https://www.citrix.com/products/citrix-adc/cpx-express.html) for [baremetal](https://github.com/citrix/citrix-k8s-ingress-controller/tree/master/deployment/baremetal) and [cloud](https://github.com/citrix/citrix-k8s-ingress-controller/tree/master/deployment) deployments.
|
||||
* F5 Networks provides [support and maintenance](https://support.f5.com/csp/article/K86859508)
|
||||
for the [F5 BIG-IP Container Ingress Services for Kubernetes](https://clouddocs.f5.com/containers/latest/userguide/kubernetes/).
|
||||
* [Gloo](https://gloo.solo.io) is an open-source ingress controller based on [Envoy](https://www.envoyproxy.io) which offers API Gateway functionality with enterprise support from [solo.io](https://www.solo.io).
|
||||
* [HAProxy Ingress](https://haproxy-ingress.github.io) is a highly customizable community-driven ingress controller for HAProxy.
|
||||
* [HAProxy Technologies](https://www.haproxy.com/) offers support and maintenance for the [HAProxy Ingress Controller for Kubernetes](https://github.com/haproxytech/kubernetes-ingress). See the [official documentation](https://www.haproxy.com/documentation/hapee/1-9r1/traffic-management/kubernetes-ingress-controller/).
|
||||
* [Istio](https://istio.io/) based ingress controller
|
||||
[Control Ingress Traffic](https://istio.io/docs/tasks/traffic-management/ingress/).
|
||||
* [Kong](https://konghq.com/) offers [community](https://discuss.konghq.com/c/kubernetes) or
|
||||
[commercial](https://konghq.com/kong-enterprise/) support and maintenance for the
|
||||
[Kong Ingress Controller for Kubernetes](https://github.com/Kong/kubernetes-ingress-controller).
|
||||
* [NGINX, Inc.](https://www.nginx.com/) offers support and maintenance for the
|
||||
[NGINX Ingress Controller for Kubernetes](https://www.nginx.com/products/nginx/kubernetes-ingress-controller).
|
||||
* [Skipper](https://opensource.zalando.com/skipper/kubernetes/ingress-controller/) HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress, designed as a library to build your custom proxy
|
||||
* [Traefik](https://github.com/traefik/traefik) is a fully featured ingress controller
|
||||
([Let's Encrypt](https://letsencrypt.org), secrets, http2, websocket), and it also comes with commercial
|
||||
support by [Traefik Labs](https://traefik.io).
|
||||
* [AKS Application Gateway Ingress Controller](https://azure.github.io/application-gateway-kubernetes-ingress/) is an ingress controller that configures the [Azure Application Gateway](https://docs.microsoft.com/azure/application-gateway/overview).
|
||||
* [Ambassador](https://www.getambassador.io/) API Gateway is an [Envoy](https://www.envoyproxy.io)-based ingress
|
||||
controller.
|
||||
* The [Citrix ingress controller](https://github.com/citrix/citrix-k8s-ingress-controller#readme) works with
|
||||
Citrix Application Delivery Controller.
|
||||
* [Contour](https://projectcontour.io/) is an [Envoy](https://www.envoyproxy.io/) based ingress controller.
|
||||
* F5 BIG-IP [Container Ingress Services for Kubernetes](https://clouddocs.f5.com/containers/latest/userguide/kubernetes/)
|
||||
lets you use an Ingress to configure F5 BIG-IP virtual servers.
|
||||
* [Gloo](https://gloo.solo.io) is an open-source ingress controller based on [Envoy](https://www.envoyproxy.io),
|
||||
which offers API gateway functionality.
|
||||
* [HAProxy Ingress](https://haproxy-ingress.github.io/) is an ingress controller for
|
||||
[HAProxy](http://www.haproxy.org/#desc).
|
||||
* The [HAProxy Ingress Controller for Kubernetes](https://github.com/haproxytech/kubernetes-ingress#readme)
|
||||
is also an ingress controller for [HAProxy](http://www.haproxy.org/#desc).
|
||||
* [Istio Ingress](https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress/)
|
||||
is an [Istio](https://istio.io/) based ingress controller.
|
||||
* The [Kong Ingress Controller for Kubernetes](https://github.com/Kong/kubernetes-ingress-controller#readme)
|
||||
is an ingress controller driving [Kong Gateway](https://konghq.com/kong/).
|
||||
* The [NGINX Ingress Controller for Kubernetes](https://www.nginx.com/products/nginx/kubernetes-ingress-controller)
|
||||
works with the [NGINX](https://www.nginx.com/resources/glossary/nginx/) webserver (as a proxy).
|
||||
* [Skipper](https://opensource.zalando.com/skipper/kubernetes/ingress-controller/) HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress, designed as a library to build your custom proxy.
|
||||
* The [Traefik Kubernetes Ingress provider](https://doc.traefik.io/traefik/providers/kubernetes-ingress/) is an
|
||||
ingress controller for the [Traefik](https://traefik.io/traefik/) proxy.
|
||||
* [Voyager](https://appscode.com/products/voyager) is an ingress controller for
|
||||
[HAProxy](http://www.haproxy.org/#desc).
|
||||
|
||||
## Using multiple Ingress controllers
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ different purposes:
|
||||
[downwardAPI](/docs/concepts/storage/volumes/#downwardapi),
|
||||
[secret](/docs/concepts/storage/volumes/#secret): inject different
|
||||
kinds of Kubernetes data into a Pod
|
||||
- [CSI ephemeral volumes](#csi-ephemeral-volume):
|
||||
- [CSI ephemeral volumes](#csi-ephemeral-volumes):
|
||||
similar to the previous volume kinds, but provided by special
|
||||
[CSI drivers](https://github.com/container-storage-interface/spec/blob/master/spec.md)
|
||||
which specifically [support this feature](https://kubernetes-csi.github.io/docs/drivers.html)
|
||||
|
||||
@@ -172,14 +172,18 @@ spec:
|
||||
# The pod template ends here
|
||||
```
|
||||
|
||||
Modifying the pod template or switching to a new pod template has no effect on the
|
||||
Pods that already exist. Pods do not receive template updates directly. Instead,
|
||||
a new Pod is created to match the revised pod template.
|
||||
Modifying the pod template or switching to a new pod template has no direct effect
|
||||
on the Pods that already exist. If you change the pod template for a workload
|
||||
resource, that resource needs to create replacement Pods that use the updated template.
|
||||
|
||||
For example, the deployment controller ensures that the running Pods match the current
|
||||
pod template for each Deployment object. If the template is updated, the Deployment has
|
||||
to remove the existing Pods and create new Pods based on the updated template. Each workload
|
||||
resource implements its own rules for handling changes to the Pod template.
|
||||
For example, the StatefulSet controller ensures that the running Pods match the current
|
||||
pod template for each StatefulSet object. If you edit the StatefulSet to change its pod
|
||||
template, the StatefulSet starts to create new Pods based on the updated template.
|
||||
Eventually, all of the old Pods are replaced with new Pods, and the update is complete.
|
||||
|
||||
Each workload resource implements its own rules for handling changes to the Pod template.
|
||||
If you want to read more about StatefulSet specifically, read
|
||||
[Update strategy](/docs/tutorials/stateful-application/basic-stateful-set/#updating-statefulsets) in the StatefulSet Basics tutorial.
|
||||
|
||||
On Nodes, the {{< glossary_tooltip term_id="kubelet" text="kubelet" >}} does not
|
||||
directly observe or manage any of the details around pod templates and updates; those
|
||||
|
||||
@@ -282,7 +282,33 @@ from the OAuth2 [token response](https://openid.net/specs/openid-connect-core-1_
|
||||
as a bearer token. See [above](#putting-a-bearer-token-in-a-request) for how the token
|
||||
is included in a request.
|
||||
|
||||
|
||||
{{< mermaid >}}
|
||||
sequenceDiagram
|
||||
participant user as User
|
||||
participant idp as Identity Provider
|
||||
participant kube as Kubectl
|
||||
participant api as API Server
|
||||
|
||||
user ->> idp: 1. Login to IdP
|
||||
activate idp
|
||||
idp -->> user: 2. Provide access_token,<br>id_token, and refresh_token
|
||||
deactivate idp
|
||||
activate user
|
||||
user ->> kube: 3. Call Kubectl<br>with --token being the id_token<br>OR add tokens to .kube/config
|
||||
deactivate user
|
||||
activate kube
|
||||
kube ->> api: 4. Authorization: Bearer...
|
||||
deactivate kube
|
||||
activate api
|
||||
api ->> api: 5. Is JWT signature valid?
|
||||
api ->> api: 6. Has the JWT expired? (iat+exp)
|
||||
api ->> api: 7. User authorized?
|
||||
api -->> kube: 8. Authorized: Perform<br>action and return result
|
||||
deactivate api
|
||||
activate kube
|
||||
kube --x user: 9. Return result
|
||||
deactivate kube
|
||||
{{< /mermaid >}}
|
||||
|
||||
1. Login to your identity provider
|
||||
2. Your identity provider will provide you with an `access_token`, `id_token` and a `refresh_token`
|
||||
@@ -328,7 +354,7 @@ tokens on behalf of another.
|
||||
Kubernetes does not provide an OpenID Connect Identity Provider.
|
||||
You can use an existing public OpenID Connect Identity Provider (such as Google, or
|
||||
[others](https://connect2id.com/products/nimbus-oauth-openid-connect-sdk/openid-connect-providers)).
|
||||
Or, you can run your own Identity Provider, such as CoreOS [dex](https://github.com/coreos/dex),
|
||||
Or, you can run your own Identity Provider, such as [dex](https://dexidp.io/),
|
||||
[Keycloak](https://github.com/keycloak/keycloak),
|
||||
CloudFoundry [UAA](https://github.com/cloudfoundry/uaa), or
|
||||
Tremolo Security's [OpenUnison](https://github.com/tremolosecurity/openunison).
|
||||
@@ -339,7 +365,7 @@ For an identity provider to work with Kubernetes it must:
|
||||
2. Run in TLS with non-obsolete ciphers
|
||||
3. Have a CA signed certificate (even if the CA is not a commercial CA or is self signed)
|
||||
|
||||
A note about requirement #3 above, requiring a CA signed certificate. If you deploy your own identity provider (as opposed to one of the cloud providers like Google or Microsoft) you MUST have your identity provider's web server certificate signed by a certificate with the `CA` flag set to `TRUE`, even if it is self signed. This is due to GoLang's TLS client implementation being very strict to the standards around certificate validation. If you don't have a CA handy, you can use [this script](https://github.com/coreos/dex/blob/1ee5920c54f5926d6468d2607c728b71cfe98092/examples/k8s/gencert.sh) from the CoreOS team to create a simple CA and a signed certificate and key pair.
|
||||
A note about requirement #3 above, requiring a CA signed certificate. If you deploy your own identity provider (as opposed to one of the cloud providers like Google or Microsoft) you MUST have your identity provider's web server certificate signed by a certificate with the `CA` flag set to `TRUE`, even if it is self signed. This is due to GoLang's TLS client implementation being very strict to the standards around certificate validation. If you don't have a CA handy, you can use [this script](https://github.com/dexidp/dex/blob/master/examples/k8s/gencert.sh) from the Dex team to create a simple CA and a signed certificate and key pair.
|
||||
Or you can use [this similar script](https://raw.githubusercontent.com/TremoloSecurity/openunison-qs-kubernetes/master/src/main/bash/makessl.sh) that generates SHA256 certs with a longer life and larger key size.
|
||||
|
||||
Setup instructions for specific systems:
|
||||
|
||||
@@ -90,6 +90,13 @@ kubectl apply -f ./my1.yaml -f ./my2.yaml # create from multiple files
|
||||
kubectl apply -f ./dir # create resource(s) in all manifest files in dir
|
||||
kubectl apply -f https://git.io/vPieo # create resource(s) from url
|
||||
kubectl create deployment nginx --image=nginx # start a single instance of nginx
|
||||
|
||||
# create a Job which prints "Hello World"
|
||||
kubectl create job hello --image=busybox -- echo "Hello World"
|
||||
|
||||
# create a CronJob that prints "Hello World" every minute
|
||||
kubectl create cronjob hello --image=busybox --schedule="*/1 * * * *" -- echo "Hello World"
|
||||
|
||||
kubectl explain pods # get the documentation for pod manifests
|
||||
|
||||
# Create multiple YAML objects from stdin
|
||||
|
||||
@@ -32,7 +32,7 @@ The cluster that `kubeadm init` and `kubeadm join` set up should be:
|
||||
- `kubeadm init`
|
||||
- `export KUBECONFIG=/etc/kubernetes/admin.conf`
|
||||
- `kubectl apply -f <network-of-choice.yaml>`
|
||||
- `kubeadm join --token <token> <master-ip>:<master-port>`
|
||||
- `kubeadm join --token <token> <endpoint>:<port>`
|
||||
- **Extendable**:
|
||||
- It should _not_ favor any particular network provider. Configuring the cluster network is out-of-scope
|
||||
- It should provide the possibility to use a config file for customizing various parameters
|
||||
@@ -206,7 +206,7 @@ Please note that:
|
||||
|
||||
1. All images will be pulled from k8s.gcr.io by default. See [using custom images](/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images) for customizing the image repository
|
||||
2. In case of kubeadm is executed in the `--dry-run` mode, static Pods files are written in a temporary folder
|
||||
3. Static Pod manifest generation for master components can be invoked individually with the [`kubeadm init phase control-plane all`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-control-plane) command
|
||||
3. Static Pod manifest generation for control plane components can be invoked individually with the [`kubeadm init phase control-plane all`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-control-plane) command
|
||||
|
||||
#### API server
|
||||
|
||||
@@ -344,7 +344,7 @@ state and make new decisions based on that data.
|
||||
Please note that:
|
||||
|
||||
1. Before saving the ClusterConfiguration, sensitive information like the token is stripped from the configuration
|
||||
2. Upload of master configuration can be invoked individually with the [`kubeadm init phase upload-config`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-upload-config) command
|
||||
2. Upload of control plane node configuration can be invoked individually with the [`kubeadm init phase upload-config`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-upload-config) command
|
||||
|
||||
### Mark the node as control-plane
|
||||
|
||||
@@ -355,7 +355,7 @@ As soon as the control plane is available, kubeadm executes following actions:
|
||||
|
||||
Please note that:
|
||||
|
||||
1. Mark control-plane phase phase can be invoked individually with the [`kubeadm init phase mark-control-plane`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-mark-master) command
|
||||
1. Mark control-plane phase phase can be invoked individually with the [`kubeadm init phase mark-control-plane`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-mark-control-plane) command
|
||||
|
||||
### Configure TLS-Bootstrapping for node joining
|
||||
|
||||
@@ -415,7 +415,7 @@ Additionally it creates a Role and a RoleBinding granting access to the ConfigMa
|
||||
|
||||
Please note that:
|
||||
|
||||
1. The access to the `cluster-info` ConfigMap _is not_ rate-limited. This may or may not be a problem if you expose your master
|
||||
1. The access to the `cluster-info` ConfigMap _is not_ rate-limited. This may or may not be a problem if you expose your cluster's API server
|
||||
to the internet; worst-case scenario here is a DoS attack where an attacker uses all the in-flight requests the kube-apiserver
|
||||
can handle to serving the `cluster-info` ConfigMap.
|
||||
|
||||
@@ -430,8 +430,8 @@ Please note that:
|
||||
|
||||
A ServiceAccount for `kube-proxy` is created in the `kube-system` namespace; then kube-proxy is deployed as a DaemonSet:
|
||||
|
||||
- The credentials (`ca.crt` and `token`) to the master come from the ServiceAccount
|
||||
- The location of the master comes from a ConfigMap
|
||||
- The credentials (`ca.crt` and `token`) to the control plane come from the ServiceAccount
|
||||
- The location (URL) of the API server comes from a ConfigMap
|
||||
- The `kube-proxy` ServiceAccount is bound to the privileges in the `system:node-proxier` ClusterRole
|
||||
|
||||
#### DNS
|
||||
|
||||
@@ -33,7 +33,7 @@ support running as a pool of interchangable resources, replicated per
|
||||
component.
|
||||
|
||||
When you deploy a cluster control plane, place replicas of
|
||||
control plane components across multple failure zones. If availability is
|
||||
control plane components across multiple failure zones. If availability is
|
||||
an important concern, select at least three failure zones and replicate
|
||||
each individual control plane component (API server, scheduler, etcd,
|
||||
cluster controller manager) across at least three failure zones.
|
||||
@@ -111,7 +111,7 @@ see [Allowed topologies](/docs/concepts/storage/storage-classes/#allowed-topolog
|
||||
## Networking
|
||||
|
||||
By itself, Kubernetes does not include zone-aware networking. You can use a
|
||||
[network plugin](docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)
|
||||
[network plugin](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/)
|
||||
to configure cluster networking, and that network solution might have zone-specific
|
||||
elements. For example, if your cloud provider supports Services with
|
||||
`type=LoadBalancer`, the load balancer might only send traffic to Pods running in the
|
||||
@@ -126,7 +126,7 @@ of different failure zones, does vary depending on exactly how your cluster is s
|
||||
## Fault recovery
|
||||
|
||||
When you set up your cluster, you might also need to consider whether and how
|
||||
your setup can restore service if all of the failure zones in a region go
|
||||
your setup can restore service if all the failure zones in a region go
|
||||
off-line at the same time. For example, do you rely on there being at least
|
||||
one node able to run Pods in a zone?
|
||||
Make sure that any cluster-critical repair work does not rely
|
||||
|
||||
+1
-1
@@ -68,7 +68,7 @@ spec:
|
||||
command:
|
||||
- powershell.exe
|
||||
- -command
|
||||
- "<#code used from https://gist.github.com/wagnerandrade/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ; ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus) } ; "
|
||||
- "<#code used from https://gist.github.com/19WAS85/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ; ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus) } ; "
|
||||
nodeSelector:
|
||||
kubernetes.io/os: windows
|
||||
```
|
||||
|
||||
+2
-2
@@ -330,8 +330,8 @@ seconds. Minimum value is 1.
|
||||
* `timeoutSeconds`: Number of seconds after which the probe times out. Defaults
|
||||
to 1 second. Minimum value is 1.
|
||||
* `successThreshold`: Minimum consecutive successes for the probe to be
|
||||
considered successful after having failed. Defaults to 1. Must be 1 for
|
||||
liveness. Minimum value is 1.
|
||||
considered successful after having failed. Defaults to 1. Must be 1 for liveness
|
||||
and startup Probes. Minimum value is 1.
|
||||
* `failureThreshold`: When a probe fails, Kubernetes will
|
||||
try `failureThreshold` times before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready.
|
||||
Defaults to 3. Minimum value is 1.
|
||||
|
||||
@@ -15,8 +15,9 @@ weight: 30
|
||||
|
||||
This page shows how to run a replicated stateful application using a
|
||||
[StatefulSet](/docs/concepts/workloads/controllers/statefulset/) controller.
|
||||
The example is a MySQL single-master topology with multiple slaves running
|
||||
asynchronous replication.
|
||||
This application is a replicated MySQL database. The example topology has a
|
||||
single primary server and multiple replicas, using asynchronous row-based
|
||||
replication.
|
||||
|
||||
{{< note >}}
|
||||
**This is not a production configuration**. MySQL settings remain on insecure defaults to keep the focus
|
||||
@@ -69,9 +70,9 @@ kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml
|
||||
```
|
||||
|
||||
This ConfigMap provides `my.cnf` overrides that let you independently control
|
||||
configuration on the MySQL master and slaves.
|
||||
In this case, you want the master to be able to serve replication logs to slaves
|
||||
and you want slaves to reject any writes that don't come via replication.
|
||||
configuration on the primary MySQL server and replicas.
|
||||
In this case, you want the primary server to be able to serve replication logs to replicas
|
||||
and you want replicas to reject any writes that don't come via replication.
|
||||
|
||||
There's nothing special about the ConfigMap itself that causes different
|
||||
portions to apply to different Pods.
|
||||
@@ -96,12 +97,12 @@ cluster and namespace.
|
||||
|
||||
The Client Service, called `mysql-read`, is a normal Service with its own
|
||||
cluster IP that distributes connections across all MySQL Pods that report
|
||||
being Ready. The set of potential endpoints includes the MySQL master and all
|
||||
slaves.
|
||||
being Ready. The set of potential endpoints includes the primary MySQL server and all
|
||||
replicas.
|
||||
|
||||
Note that only read queries can use the load-balanced Client Service.
|
||||
Because there is only one MySQL master, clients should connect directly to the
|
||||
MySQL master Pod (through its DNS entry within the Headless Service) to execute
|
||||
Because there is only one primary MySQL server, clients should connect directly to the
|
||||
primary MySQL Pod (through its DNS entry within the Headless Service) to execute
|
||||
writes.
|
||||
|
||||
### StatefulSet
|
||||
@@ -167,33 +168,33 @@ This translates the unique, stable identity provided by the StatefulSet
|
||||
controller into the domain of MySQL server IDs, which require the same
|
||||
properties.
|
||||
|
||||
The script in the `init-mysql` container also applies either `master.cnf` or
|
||||
`slave.cnf` from the ConfigMap by copying the contents into `conf.d`.
|
||||
Because the example topology consists of a single MySQL master and any number of
|
||||
slaves, the script simply assigns ordinal `0` to be the master, and everyone
|
||||
else to be slaves.
|
||||
The script in the `init-mysql` container also applies either `primary.cnf` or
|
||||
`replica.cnf` from the ConfigMap by copying the contents into `conf.d`.
|
||||
Because the example topology consists of a single primary MySQL server and any number of
|
||||
replicas, the script simply assigns ordinal `0` to be the primary server, and everyone
|
||||
else to be replicas.
|
||||
Combined with the StatefulSet controller's
|
||||
[deployment order guarantee](/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees/),
|
||||
this ensures the MySQL master is Ready before creating slaves, so they can begin
|
||||
this ensures the primary MySQL server is Ready before creating replicas, so they can begin
|
||||
replicating.
|
||||
|
||||
### Cloning existing data
|
||||
|
||||
In general, when a new Pod joins the set as a slave, it must assume the MySQL
|
||||
master might already have data on it. It also must assume that the replication
|
||||
In general, when a new Pod joins the set as a replica, it must assume the primary MySQL
|
||||
server might already have data on it. It also must assume that the replication
|
||||
logs might not go all the way back to the beginning of time.
|
||||
These conservative assumptions are the key to allow a running StatefulSet
|
||||
to scale up and down over time, rather than being fixed at its initial size.
|
||||
|
||||
The second Init Container, named `clone-mysql`, performs a clone operation on
|
||||
a slave Pod the first time it starts up on an empty PersistentVolume.
|
||||
a replica Pod the first time it starts up on an empty PersistentVolume.
|
||||
That means it copies all existing data from another running Pod,
|
||||
so its local state is consistent enough to begin replicating from the master.
|
||||
so its local state is consistent enough to begin replicating from the primary server.
|
||||
|
||||
MySQL itself does not provide a mechanism to do this, so the example uses a
|
||||
popular open-source tool called Percona XtraBackup.
|
||||
During the clone, the source MySQL server might suffer reduced performance.
|
||||
To minimize impact on the MySQL master, the script instructs each Pod to clone
|
||||
To minimize impact on the primary MySQL server, the script instructs each Pod to clone
|
||||
from the Pod whose ordinal index is one lower.
|
||||
This works because the StatefulSet controller always ensures Pod `N` is
|
||||
Ready before starting Pod `N+1`.
|
||||
@@ -206,15 +207,15 @@ server, and an `xtrabackup` container that acts as a
|
||||
[sidecar](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns).
|
||||
|
||||
The `xtrabackup` sidecar looks at the cloned data files and determines if
|
||||
it's necessary to initialize MySQL replication on the slave.
|
||||
it's necessary to initialize MySQL replication on the replica.
|
||||
If so, it waits for `mysqld` to be ready and then executes the
|
||||
`CHANGE MASTER TO` and `START SLAVE` commands with replication parameters
|
||||
extracted from the XtraBackup clone files.
|
||||
|
||||
Once a slave begins replication, it remembers its MySQL master and
|
||||
Once a replica begins replication, it remembers its primary MySQL server and
|
||||
reconnects automatically if the server restarts or the connection dies.
|
||||
Also, because slaves look for the master at its stable DNS name
|
||||
(`mysql-0.mysql`), they automatically find the master even if it gets a new
|
||||
Also, because replicas look for the primary server at its stable DNS name
|
||||
(`mysql-0.mysql`), they automatically find the primary server even if it gets a new
|
||||
Pod IP due to being rescheduled.
|
||||
|
||||
Lastly, after starting replication, the `xtrabackup` container listens for
|
||||
@@ -224,7 +225,7 @@ case the next Pod loses its PersistentVolumeClaim and needs to redo the clone.
|
||||
|
||||
## Sending client traffic
|
||||
|
||||
You can send test queries to the MySQL master (hostname `mysql-0.mysql`)
|
||||
You can send test queries to the primary MySQL server (hostname `mysql-0.mysql`)
|
||||
by running a temporary container with the `mysql:5.7` image and running the
|
||||
`mysql` client binary.
|
||||
|
||||
@@ -291,7 +292,7 @@ it running in another window so you can see the effects of the following steps.
|
||||
|
||||
## Simulating Pod and Node downtime
|
||||
|
||||
To demonstrate the increased availability of reading from the pool of slaves
|
||||
To demonstrate the increased availability of reading from the pool of replicas
|
||||
instead of a single server, keep the `SELECT @@server_id` loop from above
|
||||
running while you force a Pod out of the Ready state.
|
||||
|
||||
@@ -409,9 +410,9 @@ Now uncordon the Node to return it to a normal state:
|
||||
kubectl uncordon <node-name>
|
||||
```
|
||||
|
||||
## Scaling the number of slaves
|
||||
## Scaling the number of replicas
|
||||
|
||||
With MySQL replication, you can scale your read query capacity by adding slaves.
|
||||
With MySQL replication, you can scale your read query capacity by adding replicas.
|
||||
With StatefulSet, you can do this with a single command:
|
||||
|
||||
```shell
|
||||
|
||||
@@ -11,6 +11,7 @@ spec:
|
||||
containers:
|
||||
- name: hello
|
||||
image: busybox
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- /bin/sh
|
||||
- -c
|
||||
|
||||
@@ -5,12 +5,12 @@ metadata:
|
||||
labels:
|
||||
app: mysql
|
||||
data:
|
||||
master.cnf: |
|
||||
# Apply this config only on the master.
|
||||
primary.cnf: |
|
||||
# Apply this config only on the primary.
|
||||
[mysqld]
|
||||
log-bin
|
||||
slave.cnf: |
|
||||
# Apply this config only on slaves.
|
||||
replica.cnf: |
|
||||
# Apply this config only on replicas.
|
||||
[mysqld]
|
||||
super-read-only
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ spec:
|
||||
app: mysql
|
||||
---
|
||||
# Client service for connecting to any MySQL instance for reads.
|
||||
# For writes, you must instead connect to the master: mysql-0.mysql.
|
||||
# For writes, you must instead connect to the primary: mysql-0.mysql.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
|
||||
@@ -29,9 +29,9 @@ spec:
|
||||
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
|
||||
# Copy appropriate conf.d files from config-map to emptyDir.
|
||||
if [[ $ordinal -eq 0 ]]; then
|
||||
cp /mnt/config-map/master.cnf /mnt/conf.d/
|
||||
cp /mnt/config-map/primary.cnf /mnt/conf.d/
|
||||
else
|
||||
cp /mnt/config-map/slave.cnf /mnt/conf.d/
|
||||
cp /mnt/config-map/replica.cnf /mnt/conf.d/
|
||||
fi
|
||||
volumeMounts:
|
||||
- name: conf
|
||||
@@ -47,7 +47,7 @@ spec:
|
||||
set -ex
|
||||
# Skip the clone if data already exists.
|
||||
[[ -d /var/lib/mysql/mysql ]] && exit 0
|
||||
# Skip the clone on master (ordinal index 0).
|
||||
# Skip the clone on primary (ordinal index 0).
|
||||
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
|
||||
ordinal=${BASH_REMATCH[1]}
|
||||
[[ $ordinal -eq 0 ]] && exit 0
|
||||
@@ -108,12 +108,12 @@ spec:
|
||||
# Determine binlog position of cloned data, if any.
|
||||
if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
|
||||
# XtraBackup already generated a partial "CHANGE MASTER TO" query
|
||||
# because we're cloning from an existing slave. (Need to remove the tailing semicolon!)
|
||||
# because we're cloning from an existing replica. (Need to remove the tailing semicolon!)
|
||||
cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
|
||||
# Ignore xtrabackup_binlog_info in this case (it's useless).
|
||||
rm -f xtrabackup_slave_info xtrabackup_binlog_info
|
||||
elif [[ -f xtrabackup_binlog_info ]]; then
|
||||
# We're cloning directly from master. Parse binlog position.
|
||||
# We're cloning directly from primary. Parse binlog position.
|
||||
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
|
||||
rm -f xtrabackup_binlog_info xtrabackup_slave_info
|
||||
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
|
||||
|
||||
Reference in New Issue
Block a user