Merge remote-tracking branch 'upstream/main' into dev-1.24

This commit is contained in:
Nate W
2022-02-07 11:03:19 -08:00
152 changed files with 3401 additions and 2108 deletions
@@ -27,6 +27,11 @@ The `kube-apiserver` process accepts an argument `--encryption-provider-config`
that controls how API data is encrypted in etcd. An example configuration
is provided below.
{{< caution >}}
**IMPORTANT:** For multi-master configurations (with two or more control plane nodes) the encryption configuration file must be the same!
Otherwise, the kube-apiserver can't decrypt data stored inside the key-value store.
{{< /caution >}}
## Understanding the encryption at rest configuration.
```yaml
@@ -100,4 +100,31 @@ shown in [the example](/docs/tasks/administer-cluster/dns-custom-nameservers/#ex
The `node-local-dns` ConfigMap can also be modified directly with the stubDomain configuration
in the Corefile format. Some cloud providers might not allow modifying `node-local-dns` ConfigMap directly.
In those cases, the `kube-dns` ConfigMap can be updated.
## Setting memory limits
node-local-dns pods use memory for storing cache entries and processing queries. Since they do not watch Kubernetes objects, the cluster size or the number of Services/Endpoints do not directly affect memory usage. Memory usage is influenced by the DNS query pattern.
From [CoreDNS docs](https://github.com/coredns/deployment/blob/master/kubernetes/Scaling_CoreDNS.md),
> The default cache size is 10000 entries, which uses about 30 MB when completely filled.
This would be the memory usage for each server block (if the cache gets completely filled).
Memory usage can be reduced by specifying smaller cache sizes.
The number of concurrent queries is linked to the memory demand, because each extra
goroutine used for handling a query requires an amount of memory. You can set an upper limit
using the `max_concurrent` option in the forward plugin.
If a node-local-dns pod attempts to use more memory than is available (because of total system
resources, or because of a configured
[resource limit](/docs/concepts/configuration/manage-resources-containers/)), the operating system
may shut down that pod's container.
If this happens, the container that is terminated (“OOMKilled”) does not clean up the custom
packet filtering rules that it previously added during startup.
The node-local-dns container should get restarted (since managed as part of a DaemonSet), but this
will lead to a brief DNS downtime each time that the container fails: the packet filtering rules direct
DNS queries to a local Pod that is unhealthy.
You can determine a suitable memory limit by running node-local-dns pods without a limit and
measuring the peak usage. You can also set up and use a
[VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler)
in _recommender mode_, and then check its recommendations.
@@ -70,7 +70,7 @@ The following sysctls are supported in the _safe_ set:
- `kernel.shm_rmid_forced`,
- `net.ipv4.ip_local_port_range`,
- `net.ipv4.tcp_syncookies`,
- `net.ipv4.ping_group_range` (since Kubernetes 1.18).
- `net.ipv4.ping_group_range` (since Kubernetes 1.18),
- `net.ipv4.ip_unprivileged_port_start` (since Kubernetes 1.22).
{{< note >}}
@@ -8,10 +8,12 @@ card:
---
<!-- overview -->
Many applications rely on configuration which is used during either application initialization or runtime.
Most of the times there is a requirement to adjust values assigned to configuration parameters.
ConfigMaps is the kubernetes way to inject application pods with configuration data.
ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. This page provides a series of usage examples demonstrating how to create ConfigMaps and configure Pods using data stored in ConfigMaps.
## {{% heading "prerequisites" %}}
@@ -290,10 +290,18 @@ To enable and use token request projection, you must specify each of the followi
command line arguments to `kube-apiserver`:
* `--service-account-issuer`
It can be used as the Identifier of the service account token issuer. You can specify the `--service-account-issuer` argument multiple times, this can be useful to enable a non-disruptive change of the issuer. When this flag is specified multiple times, the first is used to generate tokens and all are used to determine which issuers are accepted. You must be running running Kubernetes v1.22 or later to be able to specify `--service-account-issuer` multiple times.
* `--service-account-key-file`
File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If specified multiple times, tokens signed by any of the specified keys are considered valid by the Kubernetes API server.
* `--service-account-signing-key-file`
Path to the file that contains the current private key of the service account token issuer. The issuer signs issued ID tokens with this private key.
* `--api-audiences` (can be omitted)
The service account token authenticator validates that tokens used against the API are bound to at least one of these audiences. If `api-audiences` is specified multiple times, tokens for any of the specified audiences are considered valid by the Kubernetes API server. If the `--service-account-issuer` flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.
{{< /note >}}
The kubelet can also project a service account token into a Pod. You can
@@ -31,7 +31,7 @@ as Windows server containers, meaning that the version of the base images does n
to match that of the host. It is, however, recommended that you use the same base image
version as your Windows Server container workloads to ensure you do not have any unused
images taking up space on the node. HostProcess containers also support
[volume mounts](./create-hostprocess-pod#volume-mounts) within the container volume.
[volume mounts](#volume-mounts) within the container volume.
### When should I use a Windows HostProcess container?
@@ -73,19 +73,20 @@ documentation for more details.
These limitations are relevant for Kubernetes v{{< skew currentVersion >}}:
- HostProcess containers require containerd 1.6 or higher
{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}.
{{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}.
- HostProcess pods can only contain HostProcess containers. This is a current limitation
of the Windows OS; non-privileged Windows containers cannot share a vNIC with the host IP namespace.
of the Windows OS; non-privileged Windows containers cannot share a vNIC with the host IP namespace.
- HostProcess containers run as a process on the host and do not have any degree of
isolation other than resource constraints imposed on the HostProcess user account. Neither
filesystem or Hyper-V isolation are supported for HostProcess containers.
- Volume mounts are supported and are mounted under the container volume. See [Volume Mounts](#volume-mounts)
isolation other than resource constraints imposed on the HostProcess user account. Neither
filesystem or Hyper-V isolation are supported for HostProcess containers.
- Volume mounts are supported and are mounted under the container volume. See
[Volume Mounts](#volume-mounts)
- A limited set of host user accounts are available for HostProcess containers by default.
See [Choosing a User Account](#choosing-a-user-account).
- Resource limits (disk, memory, cpu count) are supported in the same fashion as processes
on the host.
on the host.
- Both Named pipe mounts and Unix domain sockets are **not** supported and should instead
be accessed via their path on the host (e.g. \\\\.\\pipe\\\*)
be accessed via their path on the host (e.g. \\\\.\\pipe\\\*)
## HostProcess Pod configuration requirements
@@ -17,15 +17,11 @@ You can use it to inspect and debug container runtimes and applications on a
Kubernetes node. `crictl` and its source are hosted in the
[cri-tools](https://github.com/kubernetes-sigs/cri-tools) repository.
## {{% heading "prerequisites" %}}
`crictl` requires a Linux operating system with a CRI runtime.
<!-- steps -->
## Installing crictl
@@ -41,27 +37,37 @@ of Kubernetes. Extract it and move it to a location on your system path, such as
The `crictl` command has several subcommands and runtime flags. Use
`crictl help` or `crictl <subcommand> help` for more details.
`crictl` connects to `unix:///var/run/dockershim.sock` by default. For other
runtimes, you can set the endpoint in multiple different ways:
You can set the endpoint for `crictl` by doing one of the following:
- By setting flags `--runtime-endpoint` and `--image-endpoint`
- By setting environment variables `CONTAINER_RUNTIME_ENDPOINT` and `IMAGE_SERVICE_ENDPOINT`
- By setting the endpoint in the config file `--config=/etc/crictl.yaml`
* Set the `--runtime-endpoint` and `--image-endpoint` flags.
* Set the `CONTAINER_RUNTIME_ENDPOINT` and `IMAGE_SERVICE_ENDPOINT` environment
variables.
* Set the endpoint in the configuration file `/etc/crictl.yaml`. To specify a
different file, use the `--config=PATH_TO_FILE` flag when you run `crictl`.
{{<note>}}
If you don't set an endpoint, `crictl` attempts to connect to a list of known
endpoints, which might result in an impact to performance.
{{</note>}}
You can also specify timeout values when connecting to the server and enable or
disable debugging, by specifying `timeout` or `debug` values in the configuration
file or using the `--timeout` and `--debug` command-line flags.
To view or edit the current configuration, view or edit the contents of `/etc/crictl.yaml`.
To view or edit the current configuration, view or edit the contents of
`/etc/crictl.yaml`. For example, the configuration when using the `containerd`
container runtime would be similar to this:
```shell
cat /etc/crictl.yaml
runtime-endpoint: unix:///var/run/dockershim.sock
image-endpoint: unix:///var/run/dockershim.sock
```
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
```
To learn more about `crictl`, refer to the [`crictl`
documentation](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md).
## Example crictl commands
The following examples show some `crictl` commands and example output.
@@ -348,64 +354,9 @@ CONTAINER ID IMAGE CREATED STATE
3e025dd50a72d busybox About a minute ago Running busybox 0
```
## {{% heading "whatsnext" %}}
* [Learn more about `crictl`](https://github.com/kubernetes-sigs/cri-tools).
* [Map `docker` CLI commands to `crictl`](/reference/tools/map-crictl-dockercli/).
<!-- discussion -->
See [kubernetes-sigs/cri-tools](https://github.com/kubernetes-sigs/cri-tools)
for more information.
## Mapping from docker cli to crictl
The exact versions for below mapping table are for docker cli v1.40 and crictl v1.19.0. Please note that the list is not exhaustive. For example, it doesn't include experimental commands of docker cli.
{{< note >}}
The output format of CRICTL is similar to Docker CLI, despite some missing columns for some CLI. Make sure to check output for the specific command if your script output parsing.
{{< /note >}}
### Retrieve Debugging Information
{{< table caption="mapping from docker cli to crictl - retrieve debugging information" >}}
docker cli | crictl | Description | Unsupported Features
-- | -- | -- | --
`attach` | `attach` | Attach to a running container | `--detach-keys`, `--sig-proxy`
`exec` | `exec` | Run a command in a running container | `--privileged`, `--user`, `--detach-keys`
`images` | `images` | List images |  
`info` | `info` | Display system-wide information |  
`inspect` | `inspect`, `inspecti` | Return low-level information on a container, image or task |  
`logs` | `logs` | Fetch the logs of a container | `--details`
`ps` | `ps` | List containers |  
`stats` | `stats` | Display a live stream of container(s) resource usage statistics | Column: NET/BLOCK I/O, PIDs
`version` | `version` | Show the runtime (Docker, ContainerD, or others) version information |  
{{< /table >}}
### Perform Changes
{{< table caption="mapping from docker cli to crictl - perform changes" >}}
docker cli | crictl | Description | Unsupported Features
-- | -- | -- | --
`create` | `create` | Create a new container |  
`kill` | `stop` (timeout = 0) | Kill one or more running container | `--signal`
`pull` | `pull` | Pull an image or a repository from a registry | `--all-tags`, `--disable-content-trust`
`rm` | `rm` | Remove one or more containers |  
`rmi` | `rmi` | Remove one or more images |  
`run` | `run` | Run a command in a new container |  
`start` | `start` | Start one or more stopped containers | `--detach-keys`
`stop` | `stop` | Stop one or more running containers |  
`update` | `update` | Update configuration of one or more containers | `--restart`, `--blkio-weight` and some other resource limit not supported by CRI.
{{< /table >}}
### Supported only in crictl
{{< table caption="mapping from docker cli to crictl - supported only in crictl" >}}
crictl | Description
-- | --
`imagefsinfo` | Return image filesystem info
`inspectp` | Display the status of one or more pods
`port-forward` | Forward local port to a pod
`pods` | List pods
`runp` | Run a new pod
`rmp` | Remove one or more pods
`stopp` | Stop one or more running pods
{{< /table >}}
<!-- discussion -->
@@ -321,7 +321,7 @@ object:
metric:
name: requests-per-second
describedObject:
apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
name: main-route
target:
@@ -367,7 +367,7 @@ spec:
metric:
name: requests-per-second
describedObject:
apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
name: main-route
target:
@@ -390,7 +390,7 @@ status:
metric:
name: requests-per-second
describedObject:
apiVersion: networking.k8s.io/v1beta1
apiVersion: networking.k8s.io/v1
kind: Ingress
name: main-route
current:
@@ -56,8 +56,9 @@ Kubernetes implements horizontal pod autoscaling as a control loop that runs int
(and the default interval is 15 seconds).
Once during each period, the controller manager queries the resource utilization against the
metrics specified in each HorizontalPodAutoscaler definition. The controller manager
obtains the metrics from either the resource metrics API (for per-pod resource metrics),
metrics specified in each HorizontalPodAutoscaler definition. The controller manager
finds the target resource defined by the `scaleTargetRef`,
then selects the pods based on the target resource's `.spec.selector` labels, and obtains the metrics from either the resource metrics API (for per-pod resource metrics),
or the custom metrics API (for all other metrics).
* For per-pod resource metrics (like CPU), the controller fetches the metrics
@@ -76,16 +76,11 @@ cat <<EOF | cfssl genkey - | cfssljson -bare server
"192.0.2.24",
"10.0.34.2"
],
"CN": "system:node:my-pod.my-namespace.pod.cluster.local",
"CN": "my-pod.my-namespace.pod.cluster.local",
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"O": "system:nodes"
}
]
}
}
EOF
```
@@ -93,13 +88,13 @@ EOF
Where `192.0.2.24` is the service's cluster IP,
`my-svc.my-namespace.svc.cluster.local` is the service's DNS name,
`10.0.34.2` is the pod's IP and `my-pod.my-namespace.pod.cluster.local`
is the pod's DNS name. You should see the following output:
is the pod's DNS name. You should see the output similar to:
```
2017/03/21 06:48:17 [INFO] generate received request
2017/03/21 06:48:17 [INFO] received CSR
2017/03/21 06:48:17 [INFO] generating key: ecdsa-256
2017/03/21 06:48:17 [INFO] encoded CSR
2022/02/01 11:45:32 [INFO] generate received request
2022/02/01 11:45:32 [INFO] received CSR
2022/02/01 11:45:32 [INFO] generating key: ecdsa-256
2022/02/01 11:45:32 [INFO] encoded CSR
```
This command generates two files; it generates `server.csr` containing the PEM
@@ -120,7 +115,7 @@ metadata:
name: my-svc.my-namespace
spec:
request: $(cat server.csr | base64 | tr -d '\n')
signerName: kubernetes.io/kubelet-serving
signerName: example.com/serving
usages:
- digital signature
- key encipherment
@@ -131,7 +126,7 @@ EOF
Notice that the `server.csr` file created in step 1 is base64 encoded
and stashed in the `.spec.request` field. We are also requesting a
certificate with the "digital signature", "key encipherment", and "server
auth" key usages, signed by the `kubernetes.io/kubelet-serving` signer.
auth" key usages, signed by an example `example.com/serving` signer.
A specific `signerName` must be requested.
View documentation for [supported signer names](/docs/reference/access-authn-authz/certificate-signing-requests/#signers)
for more information.
@@ -147,14 +142,16 @@ kubectl describe csr my-svc.my-namespace
Name: my-svc.my-namespace
Labels: <none>
Annotations: <none>
CreationTimestamp: Tue, 21 Mar 2017 07:03:51 -0700
CreationTimestamp: Tue, 01 Feb 2022 11:49:15 -0500
Requesting User: yourname@example.com
Signer: example.com/serving
Status: Pending
Subject:
Common Name: my-svc.my-namespace.svc.cluster.local
Common Name: my-pod.my-namespace.pod.cluster.local
Serial Number:
Subject Alternative Names:
DNS Names: my-svc.my-namespace.svc.cluster.local
DNS Names: my-pod.my-namespace.pod.cluster.local
my-svc.my-namespace.svc.cluster.local
IP Addresses: 192.0.2.24
10.0.34.2
Events: <none>
@@ -175,30 +172,136 @@ kubectl certificate approve my-svc.my-namespace
certificatesigningrequest.certificates.k8s.io/my-svc.my-namespace approved
```
## Download the Certificate and Use It
Once the CSR is signed and approved you should see the following:
You should now see the following:
```shell
kubectl get csr
```
```none
NAME AGE REQUESTOR CONDITION
my-svc.my-namespace 10m yourname@example.com Approved,Issued
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
my-svc.my-namespace 10m example.com/serving yourname@example.com <none> Approved
```
You can download the issued certificate and save it to a `server.crt` file
by running the following:
This means the certificate request has been approved and is waiting for the
requested signer to sign it.
## Sign the Certificate Signing Request
Next, you'll play the part of a certificate signer, issue the certificate, and upload it to the API.
A signer would typically watch the Certificate Signing Request API for objects with its `signerName`,
check that they have been approved, sign certificates for those requests,
and update the API object status with the issued certificate.
### Create a Certificate Authority
First, create a signing certificate by running the following:
```shell
cat <<EOF | cfssl gencert -initca - | cfssljson -bare ca
{
"CN": "My Example Signer",
"key": {
"algo": "rsa",
"size": 2048
}
}
EOF
```
You should see the output similar to:
```none
2022/02/01 11:50:39 [INFO] generating a new CA key and certificate from CSR
2022/02/01 11:50:39 [INFO] generate received request
2022/02/01 11:50:39 [INFO] received CSR
2022/02/01 11:50:39 [INFO] generating key: rsa-2048
2022/02/01 11:50:39 [INFO] encoded CSR
2022/02/01 11:50:39 [INFO] signed certificate with serial number 263983151013686720899716354349605500797834580472
```
This produces a certificate authority key file (`ca-key.pem`) and certificate (`ca.pem`).
### Issue a Certificate
{{< codenew file="tls/server-signing-config.json" >}}
Use a `server-signing-config.json` signing configuration and the certificate authority key file
and certificate to sign the certificate request:
```shell
kubectl get csr my-svc.my-namespace -o jsonpath='{.spec.request}' | \
base64 --decode | \
cfssl sign -ca ca.pem -ca-key ca-key.pem -config server-signing-config.json - | \
cfssljson -bare ca-signed-server
```
You should see the output similar to:
```
2022/02/01 11:52:26 [INFO] signed certificate with serial number 576048928624926584381415936700914530534472870337
```
This produces a signed serving certificate file, `ca-signed-server.pem`.
### Upload the Signed Certificate
Finally, populate the signed certificate in the API object's status:
```shell
kubectl get csr my-svc.my-namespace -o json | \
jq '.status.certificate = "'$(base64 ca-signed-server.pem | tr -d '\n')'"' | \
kubectl replace --raw /apis/certificates.k8s.io/v1/certificatesigningrequests/my-svc.my-namespace/status -f -
```
{{< note >}}
This uses the command line tool [jq](https://stedolan.github.io/jq/) to populate the base64-encoded content in the `.status.certificate` field.
If you do not have `jq`, you can also save the JSON output to a file, populate this field manually, and upload the resulting file.
{{< /note >}}
Once the CSR is approved and the signed certificate is uploaded you should see the following:
```shell
kubectl get csr
```
```none
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
my-svc.my-namespace 20m example.com/serving yourname@example.com <none> Approved,Issued
```
## Download the Certificate and Use It
Now, as the requesting user, you can download the issued certificate
and save it to a `server.crt` file by running the following:
```shell
kubectl get csr my-svc.my-namespace -o jsonpath='{.status.certificate}' \
| base64 --decode > server.crt
```
Now you can use `server.crt` and `server-key.pem` as the keypair to start
your HTTPS server.
Now you can populate `server.crt` and `server-key.pem` in a secret and mount
it into a pod to use as the keypair to start your HTTPS server:
```shell
kubectl create secret tls server --cert server.crt --key server-key.pem
```
```none
secret/server created
```
Finally, you can populate `ca.pem` in a configmap and use it as the trust root
to verify the serving certificate:
```shell
kubectl create configmap example-serving-ca --from-file ca.crt=ca.pem
```
```none
configmap/example-serving-ca created
```
## Approving Certificate Signing Requests