Merge main into dev-1.22 to keep in sync

This commit is contained in:
Christopher Negus
2021-07-09 18:19:13 +00:00
448 changed files with 21785 additions and 1882 deletions
@@ -83,8 +83,11 @@ As an example, you can find detailed information about how `kube-up.sh` sets
up logging for COS image on GCP in the corresponding
[`configure-helper` script](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh).
When using a **CRI container runtime**, the kubelet is responsible for rotating the logs and managing the logging directory structure. The kubelet
sends this information to the CRI container runtime and the runtime writes the container logs to the given location. The two kubelet flags `container-log-max-size` and `container-log-max-files` can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
When using a **CRI container runtime**, the kubelet is responsible for rotating the logs and managing the logging directory structure.
The kubelet sends this information to the CRI container runtime and the runtime writes the container logs to the given location.
The two kubelet parameters [`containerLogMaxSize` and `containerLogMaxFiles`](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
in [kubelet config file](/docs/tasks/administer-cluster/kubelet-config-file/)
can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
When you run [`kubectl logs`](/docs/reference/generated/kubectl/kubectl-commands#logs) as in
the basic logging example, the kubelet on the node handles the request and
@@ -1235,10 +1235,6 @@ for secret data, so that the secrets are not stored in the clear into {{< glossa
- A user who can create a Pod that uses a secret can also see the value of that secret. Even
if the API server policy does not allow that user to read the Secret, the user could
run a Pod which exposes the secret.
- Currently, anyone with root permission on any node can read _any_ secret from the API server,
by impersonating the kubelet. It is a planned feature to only send secrets to
nodes that actually require them, to restrict the impact of a root exploit on a
single node.
## {{% heading "whatsnext" %}}
@@ -77,6 +77,20 @@ the pull policy of any object after its initial creation.
When `imagePullPolicy` is defined without a specific value, it is also set to `Always`.
### ImagePullBackOff
When a kubelet starts creating containers for a Pod using a container runtime,
it might be possible the container is in [Waiting](/docs/concepts/workloads/pods/pod-lifecycle/#container-state-waiting)
state because of `ImagePullBackOff`.
The status `ImagePullBackOff` means that a container could not start because Kubernetes
could not pull a container image (for reasons such as invalid image name, or pulling
from a private registry without `imagePullSecret`). The `BackOff` part indicates
that Kubernetes will keep trying to pull the image, with an increasing back-off delay.
Kubernetes raises the delay between each attempt until it reaches a compiled-in limit,
which is 300 seconds (5 minutes).
## Multi-architecture images with image indexes
As well as providing binary images, a container registry can also serve a [container image index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). An image index can point to multiple [image manifests](https://github.com/opencontainers/image-spec/blob/master/manifest.md) for architecture-specific versions of a container. The idea is that you can have a name for an image (for example: `pause`, `example/mycontainer`, `kube-apiserver`) and allow different systems to fetch the right binary image for the machine architecture they are using.
@@ -51,8 +51,7 @@ Some of the things that you can use an operator to automate include:
* choosing a leader for a distributed application without an internal
member election process
What might an Operator look like in more detail? Here's an example in more
detail:
What might an Operator look like in more detail? Here's an example:
1. A custom resource named SampleDB, that you can configure into the cluster.
2. A Deployment that makes sure a Pod is running that contains the
@@ -124,6 +123,7 @@ Operator.
## {{% heading "whatsnext" %}}
* Read the {{< glossary_tooltip text="CNCF" term_id="cncf" >}} [Operator White Paper](https://github.com/cncf/tag-app-delivery/blob/eece8f7307f2970f46f100f51932db106db46968/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md).
* Learn more about [Custom Resources](/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
* Find ready-made operators on [OperatorHub.io](https://operatorhub.io/) to suit your use case
* [Publish](https://operatorhub.io/) your operator for other people to use
@@ -11,7 +11,8 @@ weight: 30
{{< feature-state for_k8s_version="v1.21" state="deprecated" >}}
PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25.
PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25. For more information on the deprecation,
see [PodSecurityPolicy Deprecation: Past, Present, and Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/).
Pod Security Policies enable fine-grained authorization of pod creation and
updates.
@@ -48,13 +49,12 @@ administrator to control the following:
## Enabling Pod Security Policies
Pod security policy control is implemented as an optional (but recommended)
[admission
controller](/docs/reference/access-authn-authz/admission-controllers/#podsecuritypolicy). PodSecurityPolicies
are enforced by [enabling the admission
Pod security policy control is implemented as an optional [admission
controller](/docs/reference/access-authn-authz/admission-controllers/#podsecuritypolicy).
PodSecurityPolicies are enforced by [enabling the admission
controller](/docs/reference/access-authn-authz/admission-controllers/#how-do-i-turn-on-an-admission-control-plug-in),
but doing so without authorizing any policies **will prevent any pods from being
created** in the cluster.
but doing so without authorizing any policies **will prevent any pods from being created** in the
cluster.
Since the pod security policy API (`policy/v1beta1/podsecuritypolicy`) is
enabled independently of the admission controller, for existing clusters it is
@@ -110,7 +110,11 @@ roleRef:
name: <role name>
apiGroup: rbac.authorization.k8s.io
subjects:
# Authorize specific service accounts:
# Authorize all service accounts in a namespace (recommended):
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccounts:<authorized namespace>
# Authorize specific service accounts (not recommended):
- kind: ServiceAccount
name: <authorized service account name>
namespace: <authorized pod namespace>
@@ -139,6 +143,40 @@ Examples](/docs/reference/access-authn-authz/rbac#role-binding-examples).
For a complete example of authorizing a PodSecurityPolicy, see
[below](#example).
### Recommended Practice
PodSecurityPolicy is being replaced by a new, simplified `PodSecurity` {{< glossary_tooltip
text="admission controller" term_id="admission-controller" >}}. For more details on this change, see
[PodSecurityPolicy Deprecation: Past, Present, and
Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/). Follow these
guidelines to simplify migration from PodSecurityPolicy to the new admission controller:
1. Limit your PodSecurityPolicies to the policies defined by the [Pod Security Standards](/docs/concepts/security/pod-security-standards):
- {{< example file="policy/privileged-psp.yaml" >}}Privileged{{< /example >}}
- {{< example file="policy/baseline-psp.yaml" >}}Baseline{{< /example >}}
- {{< example file="policy/restricted-psp.yaml" >}}Restricted{{< /example >}}
2. Only bind PSPs to entire namespaces, by using the `system:serviceaccounts:<namespace>` group
(where `<namespace>` is the target namespace). For example:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
# This cluster role binding allows all pods in the "development" namespace to use the baseline PSP.
kind: ClusterRoleBinding
metadata:
name: psp-baseline-namespaces
roleRef:
kind: ClusterRole
name: psp-baseline
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
name: system:serviceaccounts:development
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts:canary
apiGroup: rbac.authorization.k8s.io
```
### Troubleshooting
@@ -661,8 +699,10 @@ Refer to the [Sysctl documentation](
## {{% heading "whatsnext" %}}
- See [PodSecurityPolicy Deprecation: Past, Present, and
Future](/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) to learn about
the future of pod security policy.
- See [Pod Security Standards](/docs/concepts/security/pod-security-standards/) for policy recommendations.
- Refer to [Pod Security Policy Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podsecuritypolicy-v1beta1-policy) for the api details.
@@ -193,7 +193,7 @@ resources based on the filesystems on the node.
If the node has a dedicated `imagefs` filesystem for container runtimes to use,
the kubelet does the following:
* If the `nodefs` filesystem meets the eviction threshlds, the kubelet garbage collects
* If the `nodefs` filesystem meets the eviction thresholds, the kubelet garbage collects
dead pods and containers.
* If the `imagefs` filesystem meets the eviction thresholds, the kubelet
deletes all unused images.
@@ -266,9 +266,23 @@ This ensures that DaemonSet pods are never evicted due to these problems.
## Taint Nodes by Condition
The node lifecycle controller automatically creates taints corresponding to
Node conditions with `NoSchedule` effect.
Similarly the scheduler does not check Node conditions; instead the scheduler checks taints. This assures that Node conditions don't affect what's scheduled onto the Node. The user can choose to ignore some of the Node's problems (represented as Node conditions) by adding appropriate Pod tolerations.
The control plane, using the node {{<glossary_tooltip text="controller" term_id="controller">}},
automatically creates taints with a `NoSchedule` effect for [node conditions](/docs/concepts/scheduling-eviction/pod-eviction#node-conditions).
The scheduler checks taints, not node conditions, when it makes scheduling
decisions. This ensures that node conditions don't directly affect scheduling.
For example, if the `DiskPressure` node condition is active, the control plane
adds the `node.kubernetes.io/disk-pressure` taint and does not schedule new pods
onto the affected node. If the `MemoryPressure` node condition is active, the
control plane adds the `node.kubernetes.io/memory-pressure` taint.
You can ignore node conditions for newly created pods by adding the corresponding
Pod tolerations. The control plane also adds the `node.kubernetes.io/memory-pressure`
toleration on pods that have a {{< glossary_tooltip text="QoS class" term_id="qos-class" >}}
other than `BestEffort`. This is because Kubernetes treats pods in the `Guaranteed`
or `Burstable` QoS classes (even pods with no memory request set) as if they are
able to cope with memory pressure, while new `BestEffort` pods are not scheduled
onto the affected node.
The DaemonSet controller automatically adds the following `NoSchedule`
tolerations to all daemons, to prevent DaemonSets from breaking.
@@ -282,7 +296,6 @@ tolerations to all daemons, to prevent DaemonSets from breaking.
Adding these tolerations ensures backward compatibility. You can also add
arbitrary tolerations to DaemonSets.
## {{% heading "whatsnext" %}}
* Read about [out of resource handling](/docs/concepts/scheduling-eviction/out-of-resource/) and how you can configure it
@@ -86,7 +86,7 @@ enforced/disallowed:
<tr>
<td>Capabilities</td>
<td>
Adding additional capabilities beyond the <a href="https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities">default set</a> must be disallowed.<br>
Adding <tt>NET_RAW</tt> or capabilities beyond the <a href="https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities">default set</a> must be disallowed.<br>
<br><b>Restricted Fields:</b><br>
spec.containers[*].securityContext.capabilities.add<br>
spec.initContainers[*].securityContext.capabilities.add<br>
@@ -194,7 +194,7 @@ well as lower-trust users.The following listed controls should be enforced/disal
<tr>
<td>Volume Types</td>
<td>
In addition to restricting HostPath volumes, the restricted profile limits usage of non-core volume types to those defined through PersistentVolumes.<br>
In addition to restricting HostPath volumes, the restricted profile limits usage of non-ephemeral volume types to those defined through PersistentVolumes.<br>
<br><b>Restricted Fields:</b><br>
spec.volumes[*].hostPath<br>
spec.volumes[*].gcePersistentDisk<br>
@@ -216,7 +216,6 @@ well as lower-trust users.The following listed controls should be enforced/disal
spec.volumes[*].portworxVolume<br>
spec.volumes[*].scaleIO<br>
spec.volumes[*].storageos<br>
spec.volumes[*].csi<br>
<br><b>Allowed Values:</b> undefined/nil<br>
</td>
</tr>
@@ -1,130 +0,0 @@
---
reviewers:
- rickypai
- thockin
title: Adding entries to Pod /etc/hosts with HostAliases
content_type: concept
weight: 60
min-kubernetes-server-version: 1.7
---
<!-- overview -->
Adding entries to a Pod's `/etc/hosts` file provides Pod-level override of hostname resolution when DNS and other options are not applicable. You can add these custom entries with the HostAliases field in PodSpec.
Modification not using HostAliases is not suggested because the file is managed by the kubelet and can be overwritten on during Pod creation/restart.
<!-- body -->
## Default hosts file content
Start an Nginx Pod which is assigned a Pod IP:
```shell
kubectl run nginx --image nginx
```
```
pod/nginx created
```
Examine a Pod IP:
```shell
kubectl get pods --output=wide
```
```
NAME READY STATUS RESTARTS AGE IP NODE
nginx 1/1 Running 0 13s 10.200.0.4 worker0
```
The hosts file content would look like this:
```shell
kubectl exec nginx -- cat /etc/hosts
```
```
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.200.0.4 nginx
```
By default, the `hosts` file only includes IPv4 and IPv6 boilerplates like
`localhost` and its own hostname.
## Adding additional entries with hostAliases
In addition to the default boilerplate, you can add additional entries to the
`hosts` file.
For example: to resolve `foo.local`, `bar.local` to `127.0.0.1` and `foo.remote`,
`bar.remote` to `10.1.2.3`, you can configure HostAliases for a Pod under
`.spec.hostAliases`:
{{< codenew file="service/networking/hostaliases-pod.yaml" >}}
You can start a Pod with that configuration by running:
```shell
kubectl apply -f https://k8s.io/examples/service/networking/hostaliases-pod.yaml
```
```
pod/hostaliases-pod created
```
Examine a Pod's details to see its IPv4 address and its status:
```shell
kubectl get pod --output=wide
```
```
NAME READY STATUS RESTARTS AGE IP NODE
hostaliases-pod 0/1 Completed 0 6s 10.200.0.5 worker0
```
The `hosts` file content looks like this:
```shell
kubectl logs hostaliases-pod
```
```
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.200.0.5 hostaliases-pod
# Entries added by HostAliases.
127.0.0.1 foo.local bar.local
10.1.2.3 foo.remote bar.remote
```
with the additional entries specified at the bottom.
## Why does the kubelet manage the hosts file? {#why-does-kubelet-manage-the-hosts-file}
The kubelet [manages](https://github.com/kubernetes/kubernetes/issues/14633) the
`hosts` file for each container of the Pod to prevent Docker from
[modifying](https://github.com/moby/moby/issues/17190) the file after the
containers have already been started.
{{< caution >}}
Avoid making manual changes to the hosts file inside a container.
If you make manual changes to the hosts file,
those changes are lost when the container exits.
{{< /caution >}}
@@ -50,7 +50,7 @@ options ndots:5
```
In summary, a pod in the _test_ namespace can successfully resolve either
`data.prod` or `data.prod.cluster.local`.
`data.prod` or `data.prod.svc.cluster.local`.
### DNS Records
@@ -189,7 +189,7 @@ and pre-created PVs, but you'll need to look at the documentation for a specific
to see its supported topology keys and examples.
{{< note >}}
If you choose to use `waitForFirstConsumer`, do not use `nodeName` in the Pod spec
If you choose to use `WaitForFirstConsumer`, do not use `nodeName` in the Pod spec
to specify node affinity. If `nodeName` is used in this case, the scheduler will be bypassed and PVC will remain in `pending` state.
Instead, you can use node selector for hostname in this case as shown below.
@@ -658,11 +658,11 @@ metadata:
provisioner: kubernetes.io/azure-disk
parameters:
storageaccounttype: Standard_LRS
kind: Shared
kind: managed
```
* `storageaccounttype`: Azure storage account Sku tier. Default is empty.
* `kind`: Possible values are `shared` (default), `dedicated`, and `managed`.
* `kind`: Possible values are `shared`, `dedicated`, and `managed` (default).
When `kind` is `shared`, all unmanaged disks are created in a few shared
storage accounts in the same resource group as the cluster. When `kind` is
`dedicated`, a new dedicated storage account will be created for the new
@@ -529,6 +529,15 @@ See the [GlusterFS example](https://github.com/kubernetes/examples/tree/{{< para
### hostPath {#hostpath}
{{< warning >}}
HostPath volumes present many security risks, and it is a best practice to avoid the use of
HostPaths when possible. When a HostPath volume must be used, it should be scoped to only the
required file or directory, and mounted as ReadOnly.
If restricting HostPath access to specific directories through AdmissionPolicy, `volumeMounts` MUST
be required to use `readOnly` mounts for the policy to be effective.
{{< /warning >}}
A `hostPath` volume mounts a file or directory from the host node's filesystem
into your Pod. This is not something that most Pods will need, but it offers a
powerful escape hatch for some applications.
@@ -558,6 +567,9 @@ The supported values for field `type` are:
Watch out when using this type of volume, because:
* HostPaths can expose privileged system credentials (such as for the Kubelet) or privileged APIs
(such as container runtime socket), which can be used for container escape or to attack other
parts of the cluster.
* Pods with identical configuration (such as created from a PodTemplate) may
behave differently on different nodes due to different files on the nodes
* The files or directories created on the underlying hosts are only writable by root. You
@@ -82,12 +82,11 @@ spec:
You can define one or multiple `topologySpreadConstraint` to instruct the kube-scheduler how to place each incoming Pod in relation to the existing Pods across your cluster. The fields are:
- **maxSkew** describes the degree to which Pods may be unevenly distributed.
It's the maximum permitted difference between the number of matching Pods in
any two topology domains of a given topology type. It must be greater than
zero. Its semantics differs according to the value of `whenUnsatisfiable`:
It must be greater than zero. Its semantics differs according to the value of `whenUnsatisfiable`:
- when `whenUnsatisfiable` equals to "DoNotSchedule", `maxSkew` is the maximum
permitted difference between the number of matching pods in the target
topology and the global minimum.
topology and the global minimum
(the minimum number of pods that match the label selector in a topology domain. For example, if you have 3 zones with 0, 2 and 3 matching pods respectively, The global minimum is 0).
- when `whenUnsatisfiable` equals to "ScheduleAnyway", scheduler gives higher
precedence to topologies that would help reduce the skew.
- **topologyKey** is the key of node labels. If two Nodes are labelled with this key and have identical values for that label, the scheduler treats both Nodes as being in the same topology. The scheduler tries to place a balanced number of Pods into each topology domain.
@@ -96,6 +95,8 @@ You can define one or multiple `topologySpreadConstraint` to instruct the kube-s
- `ScheduleAnyway` tells the scheduler to still schedule it while prioritizing nodes that minimize the skew.
- **labelSelector** is used to find matching Pods. Pods that match this label selector are counted to determine the number of Pods in their corresponding topology domain. See [Label Selectors](/docs/concepts/overview/working-with-objects/labels/#label-selectors) for more details.
When a Pod defines more than one `topologySpreadConstraint`, those constraints are ANDed: The kube-scheduler looks for a node for the incoming Pod that satisfies all the constraints.
You can read more about this field by running `kubectl explain Pod.spec.topologySpreadConstraints`.
### Example: One TopologySpreadConstraint
@@ -387,7 +388,8 @@ for more details.
## Known Limitations
- Scaling down a Deployment may result in imbalanced Pods distribution.
- There's no guarantee that the constraints remain satisfied when Pods are removed. For example, scaling down a Deployment may result in imbalanced Pods distribution.
You can use [Descheduler](https://github.com/kubernetes-sigs/descheduler) to rebalance the Pods distribution.
- Pods matched on tainted nodes are respected. See [Issue 80921](https://github.com/kubernetes/kubernetes/issues/80921)
## {{% heading "whatsnext" %}}