Merge remote-tracking branch 'upstream/master' into HEAD

This commit is contained in:
vineeth
2020-03-23 01:02:58 +05:30
298 changed files with 10720 additions and 4146 deletions
@@ -39,7 +39,7 @@ frontend and backend are connected using a Kubernetes
{{% capture lessoncontent %}}
### Creating the backend using a Deployment
## Creating the backend using a Deployment
The backend is a simple hello greeter microservice. Here is the configuration
file for the backend Deployment:
@@ -95,7 +95,7 @@ Events:
...
```
### Creating the backend Service object
## Creating the backend Service object
The key to connecting a frontend to a backend is the backend
Service. A Service creates a persistent IP address and DNS name entry
@@ -119,7 +119,7 @@ kubectl apply -f https://k8s.io/examples/service/access/hello-service.yaml
At this point, you have a backend Deployment running, and you have a
Service that can route traffic to it.
### Creating the frontend
## Creating the frontend
Now that you have your backend, you can create a frontend that connects to the backend.
The frontend connects to the backend worker Pods by using the DNS name
@@ -158,7 +158,7 @@ be to use a
so that you can change the configuration more easily.
{{< /note >}}
### Interact with the frontend Service
## Interact with the frontend Service
Once youve created a Service of type LoadBalancer, you can use this
command to find the external IP:
@@ -186,7 +186,7 @@ frontend LoadBalancer 10.51.252.116 XXX.XXX.XXX.XXX 80/TCP 1m
That IP can now be used to interact with the `frontend` service from outside the
cluster.
### Send traffic through the frontend
## Send traffic through the frontend
The frontend and backends are now connected. You can hit the endpoint
by using the curl command on the external IP of your frontend Service.
@@ -23,7 +23,7 @@ In this exercise you will use kubectl to fetch all of the Pods
running in a cluster, and format the output to pull out the list
of Containers for each.
## List all Containers in all namespaces
## List all Container images in all namespaces
- Fetch all Pods in all namespaces using `kubectl get pods --all-namespaces`
- Format the output to include only the list of Container image names
@@ -68,7 +68,7 @@ the `.items[*]` portion of the path should be omitted because a single
Pod is returned instead of a list of items.
{{< /note >}}
## List Containers by Pod
## List Container images by Pod
The formatting can be controlled further by using the `range` operation to
iterate over elements individually.
@@ -78,7 +78,7 @@ kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata
sort
```
## List Containers filtering by Pod label
## List Container images filtering by Pod label
To target only Pods matching a specific label, use the -l flag. The
following matches only Pods with labels matching `app=nginx`.
@@ -87,7 +87,7 @@ following matches only Pods with labels matching `app=nginx`.
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=nginx
```
## List Containers filtering by Pod namespace
## List Container images filtering by Pod namespace
To target only pods in a specific namespace, use the namespace flag. The
following matches only Pods in the `kube-system` namespace.
@@ -96,7 +96,7 @@ following matches only Pods in the `kube-system` namespace.
kubectl get pods --namespace kube-system -o jsonpath="{..image}"
```
## List Containers using a go-template instead of jsonpath
## List Container images using a go-template instead of jsonpath
As an alternative to jsonpath, Kubectl supports using [go-templates](https://golang.org/pkg/text/template/)
for formatting the output:
@@ -2,6 +2,7 @@
title: Use Port Forwarding to Access Applications in a Cluster
content_template: templates/task
weight: 40
min-kubernetes-server-version: v1.10
---
{{% capture overview %}}
@@ -26,104 +27,157 @@ for database debugging.
## Creating Redis deployment and service
1. Create a Redis deployment:
1. Create a Deployment that runs Redis:
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-deployment.yaml
```shell
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-deployment.yaml
```
The output of a successful command verifies that the deployment was created:
deployment.apps/redis-master created
```
deployment.apps/redis-master created
```
View the pod status to check that it is ready:
kubectl get pods
```shell
kubectl get pods
```
The output displays the pod created:
NAME READY STATUS RESTARTS AGE
redis-master-765d459796-258hz 1/1 Running 0 50s
```
NAME READY STATUS RESTARTS AGE
redis-master-765d459796-258hz 1/1 Running 0 50s
```
View the deployment status:
View the Deployment's status:
kubectl get deployment
```shell
kubectl get deployment
```
The output displays that the deployment was created:
The output displays that the Deployment was created:
NAME READY UP-TO-DATE AVAILABLE AGE
redis-master 1/1 1 1 55s
```
NAME READY UP-TO-DATE AVAILABLE AGE
redis-master 1/1 1 1 55s
```
View the replicaset status using:
The Deployment automatically manages a ReplicaSet.
View the ReplicaSet status using:
kubectl get rs
```shell
kubectl get replicaset
```
The output displays that the replicaset was created:
The output displays that the ReplicaSet was created:
NAME DESIRED CURRENT READY AGE
redis-master-765d459796 1 1 1 1m
```
NAME DESIRED CURRENT READY AGE
redis-master-765d459796 1 1 1 1m
```
2. Create a Redis service:
2. Create a Service to expose Redis on the network:
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-service.yaml
```shell
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-master-service.yaml
```
The output of a successful command verifies that the service was created:
The output of a successful command verifies that the Service was created:
service/redis-master created
```
service/redis-master created
```
Check the service created:
Check the Service created:
kubectl get svc | grep redis
```shell
kubectl get service redis-master
```
The output displays the service created:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-master ClusterIP 10.0.0.213 <none> 6379/TCP 27s
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-master ClusterIP 10.0.0.213 <none> 6379/TCP 27s
```
3. Verify that the Redis server is running in the pod and listening on port 6379:
3. Verify that the Redis server is running in the Pod, and listening on port 6379:
kubectl get pods redis-master-765d459796-258hz --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
```shell
# Change redis-master-765d459796-258hz to the name of the Pod
kubectl get pod redis-master-765d459796-258hz --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
```
The output displays the port:
The output displays the port for Redis in that Pod:
6379
```
6379
```
(this is the TCP port allocated to Redis on the internet).
## Forward a local port to a port on the Pod
1. `kubectl port-forward` allows using resource name, such as a pod name, to select a matching pod to port forward to.
## Forward a local port to a port on the pod
1. `kubectl port-forward` allows using resource name, such as a pod name, to select a matching pod to port forward to since Kubernetes v1.10.
kubectl port-forward redis-master-765d459796-258hz 7000:6379
```shell
# Change redis-master-765d459796-258hz to the name of the Pod
kubectl port-forward redis-master-765d459796-258hz 7000:6379
```
which is the same as
kubectl port-forward pods/redis-master-765d459796-258hz 7000:6379
```shell
kubectl port-forward pods/redis-master-765d459796-258hz 7000:6379
```
or
kubectl port-forward deployment/redis-master 7000:6379
```shell
kubectl port-forward deployment/redis-master 7000:6379
```
or
kubectl port-forward rs/redis-master 7000:6379
```shell
kubectl port-forward replicaset/redis-master 7000:6379
```
or
kubectl port-forward svc/redis-master 7000:6379
```shell
kubectl port-forward service/redis-master 7000:6379
```
Any of the above commands works. The output is similar to this:
I0710 14:43:38.274550 3655 portforward.go:225] Forwarding from 127.0.0.1:7000 -> 6379
I0710 14:43:38.274797 3655 portforward.go:225] Forwarding from [::1]:7000 -> 6379
```
I0710 14:43:38.274550 3655 portforward.go:225] Forwarding from 127.0.0.1:7000 -> 6379
I0710 14:43:38.274797 3655 portforward.go:225] Forwarding from [::1]:7000 -> 6379
```
2. Start the Redis command line interface:
redis-cli -p 7000
```shell
redis-cli -p 7000
```
3. At the Redis command line prompt, enter the `ping` command:
127.0.0.1:7000>ping
```
ping
```
A successful ping request returns PONG.
A successful ping request returns:
```
PONG
```
{{% /capture %}}
@@ -132,15 +186,15 @@ for database debugging.
## Discussion
Connections made to local port 7000 are forwarded to port 6379 of the pod that
is running the Redis server. With this connection in place you can use your
local workstation to debug the database that is running in the pod.
Connections made to local port 7000 are forwarded to port 6379 of the Pod that
is running the Redis server. With this connection in place, you can use your
local workstation to debug the database that is running in the Pod.
{{< warning >}}
Due to known limitations, port forward today only works for TCP protocol.
The support to UDP protocol is being tracked in
{{< note >}}
`kubectl port-forward` is implemented for TCP ports only.
The support for UDP protocol is tracked in
[issue 47862](https://github.com/kubernetes/kubernetes/issues/47862).
{{< /warning >}}
{{< /note >}}
{{% /capture %}}
@@ -148,6 +202,3 @@ The support to UDP protocol is being tracked in
{{% capture whatsnext %}}
Learn more about [kubectl port-forward](/docs/reference/generated/kubectl/kubectl-commands/#port-forward).
{{% /capture %}}
@@ -62,10 +62,10 @@ for details about addon manager and how to disable individual addons.
To mark a StorageClass as non-default, you need to change its value to `false`:
```bash
kubectl patch storageclass <your-class-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
```
where `<your-class-name>` is the name of your chosen StorageClass.
where `standard` is the name of your chosen StorageClass.
1. Mark a StorageClass as default:
@@ -73,7 +73,7 @@ for details about addon manager and how to disable individual addons.
`storageclass.kubernetes.io/is-default-class=true`.
```bash
kubectl patch storageclass <your-class-name> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl patch storageclass gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```
Please note that at most one StorageClass can be marked as default. If two
@@ -24,7 +24,7 @@ To install Kubernetes on a set of machines, consult one of the existing [Getting
## Upgrading a cluster
The current state of cluster upgrades is provider dependent, and some releases may require special care when upgrading. It is recommended that administrators consult both the [release notes](https://git.k8s.io/kubernetes/CHANGELOG.md), as well as the version specific upgrade notes prior to upgrading their clusters.
The current state of cluster upgrades is provider dependent, and some releases may require special care when upgrading. It is recommended that administrators consult both the [release notes](https://git.k8s.io/kubernetes/CHANGELOG/README.md), as well as the version specific upgrade notes prior to upgrading their clusters.
### Upgrading an Azure Kubernetes Service (AKS) cluster
@@ -265,15 +265,6 @@ work properly owing to a known issue with Alpine.
Check [here](https://github.com/kubernetes/kubernetes/issues/30215)
for more information.
## Kubernetes Federation (Multiple Zone support)
Release 1.3 introduced Cluster Federation support for multi-site Kubernetes
installations. This required some minor (backward-compatible) changes to the
way the Kubernetes cluster DNS server processes DNS queries, to facilitate
the lookup of federated services (which span multiple Kubernetes clusters).
See the [Cluster Federation Administrators' Guide](/docs/concepts/cluster-administration/federation/)
for more details on Cluster Federation and multi-site support.
## References
- [DNS for Services and Pods](/docs/concepts/services-networking/dns-pod-service/)
@@ -224,12 +224,14 @@ At this point, all requests we make to the Kubernetes cluster from the command l
Let's create some contents.
{{< codenew file="admin/snowflake-deployment.yaml" >}}
Apply the manifest to create a Deployment
```shell
kubectl run snowflake --image=k8s.gcr.io/serve_hostname --replicas=2
kubectl apply -f https://k8s.io/examples/admin/snowflake-deployment.yaml
```
We have just created a deployment whose replica size is 2 that is running the pod called `snowflake` with a basic container that just serves the hostname.
Note that `kubectl run` creates deployments only on Kubernetes cluster >= v1.2. If you are running older versions, it creates replication controllers instead.
If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/reference/generated/kubectl/kubectl-commands/#run) for more details.
```shell
kubectl get deployment
@@ -188,88 +188,22 @@ This delete is asynchronous, so for a time you will see the namespace in the `Te
To demonstrate this, let's spin up a simple Deployment and Pods in the `development` namespace.
We first check what is the current context:
```shell
kubectl config view
```
```yaml
apiVersion: v1
clusters:
cluster:
certificate-authority-data: REDACTED
server: https://130.211.122.180
name: lithe-cocoa-92103_kubernetes
contexts:
context:
cluster: lithe-cocoa-92103_kubernetes
user: lithe-cocoa-92103_kubernetes
name: lithe-cocoa-92103_kubernetes
current-context: lithe-cocoa-92103_kubernetes
kind: Config
preferences: {}
users:
name: lithe-cocoa-92103_kubernetes
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
token: 65rZW78y8HbwXXtSXuUw9DbP4FLjHi4b
name: lithe-cocoa-92103_kubernetes-basic-auth
user:
password: h5M0FtUUIflBSdI7
username: admin
```
```shell
kubectl config current-context
```
```
lithe-cocoa-92103_kubernetes
```
The next step is to define a context for the kubectl client to work in each namespace. The values of "cluster" and "user" fields are copied from the current context.
```shell
kubectl config set-context dev --namespace=development --cluster=lithe-cocoa-92103_kubernetes --user=lithe-cocoa-92103_kubernetes
kubectl config set-context prod --namespace=production --cluster=lithe-cocoa-92103_kubernetes --user=lithe-cocoa-92103_kubernetes
```
The above commands provided two request contexts you can alternate against depending on what namespace you
wish to work against.
Let's switch to operate in the `development` namespace.
```shell
kubectl config use-context dev
```
You can verify your current context by doing the following:
```shell
kubectl config current-context
dev
```
At this point, all requests we make to the Kubernetes cluster from the command line are scoped to the `development` namespace.
Let's create some contents.
```shell
kubectl run snowflake --image=k8s.gcr.io/serve_hostname --replicas=2
kubectl run snowflake --image=k8s.gcr.io/serve_hostname --replicas=2 -n=development
```
We have just created a deployment whose replica size is 2 that is running the pod called `snowflake` with a basic container that just serves the hostname.
Note that `kubectl run` creates deployments only on Kubernetes cluster >= v1.2. If you are running older versions, it creates replication controllers instead.
If you want to obtain the old behavior, use `--generator=run/v1` to create replication controllers. See [`kubectl run`](/docs/reference/generated/kubectl/kubectl-commands/#run) for more details.
```shell
kubectl get deployment
kubectl get deployment -n=development
```
```
NAME READY UP-TO-DATE AVAILABLE AGE
snowflake 2/2 2 2 2m
```
```shell
kubectl get pods -l run=snowflake
kubectl get pods -l run=snowflake -n=development
```
```
NAME READY STATUS RESTARTS AGE
@@ -281,23 +215,19 @@ This delete is asynchronous, so for a time you will see the namespace in the `Te
Let's switch to the `production` namespace and show how resources in one namespace are hidden from the other.
```shell
kubectl config use-context prod
```
The `production` namespace should be empty, and the following commands should return nothing.
```shell
kubectl get deployment
kubectl get pods
kubectl get deployment -n=production
kubectl get pods -n=production
```
Production likes to run cattle, so let's create some cattle pods.
```shell
kubectl run cattle --image=k8s.gcr.io/serve_hostname --replicas=5
kubectl run cattle --image=k8s.gcr.io/serve_hostname --replicas=5 -n=production
kubectl get deployment
kubectl get deployment -n=production
```
```
NAME READY UP-TO-DATE AVAILABLE AGE
@@ -305,7 +235,7 @@ This delete is asynchronous, so for a time you will see the namespace in the `Te
```
```shell
kubectl get pods -l run=cattle
kubectl get pods -l run=cattle -n=production
```
```
NAME READY STATUS RESTARTS AGE
@@ -2,6 +2,7 @@
reviewers:
- bowei
- zihongz
- sftim
title: Using NodeLocal DNSCache in Kubernetes clusters
content_template: templates/task
---
@@ -47,18 +48,44 @@ This is the path followed by DNS Queries after NodeLocal DNSCache is enabled:
{{< figure src="/images/docs/nodelocaldns.jpg" alt="NodeLocal DNSCache flow" title="Nodelocal DNSCache flow" caption="This image shows how NodeLocal DNSCache handles DNS queries." >}}
## Configuration
{{< note >}} The local listen IP address for NodeLocal DNSCache can be any IP in the 169.254.20.0/16 space or any other IP address that can be guaranteed to not collide with any existing IP. This document uses 169.254.20.10 as an example.
{{< /note >}}
This feature can be enabled using the command:
This feature can be enabled using the following steps:
`KUBE_ENABLE_NODELOCAL_DNS=true kubetest --up`
* Prepare a manifest similar to the sample [`nodelocaldns.yaml`](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml) and save it as `nodelocaldns.yaml.`
* Substitute the variables in the manifest with the right values:
This works for e2e clusters created on GCE. On all other environments, the following steps will setup NodeLocal DNSCache:
* kubedns=`kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}`
* domain=`<cluster-domain>`
* localdns=`<node-local-address>`
`<cluster-domain>` is "cluster.local" by default. `<node-local-address>` is the local listen IP address chosen for NodeLocal DNSCache.
* A yaml similar to [this](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml) can be applied using `kubectl create -f` command.
* No need to modify the --cluster-dns flag since NodeLocal DNSCache listens on both the kube-dns service IP as well as a link-local IP (169.254.20.10 by default)
* If kube-proxy is running in IPTABLES mode:
``` bash
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml
```
`__PILLAR__CLUSTER__DNS__` and `__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
In this mode, node-local-dns pods listen on both the kube-dns service IP as well as `<node-local-address>`, so pods can lookup DNS records using either IP address.
* If kube-proxy is running in IPVS mode:
``` bash
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml
```
In this mode, node-local-dns pods listen only on `<node-local-address>`. The node-local-dns interface cannot bind the kube-dns cluster IP since the interface used for IPVS loadbalancing already uses this address.
`__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
* Run `kubectl create -f nodelocaldns.yaml`
* If using kube-proxy in IPVS mode, `--cluster-dns` flag to kubelet needs to be modified to use `<node-local-address>` that NodeLocal DNSCache is listening on.
Otherwise, there is no need to modify the value of the `--cluster-dns` flag, since NodeLocal DNSCache listens on both the kube-dns service IP as well as `<node-local-address>`.
Once enabled, node-local-dns Pods will run in the kube-system namespace on each of the cluster nodes. This Pod runs [CoreDNS](https://github.com/coredns/coredns) in cache mode, so all CoreDNS metrics exposed by the different plugins will be available on a per-node basis.
The feature can be disabled by removing the daemonset, using `kubectl delete -f` command. On e2e clusters created on GCE, the daemonset can be removed by deleting the node-local-dns yaml from `/etc/kubernetes/addons/0-dns/nodelocaldns.yaml`
You can disable this feature by removing the DaemonSet, using `kubectl delete -f <manifest>` . You should also revert any changes you made to the kubelet configuration.
{{% /capture %}}
@@ -98,13 +98,7 @@ be configured to use the `systemd` cgroup driver.
`kube-reserved` is meant to capture resource reservation for kubernetes system
daemons like the `kubelet`, `container runtime`, `node problem detector`, etc.
It is not meant to reserve resources for system daemons that are run as pods.
`kube-reserved` is typically a function of `pod density` on the nodes. [This
performance dashboard](http://node-perf-dash.k8s.io/#/builds) exposes `cpu` and
`memory` usage profiles of `kubelet` and `docker engine` at multiple levels of
pod density. [This blog
post](https://kubernetes.io/blog/2016/11/visualize-kubelet-performance-with-node-dashboard)
explains how the dashboard can be interpreted to come up with a suitable
`kube-reserved` reservation.
`kube-reserved` is typically a function of `pod density` on the nodes.
In addition to `cpu`, `memory`, and `ephemeral-storage`, `pid` may be
specified to reserve the specified number of process IDs for
@@ -6,20 +6,20 @@ weight: 110
{{% capture overview %}}
This page shows how to configure liveness, readiness and startup probes for Containers.
This page shows how to configure liveness, readiness and startup probes for containers.
The [kubelet](/docs/admin/kubelet/) uses liveness probes to know when to
restart a Container. For example, liveness probes could catch a deadlock,
restart a container. For example, liveness probes could catch a deadlock,
where an application is running, but unable to make progress. Restarting a
Container in such a state can help to make the application more available
container in such a state can help to make the application more available
despite bugs.
The kubelet uses readiness probes to know when a Container is ready to start
accepting traffic. A Pod is considered ready when all of its Containers are ready.
The kubelet uses readiness probes to know when a container is ready to start
accepting traffic. A Pod is considered ready when all of its containers are ready.
One use of this signal is to control which Pods are used as backends for Services.
When a Pod is not ready, it is removed from Service load balancers.
The kubelet uses startup probes to know when a Container application has started.
The kubelet uses startup probes to know when a container application has started.
If such a probe is configured, it disables liveness and readiness checks until
it succeeds, making sure those probes don't interfere with the application startup.
This can be used to adopt liveness checks on slow starting containers, avoiding them
@@ -41,27 +41,27 @@ Many applications running for long periods of time eventually transition to
broken states, and cannot recover except by being restarted. Kubernetes provides
liveness probes to detect and remedy such situations.
In this exercise, you create a Pod that runs a Container based on the
In this exercise, you create a Pod that runs a container based on the
`k8s.gcr.io/busybox` image. Here is the configuration file for the Pod:
{{< codenew file="pods/probe/exec-liveness.yaml" >}}
In the configuration file, you can see that the Pod has a single Container.
In the configuration file, you can see that the Pod has a single `Container`.
The `periodSeconds` field specifies that the kubelet should perform a liveness
probe every 5 seconds. The `initialDelaySeconds` field tells the kubelet that it
should wait 5 second before performing the first probe. To perform a probe, the
kubelet executes the command `cat /tmp/healthy` in the Container. If the
command succeeds, it returns 0, and the kubelet considers the Container to be alive and
healthy. If the command returns a non-zero value, the kubelet kills the Container
kubelet executes the command `cat /tmp/healthy` in the target container. If the
command succeeds, it returns 0, and the kubelet considers the container to be alive and
healthy. If the command returns a non-zero value, the kubelet kills the container
and restarts it.
When the Container starts, it executes this command:
When the container starts, it executes this command:
```shell
/bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600"
```
For the first 30 seconds of the Container's life, there is a `/tmp/healthy` file.
For the first 30 seconds of the container's life, there is a `/tmp/healthy` file.
So during the first 30 seconds, the command `cat /tmp/healthy` returns a success
code. After 30 seconds, `cat /tmp/healthy` returns a failure code.
@@ -79,7 +79,7 @@ kubectl describe pod liveness-exec
The output indicates that no liveness probes have failed yet:
```shell
```
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
24s 24s 1 {default-scheduler } Normal Scheduled Successfully assigned liveness-exec to worker0
@@ -98,7 +98,7 @@ kubectl describe pod liveness-exec
At the bottom of the output, there are messages indicating that the liveness
probes have failed, and the containers have been killed and recreated.
```shell
```
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
37s 37s 1 {default-scheduler } Normal Scheduled Successfully assigned liveness-exec to worker0
@@ -109,7 +109,7 @@ FirstSeen LastSeen Count From SubobjectPath Type
2s 2s 1 {kubelet worker0} spec.containers{liveness} Warning Unhealthy Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
```
Wait another 30 seconds, and verify that the Container has been restarted:
Wait another 30 seconds, and verify that the container has been restarted:
```shell
kubectl get pod liveness-exec
@@ -117,7 +117,7 @@ kubectl get pod liveness-exec
The output shows that `RESTARTS` has been incremented:
```shell
```
NAME READY STATUS RESTARTS AGE
liveness-exec 1/1 Running 1 1m
```
@@ -130,23 +130,23 @@ image.
{{< codenew file="pods/probe/http-liveness.yaml" >}}
In the configuration file, you can see that the Pod has a single Container.
In the configuration file, you can see that the Pod has a single container.
The `periodSeconds` field specifies that the kubelet should perform a liveness
probe every 3 seconds. The `initialDelaySeconds` field tells the kubelet that it
should wait 3 seconds before performing the first probe. To perform a probe, the
kubelet sends an HTTP GET request to the server that is running in the Container
kubelet sends an HTTP GET request to the server that is running in the container
and listening on port 8080. If the handler for the server's `/healthz` path
returns a success code, the kubelet considers the Container to be alive and
healthy. If the handler returns a failure code, the kubelet kills the Container
returns a success code, the kubelet considers the container to be alive and
healthy. If the handler returns a failure code, the kubelet kills the container
and restarts it.
Any code greater than or equal to 200 and less than 400 indicates success. Any
other code indicates failure.
You can see the source code for the server in
[server.go](https://github.com/kubernetes/kubernetes/blob/master/test/images/agnhost/liveness/server.go).
[server.go](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/test/images/agnhost/liveness/server.go).
For the first 10 seconds that the Container is alive, the `/healthz` handler
For the first 10 seconds that the container is alive, the `/healthz` handler
returns a status of 200. After that, the handler returns a status of 500.
```go
@@ -162,9 +162,9 @@ http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
})
```
The kubelet starts performing health checks 3 seconds after the Container starts.
The kubelet starts performing health checks 3 seconds after the container starts.
So the first couple of health checks will succeed. But after 10 seconds, the health
checks will fail, and the kubelet will kill and restart the Container.
checks will fail, and the kubelet will kill and restart the container.
To try the HTTP liveness check, create a Pod:
@@ -173,21 +173,21 @@ kubectl apply -f https://k8s.io/examples/pods/probe/http-liveness.yaml
```
After 10 seconds, view Pod events to verify that liveness probes have failed and
the Container has been restarted:
the container has been restarted:
```shell
kubectl describe pod liveness-http
```
In releases prior to v1.13 (including v1.13), if the environment variable
`http_proxy` (or `HTTP_PROXY`) is set on the node where a pod is running,
`http_proxy` (or `HTTP_PROXY`) is set on the node where a Pod is running,
the HTTP liveness probe uses that proxy.
In releases after v1.13, local HTTP proxy environment variable settings do not
affect the HTTP liveness probe.
## Define a TCP liveness probe
A third type of liveness probe uses a TCP Socket. With this configuration, the
A third type of liveness probe uses a TCP socket. With this configuration, the
kubelet will attempt to open a socket to your container on the specified port.
If it can establish a connection, the container is considered healthy, if it
cant it is considered a failure.
@@ -197,7 +197,7 @@ cant it is considered a failure.
As you can see, configuration for a TCP check is quite similar to an HTTP check.
This example uses both readiness and liveness probes. The kubelet will send the
first readiness probe 5 seconds after the container starts. This will attempt to
connect to the `goproxy` container on port 8080. If the probe succeeds, the pod
connect to the `goproxy` container on port 8080. If the probe succeeds, the Pod
will be marked as ready. The kubelet will continue to run this check every 10
seconds.
@@ -351,7 +351,7 @@ port to perform the check. The kubelet sends the probe to the pods IP address
unless the address is overridden by the optional `host` field in `httpGet`. If
`scheme` field is set to `HTTPS`, the kubelet sends an HTTPS request skipping the
certificate verification. In most scenarios, you do not want to set the `host` field.
Here's one scenario where you would set it. Suppose the Container listens on 127.0.0.1
Here's one scenario where you would set it. Suppose the container listens on 127.0.0.1
and the Pod's `hostNetwork` field is true. Then `host`, under `httpGet`, should be set
to 127.0.0.1. If your pod relies on virtual hosts, which is probably the more common
case, you should not use `host`, but rather set the `Host` header in `httpHeaders`.
@@ -367,7 +367,7 @@ to resolve it.
* Learn more about
[Container Probes](/docs/concepts/workloads/pods/pod-lifecycle/#container-probes).
### Reference
You can also read the API references for:
* [Pod](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#pod-v1-core)
* [Container](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#container-v1-core)
@@ -29,6 +29,11 @@ kubectl get nodes
And verify that all of the nodes you expect to see are present and that they are all in the `Ready` state.
To get detailed information about the overall health of your cluster, you can run:
```shell
kubectl cluster-info dump
```
## Looking at logs
For now, digging deeper into the cluster requires logging into the relevant machines. Here are the locations
@@ -8,57 +8,30 @@ title: Debug Services
{{% capture overview %}}
An issue that comes up rather frequently for new installations of Kubernetes is
that a `Service` is not working properly. You've run your `Deployment` and
created a `Service`, but you get no response when you try to access it.
This document will hopefully help you to figure out what's going wrong.
that a Service is not working properly. You've run your Pods through a
Deployment (or other workload controller) and created a Service, but you
get no response when you try to access it. This document will hopefully help
you to figure out what's going wrong.
{{% /capture %}}
{{% capture body %}}
## Conventions
Throughout this doc you will see various commands that you can run. Some
commands need to be run within a `Pod`, others on a Kubernetes `Node`, and others
can run anywhere you have `kubectl` and credentials for the cluster. To make it
clear what is expected, this document will use the following conventions.
If the command "COMMAND" is expected to run in a `Pod` and produce "OUTPUT":
```shell
u@pod$ COMMAND
OUTPUT
```
If the command "COMMAND" is expected to run on a `Node` and produce "OUTPUT":
```shell
u@node$ COMMAND
OUTPUT
```
If the command is "kubectl ARGS":
```shell
kubectl ARGS
OUTPUT
```
## Running commands in a Pod
For many steps here you will want to see what a `Pod` running in the cluster
sees. The simplest way to do this is to run an interactive alpine `Pod`:
For many steps here you will want to see what a Pod running in the cluster
sees. The simplest way to do this is to run an interactive alpine Pod:
```none
kubectl run -it --rm --restart=Never alpine --image=alpine sh
/ #
```
{{< note >}}
If you don't see a command prompt, try pressing enter.
{{< /note >}}
If you already have a running `Pod` that you prefer to use, you can run a
If you already have a running Pod that you prefer to use, you can run a
command in it using:
```shell
@@ -67,21 +40,23 @@ kubectl exec <POD-NAME> -c <CONTAINER-NAME> -- <COMMAND>
## Setup
For the purposes of this walk-through, let's run some `Pods`. Since you're
probably debugging your own `Service` you can substitute your own details, or you
For the purposes of this walk-through, let's run some Pods. Since you're
probably debugging your own Service you can substitute your own details, or you
can follow along and get a second data point.
```shell
kubectl run hostnames --image=k8s.gcr.io/serve_hostname \
--labels=app=hostnames \
--port=9376 \
--replicas=3
--replicas=3
```
```none
deployment.apps/hostnames created
```
`kubectl` commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands.
{{< note >}}
This is the same as if you started the `Deployment` with the following YAML:
This is the same as if you had started the Deployment with the following
YAML:
```yaml
apiVersion: apps/v1
@@ -91,61 +66,111 @@ metadata:
spec:
selector:
matchLabels:
app: hostnames
run: hostnames
replicas: 3
template:
metadata:
labels:
app: hostnames
run: hostnames
spec:
containers:
- name: hostnames
image: k8s.gcr.io/serve_hostname
ports:
- containerPort: 9376
protocol: TCP
```
The label "run" is automatically set by `kubectl run` to the name of the
Deployment.
{{< /note >}}
Confirm your `Pods` are running:
You can confirm your Pods are running:
```shell
kubectl get pods -l app=hostnames
kubectl get pods -l run=hostnames
```
```none
NAME READY STATUS RESTARTS AGE
hostnames-632524106-bbpiw 1/1 Running 0 2m
hostnames-632524106-ly40y 1/1 Running 0 2m
hostnames-632524106-tlaok 1/1 Running 0 2m
```
You can also confirm that your Pods are serving. You can get the list of
Pod IP addresses and test them directly.
```shell
kubectl get pods -l run=hostnames \
-o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'
```
```none
10.244.0.5
10.244.0.6
10.244.0.7
```
The example container used for this walk-through simply serves its own hostname
via HTTP on port 9376, but if you are debugging your own app, you'll want to
use whatever port number your Pods are listening on.
From within a pod:
```shell
for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do
wget -qO- $ep
done
```
This should produce something like:
```
hostnames-0uton
hostnames-bvc05
hostnames-yp2kp
```
If you are not getting the responses you expect at this point, your Pods
might not be healthy or might not be listening on the port you think they are.
You might find `kubectl logs` to be useful for seeing what is happening, or
perhaps you need to `kubectl exec` directly into your Pods and debug from
there.
Assuming everything has gone to plan so far, you can start to investigate why
your Service doesn't work.
## Does the Service exist?
The astute reader will have noticed that we did not actually create a `Service`
The astute reader will have noticed that you did not actually create a Service
yet - that is intentional. This is a step that sometimes gets forgotten, and
is the first thing to check.
So what would happen if I tried to access a non-existent `Service`? Assuming you
have another `Pod` that consumes this `Service` by name you would get something
like:
What would happen if you tried to access a non-existent Service? If
you have another Pod that consumes this Service by name you would get
something like:
```shell
u@pod$ wget -O- hostnames
wget -O- hostnames
```
```none
Resolving hostnames (hostnames)... failed: Name or service not known.
wget: unable to resolve host address 'hostnames'
```
So the first thing to check is whether that `Service` actually exists:
The first thing to check is whether that Service actually exists:
```shell
kubectl get svc hostnames
```
```none
No resources found.
Error from server (NotFound): services "hostnames" not found
```
So we have a culprit, let's create the `Service`. As before, this is for the
walk-through - you can use your own `Service`'s details here.
Let's create the Service. As before, this is for the walk-through - you can
use your own Service's details here.
```shell
kubectl expose deployment hostnames --port=80 --target-port=9376
```
```none
service/hostnames exposed
```
@@ -153,11 +178,16 @@ And read it back, just to be sure:
```shell
kubectl get svc hostnames
```
```none
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hostnames ClusterIP 10.0.1.175 <none> 80/TCP 5s
```
As before, this is the same as if you had started the `Service` with YAML:
Now you know that the Service exists.
{{< note >}}
As before, this is the same as if you had started the Service with YAML:
```yaml
apiVersion: v1
@@ -166,7 +196,7 @@ metadata:
name: hostnames
spec:
selector:
app: hostnames
run: hostnames
ports:
- name: default
protocol: TCP
@@ -174,25 +204,35 @@ spec:
targetPort: 9376
```
Now you can confirm that the `Service` exists.
In order to highlight the full range of configuration, the Service you created
here uses a different port number than the Pods. For many real-world
Services, these values might be the same.
{{< /note >}}
## Does the Service work by DNS?
## Does the Service work by DNS name?
From a `Pod` in the same `Namespace`:
One of the most common ways that clients consume a Service is through a DNS
name.
From a Pod in the same Namespace:
```shell
u@pod$ nslookup hostnames
nslookup hostnames
```
```none
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: hostnames
Address 1: 10.0.1.175 hostnames.default.svc.cluster.local
```
If this fails, perhaps your `Pod` and `Service` are in different
`Namespaces`, try a namespace-qualified name:
If this fails, perhaps your Pod and Service are in different
Namespaces, try a namespace-qualified name (again, from within a Pod):
```shell
u@pod$ nslookup hostnames.default
nslookup hostnames.default
```
```none
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: hostnames.default
@@ -200,11 +240,13 @@ Address 1: 10.0.1.175 hostnames.default.svc.cluster.local
```
If this works, you'll need to adjust your app to use a cross-namespace name, or
run your app and `Service` in the same `Namespace`. If this still fails, try a
run your app and Service in the same Namespace. If this still fails, try a
fully-qualified name:
```shell
u@pod$ nslookup hostnames.default.svc.cluster.local
nslookup hostnames.default.svc.cluster.local
```
```none
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: hostnames.default.svc.cluster.local
@@ -212,18 +254,20 @@ Address 1: 10.0.1.175 hostnames.default.svc.cluster.local
```
Note the suffix here: "default.svc.cluster.local". The "default" is the
`Namespace` we're operating in. The "svc" denotes that this is a `Service`.
Namespace you're operating in. The "svc" denotes that this is a Service.
The "cluster.local" is your cluster domain, which COULD be different in your
own cluster.
You can also try this from a `Node` in the cluster:
You can also try this from a Node in the cluster:
{{< note >}}
10.0.0.10 is my DNS `Service`, yours might be different.
10.0.0.10 is the cluster's DNS Service IP, yours might be different.
{{< /note >}}
```shell
u@node$ nslookup hostnames.default.svc.cluster.local 10.0.0.10
nslookup hostnames.default.svc.cluster.local 10.0.0.10
```
```none
Server: 10.0.0.10
Address: 10.0.0.10#53
@@ -232,39 +276,49 @@ Address: 10.0.1.175
```
If you are able to do a fully-qualified name lookup but not a relative one, you
need to check that your `/etc/resolv.conf` file is correct.
need to check that your `/etc/resolv.conf` file in your Pod is correct. From
within a Pod:
```shell
u@pod$ cat /etc/resolv.conf
cat /etc/resolv.conf
```
You should see something like:
```
nameserver 10.0.0.10
search default.svc.cluster.local svc.cluster.local cluster.local example.com
options ndots:5
```
The `nameserver` line must indicate your cluster's DNS `Service`. This is
The `nameserver` line must indicate your cluster's DNS Service. This is
passed into `kubelet` with the `--cluster-dns` flag.
The `search` line must include an appropriate suffix for you to find the
`Service` name. In this case it is looking for `Services` in the local
`Namespace` (`default.svc.cluster.local`), `Services` in all `Namespaces`
(`svc.cluster.local`), and the cluster (`cluster.local`). Depending on your own
install you might have additional records after that (up to 6 total). The
cluster suffix is passed into `kubelet` with the `--cluster-domain` flag. We
assume that is "cluster.local" in this document, but yours might be different,
in which case you should change that in all of the commands above.
Service name. In this case it is looking for Services in the local
Namespace ("default.svc.cluster.local"), Services in all Namespaces
("svc.cluster.local"), and lastly for names in the cluster ("cluster.local").
Depending on your own install you might have additional records after that (up
to 6 total). The cluster suffix is passed into `kubelet` with the
`--cluster-domain` flag. Throughout this document, the cluster suffix is
assumed to be "cluster.local". Your own clusters might be configured
differently, in which case you should change that in all of the previous
commands.
The `options` line must set `ndots` high enough that your DNS client library
considers search paths at all. Kubernetes sets this to 5 by default, which is
high enough to cover all of the DNS names it generates.
### Does any Service exist in DNS?
### Does any Service work by DNS name? {#does-any-service-exist-in-dns}
If the above still fails - DNS lookups are not working for your `Service` - we
If the above still fails, DNS lookups are not working for your Service. You
can take a step back and see what else is not working. The Kubernetes master
`Service` should always work:
Service should always work. From within a Pod:
```shell
u@pod$ nslookup kubernetes.default
nslookup kubernetes.default
```
```none
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
@@ -272,34 +326,37 @@ Name: kubernetes.default
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
```
If this fails, you might need to go to the kube-proxy section of this doc, or
even go back to the top of this document and start over, but instead of
debugging your own `Service`, debug DNS.
If this fails, please see the [kube-proxy](#is-the-kube-proxy-working) section
of this document, or even go back to the top of this document and start over,
but instead of debugging your own Service, debug the DNS Service.
## Does the Service work by IP?
Assuming we can confirm that DNS works, the next thing to test is whether your
`Service` works at all. From a node in your cluster, access the `Service`'s
IP (from `kubectl get` above).
Assuming you have confirmed that DNS works, the next thing to test is whether your
Service works by its IP address. From a Pod in your cluster, access the
Service's IP (from `kubectl get` above).
```shell
u@node$ curl 10.0.1.175:80
hostnames-0uton
u@node$ curl 10.0.1.175:80
hostnames-yp2kp
u@node$ curl 10.0.1.175:80
hostnames-bvc05
for i in $(seq 1 3); do
wget -qO- 10.0.1.175:80
done
```
If your `Service` is working, you should get correct responses. If not, there
This should produce something like:
```
hostnames-0uton
hostnames-bvc05
hostnames-yp2kp
```
If your Service is working, you should get correct responses. If not, there
are a number of things that could be going wrong. Read on.
## Is the Service correct?
## Is the Service defined correctly?
It might sound silly, but you should really double and triple check that your
`Service` is correct and matches your `Pod`'s port. Read back your `Service`
Service is correct and matches your Pod's port. Read back your Service
and verify it:
```shell
@@ -316,7 +373,7 @@ kubectl get service hostnames -o json
"resourceVersion": "347189",
"creationTimestamp": "2015-07-07T15:24:29Z",
"labels": {
"app": "hostnames"
"run": "hostnames"
}
},
"spec": {
@@ -330,7 +387,7 @@ kubectl get service hostnames -o json
}
],
"selector": {
"app": "hostnames"
"run": "hostnames"
},
"clusterIP": "10.0.1.175",
"type": "ClusterIP",
@@ -342,110 +399,116 @@ kubectl get service hostnames -o json
}
```
* Is the port you are trying to access in `spec.ports[]`?
* Is the `targetPort` correct for your `Pods` (many `Pods` choose to use a different port than the `Service`)?
* If you meant it to be a numeric port, is it a number (9376) or a
string "9376"?
* If you meant it to be a named port, do your `Pods` expose a port
with the same name?
* Is the port's `protocol` the same as the `Pod`'s?
* Is the Service port you are trying to access listed in `spec.ports[]`?
* Is the `targetPort` correct for your Pods (some Pods use a different port than the Service)?
* If you meant to use a numeric port, is it a number (9376) or a string "9376"?
* If you meant to use a named port, do your Pods expose a port with the same name?
* Is the port's `protocol` correct for your Pods?
## Does the Service have any Endpoints?
If you got this far, we assume that you have confirmed that your `Service`
exists and is resolved by DNS. Now let's check that the `Pods` you ran are
actually being selected by the `Service`.
If you got this far, you have confirmed that your Service is correctly
defined and is resolved by DNS. Now let's check that the Pods you ran are
actually being selected by the Service.
Earlier we saw that the `Pods` were running. We can re-check that:
Earlier you saw that the Pods were running. You can re-check that:
```shell
kubectl get pods -l app=hostnames
kubectl get pods -l run=hostnames
```
```none
NAME READY STATUS RESTARTS AGE
hostnames-0uton 1/1 Running 0 1h
hostnames-bvc05 1/1 Running 0 1h
hostnames-yp2kp 1/1 Running 0 1h
```
The "AGE" column says that these `Pods` are about an hour old, which implies that
The `-l run=hostnames` argument is a label selector - just like our Service
has.
The "AGE" column says that these Pods are about an hour old, which implies that
they are running fine and not crashing.
The `-l app=hostnames` argument is a label selector - just like our `Service`
has. Inside the Kubernetes system is a control loop which evaluates the
selector of every `Service` and saves the results into an `Endpoints` object.
The "RESTARTS" column says that these pods are not crashing frequently or being
restarted. Frequent restarts could lead to intermittent connectivity issues.
If the restart count is high, read more about how to [debug pods](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#debugging-pods).
Inside the Kubernetes system is a control loop which evaluates the selector of
every Service and saves the results into a corresponding Endpoints object.
```shell
kubectl get endpoints hostnames
NAME ENDPOINTS
hostnames 10.244.0.5:9376,10.244.0.6:9376,10.244.0.7:9376
```
This confirms that the endpoints controller has found the correct `Pods` for
your `Service`. If the `hostnames` row is blank, you should check that the
`spec.selector` field of your `Service` actually selects for `metadata.labels`
values on your `Pods`. A common mistake is to have a typo or other error, such
as the `Service` selecting for `run=hostnames`, but the `Deployment` specifying
`app=hostnames`.
This confirms that the endpoints controller has found the correct Pods for
your Service. If the `ENDPOINTS` column is `<none>`, you should check that
the `spec.selector` field of your Service actually selects for
`metadata.labels` values on your Pods. A common mistake is to have a typo or
other error, such as the Service selecting for `app=hostnames`, but the
Deployment specifying `run=hostnames`.
## Are the Pods working?
At this point, we know that your `Service` exists and has selected your `Pods`.
Let's check that the `Pods` are actually working - we can bypass the `Service`
mechanism and go straight to the `Pods`.
At this point, you know that your Service exists and has selected your Pods.
At the beginning of this walk-through, you verified the Pods themselves.
Let's check again that the Pods are actually working - you can bypass the
Service mechanism and go straight to the Pods, as listed by the Endpoints
above.
{{< note >}}
These commands use the `Pod` port (9376), rather than the `Service` port (80).
These commands use the Pod port (9376), rather than the Service port (80).
{{< /note >}}
From within a Pod:
```shell
u@pod$ wget -qO- 10.244.0.5:9376
for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do
wget -qO- $ep
done
```
This should produce something like:
```
hostnames-0uton
pod $ wget -qO- 10.244.0.6:9376
hostnames-bvc05
u@pod$ wget -qO- 10.244.0.7:9376
hostnames-yp2kp
```
We expect each `Pod` in the `Endpoints` list to return its own hostname. If
You expect each Pod in the Endpoints list to return its own hostname. If
this is not what happens (or whatever the correct behavior is for your own
`Pods`), you should investigate what's happening there. You might find
`kubectl logs` to be useful or `kubectl exec` directly to your `Pods` and check
service from there.
Another thing to check is that your `Pods` are not crashing or being restarted.
Frequent restarts could lead to intermittent connectivity issues.
```shell
kubectl get pods -l app=hostnames
NAME READY STATUS RESTARTS AGE
hostnames-632524106-bbpiw 1/1 Running 0 2m
hostnames-632524106-ly40y 1/1 Running 0 2m
hostnames-632524106-tlaok 1/1 Running 0 2m
```
If the restart count is high, read more about how to [debug
pods](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#debugging-pods).
Pods), you should investigate what's happening there.
## Is the kube-proxy working?
If you get here, your `Service` is running, has `Endpoints`, and your `Pods`
are actually serving. At this point, the whole `Service` proxy mechanism is
If you get here, your Service is running, has Endpoints, and your Pods
are actually serving. At this point, the whole Service proxy mechanism is
suspect. Let's confirm it, piece by piece.
The default implementation of Services, and the one used on most clusters, is
kube-proxy. This is a program that runs on every node and configures one of a
small set of mechanisms for providing the Service abstraction. If your
cluster does not use kube-proxy, the following sections will not apply, and you
will have to investigate whatever implementation of Services you are using.
### Is kube-proxy running?
Confirm that `kube-proxy` is running on your `Nodes`. You should get something
like the below:
Confirm that `kube-proxy` is running on your Nodes. Running directly on a
Node, you should get something like the below:
```shell
u@node$ ps auxw | grep kube-proxy
ps auxw | grep kube-proxy
```
```none
root 4194 0.4 0.1 101864 17696 ? Sl Jul04 25:43 /usr/local/bin/kube-proxy --master=https://kubernetes-master --kubeconfig=/var/lib/kube-proxy/kubeconfig --v=2
```
Next, confirm that it is not failing something obvious, like contacting the
master. To do this, you'll have to look at the logs. Accessing the logs
depends on your `Node` OS. On some OSes it is a file, such as
depends on your Node OS. On some OSes it is a file, such as
/var/log/kube-proxy.log, while other OSes use `journalctl` to access logs. You
should see something like:
@@ -463,7 +526,7 @@ I1027 22:14:54.040223 5063 proxier.go:294] Adding new service "kube-system/ku
```
If you see error messages about not being able to contact the master, you
should double-check your `Node` configuration and installation steps.
should double-check your Node configuration and installation steps.
One of the possible reasons that `kube-proxy` cannot run correctly is that the
required `conntrack` binary cannot be found. This may happen on some Linux
@@ -472,36 +535,19 @@ installing Kubernetes from scratch. If this is the case, you need to manually
install the `conntrack` package (e.g. `sudo apt install conntrack` on Ubuntu)
and then retry.
### Is kube-proxy writing iptables rules?
Kube-proxy can run in one of a few modes. In the log listed above, the
line `Using iptables Proxier` indicates that kube-proxy is running in
"iptables" mode. The most common other mode is "ipvs". The older "userspace"
mode has largely been replaced by these.
One of the main responsibilities of `kube-proxy` is to write the `iptables`
rules which implement `Services`. Let's check that those rules are getting
written.
#### Iptables mode
The kube-proxy can run in "userspace" mode, "iptables" mode or "ipvs" mode.
Hopefully you are using the "iptables" mode or "ipvs" mode. You
should see one of the following cases.
#### Userspace
In "iptables" mode, you should see something like the following on a Node:
```shell
u@node$ iptables-save | grep hostnames
-A KUBE-PORTALS-CONTAINER -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j REDIRECT --to-ports 48577
-A KUBE-PORTALS-HOST -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j DNAT --to-destination 10.240.115.247:48577
iptables-save | grep hostnames
```
There should be 2 rules for each port on your `Service` (just one in this
example) - a "KUBE-PORTALS-CONTAINER" and a "KUBE-PORTALS-HOST". If you do
not see these, try restarting `kube-proxy` with the `-v` flag set to 4, and
then look at the logs again.
Almost nobody should be using the "userspace" mode any more, so we won't spend
more time on it here.
#### Iptables
```shell
u@node$ iptables-save | grep hostnames
```none
-A KUBE-SEP-57KPRZ3JQVENLNBR -s 10.244.3.6/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000
-A KUBE-SEP-57KPRZ3JQVENLNBR -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.3.6:9376
-A KUBE-SEP-WNBA2IHDGP2BOBGZ -s 10.244.1.7/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000
@@ -514,15 +560,20 @@ u@node$ iptables-save | grep hostnames
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -j KUBE-SEP-57KPRZ3JQVENLNBR
```
There should be 1 rule in `KUBE-SERVICES`, 1 or 2 rules per endpoint in
`KUBE-SVC-(hash)` (depending on `SessionAffinity`), one `KUBE-SEP-(hash)` chain
per endpoint, and a few rules in each `KUBE-SEP-(hash)` chain. The exact rules
will vary based on your exact config (including node-ports and load-balancers).
For each port of each Service, there should be 1 rule in `KUBE-SERVICES` and
one `KUBE-SVC-<hash>` chain. For each Pod endpoint, there should be a small
number of rules in that `KUBE-SVC-<hash>` and one `KUBE-SEP-<hash>` chain with
a small number of rules in it. The exact rules will vary based on your exact
config (including node-ports and load-balancers).
#### IPVS
#### IPVS mode
In "ipvs" mode, you should see something like the following on a Node:
```shell
u@node$ ipvsadm -ln
ipvsadm -ln
```
```none
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
...
@@ -533,14 +584,39 @@ TCP 10.0.1.175:80 rr
...
```
IPVS proxy will create a virtual server for each service address(e.g. Cluster IP, External IP, NodePort IP, Load Balancer IP etc.) and some corresponding real servers for endpoints of the service, if any. In this example, service hostnames(`10.0.1.175:80`) has 3 endpoints(`10.244.0.5:9376`, `10.244.0.6:9376`, `10.244.0.7:9376`) and you'll get results similar to above.
For each port of each Service, plus any NodePorts, external IPs, and
load-balancer IPs, kube-proxy will create a virtual server. For each Pod
endpoint, it will create corresponding real servers. In this example, service
hostnames(`10.0.1.175:80`) has 3 endpoints(`10.244.0.5:9376`,
`10.244.0.6:9376`, `10.244.0.7:9376`).
#### Userspace mode
In rare cases, you may be using "userspace" mode. From your Node:
```shell
iptables-save | grep hostnames
```
```none
-A KUBE-PORTALS-CONTAINER -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j REDIRECT --to-ports 48577
-A KUBE-PORTALS-HOST -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j DNAT --to-destination 10.240.115.247:48577
```
There should be 2 rules for each port of your Service (just one in this
example) - a "KUBE-PORTALS-CONTAINER" and a "KUBE-PORTALS-HOST".
Almost nobody should be using the "userspace" mode any more, so you won't spend
more time on it here.
### Is kube-proxy proxying?
Assuming you do see the above rules, try again to access your `Service` by IP:
Assuming you do see one the above cases, try again to access your Service by
IP from one of your Nodes:
```shell
u@node$ curl 10.0.1.175:80
curl 10.0.1.175:80
```
```none
hostnames-0uton
```
@@ -548,31 +624,36 @@ If this fails and you are using the userspace proxy, you can try accessing the
proxy directly. If you are using the iptables proxy, skip this section.
Look back at the `iptables-save` output above, and extract the
port number that `kube-proxy` is using for your `Service`. In the above
port number that `kube-proxy` is using for your Service. In the above
examples it is "48577". Now connect to that:
```shell
u@node$ curl localhost:48577
curl localhost:48577
```
```none
hostnames-yp2kp
```
If this still fails, look at the `kube-proxy` logs for specific lines like:
```shell
```none
Setting endpoints for default/hostnames:default to [10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376]
```
If you don't see those, try restarting `kube-proxy` with the `-v` flag set to 4, and
then look at the logs again.
### A Pod cannot reach itself via Service IP
### Edge case: A Pod fails to reach itself via the Service IP {#a-pod-fails-to-reach-itself-via-the-service-ip}
This might sound unlikely, but it does happen and it is supposed to work.
This can happen when the network is not properly configured for "hairpin"
traffic, usually when `kube-proxy` is running in `iptables` mode and Pods
are connected with bridge network. The `Kubelet` exposes a `hairpin-mode`
[flag](/docs/admin/kubelet/) that allows endpoints of a Service to loadbalance back to themselves
if they try to access their own Service VIP. The `hairpin-mode` flag must either be
set to `hairpin-veth` or `promiscuous-bridge`.
[flag](/docs/admin/kubelet/) that allows endpoints of a Service to loadbalance
back to themselves if they try to access their own Service VIP. The
`hairpin-mode` flag must either be set to `hairpin-veth` or
`promiscuous-bridge`.
The common steps to trouble shoot this are as follows:
@@ -581,9 +662,10 @@ You should see something like the below. `hairpin-mode` is set to
`promiscuous-bridge` in the following example.
```shell
u@node$ ps auxw|grep kubelet
ps auxw | grep kubelet
```
```none
root 3392 1.1 0.8 186804 65208 ? Sl 00:51 11:11 /usr/local/bin/kubelet --enable-debugging-handlers=true --config=/etc/kubernetes/manifests --allow-privileged=True --v=4 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --configure-cbr0=true --cgroup-root=/ --system-cgroups=/system --hairpin-mode=promiscuous-bridge --runtime-cgroups=/docker-daemon --kubelet-cgroups=/kubelet --babysit-daemons=true --max-pods=110 --serialize-image-pulls=false --outofdisk-transition-frequency=0
```
* Confirm the effective `hairpin-mode`. To do this, you'll have to look at
@@ -594,7 +676,7 @@ match `--hairpin-mode` flag due to compatibility. Check if there is any log
lines with key word `hairpin` in kubelet.log. There should be log lines
indicating the effective hairpin mode, like something below.
```shell
```none
I0629 00:51:43.648698 3252 kubelet.go:380] Hairpin mode set to "promiscuous-bridge"
```
@@ -604,6 +686,8 @@ you should see something like:
```shell
for intf in /sys/devices/virtual/net/cbr0/brif/*; do cat $intf/hairpin_mode; done
```
```none
1
1
1
@@ -615,20 +699,21 @@ has the permission to manipulate linux bridge on node. If `cbr0` bridge is
used and configured properly, you should see:
```shell
u@node$ ifconfig cbr0 |grep PROMISC
ifconfig cbr0 |grep PROMISC
```
```none
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1460 Metric:1
```
* Seek help if none of above works out.
## Seek help
If you get this far, something very strange is happening. Your `Service` is
running, has `Endpoints`, and your `Pods` are actually serving. You have DNS
working, `iptables` rules installed, and `kube-proxy` does not seem to be
misbehaving. And yet your `Service` is not working. You should probably let
us know, so we can help investigate!
If you get this far, something very strange is happening. Your Service is
running, has Endpoints, and your Pods are actually serving. You have DNS
working, and `kube-proxy` does not seem to be misbehaving. And yet your
Service is not working. Please let us know what is going on, so we can help
investigate!
Contact us on
[Slack](/docs/troubleshooting/#slack) or
@@ -2,23 +2,19 @@
reviewers:
- jessfraz
title: Inject Information into Pods Using a PodPreset
min-kubernetes-server-version: v1.10
content_template: templates/task
weight: 60
---
{{% capture overview %}}
You can use a `PodPreset` object to inject information like secrets, volume
mounts, and environment variables etc into pods at creation time.
This task shows some examples on using the `PodPreset` resource.
This page shows how to use PodPreset objects to inject information like {{< glossary_tooltip text="Secrets" term_id="secret" >}}, volume mounts, and {{< glossary_tooltip text="environment variables" term_id="container-env-variables" >}} into Pods at creation time.
{{% /capture %}}
{{% capture prerequisites %}}
Get an overview of PodPresets at
[Understanding Pod Presets](/docs/concepts/workloads/pods/podpreset/).
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
{{% /capture %}}
@@ -26,157 +22,298 @@ Get an overview of PodPresets at
{{% capture steps %}}
## Simple Pod Spec Example
## Use Pod presets to inject environment variables and volumes
This is a simple example to show how a Pod spec is modified by the Pod
Preset.
In this step, you create a preset that has a volume mount and one environment variable.
Here is the manifest for the PodPreset:
{{< codenew file="podpreset/preset.yaml" >}}
The name of a PodPreset object must be a valid
[DNS subdomain name](/docs/concepts/overview/working-with-objects/names#dns-subdomain-names).
In the manifest, you can see that the preset has an environment variable definition called `DB_PORT`
and a volume mount definition called `cache-volume` which is mounted under `/cache`. The {{< glossary_tooltip text="selector" term_id="selector" >}} specifies that
the preset will act upon any Pod that is labeled `role:frontend`.
Create the PodPreset:
```shell
kubectl apply -f https://k8s.io/examples/podpreset/preset.yaml
```
Examine the created PodPreset:
Verify that the PodPreset has been created:
```shell
kubectl get podpreset
```
```
NAME AGE
allow-database 1m
NAME CREATED AT
allow-database 2020-01-24T08:54:29Z
```
The new PodPreset will act upon any pod that has label `role: frontend`.
This manifest defines a Pod labelled `role: frontend` (matching the PodPreset's selector):
{{< codenew file="podpreset/pod.yaml" >}}
Create a pod:
Create the Pod:
```shell
kubectl create -f https://k8s.io/examples/podpreset/pod.yaml
```
List the running Pods:
Verify that the Pod is running:
```shell
kubectl get pods
```
The output shows that the Pod is running:
```
NAME READY STATUS RESTARTS AGE
website 1/1 Running 0 4m
```
**Pod spec after admission controller:**
{{< codenew file="podpreset/merged.yaml" >}}
To see above output, run the following command:
View the Pod spec altered by the admission controller in order to see the effects of the preset
having been applied:
```shell
kubectl get pod website -o yaml
```
## Pod Spec with ConfigMap Example
{{< codenew file="podpreset/merged.yaml" >}}
This is an example to show how a Pod spec is modified by the Pod Preset
that defines a `ConfigMap` for Environment Variables.
The `DB_PORT` environment variable, the `volumeMount` and the `podpreset.admission.kubernetes.io` annotation
of the Pod verify that the preset has been applied.
**User submitted pod spec:**
## Pod spec with ConfigMap example
{{< codenew file="podpreset/pod.yaml" >}}
This is an example to show how a Pod spec is modified by a Pod preset
that references a ConfigMap containing environment variables.
**User submitted `ConfigMap`:**
Here is the manifest containing the definition of the ConfigMap:
{{< codenew file="podpreset/configmap.yaml" >}}
**Example Pod Preset:**
Create the ConfigMap:
```shell
kubectl create -f https://k8s.io/examples/podpreset/configmap.yaml
```
Here is a PodPreset manifest referencing that ConfigMap:
{{< codenew file="podpreset/allow-db.yaml" >}}
**Pod spec after admission controller:**
Create the preset that references the ConfigMap:
{{< codenew file="podpreset/allow-db-merged.yaml" >}}
```shell
kubectl create -f https://k8s.io/examples/podpreset/allow-db.yaml
```
## ReplicaSet with Pod Spec Example
The following example shows that only the pod spec is modified by the Pod
Preset.
**User submitted ReplicaSet:**
{{< codenew file="podpreset/replicaset.yaml" >}}
**Example Pod Preset:**
{{< codenew file="podpreset/preset.yaml" >}}
**Pod spec after admission controller:**
Note that the ReplicaSet spec was not changed, users have to check individual pods
to validate that the PodPreset has been applied.
{{< codenew file="podpreset/replicaset-merged.yaml" >}}
## Multiple PodPreset Example
This is an example to show how a Pod spec is modified by multiple Pod
Injection Policies.
**User submitted pod spec:**
The following manifest defines a Pod matching the PodPreset for this example:
{{< codenew file="podpreset/pod.yaml" >}}
**Example Pod Preset:**
Create the Pod:
```shell
kubectl create -f https://k8s.io/examples/podpreset/pod.yaml
```
View the Pod spec altered by the admission controller in order to see the effects of the preset
having been applied:
```shell
kubectl get pod website -o yaml
```
{{< codenew file="podpreset/allow-db-merged.yaml" >}}
The `DB_PORT` environment variable and the `podpreset.admission.kubernetes.io` annotation of the Pod
verify that the preset has been applied.
## ReplicaSet with Pod spec example
This is an example to show that only Pod specs are modified by Pod presets. Other workload types
like ReplicaSets or Deployments are unaffected.
Here is the manifest for the PodPreset for this example:
{{< codenew file="podpreset/preset.yaml" >}}
**Another Pod Preset:**
Create the preset:
```shell
kubectl apply -f https://k8s.io/examples/podpreset/preset.yaml
```
This manifest defines a ReplicaSet that manages three application Pods:
{{< codenew file="podpreset/replicaset.yaml" >}}
Create the ReplicaSet:
```shell
kubectl create -f https://k8s.io/examples/podpreset/replicaset.yaml
```
Verify that the Pods created by the ReplicaSet are running:
```shell
kubectl get pods
```
The output shows that the Pods are running:
```
NAME READY STATUS RESTARTS AGE
frontend-2l94q 1/1 Running 0 2m18s
frontend-6vdgn 1/1 Running 0 2m18s
frontend-jzt4p 1/1 Running 0 2m18s
```
View the `spec` of the ReplicaSet:
```shell
kubectl get replicasets frontend -o yaml
```
{{< note >}}
The ReplicaSet object's `spec` was not changed, nor does the ReplicaSet contain a
`podpreset.admission.kubernetes.io` annotation. This is because a PodPreset only
applies to Pod objects.
To see the effects of the preset having been applied, you need to look at individual Pods.
{{< /note >}}
The command to view the specs of the affected Pods is:
```shell
kubectl get pod --selector=role=frontend -o yaml
```
{{< codenew file="podpreset/replicaset-merged.yaml" >}}
Again the `podpreset.admission.kubernetes.io` annotation of the Pods
verifies that the preset has been applied.
## Multiple Pod presets example
This is an example to show how a Pod spec is modified by multiple Pod presets.
Here is the manifest for the first PodPreset:
{{< codenew file="podpreset/preset.yaml" >}}
Create the first PodPreset for this example:
```shell
kubectl apply -f https://k8s.io/examples/podpreset/preset.yaml
```
Here is the manifest for the second PodPreset:
{{< codenew file="podpreset/proxy.yaml" >}}
**Pod spec after admission controller:**
Create the second preset:
```shell
kubectl apply -f https://k8s.io/examples/podpreset/proxy.yaml
```
Here's a manifest containing the definition of an applicable Pod (matched by two PodPresets):
{{< codenew file="podpreset/pod.yaml" >}}
Create the Pod:
```shell
kubectl create -f https://k8s.io/examples/podpreset/pod.yaml
```
View the Pod spec altered by the admission controller in order to see the effects of both presets
having been applied:
```shell
kubectl get pod website -o yaml
```
{{< codenew file="podpreset/multi-merged.yaml" >}}
## Conflict Example
The `DB_PORT` environment variable, the `proxy-volume` VolumeMount and the two `podpreset.admission.kubernetes.io`
annotations of the Pod verify that both presets have been applied.
This is an example to show how a Pod spec is not modified by the Pod Preset
when there is a conflict.
## Conflict example
**User submitted pod spec:**
This is an example to show how a Pod spec is not modified by a Pod preset when there is a conflict.
The conflict in this example consists of a `VolumeMount` in the PodPreset conflicting with a Pod that defines the same `mountPath`.
{{< codenew file="podpreset/conflict-pod.yaml" >}}
**Example Pod Preset:**
Here is the manifest for the PodPreset:
{{< codenew file="podpreset/conflict-preset.yaml" >}}
**Pod spec after admission controller will not change because of the conflict:**
Note the `mountPath` value of `/cache`.
Create the preset:
```shell
kubectl apply -f https://k8s.io/examples/podpreset/conflict-preset.yaml
```
Here is the manifest for the Pod:
{{< codenew file="podpreset/conflict-pod.yaml" >}}
**If we run `kubectl describe...` we can see the event:**
Note the volumeMount element with the same path as in the PodPreset.
Create the Pod:
```shell
kubectl describe ...
```
```
....
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Tue, 07 Feb 2017 16:56:12 -0700 Tue, 07 Feb 2017 16:56:12 -0700 1 {podpreset.admission.kubernetes.io/podpreset-allow-database } conflict Conflict on pod preset. Duplicate mountPath /cache.
kubectl create -f https://k8s.io/examples/podpreset/conflict-pod.yaml
```
## Deleting a Pod Preset
View the Pod spec:
Once you don't need a pod preset anymore, you can delete it with `kubectl`:
```shell
kubectl get pod website -o yaml
```
{{< codenew file="podpreset/conflict-pod.yaml" >}}
You can see there is no preset annotation (`podpreset.admission.kubernetes.io`). Seeing no annotation tells you that no preset has not been applied to the Pod.
However, the
[PodPreset admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podpreset)
logs a warning containing details of the conflict.
You can view the warning using `kubectl`:
```shell
kubectl -n kube-system logs -l=component=kube-apiserver
```
The output should look similar to:
```
W1214 13:00:12.987884 1 admission.go:147] conflict occurred while applying podpresets: allow-database on pod: err: merging volume mounts for allow-database has a conflict on mount path /cache:
v1.VolumeMount{Name:"other-volume", ReadOnly:false, MountPath:"/cache", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}
does not match
core.VolumeMount{Name:"cache-volume", ReadOnly:false, MountPath:"/cache", SubPath:"", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:""}
in container
```
Note the conflict message on the path for the VolumeMount.
## Deleting a PodPreset
Once you don't need a PodPreset anymore, you can delete it with `kubectl`:
```shell
kubectl delete podpreset allow-database
```
The output shows that the PodPreset was deleted:
```
podpreset "allow-database" deleted
```
@@ -119,7 +119,7 @@ metadata:
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.7.9","name":"nginx",
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
@@ -136,7 +136,7 @@ spec:
app: nginx
spec:
containers:
- image: nginx:1.7.9
- image: nginx:1.14.2
# ...
name: nginx
ports:
@@ -199,7 +199,7 @@ metadata:
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.7.9","name":"nginx",
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
@@ -216,7 +216,7 @@ spec:
app: nginx
spec:
containers:
- image: nginx:1.7.9
- image: nginx:1.14.2
# ...
name: nginx
ports:
@@ -255,7 +255,7 @@ metadata:
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.7.9","name":"nginx",
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
@@ -273,7 +273,7 @@ spec:
app: nginx
spec:
containers:
- image: nginx:1.7.9
- image: nginx:1.14.2
# ...
name: nginx
ports:
@@ -282,7 +282,7 @@ spec:
```
Update the `simple_deployment.yaml` configuration file to change the image from
`nginx:1.7.9` to `nginx:1.11.9`, and delete the `minReadySeconds` field:
`nginx:1.14.2` to `nginx:1.16.1`, and delete the `minReadySeconds` field:
{{< codenew file="application/update_deployment.yaml" >}}
@@ -303,7 +303,7 @@ The output shows the following changes to the live configuration:
* The `replicas` field retains the value of 2 set by `kubectl scale`.
This is possible because it is omitted from the configuration file.
* The `image` field has been updated to `nginx:1.11.9` from `nginx:1.7.9`.
* The `image` field has been updated to `nginx:1.16.1` from `nginx:1.14.2`.
* The `last-applied-configuration` annotation has been updated with the new image.
* The `minReadySeconds` field has been cleared.
* The `last-applied-configuration` annotation no longer contains the `minReadySeconds` field.
@@ -320,7 +320,7 @@ metadata:
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.11.9","name":"nginx",
"spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
@@ -338,7 +338,7 @@ spec:
app: nginx
spec:
containers:
- image: nginx:1.11.9 # Set by `kubectl apply`
- image: nginx:1.16.1 # Set by `kubectl apply`
# ...
name: nginx
ports:
@@ -460,7 +460,7 @@ metadata:
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.7.9","name":"nginx",
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
@@ -478,7 +478,7 @@ spec:
app: nginx
spec:
containers:
- image: nginx:1.7.9
- image: nginx:1.14.2
# ...
name: nginx
ports:
@@ -518,7 +518,7 @@ metadata:
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.11.9","name":"nginx",
"spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec:
@@ -536,7 +536,7 @@ spec:
app: nginx
spec:
containers:
- image: nginx:1.11.9 # Set by `kubectl apply`
- image: nginx:1.16.1 # Set by `kubectl apply`
# ...
name: nginx
ports:
@@ -654,7 +654,7 @@ by `name`.
# last-applied-configuration value
containers:
- name: nginx
image: nginx:1.10
image: nginx:1.16
- name: nginx-helper-a # key: nginx-helper-a; will be deleted in result
image: helper:1.3
- name: nginx-helper-b # key: nginx-helper-b; will be retained
@@ -663,7 +663,7 @@ by `name`.
# configuration file value
containers:
- name: nginx
image: nginx:1.10
image: nginx:1.16
- name: nginx-helper-b
image: helper:1.3
- name: nginx-helper-c # key: nginx-helper-c; will be added in result
@@ -672,7 +672,7 @@ by `name`.
# live configuration
containers:
- name: nginx
image: nginx:1.10
image: nginx:1.16
- name: nginx-helper-a
image: helper:1.3
- name: nginx-helper-b
@@ -684,7 +684,7 @@ by `name`.
# result after merge
containers:
- name: nginx
image: nginx:1.10
image: nginx:1.16
# Element nginx-helper-a was deleted
- name: nginx-helper-b
image: helper:1.3
@@ -779,7 +779,7 @@ spec:
app: nginx
spec:
containers:
- image: nginx:1.7.9
- image: nginx:1.14.2
imagePullPolicy: IfNotPresent # defaulted by apiserver
name: nginx
ports:
@@ -819,7 +819,7 @@ spec:
spec:
containers:
- name: nginx
image: nginx:1.7.9
image: nginx:1.14.2
ports:
- containerPort: 80
@@ -834,7 +834,7 @@ spec:
spec:
containers:
- name: nginx
image: nginx:1.7.9
image: nginx:1.14.2
ports:
- containerPort: 80
@@ -852,7 +852,7 @@ spec:
spec:
containers:
- name: nginx
image: nginx:1.7.9
image: nginx:1.14.2
ports:
- containerPort: 80
@@ -870,7 +870,7 @@ spec:
spec:
containers:
- name: nginx
image: nginx:1.7.9
image: nginx:1.14.2
ports:
- containerPort: 80
```
@@ -369,8 +369,8 @@ label, you can specify the following metric block to scale only on GET requests:
type: Object
object:
metric:
name: `http_requests`
selector: `verb=GET`
name: http_requests
selector: {matchLabels: {verb: GET}}
```
This selector uses the same syntax as the full Kubernetes label selectors. The monitoring pipeline
@@ -153,14 +153,14 @@ from the [`kubectl` reference](/docs/reference/generated/kubectl/kubectl-command
## Walkthrough
Let's say you were running version 1.7.9 of nginx:
Let's say you were running version 1.14.2 of nginx:
{{< codenew file="controllers/replication-nginx-1.7.9.yaml" >}}
{{< codenew file="controllers/replication-nginx-1.14.2.yaml" >}}
To update to version 1.9.1, you can use [`kubectl rolling-update --image`](https://git.k8s.io/community/contributors/design-proposals/cli/simple-rolling-update.md) to specify the new image:
To update to version 1.16.1, you can use [`kubectl rolling-update --image`](https://git.k8s.io/community/contributors/design-proposals/cli/simple-rolling-update.md) to specify the new image:
```shell
kubectl rolling-update my-nginx --image=nginx:1.9.1
kubectl rolling-update my-nginx --image=nginx:1.16.1
```
```
Created my-nginx-ccba8fbd8cc8160970f63f9a2696fc46
@@ -213,7 +213,7 @@ This is one example where the immutability of containers is a huge asset.
If you need to update more than just the image (e.g., command arguments, environment variables), you can create a new replication controller, with a new name and distinguishing label value, such as:
{{< codenew file="controllers/replication-nginx-1.9.2.yaml" >}}
{{< codenew file="controllers/replication-nginx-1.16.1.yaml" >}}
and roll it out:
@@ -34,7 +34,7 @@ This page shows how to run an application using a Kubernetes Deployment object.
You can run an application by creating a Kubernetes Deployment object, and you
can describe a Deployment in a YAML file. For example, this YAML file describes
a Deployment that runs the nginx:1.7.9 Docker image:
a Deployment that runs the nginx:1.14.2 Docker image:
{{< codenew file="application/deployment.yaml" >}}
@@ -64,7 +64,7 @@ a Deployment that runs the nginx:1.7.9 Docker image:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.7.9
Image: nginx:1.14.2
Port: 80/TCP
Environment: <none>
Mounts: <none>
@@ -385,6 +385,27 @@ However, the kubectl completion script depends on [**bash-completion**](https://
there are two versions of bash-completion, v1 and v2. V1 is for Bash 3.2 (which is the default on macOS), and v2 is for Bash 4.1+. The kubectl completion script **doesn't work** correctly with bash-completion v1 and Bash 3.2. It requires **bash-completion v2** and **Bash 4.1+**. Thus, to be able to correctly use kubectl completion on macOS, you have to install and use Bash 4.1+ ([*instructions*](https://itnext.io/upgrading-bash-on-macos-7138bd1066ba)). The following instructions assume that you use Bash 4.1+ (that is, any Bash version of 4.1 or newer).
{{< /warning >}}
### Upgrade Bash
The instructions here assume you use Bash 4.1+. You can check your Bash's version by running:
```shell
echo $BASH_VERSION
```
If it is too old, you can install/upgrade it using Homebrew:
```shell
brew install bash
```
Reload your shell and verify that the desired version is being used:
```shell
echo $BASH_VERSION $SHELL
```
Homebrew usually installs it at `/usr/local/bin/bash`.
### Install bash-completion
@@ -86,6 +86,12 @@ The `none` VM driver can result in security and data loss issues.
Before using `--vm-driver=none`, consult [this documentation](https://minikube.sigs.k8s.io/docs/reference/drivers/none/) for more information.
{{< /caution >}}
Minikube also supports a `vm-driver=podman` similar to the Docker driver. Podman run as superuser privilege (root user) is the best way to ensure that your containers have full access to any feature available on your system.
{{< caution >}}
The `podman` driver requires running the containers as root because regular user accounts dont have full access to all operating system features that their containers might need to run.
{{< /caution >}}
### Install Minikube using a package
There are *experimental* packages for Minikube available; you can find Linux (AMD64) packages