whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Do a top-to-bottom review of "debug-services" (#19152)

Browse Source 
* Do a top-to-bottom review of "debug-services"

I frequently refer people to this doc, so I thought it would make sense
to review it.  Overall ot was pretty up to date. I did a cleanup pass
and moved a few things around.  Mostly cosmetic.

* Fix backticked resources in "debug-services"

* Fix code vs output blocks in "debug-services"

* Remove prompts from code blocks

* s/we/you/

* Apply feedback edits
This commit is contained in:
Tim Hockin
2020-03-10 07:17:37 -07:00
committed by GitHub
parent dade10e454
commit 4a02480867
1 changed files with 296 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/tasks/debug-application-cluster/debug-service.md
+296 -211
View File
@@ -8,57 +8,30 @@ title: Debug Services
{{% capture overview %}} {{% capture overview %}}
An issue that comes up rather frequently for new installations of Kubernetes is An issue that comes up rather frequently for new installations of Kubernetes is
that a `Service` is not working properly. You've run your `Deployment` and that a Service is not working properly. You've run your Pods through a
created a `Service`, but you get no response when you try to access it. Deployment (or other workload controller) and created a Service, but you
This document will hopefully help you to figure out what's going wrong. get no response when you try to access it. This document will hopefully help
you to figure out what's going wrong.
{{% /capture %}} {{% /capture %}}
{{% capture body %}} {{% capture body %}}
## Conventions
Throughout this doc you will see various commands that you can run. Some
commands need to be run within a `Pod`, others on a Kubernetes `Node`, and others
can run anywhere you have `kubectl` and credentials for the cluster. To make it
clear what is expected, this document will use the following conventions.
If the command "COMMAND" is expected to run in a `Pod` and produce "OUTPUT":
```shell
u@pod$ COMMAND
OUTPUT
```
If the command "COMMAND" is expected to run on a `Node` and produce "OUTPUT":
```shell
u@node$ COMMAND
OUTPUT
```
If the command is "kubectl ARGS":
```shell
kubectl ARGS
OUTPUT
```
## Running commands in a Pod ## Running commands in a Pod
For many steps here you will want to see what a `Pod` running in the cluster For many steps here you will want to see what a Pod running in the cluster
sees. The simplest way to do this is to run an interactive alpine `Pod`: sees. The simplest way to do this is to run an interactive alpine Pod:
```none ```none
kubectl run -it --rm --restart=Never alpine --image=alpine sh kubectl run -it --rm --restart=Never alpine --image=alpine sh
/ #
``` ```
{{< note >}} {{< note >}}
If you don't see a command prompt, try pressing enter. If you don't see a command prompt, try pressing enter.
{{< /note >}} {{< /note >}}
If you already have a running `Pod` that you prefer to use, you can run a If you already have a running Pod that you prefer to use, you can run a
command in it using: command in it using:
```shell ```shell
@@ -67,21 +40,23 @@ kubectl exec <POD-NAME> -c <CONTAINER-NAME> -- <COMMAND>
## Setup ## Setup
For the purposes of this walk-through, let's run some `Pods`. Since you're For the purposes of this walk-through, let's run some Pods. Since you're
probably debugging your own `Service` you can substitute your own details, or you probably debugging your own Service you can substitute your own details, or you
can follow along and get a second data point. can follow along and get a second data point.
```shell ```shell
kubectl run hostnames --image=k8s.gcr.io/serve_hostname \ kubectl run hostnames --image=k8s.gcr.io/serve_hostname \
--labels=app=hostnames \ --replicas=3
--port=9376 \ ```
--replicas=3 ```none
deployment.apps/hostnames created deployment.apps/hostnames created
``` ```
`kubectl` commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands. `kubectl` commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands.
{{< note >}} {{< note >}}
This is the same as if you started the `Deployment` with the following YAML: This is the same as if you had started the Deployment with the following
YAML:
```yaml ```yaml
apiVersion: apps/v1 apiVersion: apps/v1
@@ -91,61 +66,111 @@ metadata:
spec: spec:
selector: selector:
matchLabels: matchLabels:
app: hostnames run: hostnames
replicas: 3 replicas: 3
template: template:
metadata: metadata:
labels: labels:
app: hostnames run: hostnames
spec: spec:
containers: containers:
- name: hostnames - name: hostnames
image: k8s.gcr.io/serve_hostname image: k8s.gcr.io/serve_hostname
ports:
- containerPort: 9376
protocol: TCP
``` ```
The label "run" is automatically set by `kubectl run` to the name of the
Deployment.
{{< /note >}} {{< /note >}}
Confirm your `Pods` are running: You can confirm your Pods are running:
```shell ```shell
kubectl get pods -l app=hostnames kubectl get pods -l run=hostnames
```
```none
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
hostnames-632524106-bbpiw 1/1 Running 0 2m hostnames-632524106-bbpiw 1/1 Running 0 2m
hostnames-632524106-ly40y 1/1 Running 0 2m hostnames-632524106-ly40y 1/1 Running 0 2m
hostnames-632524106-tlaok 1/1 Running 0 2m hostnames-632524106-tlaok 1/1 Running 0 2m
``` ```
You can also confirm that your Pods are serving. You can get the list of
Pod IP addresses and test them directly.
```shell
kubectl get pods -l run=hostnames \
-o go-template='{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'
```
```none
10.244.0.5
10.244.0.6
10.244.0.7
```
The example container used for this walk-through simply serves its own hostname
via HTTP on port 9376, but if you are debugging your own app, you'll want to
use whatever port number your Pods are listening on.
From within a pod:
```shell
for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do
wget -qO- $ep
done
```
This should produce something like:
```
hostnames-0uton
hostnames-bvc05
hostnames-yp2kp
```
If you are not getting the responses you expect at this point, your Pods
might not be healthy or might not be listening on the port you think they are.
You might find `kubectl logs` to be useful for seeing what is happening, or
perhaps you need to `kubectl exec` directly into your Pods and debug from
there.
Assuming everything has gone to plan so far, you can start to investigate why
your Service doesn't work.
## Does the Service exist? ## Does the Service exist?
The astute reader will have noticed that we did not actually create a `Service` The astute reader will have noticed that you did not actually create a Service
yet - that is intentional. This is a step that sometimes gets forgotten, and yet - that is intentional. This is a step that sometimes gets forgotten, and
is the first thing to check. is the first thing to check.
So what would happen if I tried to access a non-existent `Service`? Assuming you What would happen if you tried to access a non-existent Service? If
have another `Pod` that consumes this `Service` by name you would get something you have another Pod that consumes this Service by name you would get
like: something like:
```shell ```shell
u@pod$ wget -O- hostnames wget -O- hostnames
```
```none
Resolving hostnames (hostnames)... failed: Name or service not known. Resolving hostnames (hostnames)... failed: Name or service not known.
wget: unable to resolve host address 'hostnames' wget: unable to resolve host address 'hostnames'
``` ```
So the first thing to check is whether that `Service` actually exists: The first thing to check is whether that Service actually exists:
```shell ```shell
kubectl get svc hostnames kubectl get svc hostnames
```
```none
No resources found. No resources found.
Error from server (NotFound): services "hostnames" not found Error from server (NotFound): services "hostnames" not found
``` ```
So we have a culprit, let's create the `Service`. As before, this is for the Let's create the Service. As before, this is for the walk-through - you can
walk-through - you can use your own `Service`'s details here. use your own Service's details here.
```shell ```shell
kubectl expose deployment hostnames --port=80 --target-port=9376 kubectl expose deployment hostnames --port=80 --target-port=9376
```
```none
service/hostnames exposed service/hostnames exposed
``` ```
@@ -153,11 +178,16 @@ And read it back, just to be sure:
```shell ```shell
kubectl get svc hostnames kubectl get svc hostnames
```
```none
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hostnames ClusterIP 10.0.1.175 <none> 80/TCP 5s hostnames ClusterIP 10.0.1.175 <none> 80/TCP 5s
``` ```
As before, this is the same as if you had started the `Service` with YAML: Now you know that the Service exists.
{{< note >}}
As before, this is the same as if you had started the Service with YAML:
```yaml ```yaml
apiVersion: v1 apiVersion: v1
@@ -166,7 +196,7 @@ metadata:
name: hostnames name: hostnames
spec: spec:
selector: selector:
app: hostnames run: hostnames
ports: ports:
- name: default - name: default
protocol: TCP protocol: TCP
@@ -174,25 +204,35 @@ spec:
targetPort: 9376 targetPort: 9376
``` ```
Now you can confirm that the `Service` exists. In order to highlight the full range of configuration, the Service you created
here uses a different port number than the Pods. For many real-world
Services, these values might be the same.
{{< /note >}}
## Does the Service work by DNS? ## Does the Service work by DNS name?
From a `Pod` in the same `Namespace`: One of the most common ways that clients consume a Service is through a DNS
name.
From a Pod in the same Namespace:
```shell ```shell
u@pod$ nslookup hostnames nslookup hostnames
```
```none
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: hostnames Name: hostnames
Address 1: 10.0.1.175 hostnames.default.svc.cluster.local Address 1: 10.0.1.175 hostnames.default.svc.cluster.local
``` ```
If this fails, perhaps your `Pod` and `Service` are in different If this fails, perhaps your Pod and Service are in different
`Namespaces`, try a namespace-qualified name: Namespaces, try a namespace-qualified name (again, from within a Pod):
```shell ```shell
u@pod$ nslookup hostnames.default nslookup hostnames.default
```
```none
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: hostnames.default Name: hostnames.default
@@ -200,11 +240,13 @@ Address 1: 10.0.1.175 hostnames.default.svc.cluster.local
``` ```
If this works, you'll need to adjust your app to use a cross-namespace name, or If this works, you'll need to adjust your app to use a cross-namespace name, or
run your app and `Service` in the same `Namespace`. If this still fails, try a run your app and Service in the same Namespace. If this still fails, try a
fully-qualified name: fully-qualified name:
```shell ```shell
u@pod$ nslookup hostnames.default.svc.cluster.local nslookup hostnames.default.svc.cluster.local
```
```none
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: hostnames.default.svc.cluster.local Name: hostnames.default.svc.cluster.local
@@ -212,18 +254,20 @@ Address 1: 10.0.1.175 hostnames.default.svc.cluster.local
``` ```
Note the suffix here: "default.svc.cluster.local". The "default" is the Note the suffix here: "default.svc.cluster.local". The "default" is the
`Namespace` we're operating in. The "svc" denotes that this is a `Service`. Namespace you're operating in. The "svc" denotes that this is a Service.
The "cluster.local" is your cluster domain, which COULD be different in your The "cluster.local" is your cluster domain, which COULD be different in your
own cluster. own cluster.
You can also try this from a `Node` in the cluster: You can also try this from a Node in the cluster:
{{< note >}} {{< note >}}
10.0.0.10 is my DNS `Service`, yours might be different. 10.0.0.10 is the cluster's DNS Service IP, yours might be different.
{{< /note >}} {{< /note >}}
```shell ```shell
u@node$ nslookup hostnames.default.svc.cluster.local 10.0.0.10 nslookup hostnames.default.svc.cluster.local 10.0.0.10
```
```none
Server: 10.0.0.10 Server: 10.0.0.10
Address: 10.0.0.10#53 Address: 10.0.0.10#53
@@ -232,39 +276,49 @@ Address: 10.0.1.175
``` ```
If you are able to do a fully-qualified name lookup but not a relative one, you If you are able to do a fully-qualified name lookup but not a relative one, you
need to check that your `/etc/resolv.conf` file is correct. need to check that your `/etc/resolv.conf` file in your Pod is correct. From
within a Pod:
```shell ```shell
u@pod$ cat /etc/resolv.conf cat /etc/resolv.conf
```
You should see something like:
```
nameserver 10.0.0.10 nameserver 10.0.0.10
search default.svc.cluster.local svc.cluster.local cluster.local example.com search default.svc.cluster.local svc.cluster.local cluster.local example.com
options ndots:5 options ndots:5
``` ```
The `nameserver` line must indicate your cluster's DNS `Service`. This is The `nameserver` line must indicate your cluster's DNS Service. This is
passed into `kubelet` with the `--cluster-dns` flag. passed into `kubelet` with the `--cluster-dns` flag.
The `search` line must include an appropriate suffix for you to find the The `search` line must include an appropriate suffix for you to find the
`Service` name. In this case it is looking for `Services` in the local Service name. In this case it is looking for Services in the local
`Namespace` (`default.svc.cluster.local`), `Services` in all `Namespaces` Namespace ("default.svc.cluster.local"), Services in all Namespaces
(`svc.cluster.local`), and the cluster (`cluster.local`). Depending on your own ("svc.cluster.local"), and lastly for names in the cluster ("cluster.local").
install you might have additional records after that (up to 6 total). The Depending on your own install you might have additional records after that (up
cluster suffix is passed into `kubelet` with the `--cluster-domain` flag. We to 6 total). The cluster suffix is passed into `kubelet` with the
assume that is "cluster.local" in this document, but yours might be different, `--cluster-domain` flag. Throughout this document, the cluster suffix is
in which case you should change that in all of the commands above. assumed to be "cluster.local". Your own clusters might be configured
differently, in which case you should change that in all of the previous
commands.
The `options` line must set `ndots` high enough that your DNS client library The `options` line must set `ndots` high enough that your DNS client library
considers search paths at all. Kubernetes sets this to 5 by default, which is considers search paths at all. Kubernetes sets this to 5 by default, which is
high enough to cover all of the DNS names it generates. high enough to cover all of the DNS names it generates.
### Does any Service exist in DNS? ### Does any Service work by DNS name? {#does-any-service-exist-in-dns}
If the above still fails - DNS lookups are not working for your `Service` - we If the above still fails, DNS lookups are not working for your Service. You
can take a step back and see what else is not working. The Kubernetes master can take a step back and see what else is not working. The Kubernetes master
`Service` should always work: Service should always work. From within a Pod:
```shell ```shell
u@pod$ nslookup kubernetes.default nslookup kubernetes.default
```
```none
Server: 10.0.0.10 Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
@@ -272,34 +326,37 @@ Name: kubernetes.default
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
``` ```
If this fails, you might need to go to the kube-proxy section of this doc, or If this fails, please see the [kube-proxy](#is-the-kube-proxy-working) section
even go back to the top of this document and start over, but instead of of this document, or even go back to the top of this document and start over,
debugging your own `Service`, debug DNS. but instead of debugging your own Service, debug the DNS Service.
## Does the Service work by IP? ## Does the Service work by IP?
Assuming we can confirm that DNS works, the next thing to test is whether your Assuming you have confirmed that DNS works, the next thing to test is whether your
`Service` works at all. From a node in your cluster, access the `Service`'s Service works by its IP address. From a Pod in your cluster, access the
IP (from `kubectl get` above). Service's IP (from `kubectl get` above).
```shell ```shell
u@node$ curl 10.0.1.175:80 for i in $(seq 1 3); do
hostnames-0uton wget -qO- 10.0.1.175:80
done
u@node$ curl 10.0.1.175:80
hostnames-yp2kp
u@node$ curl 10.0.1.175:80
hostnames-bvc05
``` ```
If your `Service` is working, you should get correct responses. If not, there This should produce something like:
```
hostnames-0uton
hostnames-bvc05
hostnames-yp2kp
```
If your Service is working, you should get correct responses. If not, there
are a number of things that could be going wrong. Read on. are a number of things that could be going wrong. Read on.
## Is the Service correct? ## Is the Service defined correctly?
It might sound silly, but you should really double and triple check that your It might sound silly, but you should really double and triple check that your
`Service` is correct and matches your `Pod`'s port. Read back your `Service` Service is correct and matches your Pod's port. Read back your Service
and verify it: and verify it:
```shell ```shell
@@ -316,7 +373,7 @@ kubectl get service hostnames -o json
"resourceVersion": "347189", "resourceVersion": "347189",
"creationTimestamp": "2015-07-07T15:24:29Z", "creationTimestamp": "2015-07-07T15:24:29Z",
"labels": { "labels": {
"app": "hostnames" "run": "hostnames"
} }
}, },
"spec": { "spec": {
@@ -330,7 +387,7 @@ kubectl get service hostnames -o json
} }
], ],
"selector": { "selector": {
"app": "hostnames" "run": "hostnames"
}, },
"clusterIP": "10.0.1.175", "clusterIP": "10.0.1.175",
"type": "ClusterIP", "type": "ClusterIP",
@@ -342,110 +399,116 @@ kubectl get service hostnames -o json
} }
``` ```
* Is the port you are trying to access in `spec.ports[]`? * Is the Service port you are trying to access listed in `spec.ports[]`?
* Is the `targetPort` correct for your `Pods` (many `Pods` choose to use a different port than the `Service`)? * Is the `targetPort` correct for your Pods (some Pods use a different port than the Service)?
* If you meant it to be a numeric port, is it a number (9376) or a * If you meant to use a numeric port, is it a number (9376) or a string "9376"?
string "9376"? * If you meant to use a named port, do your Pods expose a port with the same name?
* If you meant it to be a named port, do your `Pods` expose a port * Is the port's `protocol` correct for your Pods?
with the same name?
* Is the port's `protocol` the same as the `Pod`'s?
## Does the Service have any Endpoints? ## Does the Service have any Endpoints?
If you got this far, we assume that you have confirmed that your `Service` If you got this far, you have confirmed that your Service is correctly
exists and is resolved by DNS. Now let's check that the `Pods` you ran are defined and is resolved by DNS. Now let's check that the Pods you ran are
actually being selected by the `Service`. actually being selected by the Service.
Earlier we saw that the `Pods` were running. We can re-check that: Earlier you saw that the Pods were running. You can re-check that:
```shell ```shell
kubectl get pods -l app=hostnames kubectl get pods -l run=hostnames
```
```none
NAME READY STATUS RESTARTS AGE NAME READY STATUS RESTARTS AGE
hostnames-0uton 1/1 Running 0 1h hostnames-0uton 1/1 Running 0 1h
hostnames-bvc05 1/1 Running 0 1h hostnames-bvc05 1/1 Running 0 1h
hostnames-yp2kp 1/1 Running 0 1h hostnames-yp2kp 1/1 Running 0 1h
``` ```
The "AGE" column says that these `Pods` are about an hour old, which implies that The `-l run=hostnames` argument is a label selector - just like our Service
has.
The "AGE" column says that these Pods are about an hour old, which implies that
they are running fine and not crashing. they are running fine and not crashing.
The `-l app=hostnames` argument is a label selector - just like our `Service` The "RESTARTS" column says that these pods are not crashing frequently or being
has. Inside the Kubernetes system is a control loop which evaluates the restarted. Frequent restarts could lead to intermittent connectivity issues.
selector of every `Service` and saves the results into an `Endpoints` object. If the restart count is high, read more about how to [debug pods](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#debugging-pods).
Inside the Kubernetes system is a control loop which evaluates the selector of
every Service and saves the results into a corresponding Endpoints object.
```shell ```shell
kubectl get endpoints hostnames kubectl get endpoints hostnames
NAME ENDPOINTS NAME ENDPOINTS
hostnames 10.244.0.5:9376,10.244.0.6:9376,10.244.0.7:9376 hostnames 10.244.0.5:9376,10.244.0.6:9376,10.244.0.7:9376
``` ```
This confirms that the endpoints controller has found the correct `Pods` for This confirms that the endpoints controller has found the correct Pods for
your `Service`. If the `hostnames` row is blank, you should check that the your Service. If the `ENDPOINTS` column is `<none>`, you should check that
`spec.selector` field of your `Service` actually selects for `metadata.labels` the `spec.selector` field of your Service actually selects for
values on your `Pods`. A common mistake is to have a typo or other error, such `metadata.labels` values on your Pods. A common mistake is to have a typo or
as the `Service` selecting for `run=hostnames`, but the `Deployment` specifying other error, such as the Service selecting for `app=hostnames`, but the
`app=hostnames`. Deployment specifying `run=hostnames`.
## Are the Pods working? ## Are the Pods working?
At this point, we know that your `Service` exists and has selected your `Pods`. At this point, you know that your Service exists and has selected your Pods.
Let's check that the `Pods` are actually working - we can bypass the `Service` At the beginning of this walk-through, you verified the Pods themselves.
mechanism and go straight to the `Pods`. Let's check again that the Pods are actually working - you can bypass the
Service mechanism and go straight to the Pods, as listed by the Endpoints
above.
{{< note >}} {{< note >}}
These commands use the `Pod` port (9376), rather than the `Service` port (80). These commands use the Pod port (9376), rather than the Service port (80).
{{< /note >}} {{< /note >}}
From within a Pod:
```shell ```shell
u@pod$ wget -qO- 10.244.0.5:9376 for ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376; do
wget -qO- $ep
done
```
This should produce something like:
```
hostnames-0uton hostnames-0uton
pod $ wget -qO- 10.244.0.6:9376
hostnames-bvc05 hostnames-bvc05
u@pod$ wget -qO- 10.244.0.7:9376
hostnames-yp2kp hostnames-yp2kp
``` ```
We expect each `Pod` in the `Endpoints` list to return its own hostname. If You expect each Pod in the Endpoints list to return its own hostname. If
this is not what happens (or whatever the correct behavior is for your own this is not what happens (or whatever the correct behavior is for your own
`Pods`), you should investigate what's happening there. You might find Pods), you should investigate what's happening there.
`kubectl logs` to be useful or `kubectl exec` directly to your `Pods` and check
service from there.
Another thing to check is that your `Pods` are not crashing or being restarted.
Frequent restarts could lead to intermittent connectivity issues.
```shell
kubectl get pods -l app=hostnames
NAME READY STATUS RESTARTS AGE
hostnames-632524106-bbpiw 1/1 Running 0 2m
hostnames-632524106-ly40y 1/1 Running 0 2m
hostnames-632524106-tlaok 1/1 Running 0 2m
```
If the restart count is high, read more about how to [debug
pods](/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#debugging-pods).
## Is the kube-proxy working? ## Is the kube-proxy working?
If you get here, your `Service` is running, has `Endpoints`, and your `Pods` If you get here, your Service is running, has Endpoints, and your Pods
are actually serving. At this point, the whole `Service` proxy mechanism is are actually serving. At this point, the whole Service proxy mechanism is
suspect. Let's confirm it, piece by piece. suspect. Let's confirm it, piece by piece.
The default implementation of Services, and the one used on most clusters, is
kube-proxy. This is a program that runs on every node and configures one of a
small set of mechanisms for providing the Service abstraction. If your
cluster does not use kube-proxy, the following sections will not apply, and you
will have to investigate whatever implementation of Services you are using.
### Is kube-proxy running? ### Is kube-proxy running?
Confirm that `kube-proxy` is running on your `Nodes`. You should get something Confirm that `kube-proxy` is running on your Nodes. Running directly on a
like the below: Node, you should get something like the below:
```shell ```shell
u@node$ ps auxw | grep kube-proxy ps auxw | grep kube-proxy
```
```none
root 4194 0.4 0.1 101864 17696 ? Sl Jul04 25:43 /usr/local/bin/kube-proxy --master=https://kubernetes-master --kubeconfig=/var/lib/kube-proxy/kubeconfig --v=2 root 4194 0.4 0.1 101864 17696 ? Sl Jul04 25:43 /usr/local/bin/kube-proxy --master=https://kubernetes-master --kubeconfig=/var/lib/kube-proxy/kubeconfig --v=2
``` ```
Next, confirm that it is not failing something obvious, like contacting the Next, confirm that it is not failing something obvious, like contacting the
master. To do this, you'll have to look at the logs. Accessing the logs master. To do this, you'll have to look at the logs. Accessing the logs
depends on your `Node` OS. On some OSes it is a file, such as depends on your Node OS. On some OSes it is a file, such as
/var/log/kube-proxy.log, while other OSes use `journalctl` to access logs. You /var/log/kube-proxy.log, while other OSes use `journalctl` to access logs. You
should see something like: should see something like:
@@ -463,7 +526,7 @@ I1027 22:14:54.040223 5063 proxier.go:294] Adding new service "kube-system/ku
``` ```
If you see error messages about not being able to contact the master, you If you see error messages about not being able to contact the master, you
should double-check your `Node` configuration and installation steps. should double-check your Node configuration and installation steps.
One of the possible reasons that `kube-proxy` cannot run correctly is that the One of the possible reasons that `kube-proxy` cannot run correctly is that the
required `conntrack` binary cannot be found. This may happen on some Linux required `conntrack` binary cannot be found. This may happen on some Linux
@@ -472,36 +535,19 @@ installing Kubernetes from scratch. If this is the case, you need to manually
install the `conntrack` package (e.g. `sudo apt install conntrack` on Ubuntu) install the `conntrack` package (e.g. `sudo apt install conntrack` on Ubuntu)
and then retry. and then retry.
### Is kube-proxy writing iptables rules? Kube-proxy can run in one of a few modes. In the log listed above, the
line `Using iptables Proxier` indicates that kube-proxy is running in
"iptables" mode. The most common other mode is "ipvs". The older "userspace"
mode has largely been replaced by these.
One of the main responsibilities of `kube-proxy` is to write the `iptables` #### Iptables mode
rules which implement `Services`. Let's check that those rules are getting
written.
The kube-proxy can run in "userspace" mode, "iptables" mode or "ipvs" mode. In "iptables" mode, you should see something like the following on a Node:
Hopefully you are using the "iptables" mode or "ipvs" mode. You
should see one of the following cases.
#### Userspace
```shell ```shell
u@node$ iptables-save | grep hostnames iptables-save | grep hostnames
-A KUBE-PORTALS-CONTAINER -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j REDIRECT --to-ports 48577
-A KUBE-PORTALS-HOST -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j DNAT --to-destination 10.240.115.247:48577
``` ```
```none
There should be 2 rules for each port on your `Service` (just one in this
example) - a "KUBE-PORTALS-CONTAINER" and a "KUBE-PORTALS-HOST". If you do
not see these, try restarting `kube-proxy` with the `-v` flag set to 4, and
then look at the logs again.
Almost nobody should be using the "userspace" mode any more, so we won't spend
more time on it here.
#### Iptables
```shell
u@node$ iptables-save | grep hostnames
-A KUBE-SEP-57KPRZ3JQVENLNBR -s 10.244.3.6/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000 -A KUBE-SEP-57KPRZ3JQVENLNBR -s 10.244.3.6/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000
-A KUBE-SEP-57KPRZ3JQVENLNBR -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.3.6:9376 -A KUBE-SEP-57KPRZ3JQVENLNBR -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.3.6:9376
-A KUBE-SEP-WNBA2IHDGP2BOBGZ -s 10.244.1.7/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000 -A KUBE-SEP-WNBA2IHDGP2BOBGZ -s 10.244.1.7/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000
@@ -514,15 +560,20 @@ u@node$ iptables-save | grep hostnames
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -j KUBE-SEP-57KPRZ3JQVENLNBR -A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -j KUBE-SEP-57KPRZ3JQVENLNBR
``` ```
There should be 1 rule in `KUBE-SERVICES`, 1 or 2 rules per endpoint in For each port of each Service, there should be 1 rule in `KUBE-SERVICES` and
`KUBE-SVC-(hash)` (depending on `SessionAffinity`), one `KUBE-SEP-(hash)` chain one `KUBE-SVC-<hash>` chain. For each Pod endpoint, there should be a small
per endpoint, and a few rules in each `KUBE-SEP-(hash)` chain. The exact rules number of rules in that `KUBE-SVC-<hash>` and one `KUBE-SEP-<hash>` chain with
will vary based on your exact config (including node-ports and load-balancers). a small number of rules in it. The exact rules will vary based on your exact
config (including node-ports and load-balancers).
#### IPVS #### IPVS mode
In "ipvs" mode, you should see something like the following on a Node:
```shell ```shell
u@node$ ipvsadm -ln ipvsadm -ln
```
```none
Prot LocalAddress:Port Scheduler Flags Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn -> RemoteAddress:Port Forward Weight ActiveConn InActConn
... ...
@@ -533,14 +584,39 @@ TCP 10.0.1.175:80 rr
... ...
``` ```
IPVS proxy will create a virtual server for each service address(e.g. Cluster IP, External IP, NodePort IP, Load Balancer IP etc.) and some corresponding real servers for endpoints of the service, if any. In this example, service hostnames(`10.0.1.175:80`) has 3 endpoints(`10.244.0.5:9376`, `10.244.0.6:9376`, `10.244.0.7:9376`) and you'll get results similar to above. For each port of each Service, plus any NodePorts, external IPs, and
load-balancer IPs, kube-proxy will create a virtual server. For each Pod
endpoint, it will create corresponding real servers. In this example, service
hostnames(`10.0.1.175:80`) has 3 endpoints(`10.244.0.5:9376`,
`10.244.0.6:9376`, `10.244.0.7:9376`).
#### Userspace mode
In rare cases, you may be using "userspace" mode. From your Node:
```shell
iptables-save | grep hostnames
```
```none
-A KUBE-PORTALS-CONTAINER -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j REDIRECT --to-ports 48577
-A KUBE-PORTALS-HOST -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j DNAT --to-destination 10.240.115.247:48577
```
There should be 2 rules for each port of your Service (just one in this
example) - a "KUBE-PORTALS-CONTAINER" and a "KUBE-PORTALS-HOST".
Almost nobody should be using the "userspace" mode any more, so you won't spend
more time on it here.
### Is kube-proxy proxying? ### Is kube-proxy proxying?
Assuming you do see the above rules, try again to access your `Service` by IP: Assuming you do see one the above cases, try again to access your Service by
IP from one of your Nodes:
```shell ```shell
u@node$ curl 10.0.1.175:80 curl 10.0.1.175:80
```
```none
hostnames-0uton hostnames-0uton
``` ```
@@ -548,31 +624,36 @@ If this fails and you are using the userspace proxy, you can try accessing the
proxy directly. If you are using the iptables proxy, skip this section. proxy directly. If you are using the iptables proxy, skip this section.
Look back at the `iptables-save` output above, and extract the Look back at the `iptables-save` output above, and extract the
port number that `kube-proxy` is using for your `Service`. In the above port number that `kube-proxy` is using for your Service. In the above
examples it is "48577". Now connect to that: examples it is "48577". Now connect to that:
```shell ```shell
u@node$ curl localhost:48577 curl localhost:48577
```
```none
hostnames-yp2kp hostnames-yp2kp
``` ```
If this still fails, look at the `kube-proxy` logs for specific lines like: If this still fails, look at the `kube-proxy` logs for specific lines like:
```shell ```none
Setting endpoints for default/hostnames:default to [10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376] Setting endpoints for default/hostnames:default to [10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376]
``` ```
If you don't see those, try restarting `kube-proxy` with the `-v` flag set to 4, and If you don't see those, try restarting `kube-proxy` with the `-v` flag set to 4, and
then look at the logs again. then look at the logs again.
### A Pod cannot reach itself via Service IP ### Edge case: A Pod fails to reach itself via the Service IP {#a-pod-fails-to-reach-itself-via-the-service-ip}
This might sound unlikely, but it does happen and it is supposed to work.
This can happen when the network is not properly configured for "hairpin" This can happen when the network is not properly configured for "hairpin"
traffic, usually when `kube-proxy` is running in `iptables` mode and Pods traffic, usually when `kube-proxy` is running in `iptables` mode and Pods
are connected with bridge network. The `Kubelet` exposes a `hairpin-mode` are connected with bridge network. The `Kubelet` exposes a `hairpin-mode`
[flag](/docs/admin/kubelet/) that allows endpoints of a Service to loadbalance back to themselves [flag](/docs/admin/kubelet/) that allows endpoints of a Service to loadbalance
if they try to access their own Service VIP. The `hairpin-mode` flag must either be back to themselves if they try to access their own Service VIP. The
set to `hairpin-veth` or `promiscuous-bridge`. `hairpin-mode` flag must either be set to `hairpin-veth` or
`promiscuous-bridge`.
The common steps to trouble shoot this are as follows: The common steps to trouble shoot this are as follows:
@@ -581,9 +662,10 @@ You should see something like the below. `hairpin-mode` is set to
`promiscuous-bridge` in the following example. `promiscuous-bridge` in the following example.
```shell ```shell
u@node$ ps auxw|grep kubelet ps auxw | grep kubelet
```
```none
root 3392 1.1 0.8 186804 65208 ? Sl 00:51 11:11 /usr/local/bin/kubelet --enable-debugging-handlers=true --config=/etc/kubernetes/manifests --allow-privileged=True --v=4 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --configure-cbr0=true --cgroup-root=/ --system-cgroups=/system --hairpin-mode=promiscuous-bridge --runtime-cgroups=/docker-daemon --kubelet-cgroups=/kubelet --babysit-daemons=true --max-pods=110 --serialize-image-pulls=false --outofdisk-transition-frequency=0 root 3392 1.1 0.8 186804 65208 ? Sl 00:51 11:11 /usr/local/bin/kubelet --enable-debugging-handlers=true --config=/etc/kubernetes/manifests --allow-privileged=True --v=4 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --configure-cbr0=true --cgroup-root=/ --system-cgroups=/system --hairpin-mode=promiscuous-bridge --runtime-cgroups=/docker-daemon --kubelet-cgroups=/kubelet --babysit-daemons=true --max-pods=110 --serialize-image-pulls=false --outofdisk-transition-frequency=0
``` ```
* Confirm the effective `hairpin-mode`. To do this, you'll have to look at * Confirm the effective `hairpin-mode`. To do this, you'll have to look at
@@ -594,7 +676,7 @@ match `--hairpin-mode` flag due to compatibility. Check if there is any log
lines with key word `hairpin` in kubelet.log. There should be log lines lines with key word `hairpin` in kubelet.log. There should be log lines
indicating the effective hairpin mode, like something below. indicating the effective hairpin mode, like something below.
```shell ```none
I0629 00:51:43.648698 3252 kubelet.go:380] Hairpin mode set to "promiscuous-bridge" I0629 00:51:43.648698 3252 kubelet.go:380] Hairpin mode set to "promiscuous-bridge"
``` ```
@@ -604,6 +686,8 @@ you should see something like:
```shell ```shell
for intf in /sys/devices/virtual/net/cbr0/brif/*; do cat $intf/hairpin_mode; done for intf in /sys/devices/virtual/net/cbr0/brif/*; do cat $intf/hairpin_mode; done
```
```none
1 1
1 1
1 1
@@ -615,20 +699,21 @@ has the permission to manipulate linux bridge on node. If `cbr0` bridge is
used and configured properly, you should see: used and configured properly, you should see:
```shell ```shell
u@node$ ifconfig cbr0 |grep PROMISC ifconfig cbr0 |grep PROMISC
```
```none
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1460 Metric:1 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1460 Metric:1
``` ```
* Seek help if none of above works out. * Seek help if none of above works out.
## Seek help ## Seek help
If you get this far, something very strange is happening. Your `Service` is If you get this far, something very strange is happening. Your Service is
running, has `Endpoints`, and your `Pods` are actually serving. You have DNS running, has Endpoints, and your Pods are actually serving. You have DNS
working, `iptables` rules installed, and `kube-proxy` does not seem to be working, and `kube-proxy` does not seem to be misbehaving. And yet your
misbehaving. And yet your `Service` is not working. You should probably let Service is not working. Please let us know what is going on, so we can help
us know, so we can help investigate! investigate!
Contact us on Contact us on
[Slack](/docs/troubleshooting/#slack) or [Slack](/docs/troubleshooting/#slack) or