whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Syntax highlighting

Browse Source
This commit is contained in:
John Mulhausen
2016-02-16 17:53:42 -08:00
parent bbeb111863
commit ee72211075
115 changed files with 1922 additions and 1919 deletions
Download Patch File Download Diff File Expand all files Collapse all files
v1.1/docs/user-guide/debugging-services.md
+62 -62
View File
@@ -18,30 +18,30 @@ clear what is expected, this document will use the following conventions.
If the command "COMMAND" is expected to run in a `Pod` and produce "OUTPUT":
{% highlight console %}
```shell
u@pod$ COMMAND
OUTPUT
{% endhighlight %}
```
If the command "COMMAND" is expected to run on a `Node` and produce "OUTPUT":
{% highlight console %}
```shell
u@node$ COMMAND
OUTPUT
{% endhighlight %}
```
If the command is "kubectl ARGS":
{% highlight console %}
```shell
$ kubectl ARGS
OUTPUT
{% endhighlight %}
```
## Running commands in a Pod
@@ -49,7 +49,7 @@ For many steps here you will want to see what a `Pod` running in the cluster
sees. Kubernetes does not directly support interactive `Pod`s (yet), but you can
approximate it:
{% highlight console %}
```shell
$ cat <<EOF | kubectl create -f -
apiVersion: v1
@@ -66,25 +66,25 @@ spec:
EOF
pods/busybox-sleep
{% endhighlight %}
```
Now, when you need to run a command (even an interactive shell) in a `Pod`-like
context, use:
{% highlight console %}
```shell
$ kubectl exec busybox-sleep -- <COMMAND>
{% endhighlight %}
```
or
{% highlight console %}
```shell
$ kubectl exec -ti busybox-sleep sh
/ #
{% endhighlight %}
```
## Setup
@@ -92,7 +92,7 @@ For the purposes of this walk-through, let's run some `Pod`s. Since you're
probably debugging your own `Service` you can substitute your own details, or you
can follow along and get a second data point.
{% highlight console %}
```shell
$ kubectl run hostnames --image=gcr.io/google_containers/serve_hostname \
--labels=app=hostnames \
@@ -101,12 +101,12 @@ $ kubectl run hostnames --image=gcr.io/google_containers/serve_hostname \
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
hostnames hostnames gcr.io/google_containers/serve_hostname app=hostnames 3
{% endhighlight %}
```
Note that this is the same as if you had started the `ReplicationController` with
the following YAML:
{% highlight yaml %}
```yaml
apiVersion: v1
kind: ReplicationController
@@ -128,11 +128,11 @@ spec:
- containerPort: 9376
protocol: TCP
{% endhighlight %}
```
Confirm your `Pod`s are running:
{% highlight console %}
```shell
$ kubectl get pods -l app=hostnames
NAME READY STATUS RESTARTS AGE
@@ -140,7 +140,7 @@ hostnames-0uton 1/1 Running 0 12s
hostnames-bvc05 1/1 Running 0 12s
hostnames-yp2kp 1/1 Running 0 12s
{% endhighlight %}
```
## Does the Service exist?
@@ -152,53 +152,53 @@ So what would happen if I tried to access a non-existent `Service`? Assuming yo
have another `Pod` that consumes this `Service` by name you would get something
like:
{% highlight console %}
```shell
u@pod$ wget -qO- hostnames
wget: bad address 'hostname'
{% endhighlight %}
```
or:
{% highlight console %}
```shell
u@pod$ echo $HOSTNAMES_SERVICE_HOST
{% endhighlight %}
```
So the first thing to check is whether that `Service` actually exists:
{% highlight console %}
```shell
$ kubectl get svc hostnames
Error from server: service "hostnames" not found
{% endhighlight %}
```
So we have a culprit, let's create the `Service`. As before, this is for the
walk-through - you can use your own `Service`'s details here.
{% highlight console %}
```shell
$ kubectl expose rc hostnames --port=80 --target-port=9376
service "hostnames" exposed
{% endhighlight %}
```
And read it back, just to be sure:
{% highlight console %}
```shell
$ kubectl get svc hostnames
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
hostnames 10.0.0.1 <none> 80/TCP run=hostnames 1h
{% endhighlight %}
```
As before, this is the same as if you had started the `Service` with YAML:
{% highlight yaml %}
```yaml
apiVersion: v1
kind: Service
@@ -213,7 +213,7 @@ spec:
port: 80
targetPort: 9376
{% endhighlight %}
```
Now you can confirm that the `Service` exists.
@@ -221,7 +221,7 @@ Now you can confirm that the `Service` exists.
From a `Pod` in the same `Namespace`:
{% highlight console %}
```shell
u@pod$ nslookup hostnames
Server: 10.0.0.10
@@ -230,12 +230,12 @@ Address: 10.0.0.10#53
Name: hostnames
Address: 10.0.1.175
{% endhighlight %}
```
If this fails, perhaps your `Pod` and `Service` are in different
`Namespace`s, try a namespace-qualified name:
{% highlight console %}
```shell
u@pod$ nslookup hostnames.default
Server: 10.0.0.10
@@ -244,12 +244,12 @@ Address: 10.0.0.10#53
Name: hostnames.default
Address: 10.0.1.175
{% endhighlight %}
```
If this works, you'll need to ensure that `Pod`s and `Service`s run in the same
`Namespace`. If this still fails, try a fully-qualified name:
{% highlight console %}
```shell
u@pod$ nslookup hostnames.default.svc.cluster.local
Server: 10.0.0.10
@@ -258,7 +258,7 @@ Address: 10.0.0.10#53
Name: hostnames.default.svc.cluster.local
Address: 10.0.1.175
{% endhighlight %}
```
Note the suffix here: "default.svc.cluster.local". The "default" is the
`Namespace` we're operating in. The "svc" denotes that this is a `Service`.
@@ -267,7 +267,7 @@ The "cluster.local" is your cluster domain.
You can also try this from a `Node` in the cluster (note: 10.0.0.10 is my DNS
`Service`):
{% highlight console %}
```shell
u@node$ nslookup hostnames.default.svc.cluster.local 10.0.0.10
Server: 10.0.0.10
@@ -276,7 +276,7 @@ Address: 10.0.0.10#53
Name: hostnames.default.svc.cluster.local
Address: 10.0.1.175
{% endhighlight %}
```
If you are able to do a fully-qualified name lookup but not a relative one, you
need to check that your `kubelet` is running with the right flags.
@@ -291,7 +291,7 @@ If the above still fails - DNS lookups are not working for your `Service` - we
can take a step back and see what else is not working. The Kubernetes master
`Service` should always work:
{% highlight console %}
```shell
u@pod$ nslookup kubernetes.default
Server: 10.0.0.10
@@ -300,7 +300,7 @@ Address 1: 10.0.0.10
Name: kubernetes
Address 1: 10.0.0.1
{% endhighlight %}
```
If this fails, you might need to go to the kube-proxy section of this doc, or
even go back to the top of this document and start over, but instead of
@@ -311,7 +311,7 @@ debugging your own `Service`, debug DNS.
The next thing to test is whether your `Service` works at all. From a
`Node` in your cluster, access the `Service`'s IP (from `kubectl get` above).
{% highlight console %}
```shell
u@node$ curl 10.0.1.175:80
hostnames-0uton
@@ -322,7 +322,7 @@ hostnames-yp2kp
u@node$ curl 10.0.1.175:80
hostnames-bvc05
{% endhighlight %}
```
If your `Service` is working, you should get correct responses. If not, there
are a number of things that could be going wrong. Read on.
@@ -333,7 +333,7 @@ It might sound silly, but you should really double and triple check that your
`Service` is correct and matches your `Pods`. Read back your `Service` and
verify it:
{% highlight console %}
```shell
$ kubectl get service hostnames -o json
{
@@ -372,7 +372,7 @@ $ kubectl get service hostnames -o json
}
}
{% endhighlight %}
```
Is the port you are trying to access in `spec.ports[]`? Is the `targetPort`
correct for your `Pod`s? If you meant it to be a numeric port, is it a number
@@ -388,7 +388,7 @@ actually being selected by the `Service`.
Earlier we saw that the `Pod`s were running. We can re-check that:
{% highlight console %}
```shell
$ kubectl get pods -l app=hostnames
NAME READY STATUS RESTARTS AGE
@@ -396,7 +396,7 @@ hostnames-0uton 1/1 Running 0 1h
hostnames-bvc05 1/1 Running 0 1h
hostnames-yp2kp 1/1 Running 0 1h
{% endhighlight %}
```
The "AGE" column says that these `Pod`s are about an hour old, which implies that
they are running fine and not crashing.
@@ -405,13 +405,13 @@ The `-l app=hostnames` argument is a label selector - just like our `Service`
has. Inside the Kubernetes system is a control loop which evaluates the
selector of every `Service` and save the results into an `Endpoints` object.
{% highlight console %}
```shell
$ kubectl get endpoints hostnames
NAME ENDPOINTS
hostnames 10.244.0.5:9376,10.244.0.6:9376,10.244.0.7:9376
{% endhighlight %}
```
This confirms that the control loop has found the correct `Pod`s for your
`Service`. If the `hostnames` row is blank, you should check that the
@@ -424,7 +424,7 @@ At this point, we know that your `Service` exists and has selected your `Pod`s.
Let's check that the `Pod`s are actually working - we can bypass the `Service`
mechanism and go straight to the `Pod`s.
{% highlight console %}
```shell
u@pod$ wget -qO- 10.244.0.5:9376
hostnames-0uton
@@ -435,7 +435,7 @@ hostnames-bvc05
u@pod$ wget -qO- 10.244.0.7:9376
hostnames-yp2kp
{% endhighlight %}
```
We expect each `Pod` in the `Endpoints` list to return its own hostname. If
this is not what happens (or whatever the correct behavior is for your own
@@ -454,12 +454,12 @@ suspect. Let's confirm it, piece by piece.
Confirm that `kube-proxy` is running on your `Node`s. You should get something
like the below:
{% highlight console %}
```shell
u@node$ ps auxw | grep kube-proxy
root 4194 0.4 0.1 101864 17696 ? Sl Jul04 25:43 /usr/local/bin/kube-proxy --master=https://kubernetes-master --kubeconfig=/var/lib/kube-proxy/kubeconfig --v=2
{% endhighlight %}
```
Next, confirm that it is not failing something obvious, like contacting the
master. To do this, you'll have to look at the logs. Accessing the logs
@@ -467,7 +467,7 @@ depends on your `Node` OS. On some OSes it is a file, such as
/var/log/kube-proxy.log, while other OSes use `journalctl` to access logs. You
should see something like:
{% highlight console %}
```shell
I0707 17:34:53.945651 30031 server.go:88] Running in resource-only container "/kube-proxy"
I0707 17:34:53.945921 30031 proxier.go:121] Setting proxy IP to 10.240.115.247 and initializing iptables
@@ -488,7 +488,7 @@ I0707 17:34:54.903107 30031 proxysocket.go:130] Accepted TCP connection from 1
I0707 17:35:46.015868 30031 proxysocket.go:246] New UDP connection from 10.244.3.2:57493
I0707 17:35:46.017061 30031 proxysocket.go:246] New UDP connection from 10.244.3.2:55471
{% endhighlight %}
```
If you see error messages about not being able to contact the master, you
should double-check your `Node` configuration and installation steps.
@@ -499,13 +499,13 @@ One of the main responsibilities of `kube-proxy` is to write the `iptables`
rules which implement `Service`s. Let's check that those rules are getting
written.
{% highlight console %}
```shell
u@node$ iptables-save | grep hostnames
-A KUBE-PORTALS-CONTAINER -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j REDIRECT --to-ports 48577
-A KUBE-PORTALS-HOST -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames:default" -m tcp --dport 80 -j DNAT --to-destination 10.240.115.247:48577
{% endhighlight %}
```
There should be 2 rules for each port on your `Service` (just one in this
example) - a "KUBE-PORTALS-CONTAINER" and a "KUBE-PORTALS-HOST". If you do
@@ -516,32 +516,32 @@ then look at the logs again.
Assuming you do see the above rules, try again to access your `Service` by IP:
{% highlight console %}
```shell
u@node$ curl 10.0.1.175:80
hostnames-0uton
{% endhighlight %}
```
If this fails, we can try accessing the proxy directly. Look back at the
`iptables-save` output above, and extract the port number that `kube-proxy` is
using for your `Service`. In the above examples it is "48577". Now connect to
that:
{% highlight console %}
```shell
u@node$ curl localhost:48577
hostnames-yp2kp
{% endhighlight %}
```
If this still fails, look at the `kube-proxy` logs for specific lines like:
{% highlight console %}
```shell
Setting endpoints for default/hostnames:default to [10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376]
{% endhighlight %}
```
If you don't see those, try restarting `kube-proxy` with the `-V` flag set to 4, and
then look at the logs again.