diff --git a/docs/admin/dns.md b/docs/admin/dns.md index cc132201aa..b87d3196bc 100644 --- a/docs/admin/dns.md +++ b/docs/admin/dns.md @@ -124,7 +124,7 @@ With v1.3, the following annotations are deprecated: `pod.beta.kubernetes.io/hos ## How do I test if it is working? -### Create a simple Pod to use as a test environment. +### Create a simple Pod to use as a test environment Create a file named busybox.yaml with the following contents: @@ -152,7 +152,7 @@ Then create a pod using this file: kubectl create -f busybox.yaml ``` -### Wait for this pod to go into the running state. +### Wait for this pod to go into the running state You can get its status with: ``` @@ -165,7 +165,7 @@ NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 ``` -### Validate DNS works +### Validate that DNS is working Once that pod is running, you can exec nslookup in that environment: @@ -185,6 +185,115 @@ Address 1: 10.0.0.1 If you see that, DNS is working correctly. +### Troubleshooting Tips + +If the nslookup command fails, check the following: + +#### Check the local DNS configuration first +Take a look inside the resolv.conf file. (See "Inheriting DNS from the node" and "Known issues" below for more information) + +``` +cat /etc/resolv.conf +``` + +Verify that the search path and name server are set up like the following (note that seach path may vary for different cloud providers): + +``` +search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal +nameserver 10.0.0.10 +options ndots:5 +``` + +#### Quick diagnosis + +Errors such as the following indicate a problem with the kube-dns add-on or associated Services: + +``` +$ kubectl exec busybox -- nslookup kubernetes.default +Server: 10.0.0.10 +Address 1: 10.0.0.10 + +nslookup: can't resolve 'kubernetes.default' +``` + +or + +``` +$ kubectl exec busybox -- nslookup kubernetes.default +Server: 10.0.0.10 +Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local + +nslookup: can't resolve 'kubernetes.default' +``` + +#### Check if the DNS pod is running + +Use the kubectl get pods command to verify that the DNS pod is running. + +``` +kubectl get pods --namespace=kube-system -l k8s-app=kube-dns +``` + +You should see something like: + +``` +NAME READY STATUS RESTARTS AGE +... +kube-dns-v19-ezo1y 3/3 Running 0 1h +... +``` + +If you see that no pod is running or that the pod has failed/completed, the dns add-on may not be deployed by default in your current environment and you will have to deploy it manually. + +#### Check for Errors in the DNS pod + +Use `kubectl logs` command to see logs for the DNS daemons. + +``` +kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns +kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq +kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz +``` + +See if there is any suspicious log. W, E, F letter at the beginning represent Warning, Error and Failure. Please search for entries that have these as the logging level and use [kubernetes issues](https://github.com/kubernetes/kubernetes/issues) to report unexpected errors. + +#### Is dns service up? + +Verify that the DNS service is up by using the `kubectl get service` command. + +``` +kubectl get svc --namespace=kube-system +``` + +You should see: + +``` +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +... +kube-dns 10.0.0.10 53/UDP,53/TCP 1h +... +``` + +If you have created the service or in the case it should be created by default but it does not appear, see this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) for more information. + +#### Are dns endpoints exposed? + +You can verify that dns endpoints are exposed by using the `kubectl get endpoints` command. + +``` +kubectl get ep kube-dns --namespace=kube-system +``` + +You should see something like: +``` +NAME ENDPOINTS AGE +kube-dns 10.180.3.17:53,10.180.3.17:53 1h +``` + +If you do not see the endpoints, see endpoints section in the [debugging services documentation](http://kubernetes.io/docs/user-guide/debugging-services/). + +For additional Kubernetes DNS examples, see the [cluster-dns examples](https://github.com/kubernetes/kubernetes/tree/master/examples/cluster-dns) in the Kubernetes GitHub repository. + ## Kubernetes Federation (Multiple Zone support) Release 1.3 introduced Cluster Federation support for multi-site @@ -213,6 +322,34 @@ the flag `--cluster-domain=` The Kubernetes cluster DNS server (based off the [SkyDNS](https://github.com/skynetservices/skydns) library) supports forward lookups (A records), service lookups (SRV records) and reverse IP address lookups (PTR records). +## Inheriting DNS from the node +When running a pod, kubelet will prepend the cluster DNS server and search +paths to the node's own DNS settings. If the node is able to resolve DNS names +specific to the larger environment, pods should be able to, also. See "Known +issues" below for a caveat. + +If you don't want this, or if you want a different DNS config for pods, you can +use the kubelet's `--resolv-conf` flag. Setting it to "" means that pods will +not inherit DNS. Setting it to a valid file path means that kubelet will use +this file instead of `/etc/resolv.conf` for DNS inheritance. + +## Known issues +Kubernetes installs do not configure the nodes' resolv.conf files to use the +cluster DNS by default, because that process is inherently distro-specific. +This should probably be implemented eventually. + +Linux's libc is impossibly stuck ([see this bug from +2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just +3 DNS `nameserver` records and 6 DNS `search` records. Kubernetes needs to +consume 1 `nameserver` record and 3 `search` records. This means that if a +local installation already uses 3 `nameserver`s or uses more than 3 `search`es, +some of those settings will be lost. As a partial workaround, the node can run +`dnsmasq` which will provide more `nameserver` entries, but not more `search` +entries. You can also use kubelet's `--resolv-conf` flag. + +If you are using Alpine version 3.3 or earlier as your base image, dns may not +work properly owing to a known issue with Alpine. Check [here](https://github.com/kubernetes/kubernetes/issues/30215) +for more information. ## References diff --git a/docs/user-guide/federation/federated-ingress.md b/docs/user-guide/federation/federated-ingress.md index 87965a3fc7..6198de1817 100644 --- a/docs/user-guide/federation/federated-ingress.md +++ b/docs/user-guide/federation/federated-ingress.md @@ -64,12 +64,12 @@ healthy backend service endpoint at all times, even in the event of pod, cluster, availability zone or regional outages. -Note that in the - case of Google Cloud, the logical L7 load balancer is not a single physical device (which - would present both a single point of failure, and a single global - network routing choke point), but rather a [truly global, highly available - load balancing managed service](https://cloud.google.com/load-balancing/), - globally reachable via a single, static IP address. +Note that in the case of Google Cloud, the logical L7 load balancer is +not a single physical device (which would present both a single point +of failure, and a single global network routing choke point), but +rather a +[truly global, highly available load balancing managed service](https://cloud.google.com/load-balancing/), +globally reachable via a single, static IP address. Clients inside your federated Kubernetes clusters (i.e. Pods) will be automatically routed to the cluster-local shard of the Federated Service @@ -86,13 +86,13 @@ You can create a federated ingress in any of the usual ways, for example using k ``` shell kubectl --context=federation-cluster create -f myingress.yaml ``` - +For example ingress YAML configurations, see the [Ingress User Guide](/docs/user-guide/ingress/) The '--context=federation-cluster' flag tells kubectl to submit the request to the Federation API endpoint, with the appropriate credentials. If you have not yet configured such a context, visit the [federation admin guide](/docs/admin/federation/) or one of the [administration tutorials](https://github.com/kelseyhightower/kubernetes-cluster-federation) -to find out how to do so. TODO: Update links +to find out how to do so. As described above, the Federated Ingress will automatically create and maintain matching Kubernetes ingresses in all of the clusters @@ -147,17 +147,28 @@ Events: 2m 2m 1 {loadbalancer-controller } Normal CREATE ip: 130.211.5.194 ``` -Note the address of your Federated Ingress +Note that: + +1. the address of your Federated Ingress corresponds with the address of all of the underlying Kubernetes ingresses (once these have been allocated - this may take up to a few minutes). - -Note also that we have not yet provisioned any backend Pods to receive +2. we have not yet provisioned any backend Pods to receive the network traffic directed to this ingress (i.e. 'Service Endpoints' behind the service backing the Ingress), so the Federated Ingress does not yet consider these to be healthy shards and will not direct traffic to any of these clusters. +3. the federation control system will +automatically reconfigure the load balancer controllers in all of the +clusters in your federation to make them consistent, and allow +them to share global load balancers. But this reconfiguration can +only complete successfully if there are no pre-existing Ingresses in +those clusters (this is a safety feature to prevent accidental +breakage of existing ingresses). So to ensure that your federated +ingresses function correctly, either start with new, empty clusters, or make +sure that you delete (and recreate if necessary) all pre-existing +Ingresses in the clusters comprising your federation. -## Adding backend services and pods +#Adding backend services and pods To render the underlying ingress shards healthy, we need to add backend Pods behind the service upon which the Ingress is based. There are several ways to achieve this, but @@ -175,6 +186,16 @@ kubectl --context=federation-cluster create -f services/nginx.yaml kubectl --context=federation-cluster create -f myreplicaset.yaml ``` +Note that in order for your federated ingress to work correctly on +Google Cloud, the node ports of all of the underlying cluster-local +services need to be identical. If you're using a federated service +this is easy to do. Simply pick a node port that is not already +being used in any of your clusters, and add that to the spec of your +federated service. If you do not specify a node port for your +federated service, each cluster will choose it's own node port for +its cluster-local shard of the service, and these will probably end +up being different, which is not what you want. + You can verify this by checking in each of the underlying clusters, for example: ``` shell @@ -258,6 +279,35 @@ Check that: `service-controller` or `replicaset-controller`, errors in the output of `kubectl logs federation-controller-manager --namespace federation`). +#### I can create a federated ingress successfully, but request load is not correctly distributed across the underlying clusters + +Check that: + +1. the services underlying your federated ingress in each cluster have + identical node ports. See [above](#creating_a_federated_ingress) for further explanation. +2. the load balancer controllers in each of your clusters are of the + correct type ("GLBC") and have been correctly reconfigured by the + federation control plane to share a global GCE load balancer (this + should happen automatically). If they of the correct type, and + have been correctly reconfigured, the UID data item in the GLBC + configmap in each cluster will be identical across all clusters. + See + [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) + for further details. + If this is not the case, check the logs of your federation + controller manager to determine why this automated reconfiguration + might be failing. +3. no ingresses have been manually created in any of your clusters before the above + reconfiguration of the load balancer controller completed + successfully. Ingresses created before the reconfiguration of + your GLBC will interfere with the behavior of your federated + ingresses created after the reconfiguration (see + [the GLBC docs](https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#changing-the-cluster-uid) + for further information. To remedy this, + delete any ingresses created before the cluster joined the + federation (and had it's GLBC reconfigured), and recreate them if + necessary. + #### This troubleshooting guide did not help me solve my problem Please use one of our [support channels](http://kubernetes.io/docs/troubleshooting/) to seek assistance.