diff --git a/_data/guides.yml b/_data/guides.yml index 38ee10b438..2d79fb00dd 100644 --- a/_data/guides.yml +++ b/_data/guides.yml @@ -164,6 +164,7 @@ toc: - docs/getting-started-guides/ubuntu/troubleshooting.md - docs/getting-started-guides/ubuntu/decommissioning.md - docs/getting-started-guides/ubuntu/calico.md + - docs/getting-started-guides/ubuntu/operational-considerations.md - docs/getting-started-guides/ubuntu/glossary.md - docs/getting-started-guides/ubuntu/local.md - docs/getting-started-guides/ubuntu/logging.md diff --git a/docs/getting-started-guides/ubuntu/index.md b/docs/getting-started-guides/ubuntu/index.md index 5e706f0add..d54b94a88c 100644 --- a/docs/getting-started-guides/ubuntu/index.md +++ b/docs/getting-started-guides/ubuntu/index.md @@ -43,6 +43,7 @@ These are more in-depth guides for users choosing to run Kubernetes in productio - [Storage](/docs/getting-started-guides/ubuntu/storage) - [Troubleshooting](/docs/getting-started-guides/ubuntu/troubleshooting) - [Decommissioning](/docs/getting-started-guides/ubuntu/decommissioning) + - [Operational Considerations](/docs/getting-started-guides/ubuntu/operational-considerations) - [Glossary](/docs/getting-started-guides/ubuntu/glossary) ## Developer Guides diff --git a/docs/getting-started-guides/ubuntu/logging.md b/docs/getting-started-guides/ubuntu/logging.md index b7ae3c9a10..ae7237a0a7 100644 --- a/docs/getting-started-guides/ubuntu/logging.md +++ b/docs/getting-started-guides/ubuntu/logging.md @@ -18,3 +18,43 @@ The `juju debug-log` will show all of the consolidated logs of all the Juju agen See the [Juju documentation](https://jujucharms.com/docs/stable/troubleshooting-logs) for more information. +## Managing log verbosity + +Log verbosity in Juju is set at the model level. You can adjust it at any time: + +``` +juju add-model k8s-development --config logging-config='=DEBUG;unit=DEBUG' +``` + +and later + +``` +juju config-model k8s-production --config logging-config='=ERROR;unit=ERROR' +``` + +In addition, the jujud daemon is started in debug mode by default on all controllers. To remove that behavior edit ```/var/lib/juju/init/jujud-machine-0/exec-start.sh``` on the controller node and comment the ```--debug``` section. + +It then contains: + +``` +#!/usr/bin/env bash + +# Set up logging. +touch '/var/log/juju/machine-0.log' +chown syslog:syslog '/var/log/juju/machine-0.log' +chmod 0600 '/var/log/juju/machine-0.log' +exec >> '/var/log/juju/machine-0.log' +exec 2>&1 + +# Run the script. +'/var/lib/juju/tools/machine-0/jujud' machine --data-dir '/var/lib/juju' --machine-id 0 # --debug +``` + +Then restart the service with: + +``` +sudo systemctl restart jujud-machine-0.service +``` + +See the [official documentation](https://jujucharms.com/docs/stable/models-config) for more information about logging and other model settings in Juju. + diff --git a/docs/getting-started-guides/ubuntu/operational-considerations.md b/docs/getting-started-guides/ubuntu/operational-considerations.md new file mode 100644 index 0000000000..e6c6a02097 --- /dev/null +++ b/docs/getting-started-guides/ubuntu/operational-considerations.md @@ -0,0 +1,160 @@ +--- +title: Operational Considerations +--- + +{% capture overview %} +This page gives recommendations and hints for people managing long lived clusters +{% endcapture %} +{% capture prerequisites %} +This page assumes you understand the basics of Juju and Kubernetes. +{% endcapture %} + +{% capture steps %} + +## Managing Juju + +### Sizing your controller node + +The Juju Controller: + +* requires about 2 to 2.5GB RAM to operate. +* uses a MongoDB database as a storage backend for the configuration and state of the cluster. This database can grow significantly, and can also be the biggest consumer of CPU cycles on the instance +* aggregates and stores the log data of all services and units. Therefore, significant storage is needed for long lived models. If your intention is to keep the cluster running, make sure to provision at least 64GB for the logs. + +To bootstrap a controller with constraints run the following command: + +``` +juju bootstrap --contraints "mem=8GB cpu-cores=4 root-disk=128G" +``` + +Juju will select the cheapest instance type matching your constraints on your target cloud. You can also use the ```instance-type``` constraint in conjunction with ```root-disk``` for strict control. For more information about the constraints available, refer to the [official documentation](https://jujucharms.com/docs/stable/reference-constraints) + +Additional information about logging can be found in the [logging section](/docs/getting-started-guides/ubuntu/logging) + +### SSHing into the Controller Node + +By default, Juju will create a pair of SSH keys that it will use to automate the connection to units. They are stored on the client node in ```~/.local/share/juju/ssh/``` + +After deployment, Juju Controller is a "silent unit" that acts as a proxy between the client and the deployed applications. Nevertheless it can be useful to SSH into it. + +First you need to understand your environment, especially if you run several Juju models and controllers. Run + +``` +juju list-models --all +$ juju models --all +Controller: k8s + +Model Cloud/Region Status Machines Cores Access Last connection +admin/controller lxd/localhost available 1 - admin just now +admin/default lxd/localhost available 0 - admin 2017-01-23 +admin/whale* lxd/localhost available 6 - admin 3 minutes ago + +``` + +The first line ```Controller: k8s``` refers to how you bootstrapped. + +Then you will see 2, 3 or more models listed below. + +* admin/controller is the default model that hosts all controller units of juju +* admin/default is created by default as the primary model to host the user application, such as the Kubernetes cluster +* admin/whale is an additional model created if you use conjure-up as an overlay on top of Juju. + +Now to ssh into a controller node, you first ask Juju to switch context, then ssh as you would with a normal unit: + +``` +juju switch controller +``` + +At this stage, you can query the controller model as well: + +``` +juju status +Model Controller Cloud/Region Version +controller k8s lxd/localhost 2.0.2 + +App Version Status Scale Charm Store Rev OS Notes + +Unit Workload Agent Machine Public address Ports Message + +Machine State DNS Inst id Series AZ +0 started 10.191.22.15 juju-2a5ed8-0 xenial +``` + +Note that if you had bootstrapped in HA mode, you would see several machines listed. + +Now ssh-ing into the controller follows the same semantic as classic Juju commands: + +``` +$ juju ssh 0 +Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.8.0-34-generic x86_64) + + * Documentation: https://help.ubuntu.com + * Management: https://landscape.canonical.com + * Support: https://ubuntu.com/advantage + + Get cloud support with Ubuntu Advantage Cloud Guest: + http://www.ubuntu.com/business/services/cloud + +0 packages can be updated. +0 updates are security updates. + + +Last login: Tue Jan 24 16:38:13 2017 from 10.191.22.1 +ubuntu@juju-2a5ed8-0:~$ +``` + +When you are done and want to come back to your initial model, exit the controller and + + +Then if you need to switch back to your cluster and ssh into the units, run + +``` +juju switch default +``` + +## Managing your Kubernetes cluster + +### Running privileged containers + +By default juju-deployed clusters do not support running privileged containers. If you need them, you have to edit ```/etc/default/kube-apiserver``` on the master nodes, and ```/etc/default/kubelet``` on your worker nodes. + +On Kubernetes Core or on small deployment, run the following commands from the Juju client: + +#### Manually + +1. Update the Master + +``` +juju ssh kubernetes-master/0 "sudo sed -i 's/KUBE_API_ARGS=\"/KUBE_API_ARGS=\"--allow-privileged\ /' /etc/default/kube-apiserver && sudo systemctl restart kube-apiserver.service" +``` + +2. Update the Worker(s) + +``` +juju ssh kubernetes-worker/0 "sudo sed -i 's/KUBELET_ARGS=\"/KUBELET_ARGS=\"--allow-privileged\ /' /etc/default/kubelet && sudo systemctl restart kubelet.service" +``` + +#### Programmatically + +If the deployment is larger the following commands will run on all units successively: + +1. Update all Masters + +``` +juju show-status kubernetes-master --format json | \ + jq --raw-output '.applications."kubernetes-master".units | keys[]' | \ + xargs -I UNIT juju ssh UNIT "sudo sed -i 's/KUBE_API_ARGS=\"/KUBE_API_ARGS=\"--allow-privileged\ /' /etc/default/kube-apiserver && sudo systemctl restart kube-apiserver.service" +``` + +2. Update all workers + +``` +juju show-status kubernetes-worker --format json | \ + jq --raw-output '.applications."kubernetes-worker".units | keys[]' | \ + xargs -I UNIT juju ssh UNIT "sudo sed -i 's/KUBELET_ARGS=\"/KUBELET_ARGS=\"--allow-privileged\ /' /etc/default/kubelet && sudo systemctl restart kubelet.service" +``` + + +{% endcapture %} + +{% include templates/task.md %} \ No newline at end of file diff --git a/docs/getting-started-guides/ubuntu/troubleshooting.md b/docs/getting-started-guides/ubuntu/troubleshooting.md index e8c055b17a..44b3a34346 100644 --- a/docs/getting-started-guides/ubuntu/troubleshooting.md +++ b/docs/getting-started-guides/ubuntu/troubleshooting.md @@ -105,6 +105,102 @@ charm unit data, etc. Additional application-specific information may be included as well. ## Common Problems +### Load Balancer interfering with Helm + +This section assumes you have a working deployment of Kubernetes via Juju using a Load Balancer for the API, and that you are using Helm to deploy charts. + +To deploy Helm you will have run: + +``` +helm init +$HELM_HOME has been configured at /home/ubuntu/.helm +Tiller (the helm server side component) has been installed into your Kubernetes Cluster. +Happy Helming! +``` + +Then when using helm you may see one of the following errors: + +* Helm doesn't get the version from the Tiller server + +``` +helm version +Client: &version.Version{SemVer:"v2.1.3", GitCommit:"5cbc48fb305ca4bf68c26eb8d2a7eb363227e973", GitTreeState:"clean"} +Error: cannot connect to Tiller +``` + +* Helm cannot install your chart + +``` +helm install --debug +Error: forwarding ports: error upgrading connection: Upgrade request required +``` + +This is caused by the API load balancer not forwarding ports in the context of the helm client-server relationship. To deploy using helm, you will need to follow these steps: + +1. Expose the Kubernetes Master service + +``` +juju expose kubernetes-master +``` + +2. Identify the public IP address of one of your masters + +``` +juju status kubernetes-master +Model Controller Cloud/Region Version +production k8s-admin aws/us-east-1 2.0.0 + +App Version Status Scale Charm Store Rev OS Notes +flannel 0.6.1 active 1 flannel jujucharms 7 ubuntu +kubernetes-master 1.5.1 active 1 kubernetes-master jujucharms 10 ubuntu exposed + +Unit Workload Agent Machine Public address Ports Message +kubernetes-master/0* active idle 5 54.210.100.102 6443/tcp Kubernetes master running. + flannel/0 active idle 54.210.100.102 Flannel subnet 10.1.50.1/24 + +Machine State DNS Inst id Series AZ +5 started 54.210.100.102 i-002b7150639eb183b xenial us-east-1a + +Relation Provides Consumes Type +certificates easyrsa kubernetes-master regular +etcd etcd flannel regular +etcd etcd kubernetes-master regular +cni flannel kubernetes-master regular +loadbalancer kubeapi-load-balancer kubernetes-master regular +cni kubernetes-master flannel subordinate +cluster-dns kubernetes-master kubernetes-worker regular +cni kubernetes-worker flannel subordinate +``` + +In this context the public IP address is 54.210.100.102. + +If you want to access this data programmatically you can use the JSON output: + +``` +juju show-status kubernetes-master --format json | jq --raw-output '.applications."kubernetes-master".units | keys[]' +54.210.100.102 +``` + +3. Update the kubeconfig file + +Identify the kubeconfig file or section used for this cluster, and edit the server configuration. + +By default, it will look like ```https://54.213.123.123:443```. Replace it with the Kubernetes Master endpoint ```https://54.210.100.102:6443``` and save. + +Note that the default port used by CDK for the Kubernetes Master API is 6443 while the port exposed by the load balancer is 443. + +4. Start helming again! + +``` +helm install --debug +Created tunnel using local port: '36749' +SERVER: "localhost:36749" +CHART PATH: /home/ubuntu/.helm/ +NAME: +... +... + +``` ## etcd diff --git a/docs/tutorials/stateless-application/hello-minikube.md b/docs/tutorials/stateless-application/hello-minikube.md index a4d9af0e17..187e059b0a 100644 --- a/docs/tutorials/stateless-application/hello-minikube.md +++ b/docs/tutorials/stateless-application/hello-minikube.md @@ -88,7 +88,7 @@ If a proxy server is required, use the following method to start Minikube cluste minikube start --vm-driver=xhyve --docker-env HTTP_PROXY=http://your-http-proxy-host:your-http-proxy-port --docker-env HTTPS_PROXY=http(s)://your-https-proxy-host:your-https-proxy-port ``` -The `--vm-driver=xyhve` flag specifies that you are using Docker for Mac. The +The `--vm-driver=xhyve` flag specifies that you are using Docker for Mac. The default VM driver is VirtualBox. Now set the Minikube context. The context is what determines which cluster diff --git a/docs/user-guide/persistent-volumes/index.md b/docs/user-guide/persistent-volumes/index.md index a1260ac22d..d6245e8198 100644 --- a/docs/user-guide/persistent-volumes/index.md +++ b/docs/user-guide/persistent-volumes/index.md @@ -543,7 +543,7 @@ and need persistent storage, we recommend that you use the following pattern: - Do include PersistentVolumeClaim objects in your bundle of config (alongside Deployments, ConfigMaps, etc). - Do not include PersistentVolume objects in the config, since the user instantiating the config may not have permission to create PersistentVolumes. -- Give the user the option of providing a storage class name when instantating the template. +- Give the user the option of providing a storage class name when instantiating the template. - If the user provides a storage class name, and the cluster is version 1.4 or newer, put that value into the `volume.beta.kubernetes.io/storage-class` annotation of the PVC. This will cause the PVC to match the right storage class if the cluster has StorageClasses enabled by the admin. - If the user does not provide a storage class name or the cluster is version 1.3, then instead put a `volume.alpha.kubernetes.io/storage-class: default` annotation on the PVC. diff --git a/docs/user-guide/secrets/index.md b/docs/user-guide/secrets/index.md index 8388dac53f..9723e00aee 100644 --- a/docs/user-guide/secrets/index.md +++ b/docs/user-guide/secrets/index.md @@ -763,7 +763,7 @@ make that key begin with a dot. For example, when the following secret is mount { "name": "dotfile-test-container", "image": "gcr.io/google_containers/busybox", - "command": "ls -l /etc/secret-volume", + "command": [ "ls", "-l", "/etc/secret-volume" ], "volumeMounts": [ { "name": "secret-volume",