whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Convert site to Hugo (#8316)

Browse Source 
This commit converts content and layout to use Hugo.
This commit is contained in:
Bjørn Erik Pedersen
2018-05-05 18:00:51 +02:00
committed by k8s-ci-robot
parent 7745f0e0c5
commit 7f3b633aa0
2327 changed files with 10928 additions and 171503 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/getting-started-guides/OWNERS
+3
View File
@@ -0,0 +1,3 @@
reviewers:
- errordeveloper
content/en/docs/getting-started-guides/_index.md
Executable
+5
View File
@@ -0,0 +1,5 @@
---
title: "Independent Solutions"
weight: 50
---
content/en/docs/getting-started-guides/alibaba-cloud.md
+20
View File
@@ -0,0 +1,20 @@
---
reviewers:
- colemickens
- brendandburns
title: Running Kubernetes on Alibaba Cloud
---
## Alibaba Cloud Container Service
The [Alibaba Cloud Container Service](https://www.aliyun.com/product/containerservice) lets you run and manage Docker applications on a cluster of Alibaba Cloud ECS instances. It supports the popular open source container orchestrators: Docker Swarm and Kubernetes.
To simplify cluster deployment and management, use [Kubernetes Support for Alibaba Cloud Container Service](https://www.aliyun.com/solution/kubernetes/). You can get started quickly by following the [Kubernetes walk-through](https://help.aliyun.com/document_detail/53751.html), and there are some [tutorials for Kubernetes Support on Alibaba Cloud](https://yq.aliyun.com/teams/11/type_blog-cid_200-page_1) in Chinese.
To use custom binaries or open source Kubernetes, follow the instructions below.
## Custom Deployments
The source code for [Kubernetes with Alibaba Cloud provider implementation](https://github.com/AliyunContainerService/kubernetes) is open source and available on GitHub.
For more information, see "[Quick deployment of Kubernetes - VPC environment on Alibaba Cloud](https://www.alibabacloud.com/forum/read-830)" in English and [Chinese](https://yq.aliyun.com/articles/66474).
content/en/docs/getting-started-guides/alternatives.md
+10
View File
@@ -0,0 +1,10 @@
---
reviewers:
- pwittrock
title: Deprecated Alternatives
---
# *Stop. These guides are superseded by [Minikube](../minikube/). They are only listed here for completeness.*
* [Using Vagrant](https://git.k8s.io/community/contributors/devel/vagrant.md)
* *Advanced:* [Directly using Kubernetes raw binaries (Linux Only)](https://git.k8s.io/community/contributors/devel/running-locally.md)
content/en/docs/getting-started-guides/aws.md
+80
View File
@@ -0,0 +1,80 @@
---
reviewers:
- justinsb
- clove
title: Running Kubernetes on AWS EC2
---
{{< toc >}}
## Supported Production Grade Tools
* [conjure-up](/docs/getting-started-guides/ubuntu/) is an open-source installer for Kubernetes that creates Kubernetes clusters with native AWS integrations on Ubuntu.
* [Kubernetes Operations](https://github.com/kubernetes/kops) - Production Grade K8s Installation, Upgrades, and Management. Supports running Debian, Ubuntu, CentOS, and RHEL in AWS.
* [CoreOS Tectonic](https://coreos.com/tectonic/) includes the open-source [Tectonic Installer](https://github.com/coreos/tectonic-installer) that creates Kubernetes clusters with Container Linux nodes on AWS.
* CoreOS originated and the Kubernetes Incubator maintains [a CLI tool, `kube-aws`](https://github.com/kubernetes-incubator/kube-aws), that creates and manages Kubernetes clusters with [Container Linux](https://coreos.com/why/) nodes, using AWS tools: EC2, CloudFormation and Autoscaling.
---
## Getting started with your cluster
### Command line administration tool: `kubectl`
The cluster startup script will leave you with a `kubernetes` directory on your workstation.
Alternately, you can download the latest Kubernetes release from [this page](https://github.com/kubernetes/kubernetes/releases).
Next, add the appropriate binary folder to your `PATH` to access kubectl:
```shell
# OS X
export PATH=<path/to/kubernetes-directory>/platforms/darwin/amd64:$PATH
# Linux
export PATH=<path/to/kubernetes-directory>/platforms/linux/amd64:$PATH
```
An up-to-date documentation page for this tool is available here: [kubectl manual](/docs/user-guide/kubectl/)
By default, `kubectl` will use the `kubeconfig` file generated during the cluster startup for authenticating against the API.
For more information, please read [kubeconfig files](/docs/tasks/access-application-cluster/configure-access-multiple-clusters/)
### Examples
See [a simple nginx example](/docs/tasks/run-application/run-stateless-application-deployment/) to try out your new cluster.
The "Guestbook" application is another popular example to get started with Kubernetes: [guestbook example](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/)
For more complete applications, please look in the [examples directory](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/)
## Scaling the cluster
Adding and removing nodes through `kubectl` is not supported. You can still scale the amount of nodes manually through adjustments of the 'Desired' and 'Max' properties within the [Auto Scaling Group](http://docs.aws.amazon.com/autoscaling/latest/userguide/as-manual-scaling.html), which was created during the installation.
## Tearing down the cluster
Make sure the environment variables you used to provision your cluster are still exported, then call the following script inside the
`kubernetes` directory:
```shell
cluster/kube-down.sh
```
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------------- | ---------- | --------------------------------------------- | ---------| ----------------------------
AWS | kops | Debian | k8s (VPC) | [docs](https://github.com/kubernetes/kops) | | Community ([@justinsb](https://github.com/justinsb))
AWS | CoreOS | CoreOS | flannel | [docs](/docs/getting-started-guides/aws) | | Community
AWS | Juju | Ubuntu | flannel, calico, canal | [docs](/docs/getting-started-guides/ubuntu) | 100% | Commercial, Community
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
## Further reading
Please see the [Kubernetes docs](/docs/) for more details on administering
and using a Kubernetes cluster.
content/en/docs/getting-started-guides/azure.md
+39
View File
@@ -0,0 +1,39 @@
---
reviewers:
- colemickens
- brendandburns
title: Running Kubernetes on Azure
---
## Azure Container Service
The [Azure Container Service](https://azure.microsoft.com/en-us/services/container-service/) offers simple
deployments of one of three open source orchestrators: DC/OS, Swarm, and Kubernetes clusters.
For an example of deploying a Kubernetes cluster onto Azure via the Azure Container Service:
**[Microsoft Azure Container Service - Kubernetes Walkthrough](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes)**
## Custom Deployments: ACS-Engine
The core of the Azure Container Service is **open source** and available on GitHub for the community
to use and contribute to: **[ACS-Engine](https://github.com/Azure/acs-engine)**.
ACS-Engine is a good choice if you need to make customizations to the deployment beyond what the Azure Container
Service officially supports. These customizations include deploying into existing virtual networks, utilizing multiple
agent pools, and more. Some community contributions to ACS-Engine may even become features of the Azure Container Service.
The input to ACS-Engine is similar to the ARM template syntax used to deploy a cluster directly with the Azure Container Service.
The resulting output is an Azure Resource Manager Template that can then be checked into source control and can then be used
to deploy Kubernetes clusters into Azure.
You can get started quickly by following the **[ACS-Engine Kubernetes Walkthrough](https://github.com/Azure/acs-engine/blob/master/docs/kubernetes.md)**.
## CoreOS Tectonic for Azure
The CoreOS Tectonic Installer for Azure is **open source** and available on GitHub for the community to use and contribute to: **[Tectonic Installer](https://github.com/coreos/tectonic-installer)**.
Tectonic Installer is a good choice when you need to make cluster customizations as it is built on [Hashicorp's Terraform](https://www.terraform.io/docs/providers/azurerm/) Azure Resource Manager (ARM) provider. This enables users to customize or integrate using familiar Terraform tooling.
You can get started using the [Tectonic Installer for Azure Guide](https://coreos.com/tectonic/docs/latest/install/azure/azure-terraform.html).
content/en/docs/getting-started-guides/clc.md
+341
View File
@@ -0,0 +1,341 @@
---
title: Running Kubernetes on CenturyLink Cloud
---
{: toc}
These scripts handle the creation, deletion and expansion of Kubernetes clusters on CenturyLink Cloud.
You can accomplish all these tasks with a single command. We have made the Ansible playbooks used to perform these tasks available [here](https://github.com/CenturyLinkCloud/adm-kubernetes-on-clc/blob/master/ansible/README.md).
## Find Help
If you run into any problems or want help with anything, we are here to help. Reach out to use via any of the following ways:
- Submit a github issue
- Send an email to Kubernetes AT ctl DOT io
- Visit [http://info.ctl.io/kubernetes](http://info.ctl.io/kubernetes)
## Clusters of VMs or Physical Servers, your choice.
- We support Kubernetes clusters on both Virtual Machines or Physical Servers. If you want to use physical servers for the worker nodes (minions), simple use the --minion_type=bareMetal flag.
- For more information on physical servers, visit: [https://www.ctl.io/bare-metal/](https://www.ctl.io/bare-metal/)
- Physical serves are only available in the VA1 and GB3 data centers.
- VMs are available in all 13 of our public cloud locations
## Requirements
The requirements to run this script are:
- A linux administrative host (tested on ubuntu and OSX)
- python 2 (tested on 2.7.11)
- pip (installed with python as of 2.7.9)
- git
- A CenturyLink Cloud account with rights to create new hosts
- An active VPN connection to the CenturyLink Cloud from your linux host
## Script Installation
After you have all the requirements met, please follow these instructions to install this script.
1) Clone this repository and cd into it.
```shell
git clone https://github.com/CenturyLinkCloud/adm-kubernetes-on-clc
```
2) Install all requirements, including
* Ansible
* CenturyLink Cloud SDK
* Ansible Modules
```shell
sudo pip install -r ansible/requirements.txt
```
3) Create the credentials file from the template and use it to set your ENV variables
```shell
cp ansible/credentials.sh.template ansible/credentials.sh
vi ansible/credentials.sh
source ansible/credentials.sh
```
4) Grant your machine access to the CenturyLink Cloud network by using a VM inside the network or [ configuring a VPN connection to the CenturyLink Cloud network.](https://www.ctl.io/knowledge-base/network/how-to-configure-client-vpn/)
#### Script Installation Example: Ubuntu 14 Walkthrough
If you use an ubuntu 14, for your convenience we have provided a step by step
guide to install the requirements and install the script.
```shell
# system
apt-get update
apt-get install -y git python python-crypto
curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py
# installing this repository
mkdir -p ~home/k8s-on-clc
cd ~home/k8s-on-clc
git clone https://github.com/CenturyLinkCloud/adm-kubernetes-on-clc.git
cd adm-kubernetes-on-clc/
pip install -r requirements.txt
# getting started
cd ansible
cp credentials.sh.template credentials.sh; vi credentials.sh
source credentials.sh
```
## Cluster Creation
To create a new Kubernetes cluster, simply run the ```kube-up.sh``` script. A complete
list of script options and some examples are listed below.
```shell
CLC_CLUSTER_NAME=[name of kubernetes cluster]
cd ./adm-kubernetes-on-clc
bash kube-up.sh -c="$CLC_CLUSTER_NAME"
```
It takes about 15 minutes to create the cluster. Once the script completes, it
will output some commands that will help you setup kubectl on your machine to
point to the new cluster.
When the cluster creation is complete, the configuration files for it are stored
locally on your administrative host, in the following directory
```shell
> CLC_CLUSTER_HOME=$HOME/.clc_kube/$CLC_CLUSTER_NAME/
```
#### Cluster Creation: Script Options
```shell
Usage: kube-up.sh [OPTIONS]
Create servers in the CenturyLinkCloud environment and initialize a Kubernetes cluster
Environment variables CLC_V2_API_USERNAME and CLC_V2_API_PASSWD must be set in
order to access the CenturyLinkCloud API
All options (both short and long form) require arguments, and must include "="
between option name and option value.
-h (--help) display this help and exit
-c= (--clc_cluster_name=) set the name of the cluster, as used in CLC group names
-t= (--minion_type=) standard -> VM (default), bareMetal -> physical]
-d= (--datacenter=) VA1 (default)
-m= (--minion_count=) number of kubernetes minion nodes
-mem= (--vm_memory=) number of GB ram for each minion
-cpu= (--vm_cpu=) number of virtual cps for each minion node
-phyid= (--server_conf_id=) physical server configuration id, one of
physical_server_20_core_conf_id
physical_server_12_core_conf_id
physical_server_4_core_conf_id (default)
-etcd_separate_cluster=yes create a separate cluster of three etcd nodes,
otherwise run etcd on the master node
```
## Cluster Expansion
To expand an existing Kubernetes cluster, run the ```add-kube-node.sh```
script. A complete list of script options and some examples are listed [below](#cluster-expansion-script-options).
This script must be run from the same host that created the cluster (or a host
that has the cluster artifact files stored in ```~/.clc_kube/$cluster_name```).
```shell
cd ./adm-kubernetes-on-clc
bash add-kube-node.sh -c="name_of_kubernetes_cluster" -m=2
```
#### Cluster Expansion: Script Options
```shell
Usage: add-kube-node.sh [OPTIONS]
Create servers in the CenturyLinkCloud environment and add to an
existing CLC kubernetes cluster
Environment variables CLC_V2_API_USERNAME and CLC_V2_API_PASSWD must be set in
order to access the CenturyLinkCloud API
-h (--help) display this help and exit
-c= (--clc_cluster_name=) set the name of the cluster, as used in CLC group names
-m= (--minion_count=) number of kubernetes minion nodes to add
```
## Cluster Deletion
There are two ways to delete an existing cluster:
1) Use our python script:
```shell
python delete_cluster.py --cluster=clc_cluster_name --datacenter=DC1
```
2) Use the CenturyLink Cloud UI. To delete a cluster, log into the CenturyLink
Cloud control portal and delete the parent server group that contains the
Kubernetes Cluster. We hope to add a scripted option to do this soon.
## Examples
Create a cluster with name of k8s_1, 1 master node and 3 worker minions (on physical machines), in VA1
```shell
bash kube-up.sh --clc_cluster_name=k8s_1 --minion_type=bareMetal --minion_count=3 --datacenter=VA1
```
Create a cluster with name of k8s_2, an ha etcd cluster on 3 VMs and 6 worker minions (on VMs), in VA1
```shell
bash kube-up.sh --clc_cluster_name=k8s_2 --minion_type=standard --minion_count=6 --datacenter=VA1 --etcd_separate_cluster=yes
```
Create a cluster with name of k8s_3, 1 master node, and 10 worker minions (on VMs) with higher mem/cpu, in UC1:
```shell
bash kube-up.sh --clc_cluster_name=k8s_3 --minion_type=standard --minion_count=10 --datacenter=VA1 -mem=6 -cpu=4
```
## Cluster Features and Architecture
We configure the Kubernetes cluster with the following features:
* KubeDNS: DNS resolution and service discovery
* Heapster/InfluxDB: For metric collection. Needed for Grafana and auto-scaling.
* Grafana: Kubernetes/Docker metric dashboard
* KubeUI: Simple web interface to view Kubernetes state
* Kube Dashboard: New web interface to interact with your cluster
We use the following to create the Kubernetes cluster:
* Kubernetes 1.1.7
* Ubuntu 14.04
* Flannel 0.5.4
* Docker 1.9.1-0~trusty
* Etcd 2.2.2
## Optional add-ons
* Logging: We offer an integrated centralized logging ELK platform so that all
Kubernetes and docker logs get sent to the ELK stack. To install the ELK stack
and configure Kubernetes to send logs to it, follow [the log
aggregation documentation](https://github.com/CenturyLinkCloud/adm-kubernetes-on-clc/blob/master/log_aggregration.md). Note: We don't install this by default as
the footprint isn't trivial.
## Cluster management
The most widely used tool for managing a Kubernetes cluster is the command-line
utility ```kubectl```. If you do not already have a copy of this binary on your
administrative machine, you may run the script ```install_kubectl.sh``` which will
download it and install it in ```/usr/bin/local```.
The script requires that the environment variable ```CLC_CLUSTER_NAME``` be defined
```install_kubectl.sh``` also writes a configuration file which will embed the necessary
authentication certificates for the particular cluster. The configuration file is
written to the ```${CLC_CLUSTER_HOME}/kube``` directory
```shell
export KUBECONFIG=${CLC_CLUSTER_HOME}/kube/config
kubectl version
kubectl cluster-info
```
### Accessing the cluster programmatically
It's possible to use the locally stored client certificates to access the apiserver. For example, you may want to use any of the [Kubernetes API client libraries](/docs/reference/client-libraries/) to program against your Kubernetes cluster in the programming language of your choice.
To demonstrate how to use these locally stored certificates, we provide the following example of using ```curl``` to communicate to the master apiserver via https:
```shell
curl \
--cacert ${CLC_CLUSTER_HOME}/pki/ca.crt \
--key ${CLC_CLUSTER_HOME}/pki/kubecfg.key \
--cert ${CLC_CLUSTER_HOME}/pki/kubecfg.crt https://${MASTER_IP}:6443
```
But please note, this *does not* work out of the box with the ```curl``` binary
distributed with OSX.
### Accessing the cluster with a browser
We install [the kubernetes dashboard](/docs/tasks/web-ui-dashboard/). When you
create a cluster, the script should output URLs for these interfaces like this:
kubernetes-dashboard is running at ```https://${MASTER_IP}:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy```.
Note on Authentication to the UIs: The cluster is set up to use basic
authentication for the user _admin_. Hitting the url at
```https://${MASTER_IP}:6443``` will require accepting the self-signed certificate
from the apiserver, and then presenting the admin password written to file at:
```> _${CLC_CLUSTER_HOME}/kube/admin_password.txt_```
### Configuration files
Various configuration files are written into the home directory *CLC_CLUSTER_HOME* under
```.clc_kube/${CLC_CLUSTER_NAME}``` in several subdirectories. You can use these files
to access the cluster from machines other than where you created the cluster from.
* ```config/```: Ansible variable files containing parameters describing the master and minion hosts
* ```hosts/```: hosts files listing access information for the ansible playbooks
* ```kube/```: ```kubectl``` configuration files, and the basic-authentication password for admin access to the Kubernetes API
* ```pki/```: public key infrastructure files enabling TLS communication in the cluster
* ```ssh/```: SSH keys for root access to the hosts
## ```kubectl``` usage examples
There are a great many features of _kubectl_. Here are a few examples
List existing nodes, pods, services and more, in all namespaces, or in just one:
```shell
kubectl get nodes
kubectl get --all-namespaces services
kubectl get --namespace=kube-system replicationcontrollers
```
The Kubernetes API server exposes services on web URLs, which are protected by requiring
client certificates. If you run a kubectl proxy locally, ```kubectl``` will provide
the necessary certificates and serve locally over http.
```shell
kubectl proxy -p 8001
```
Then, you can access urls like ```http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/``` without the need for client certificates in your browser.
## What Kubernetes features do not work on CenturyLink Cloud
These are the known items that don't work on CenturyLink cloud but do work on other cloud providers:
- At this time, there is no support services of the type [LoadBalancer](/docs/tasks/access-application-cluster/create-external-load-balancer/). We are actively working on this and hope to publish the changes sometime around April 2016.
- At this time, there is no support for persistent storage volumes provided by
CenturyLink Cloud. However, customers can bring their own persistent storage
offering. We ourselves use Gluster.
## Ansible Files
If you want more information about our Ansible files, please [read this file](https://github.com/CenturyLinkCloud/adm-kubernetes-on-clc/blob/master/ansible/README.md)
## Further reading
Please see the [Kubernetes docs](/docs/) for more details on administering
and using a Kubernetes cluster.
content/en/docs/getting-started-guides/cloudstack.md
+93
View File
@@ -0,0 +1,93 @@
---
reviewers:
- thockin
title: Cloudstack
---
[CloudStack](https://cloudstack.apache.org/) is a software to build public and private clouds based on hardware virtualization principles (traditional IaaS). To deploy Kubernetes on CloudStack there are several possibilities depending on the Cloud being used and what images are made available. CloudStack also has a vagrant plugin available, hence Vagrant could be used to deploy Kubernetes either using the existing shell provisioner or using new Salt based recipes.
[CoreOS](http://coreos.com) templates for CloudStack are built [nightly](http://stable.release.core-os.net/amd64-usr/current/). CloudStack operators need to [register](http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/templates.html) this template in their cloud before proceeding with these Kubernetes deployment instructions.
This guide uses a single [Ansible playbook](https://github.com/apachecloudstack/k8s), which is completely automated and can deploy Kubernetes on a CloudStack based Cloud using CoreOS images. The playbook, creates an ssh key pair, creates a security group and associated rules and finally starts coreOS instances configured via cloud-init.
{{< toc >}}
## Prerequisites
$ sudo apt-get install -y python-pip libssl-dev
$ sudo pip install cs
$ sudo pip install sshpubkeys
$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt-get update
$ sudo apt-get install ansible
On CloudStack server you also have to install libselinux-python :
yum install libselinux-python
[_cs_](https://github.com/exoscale/cs) is a python module for the CloudStack API.
Set your CloudStack endpoint, API keys and HTTP method used.
You can define them as environment variables: `CLOUDSTACK_ENDPOINT`, `CLOUDSTACK_KEY`, `CLOUDSTACK_SECRET` and `CLOUDSTACK_METHOD`.
Or create a `~/.cloudstack.ini` file:
[cloudstack]
endpoint = <your cloudstack api endpoint>
key = <your api access key>
secret = <your api secret key>
method = post
We need to use the http POST method to pass the _large_ userdata to the coreOS instances.
### Clone the playbook
$ git clone https://github.com/apachecloudstack/k8s
$ cd kubernetes-cloudstack
### Create a Kubernetes cluster
You simply need to run the playbook.
$ ansible-playbook k8s.yml
Some variables can be edited in the `k8s.yml` file.
vars:
ssh_key: k8s
k8s_num_nodes: 2
k8s_security_group_name: k8s
k8s_node_prefix: k8s2
k8s_template: <templatename>
k8s_instance_type: <serviceofferingname>
This will start a Kubernetes master node and a number of compute nodes (by default 2).
The `instance_type` and `template` are specific, edit them to specify your CloudStack cloud specific template and instance type (i.e. service offering).
Check the tasks and templates in `roles/k8s` if you want to modify anything.
Once the playbook as finished, it will print out the IP of the Kubernetes master:
TASK: [k8s | debug msg='k8s master IP is {{ k8s_master.default_ip }}'] ********
SSH to it using the key that was created and using the _core_ user and you can list the machines in your cluster:
$ ssh -i ~/.ssh/id_rsa_k8s core@<master IP>
$ fleetctl list-machines
MACHINE IP METADATA
a017c422... <node #1 IP> role=node
ad13bf84... <master IP> role=master
e9af8293... <node #2 IP> role=node
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
CloudStack | Ansible | CoreOS | flannel | [docs](/docs/getting-started-guides/cloudstack/) | | Community ([@Guiques](https://github.com/ltupin/))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
content/en/docs/getting-started-guides/coreos/OWNERS
+3
View File
@@ -0,0 +1,3 @@
reviewers:
- errordeveloper
content/en/docs/getting-started-guides/coreos/_index.md
+92
View File
@@ -0,0 +1,92 @@
---
title: CoreOS
toc_hide: true
---
{{< toc >}}
There are multiple guides on running Kubernetes with [CoreOS](https://coreos.com/kubernetes/docs/latest/):
### Official CoreOS Guides
These guides are maintained by CoreOS and deploy Kubernetes the "CoreOS Way" with full TLS, the DNS add-on, and more. These guides pass Kubernetes conformance testing and we encourage you to [test this yourself](https://coreos.com/kubernetes/docs/latest/conformance-tests.html).
[**AWS Multi-Node**](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html)
Guide and CLI tool for setting up a multi-node cluster on AWS. CloudFormation is used to set up a master and multiple workers in auto-scaling groups.
<hr/>
[**Bare Metal Multi-Node**](https://coreos.com/kubernetes/docs/latest/kubernetes-on-baremetal.html#automated-provisioning)
Guide and HTTP/API service for PXE booting and provisioning a multi-node cluster on bare metal. [Ignition](https://coreos.com/ignition/docs/latest/) is used to provision a master and multiple workers on the first boot from disk.
[**Vagrant Multi-Node**](https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant.html)
Guide to setting up a multi-node cluster on Vagrant. The deployer can independently configure the number of etcd nodes, master nodes, and worker nodes to bring up a fully HA control plane.
<hr/>
[**Vagrant Single-Node**](https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html)
The quickest way to set up a Kubernetes development environment locally. As easy as `git clone`, `vagrant up` and configuring `kubectl`.
<hr/>
[**Full Step by Step Guide**](https://coreos.com/kubernetes/docs/latest/getting-started.html)
A generic guide to setting up an HA cluster on any cloud or bare metal, with full TLS. Repeat the master or worker steps to configure more machines of that role.
### Community Guides
These guides are maintained by community members, cover specific platforms and use cases, and experiment with different ways of configuring Kubernetes on CoreOS.
[**Easy Multi-node Cluster on Google Compute Engine**](https://github.com/rimusz/coreos-multi-node-k8s-gce/blob/master/README.md)
Scripted installation of a single master, multi-worker cluster on GCE. Kubernetes components are managed by [fleet](https://github.com/coreos/fleet).
<hr/>
[**Multi-node cluster using cloud-config and Weave on Vagrant**](https://github.com/errordeveloper/weave-demos/blob/master/poseidon/README.md)
Configure a Vagrant-based cluster of 3 machines with networking provided by Weave.
<hr/>
[**Multi-node cluster using cloud-config and Vagrant**](https://github.com/pires/kubernetes-vagrant-coreos-cluster/blob/master/README.md)
Configure a single master, multi-worker cluster locally, running on your choice of hypervisor: VirtualBox, Parallels, or VMware
<hr/>
[**Single-node cluster using a small OS X App**](https://github.com/rimusz/kube-solo-osx/blob/master/README.md)
Guide to running a solo cluster (master + worker) controlled by an OS X menubar application. Uses xhyve + CoreOS under the hood.
<hr/>
[**Multi-node cluster with Vagrant and fleet units using a small OS X App**](https://github.com/rimusz/coreos-osx-gui-kubernetes-cluster/blob/master/README.md)
Guide to running a single master, multi-worker cluster controlled by an OS X menubar application. Uses Vagrant under the hood.
<hr/>
[**Multi-node cluster using cloud-config, CoreOS and VMware ESXi**](https://github.com/xavierbaude/VMware-coreos-multi-nodes-Kubernetes)
Configure a single master, single worker cluster on VMware ESXi.
<hr/>
[**Single/Multi-node cluster using cloud-config, CoreOS and Foreman**](https://github.com/johscheuer/theforeman-coreos-kubernetes)
Configure a standalone Kubernetes or a Kubernetes cluster with [Foreman](https://theforeman.org).
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
GCE | CoreOS | CoreOS | flannel | [docs](/docs/getting-started-guides/coreos) | | Community ([@pires](https://github.com/pires))
Vagrant | CoreOS | CoreOS | flannel | [docs](/docs/getting-started-guides/coreos) | | Community ([@pires](https://github.com/pires), [@AntonioMeireles](https://github.com/AntonioMeireles))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
content/en/docs/getting-started-guides/coreos/cloud-configs/master.yaml
+142
View File
@@ -0,0 +1,142 @@
#cloud-config
---
write-files:
- path: /etc/conf.d/nfs
permissions: '0644'
content: |
OPTS_RPC_MOUNTD=""
- path: /opt/bin/wupiao
permissions: '0755'
content: |
#!/bin/bash
# [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen
[ -n "$1" ] && \
until curl -o /dev/null -sIf http://${1}; do \
sleep 1 && echo .;
done;
exit $?
hostname: master
coreos:
etcd2:
name: master
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
initial-cluster-token: k8s_etcd
listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001
initial-advertise-peer-urls: http://$private_ipv4:2380
initial-cluster: master=http://$private_ipv4:2380
initial-cluster-state: new
fleet:
metadata: "role=master"
units:
- name: etcd2.service
command: start
- name: generate-serviceaccount-key.service
command: start
content: |
[Unit]
Description=Generate service-account key file
[Service]
ExecStartPre=-/usr/bin/mkdir -p /opt/bin
ExecStart=/bin/openssl genrsa -out /opt/bin/kube-serviceaccount.key 2048 2>/dev/null
RemainAfterExit=yes
Type=oneshot
- name: setup-network-environment.service
command: start
content: |
[Unit]
Description=Setup Network Environment
Documentation=https://github.com/kelseyhightower/setup-network-environment
Requires=network-online.target
After=network-online.target
[Service]
ExecStartPre=-/usr/bin/mkdir -p /opt/bin
ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment
ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment
ExecStart=/opt/bin/setup-network-environment
RemainAfterExit=yes
Type=oneshot
- name: fleet.service
command: start
- name: flanneld.service
command: start
drop-ins:
- name: 50-network-config.conf
content: |
[Unit]
Requires=etcd2.service
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
- name: docker.service
command: start
- name: kube-apiserver.service
command: start
content: |
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
Requires=setup-network-environment.service etcd2.service generate-serviceaccount-key.service
After=setup-network-environment.service etcd2.service generate-serviceaccount-key.service
[Service]
EnvironmentFile=/etc/network-environment
ExecStartPre=-/usr/bin/mkdir -p /opt/bin
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-apiserver -z /opt/bin/kube-apiserver https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-apiserver
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-apiserver
ExecStartPre=/opt/bin/wupiao 127.0.0.1:2379/v2/machines
ExecStart=/opt/bin/kube-apiserver \
--service-account-key-file=/opt/bin/kube-serviceaccount.key \
--service-account-lookup=false \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota \
--runtime-config=api/v1 \
--allow-privileged=true \
--insecure-bind-address=0.0.0.0 \
--insecure-port=8080 \
--kubelet-https=true \
--secure-port=6443 \
--service-cluster-ip-range=10.100.0.0/16 \
--etcd-servers=http://127.0.0.1:2379 \
--public-address-override=${DEFAULT_IPV4} \
--logtostderr=true
Restart=always
RestartSec=10
- name: kube-controller-manager.service
command: start
content: |
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
Requires=kube-apiserver.service
After=kube-apiserver.service
[Service]
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-controller-manager -z /opt/bin/kube-controller-manager https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-controller-manager
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-controller-manager
ExecStart=/opt/bin/kube-controller-manager \
--service-account-private-key-file=/opt/bin/kube-serviceaccount.key \
--master=127.0.0.1:8080 \
--logtostderr=true
Restart=always
RestartSec=10
- name: kube-scheduler.service
command: start
content: |
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
Requires=kube-apiserver.service
After=kube-apiserver.service
[Service]
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-scheduler -z /opt/bin/kube-scheduler https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-scheduler
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-scheduler
ExecStart=/opt/bin/kube-scheduler --master=127.0.0.1:8080
Restart=always
RestartSec=10
update:
group: alpha
reboot-strategy: off
content/en/docs/getting-started-guides/coreos/cloud-configs/node.yaml
+93
View File
@@ -0,0 +1,93 @@
#cloud-config
write-files:
- path: /opt/bin/wupiao
permissions: '0755'
content: |
#!/bin/bash
# [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen
[ -n "$1" ] && [ -n "$2" ] && while ! curl --output /dev/null \
--silent --head --fail \
http://${1}:${2}; do sleep 1 && echo -n .; done;
exit $?
coreos:
etcd2:
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
advertise-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
initial-cluster: master=http://<master-private-ip>:2380
proxy: on
fleet:
metadata: "role=node"
units:
- name: etcd2.service
command: start
- name: fleet.service
command: start
- name: flanneld.service
command: start
- name: docker.service
command: start
- name: setup-network-environment.service
command: start
content: |
[Unit]
Description=Setup Network Environment
Documentation=https://github.com/kelseyhightower/setup-network-environment
Requires=network-online.target
After=network-online.target
[Service]
ExecStartPre=-/usr/bin/mkdir -p /opt/bin
ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment
ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment
ExecStart=/opt/bin/setup-network-environment
RemainAfterExit=yes
Type=oneshot
- name: kube-proxy.service
command: start
content: |
[Unit]
Description=Kubernetes Proxy
Documentation=https://github.com/kubernetes/kubernetes
Requires=setup-network-environment.service
After=setup-network-environment.service
[Service]
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-proxy -z /opt/bin/kube-proxy https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-proxy
ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-proxy
# wait for kubernetes master to be up and ready
ExecStartPre=/opt/bin/wupiao <master-private-ip> 8080
ExecStart=/opt/bin/kube-proxy \
--master=<master-private-ip>:8080 \
--logtostderr=true
Restart=always
RestartSec=10
- name: kube-kubelet.service
command: start
content: |
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
Requires=setup-network-environment.service
After=setup-network-environment.service
[Service]
EnvironmentFile=/etc/network-environment
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kubelet -z /opt/bin/kubelet https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kubelet
ExecStartPre=/usr/bin/chmod +x /opt/bin/kubelet
# wait for kubernetes master to be up and ready
ExecStartPre=/opt/bin/wupiao <master-private-ip> 8080
ExecStart=/opt/bin/kubelet \
--address=0.0.0.0 \
--port=10250 \
--hostname-override=${DEFAULT_IPV4} \
--api-servers=<master-private-ip>:8080 \
--allow-privileged=true \
--logtostderr=true \
--cadvisor-port=4194 \
--healthz-bind-address=0.0.0.0 \
--healthz-port=10248
Restart=always
RestartSec=10
update:
group: alpha
reboot-strategy: off
content/en/docs/getting-started-guides/dcos.md
+16
View File
@@ -0,0 +1,16 @@
---
reviewers:
- smugcloud
title: Kubernetes on DCOS
---
Mesosphere provides an easy option to provision Kubernetes onto [DC/OS](https://mesosphere.com/product/), offering:
* Pure upstream Kubernetes
* Single-click cluster provisioning
* Highly available and secure by default
* Kubernetes running alongside fast-data platforms (e.g. Akka, Cassandra, Kafka, Spark)
## Official Mesosphere Guide
The canonical source of getting started on DC/OS is located in the [quickstart repo](https://github.com/mesosphere/dcos-kubernetes-quickstart).
content/en/docs/getting-started-guides/fedora/OWNERS
+5
View File
@@ -0,0 +1,5 @@
reviewers:
- aveshagarwal
- eparis
- thockin
content/en/docs/getting-started-guides/fedora/_index.md
Executable
+5
View File
@@ -0,0 +1,5 @@
---
title: "Bare Metal"
weight: 60
---
content/en/docs/getting-started-guides/fedora/fedora_manual_config.md
+184
View File
@@ -0,0 +1,184 @@
---
reviewers:
- aveshagarwal
- eparis
- thockin
title: Fedora (Single Node)
---
{{< toc >}}
## Prerequisites
1. You need 2 or more machines with Fedora installed. These can be either bare metal machines or virtual machines.
## Instructions
This is a getting started guide for Fedora. It is a manual configuration so you understand all the underlying packages / services / ports, etc...
This guide will only get ONE node (previously minion) working. Multiple nodes require a functional [networking configuration](/docs/concepts/cluster-administration/networking/) done outside of Kubernetes. Although the additional Kubernetes configuration requirements should be obvious.
The Kubernetes package provides a few services: kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, kube-proxy. These services are managed by systemd and the configuration resides in a central location: `/etc/kubernetes`. We will break the services up between the hosts. The first host, fed-master, will be the Kubernetes master. This host will run the kube-apiserver, kube-controller-manager, and kube-scheduler. In addition, the master will also run _etcd_ (not needed if _etcd_ runs on a different host but this guide assumes that _etcd_ and Kubernetes master run on the same host). The remaining host, fed-node will be the node and run kubelet, proxy and docker.
**System Information:**
Hosts:
```conf
fed-master = 192.168.121.9
fed-node = 192.168.121.65
```
**Prepare the hosts:**
* Install Kubernetes on all hosts - fed-{master,node}. This will also pull in docker. Also install etcd on fed-master. This guide has been tested with Kubernetes-0.18 and beyond.
* Running on AWS EC2 with RHEL 7.2, you need to enable "extras" repository for yum by editing `/etc/yum.repos.d/redhat-rhui.repo` and changing the `enable=0` to `enable=1` for extras.
```shell
dnf -y install kubernetes
```
* Install etcd
```shell
dnf -y install etcd
```
* Add master and node to `/etc/hosts` on all machines (not needed if hostnames already in DNS). Make sure that communication works between fed-master and fed-node by using a utility such as ping.
```shell
echo "192.168.121.9 fed-master
192.168.121.65 fed-node" >> /etc/hosts
```
* Edit `/etc/kubernetes/config` (which should be the same on all hosts) to set
the name of the master server:
```shell
# Comma separated list of nodes in the etcd cluster
KUBE_MASTER="--master=http://fed-master:8080"
```
* Disable the firewall on both the master and node, as Docker does not play well with other firewall rule managers. Please note that iptables.service does not exist on the default Fedora Server install.
```shell
systemctl mask firewalld.service
systemctl stop firewalld.service
systemctl disable iptables.service
systemctl stop iptables.service
```
**Configure the Kubernetes services on the master.**
* Edit `/etc/kubernetes/apiserver` to appear as such. The service-cluster-ip-range IP addresses must be an unused block of addresses, not used anywhere else. They do not need to be routed or assigned to anything.
```shell
# The address on the local server to listen to.
KUBE_API_ADDRESS="--address=0.0.0.0"
# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"
# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
# Add your own!
KUBE_API_ARGS=""
```
* Edit `/etc/etcd/etcd.conf` to let etcd listen on all available IPs instead of 127.0.0.1. If you have not done this, you might see an error such as "connection refused".
```shell
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
```
* Start the appropriate services on master:
```shell
for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
```
**Configure the Kubernetes services on the node.**
***We need to configure the kubelet on the node.***
* Edit `/etc/kubernetes/kubelet` to appear as such:
```shell
###
# Kubernetes kubelet (node) config
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=fed-node"
# location of the api-server
KUBELET_ARGS="--cgroup-driver=systemd --kubeconfig=/etc/kubernetes/master-kubeconfig.yaml --require-kubeconfig"
# Add your own!
KUBELET_ARGS=""
```
```yaml
kind: Config
clusters:
- name: local
cluster:
server: http://fed-master:8080
users:
- name: kubelet
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
```
* Start the appropriate services on the node (fed-node).
```shell
for SERVICES in kube-proxy kubelet docker; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
```
* Check to make sure now the cluster can see the fed-node on fed-master, and its status changes to _Ready_.
```shell
kubectl get nodes
NAME STATUS AGE VERSION
fed-node Ready 4h
```
* Deletion of nodes:
To delete _fed-node_ from your Kubernetes cluster, one should run the following on fed-master (Please do not do it, it is just for information):
```shell
kubectl delete -f ./node.json
```
*You should be finished!*
**The cluster should be running! Launch a test pod.**
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Bare-metal | custom | Fedora | _none_ | [docs](/docs/getting-started-guides/fedora/fedora_manual_config) | | Project
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
content/en/docs/getting-started-guides/fedora/flannel_multi_node_cluster.md
+192
View File
@@ -0,0 +1,192 @@
---
reviewers:
- dchen1107
- erictune
- thockin
title: Fedora (Multi Node)
---
{{< toc >}}
This document describes how to deploy Kubernetes on multiple hosts to set up a multi-node cluster and networking with flannel. Follow fedora [getting started guide](/docs/getting-started-guides/fedora/fedora_manual_config/) to setup 1 master (fed-master) and 2 or more nodes. Make sure that all nodes have different names (fed-node1, fed-node2 and so on) and labels (fed-node1-label, fed-node2-label, and so on) to avoid any conflict. Also make sure that the Kubernetes master host is running etcd, kube-controller-manager, kube-scheduler, and kube-apiserver services, and the nodes are running docker, kube-proxy and kubelet services. Now install flannel on Kubernetes nodes. Flannel on each node configures an overlay network that docker uses. Flannel runs on each node to setup a unique class-C container network.
## Prerequisites
You need 2 or more machines with Fedora installed.
## Master Setup
**Perform following commands on the Kubernetes master**
* Configure flannel by creating a `flannel-config.json` in your current directory on fed-master. Flannel provides udp and vxlan among other overlay networking backend options. In this guide, we choose kernel based vxlan backend. The contents of the json are:
```json
{
"Network": "18.16.0.0/16",
"SubnetLen": 24,
"Backend": {
"Type": "vxlan",
"VNI": 1
}
}
```
**NOTE:** Choose an IP range that is *NOT* part of the public IP address range.
Add the configuration to the etcd server on fed-master.
```shell
etcdctl set /coreos.com/network/config < flannel-config.json
```
* Verify that the key exists in the etcd server on fed-master.
```shell
etcdctl get /coreos.com/network/config
```
## Node Setup
**Perform following commands on all Kubernetes nodes**
Install the flannel package
```shell
# dnf -y install flannel
```
Edit the flannel configuration file /etc/sysconfig/flanneld as follows:
```shell
# Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
FLANNEL_ETCD="http://fed-master:2379"
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/coreos.com/network"
# Any additional options that you want to pass
FLANNEL_OPTIONS=""
```
**Note:** By default, flannel uses the interface for the default route. If you have multiple interfaces and would like to use an interface other than the default route one, you could add "-iface=" to FLANNEL_OPTIONS. For additional options, run `flanneld --help` on command line.
Enable the flannel service.
```shell
systemctl enable flanneld
```
If docker is not running, then starting flannel service is enough and skip the next step.
```shell
systemctl start flanneld
```
If docker is already running, then stop docker, delete docker bridge (docker0), start flanneld and restart docker as follows. Another alternative is to just reboot the system (`systemctl reboot`).
```shell
systemctl stop docker
ip link delete docker0
systemctl start flanneld
systemctl start docker
```
## **Test the cluster and flannel configuration**
Now check the interfaces on the nodes. Notice there is now a flannel.1 interface, and the ip addresses of docker0 and flannel.1 interfaces are in the same network. You will notice that docker0 is assigned a subnet (18.16.29.0/24 as shown below) on each Kubernetes node out of the IP range configured above. A working output should look like this:
```shell
# ip -4 a|grep inet
inet 127.0.0.1/8 scope host lo
inet 192.168.122.77/24 brd 192.168.122.255 scope global dynamic eth0
inet 18.16.29.0/16 scope global flannel.1
inet 18.16.29.1/24 scope global docker0
```
From any node in the cluster, check the cluster members by issuing a query to etcd server via curl (only partial output is shown using `grep -E "\{|\}|key|value"`). If you set up a 1 master and 3 nodes cluster, you should see one block for each node showing the subnets they have been assigned. You can associate those subnets to each node by the MAC address (VtepMAC) and IP address (Public IP) that is listed in the output.
```shell
curl -s http://fed-master:2379/v2/keys/coreos.com/network/subnets | python -mjson.tool
```
```json
{
"node": {
"key": "/coreos.com/network/subnets",
{
"key": "/coreos.com/network/subnets/18.16.29.0-24",
"value": "{\"PublicIP\":\"192.168.122.77\",\"BackendType\":\"vxlan\",\"BackendData\":{\"VtepMAC\":\"46:f1:d0:18:d0:65\"}}"
},
{
"key": "/coreos.com/network/subnets/18.16.83.0-24",
"value": "{\"PublicIP\":\"192.168.122.36\",\"BackendType\":\"vxlan\",\"BackendData\":{\"VtepMAC\":\"ca:38:78:fc:72:29\"}}"
},
{
"key": "/coreos.com/network/subnets/18.16.90.0-24",
"value": "{\"PublicIP\":\"192.168.122.127\",\"BackendType\":\"vxlan\",\"BackendData\":{\"VtepMAC\":\"92:e2:80:ba:2d:4d\"}}"
}
}
}
```
From all nodes, review the `/run/flannel/subnet.env` file. This file was generated automatically by flannel.
```shell
# cat /run/flannel/subnet.env
FLANNEL_SUBNET=18.16.29.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
```
At this point, we have etcd running on the Kubernetes master, and flannel / docker running on Kubernetes nodes. Next steps are for testing cross-host container communication which will confirm that docker and flannel are configured properly.
Issue the following commands on any 2 nodes:
```shell
# docker run -it fedora:latest bash
bash-4.3#
```
This will place you inside the container. Install iproute and iputils packages to install ip and ping utilities. Due to a [bug](https://bugzilla.redhat.com/show_bug.cgi?id=1142311), it is required to modify capabilities of ping binary to work around "Operation not permitted" error.
```shell
bash-4.3# dnf -y install iproute iputils
bash-4.3# setcap cap_net_raw-ep /usr/bin/ping
```
Now note the IP address on the first node:
```shell
bash-4.3# ip -4 a l eth0 | grep inet
inet 18.16.29.4/24 scope global eth0
```
And also note the IP address on the other node:
```shell
bash-4.3# ip a l eth0 | grep inet
inet 18.16.90.4/24 scope global eth0
```
Now ping from the first node to the other node:
```shell
bash-4.3# ping 18.16.90.4
PING 18.16.90.4 (18.16.90.4) 56(84) bytes of data.
64 bytes from 18.16.90.4: icmp_seq=1 ttl=62 time=0.275 ms
64 bytes from 18.16.90.4: icmp_seq=2 ttl=62 time=0.372 ms
```
Now Kubernetes multi-node cluster is set up with overlay networking set up by flannel.
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Bare-metal | custom | Fedora | flannel | [docs](/docs/getting-started-guides/fedora/flannel_multi_node_cluster/) | | Community ([@aveshagarwal](https://github.com/aveshagarwal))
libvirt | custom | Fedora | flannel | [docs](/docs/getting-started-guides/fedora/flannel_multi_node_cluster/) | | Community ([@aveshagarwal](https://github.com/aveshagarwal))
KVM | custom | Fedora | flannel | [docs](/docs/getting-started-guides/fedora/flannel_multi_node_cluster/) | | Community ([@aveshagarwal](https://github.com/aveshagarwal))
content/en/docs/getting-started-guides/gce.md
+216
View File
@@ -0,0 +1,216 @@
---
reviewers:
- brendandburns
- jbeda
- mikedanese
- thockin
title: Running Kubernetes on Google Compute Engine
---
The example below creates a Kubernetes cluster with 4 worker node Virtual Machines and a master Virtual Machine (i.e. 5 VMs in your cluster). This cluster is set up and controlled from your workstation (or wherever you find convenient).
{{< toc >}}
### Before you start
If you want a simplified getting started experience and GUI for managing clusters, please consider trying [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) for hosted cluster installation and management.
For an easy way to experiment with the Kubernetes development environment, click the button below
to open a Google Cloud Shell with an auto-cloned copy of the Kubernetes source repo.
[![Open in Cloud Shell](http://gstatic.com/cloudssh/images/open-btn.png)](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/kubernetes/kubernetes&page=editor&open_in_editor=README.md)
If you want to use custom binaries or pure open source Kubernetes, please continue with the instructions below.
### Prerequisites
1. You need a Google Cloud Platform account with billing enabled. Visit the [Google Developers Console](https://console.cloud.google.com) for more details.
1. Install `gcloud` as necessary. `gcloud` can be installed as a part of the [Google Cloud SDK](https://cloud.google.com/sdk/).
1. Enable the [Compute Engine Instance Group Manager API](https://console.developers.google.com/apis/api/replicapool.googleapis.com/overview) in the [Google Cloud developers console](https://console.developers.google.com/apis/library).
1. Make sure that gcloud is set to use the Google Cloud Platform project you want. You can check the current project using `gcloud config list project` and change it via `gcloud config set project <project-id>`.
1. Make sure you have credentials for GCloud by running `gcloud auth login`.
1. (Optional) In order to make API calls against GCE, you must also run `gcloud auth application-default login`.
1. Make sure you can start up a GCE VM from the command line. At least make sure you can do the [Create an instance](https://cloud.google.com/compute/docs/instances/#startinstancegcloud) part of the GCE Quickstart.
1. Make sure you can SSH into the VM without interactive prompts. See the [Log in to the instance](https://cloud.google.com/compute/docs/instances/#sshing) part of the GCE Quickstart.
### Starting a cluster
You can install a client and start a cluster with either one of these commands (we list both in case only one is installed on your machine):
```shell
curl -sS https://get.k8s.io | bash
```
or
```shell
wget -q -O - https://get.k8s.io | bash
```
Once this command completes, you will have a master VM and four worker VMs, running as a Kubernetes cluster.
By default, some containers will already be running on your cluster. Containers like `fluentd` provide [logging](/docs/concepts/cluster-administration/logging/), while `heapster` provides [monitoring](http://releases.k8s.io/master/cluster/addons/cluster-monitoring/README.md) services.
The script run by the commands above creates a cluster with the name/prefix "kubernetes". It defines one specific cluster config, so you can't run it more than once.
Alternately, you can download and install the latest Kubernetes release from [this page](https://github.com/kubernetes/kubernetes/releases), then run the `<kubernetes>/cluster/kube-up.sh` script to start the cluster:
```shell
cd kubernetes
cluster/kube-up.sh
```
If you want more than one cluster running in your project, want to use a different name, or want a different number of worker nodes, see the `<kubernetes>/cluster/gce/config-default.sh` file for more fine-grained configuration before you start up your cluster.
If you run into trouble, please see the section on [troubleshooting](/docs/getting-started-guides/gce/#troubleshooting), post to the
[kubernetes-users group](https://groups.google.com/forum/#!forum/kubernetes-users), or come ask questions on [Slack](/docs/troubleshooting/#slack).
The next few steps will show you:
1. How to set up the command line client on your workstation to manage the cluster
1. Examples of how to use the cluster
1. How to delete the cluster
1. How to start clusters with non-default options (like larger clusters)
### Installing the Kubernetes command line tools on your workstation
The cluster startup script will leave you with a running cluster and a `kubernetes` directory on your workstation.
The [kubectl](/docs/user-guide/kubectl/) tool controls the Kubernetes cluster
manager. It lets you inspect your cluster resources, create, delete, and update
components, and much more. You will use it to look at your new cluster and bring
up example apps.
You can use `gcloud` to install the `kubectl` command-line tool on your workstation:
gcloud components install kubectl
**Note:** The kubectl version bundled with `gcloud` may be older than the one
downloaded by the get.k8s.io install script. See [Installing kubectl](/docs/tasks/kubectl/install/)
document to see how you can set up the latest `kubectl` on your workstation.
### Getting started with your cluster
#### Inspect your cluster
Once `kubectl` is in your path, you can use it to look at your cluster. E.g., running:
```shell
$ kubectl get --all-namespaces services
```
should show a set of [services](/docs/user-guide/services) that look something like this:
```shell
NAMESPACE NAME CLUSTER_IP EXTERNAL_IP PORT(S) AGE
default kubernetes 10.0.0.1 <none> 443/TCP 1d
kube-system kube-dns 10.0.0.2 <none> 53/TCP,53/UDP 1d
kube-system kube-ui 10.0.0.3 <none> 80/TCP 1d
...
```
Similarly, you can take a look at the set of [pods](/docs/user-guide/pods) that were created during cluster startup.
You can do this via the
```shell
$ kubectl get --all-namespaces pods
```
command.
You'll see a list of pods that looks something like this (the name specifics will be different):
```shell
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system fluentd-cloud-logging-kubernetes-minion-63uo 1/1 Running 0 14m
kube-system fluentd-cloud-logging-kubernetes-minion-c1n9 1/1 Running 0 14m
kube-system fluentd-cloud-logging-kubernetes-minion-c4og 1/1 Running 0 14m
kube-system fluentd-cloud-logging-kubernetes-minion-ngua 1/1 Running 0 14m
kube-system kube-dns-v5-7ztia 3/3 Running 0 15m
kube-system kube-ui-v1-curt1 1/1 Running 0 15m
kube-system monitoring-heapster-v5-ex4u3 1/1 Running 1 15m
kube-system monitoring-influx-grafana-v1-piled 2/2 Running 0 15m
```
Some of the pods may take a few seconds to start up (during this time they'll show `Pending`), but check that they all show as `Running` after a short period.
#### Run some examples
Then, see [a simple nginx example](/docs/user-guide/simple-nginx) to try out your new cluster.
For more complete applications, please look in the [examples directory](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/). The [guestbook example](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/guestbook/) is a good "getting started" walkthrough.
### Tearing down the cluster
To remove/delete/teardown the cluster, use the `kube-down.sh` script.
```shell
cd kubernetes
cluster/kube-down.sh
```
Likewise, the `kube-up.sh` in the same directory will bring it back up. You do not need to rerun the `curl` or `wget` command: everything needed to setup the Kubernetes cluster is now on your workstation.
### Customizing
The script above relies on Google Storage to stage the Kubernetes release. It
then will start (by default) a single master VM along with 4 worker VMs. You
can tweak some of these parameters by editing `kubernetes/cluster/gce/config-default.sh`
You can view a transcript of a successful cluster creation
[here](https://gist.github.com/satnam6502/fc689d1b46db9772adea).
### Troubleshooting
#### Project settings
You need to have the Google Cloud Storage API, and the Google Cloud Storage
JSON API enabled. It is activated by default for new projects. Otherwise, it
can be done in the Google Cloud Console. See the [Google Cloud Storage JSON
API Overview](https://cloud.google.com/storage/docs/json_api/) for more
details.
Also ensure that-- as listed in the [Prerequisites section](#prerequisites)-- you've enabled the `Compute Engine Instance Group Manager API`, and can start up a GCE VM from the command line as in the [GCE Quickstart](https://cloud.google.com/compute/docs/quickstart) instructions.
#### Cluster initialization hang
If the Kubernetes startup script hangs waiting for the API to be reachable, you can troubleshoot by SSHing into the master and node VMs and looking at logs such as `/var/log/startupscript.log`.
**Once you fix the issue, you should run `kube-down.sh` to cleanup** after the partial cluster creation, before running `kube-up.sh` to try again.
#### SSH
If you're having trouble SSHing into your instances, ensure the GCE firewall
isn't blocking port 22 to your VMs. By default, this should work but if you
have edited firewall rules or created a new non-default network, you'll need to
expose it: `gcloud compute firewall-rules create default-ssh --network=<network-name>
--description "SSH allowed from anywhere" --allow tcp:22`
Additionally, your GCE SSH key must either have no passcode or you need to be
using `ssh-agent`.
#### Networking
The instances must be able to connect to each other using their private IP. The
script uses the "default" network which should have a firewall rule called
"default-allow-internal" which allows traffic on any port on the private IPs.
If this rule is missing from the default network or if you change the network
being used in `cluster/config-default.sh` create a new rule with the following
field values:
* Source Ranges: `10.0.0.0/8`
* Allowed Protocols and Port: `tcp:1-65535;udp:1-65535;icmp`
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
GCE | Saltstack | Debian | GCE | [docs](/docs/getting-started-guides/gce/) | | Project
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
## Further reading
Please see the [Kubernetes docs](/docs/) for more details on administering
and using a Kubernetes cluster.
content/en/docs/getting-started-guides/kops.md
+165
View File
@@ -0,0 +1,165 @@
---
title: Installing Kubernetes on AWS with kops
---
## Overview
This quickstart shows you how to easily install a Kubernetes cluster on AWS.
It uses a tool called [`kops`](https://github.com/kubernetes/kops).
kops is an opinionated provisioning system:
* Fully automated installation
* Uses DNS to identify clusters
* Self-healing: everything runs in Auto-Scaling Groups
* Limited OS support (Debian preferred, Ubuntu 16.04 supported, early support for CentOS & RHEL)
* High-Availability support
* Can directly provision, or generate terraform manifests
If your opinions differ from these you may prefer to build your own cluster using [kubeadm](/docs/admin/kubeadm/) as
a building block. kops builds on the kubeadm work.
## Creating a cluster
### (1/5) Install kops
#### Requirements
You must have [kubectl](/docs/tasks/tools/install-kubectl/) installed in order for kops to work.
#### Installation
Download kops from the [releases page](https://github.com/kubernetes/kops/releases) (it is also easy to build from source):
On MacOS:
```
curl -OL https://github.com/kubernetes/kops/releases/download/1.8.0/kops-darwin-amd64
chmod +x kops-darwin-amd64
mv kops-darwin-amd64 /usr/local/bin/kops
# you can also install using Homebrew
brew update && brew install kops
```
On Linux:
```
wget https://github.com/kubernetes/kops/releases/download/1.8.0/kops-linux-amd64
chmod +x kops-linux-amd64
mv kops-linux-amd64 /usr/local/bin/kops
```
### (2/5) Create a route53 domain for your cluster
kops uses DNS for discovery, both inside the cluster and so that you can reach the kubernetes API server
from clients.
kops has a strong opinion on the cluster name: it should be a valid DNS name. By doing so you will
no longer get your clusters confused, you can share clusters with your colleagues unambiguously,
and you can reach them without relying on remembering an IP address.
You can, and probably should, use subdomains to divide your clusters. As our example we will use
`useast1.dev.example.com`. The API server endpoint will then be `api.useast1.dev.example.com`.
A Route53 hosted zone can serve subdomains. Your hosted zone could be `useast1.dev.example.com`,
but also `dev.example.com` or even `example.com`. kops works with any of these, so typically
you choose for organization reasons (e.g. you are allowed to create records under `dev.example.com`,
but not under `example.com`).
Let's assume you're using `dev.example.com` as your hosted zone. You create that hosted zone using
the [normal process](http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingNewSubdomain.html), or
with a command such as `aws route53 create-hosted-zone --name dev.example.com --caller-reference 1`.
You must then set up your NS records in the parent domain, so that records in the domain will resolve. Here,
you would create NS records in `example.com` for `dev`. If it is a root domain name you would configure the NS
records at your domain registrar (e.g. `example.com` would need to be configured where you bought `example.com`).
This step is easy to mess up (it is the #1 cause of problems!) You can double-check that
your cluster is configured correctly if you have the dig tool by running:
`dig NS dev.example.com`
You should see the 4 NS records that Route53 assigned your hosted zone.
### (3/5) Create an S3 bucket to store your clusters state
kops lets you manage your clusters even after installation. To do this, it must keep track of the clusters
that you have created, along with their configuration, the keys they are using etc. This information is stored
in an S3 bucket. S3 permissions are used to control access to the bucket.
Multiple clusters can use the same S3 bucket, and you can share an S3 bucket between your colleagues that
administer the same clusters - this is much easier than passing around kubecfg files. But anyone with access
to the S3 bucket will have administrative access to all your clusters, so you don't want to share it beyond
the operations team.
So typically you have one S3 bucket for each ops team (and often the name will correspond
to the name of the hosted zone above!)
In our example, we chose `dev.example.com` as our hosted zone, so let's pick `clusters.dev.example.com` as
the S3 bucket name.
* Export `AWS_PROFILE` (if you need to select a profile for the AWS CLI to work)
* Create the S3 bucket using `aws s3 mb s3://clusters.dev.example.com`
* You can `export KOPS_STATE_STORE=s3://clusters.dev.example.com` and then kops will use this location by default.
We suggest putting this in your bash profile or similar.
### (4/5) Build your cluster configuration
Run "kops create cluster" to create your cluster configuration:
`kops create cluster --zones=us-east-1c useast1.dev.example.com`
kops will create the configuration for your cluster. Note that it _only_ creates the configuration, it does
not actually create the cloud resources - you'll do that in the next step with a `kops update cluster`. This
give you an opportunity to review the configuration or change it.
It prints commands you can use to explore further:
* List your clusters with: `kops get cluster`
* Edit this cluster with: `kops edit cluster useast1.dev.example.com`
* Edit your node instance group: `kops edit ig --name=useast1.dev.example.com nodes`
* Edit your master instance group: `kops edit ig --name=useast1.dev.example.com master-us-east-1c`
If this is your first time using kops, do spend a few minutes to try those out! An instance group is a
set of instances, which will be registered as kubernetes nodes. On AWS this is implemented via auto-scaling-groups.
You can have several instance groups, for example if you wanted nodes that are a mix of spot and on-demand instances, or
GPU and non-GPU instances.
### (5/5) Create the cluster in AWS
Run "kops update cluster" to create your cluster in AWS:
`kops update cluster useast1.dev.example.com --yes`
That takes a few seconds to run, but then your cluster will likely take a few minutes to actually be ready.
`kops update cluster` will be the tool you'll use whenever you change the configuration of your cluster; it
applies the changes you have made to the configuration to your cluster - reconfiguring AWS or kubernetes as needed.
For example, after you `kops edit ig nodes`, then `kops update cluster --yes` to apply your configuration, and
sometimes you will also have to `kops rolling-update cluster` to roll out the configuration immediately.
Without `--yes`, `kops update cluster` will show you a preview of what it is going to do. This is handy
for production clusters!
### Explore other add-ons
See the [list of add-ons](/docs/concepts/cluster-administration/addons/) to explore other add-ons, including tools for logging, monitoring, network policy, visualization &amp; control of your Kubernetes cluster.
## What's next
* Learn more about Kubernetes [concepts](/docs/concepts/) and [`kubectl`](/docs/user-guide/kubectl-overview/).
* Learn about `kops` [advanced usage](https://github.com/kubernetes/kops)
## Cleanup
* To delete your cluster: `kops delete cluster useast1.dev.example.com --yes`
## Feedback
* Slack Channel: [#sig-aws](https://kubernetes.slack.com/messages/sig-aws/) has a lot of kops users
* [GitHub Issues](https://github.com/kubernetes/kops/issues)
content/en/docs/getting-started-guides/kubespray.md
+104
View File
@@ -0,0 +1,104 @@
---
title: Installing Kubernetes On-premises/Cloud Providers with Kubespray
---
## Overview
This quickstart helps to install a Kubernetes cluster hosted on GCE, Azure, OpenStack, AWS, or Baremetal with [Kubespray](https://github.com/kubernetes-incubator/kubespray).
Kubespray is a composition of [Ansible](http://docs.ansible.com/) playbooks, [inventory](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/ansible.md), provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks. Kubespray provides:
* a highly available cluster
* composable attributes
* support for most popular Linux distributions (CoreOS, Debian Jessie, Ubuntu 16.04, CentOS/RHEL 7, Fedora/CentOS Atomic)
* continuous integration tests
To choose a tool which best fits your use case, read [this comparison](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/comparisons.md) to [kubeadm](/docs/admin/kubeadm/) and [kops](../kops).
## Creating a cluster
### (1/5) Meet the underlay [requirements](https://github.com/kubernetes-incubator/kubespray#requirements)
Provision servers with the following requirements:
* `Ansible v2.4` (or newer)
* `Jinja 2.9` (or newer)
* `python-netaddr` installed on the machine that running Ansible commands
* Target servers must have access to the Internet in order to pull docker images
* Target servers are configured to allow IPv4 forwarding
* Target servers have SSH connectivity ( tcp/22 ) directly to your nodes or through a bastion host/ssh jump box
* Target servers have a privileged user
* Your SSH key must be copied to all the servers that are part of your inventory
* Firewall rules configured properly to allow Ansible and Kubernetes components to communicate
* If using a cloud provider, you must have the appropriate credentials available and exported as environment variables
Kubespray provides the following utilities to help provision your environment:
* [Terraform](https://www.terraform.io/) scripts for the following cloud providers:
* [AWS](https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/terraform/aws)
* [OpenStack](https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/terraform/openstack)
### (2/5) Compose an inventory file
After you provision your servers, create an [inventory file for Ansible](http://docs.ansible.com/ansible/intro_inventory.html). You can do this manually or via a dynamic inventory script. For more information, see "[Building your own inventory](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md#building-your-own-inventory)".
### (3/5) Plan your cluster deployment
Kubespray provides the ability to customize many aspects of the deployment:
* CNI (networking) plugins
* DNS configuration
* Choice of control plane: native/binary or containerized with docker or rkt)
* Component versions
* Calico route reflectors
* Component runtime options
* Certificate generation methods
Kubespray customizations can be made to a [variable file](http://docs.ansible.com/ansible/playbooks_variables.html). If you are just getting started with Kubespray, consider using the Kubespray defaults to deploy your cluster and explore Kubernetes.
### (4/5) Deploy a Cluster
Next, deploy your cluster:
Cluster deployment using [ansible-playbook](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md#starting-custom-deployment).
```console
ansible-playbook -i your/inventory/hosts.ini cluster.yml -b -v \
--private-key=~/.ssh/private_key
```
Large deployments (100+ nodes) may require [specific adjustments](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/large-deployments.md) for best results.
### (5/5) Verify the deployment
Kubespray provides a way to verify inter-pod connectivity and DNS resolve with [Netchecker](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/netcheck.md). Netchecker ensures the netchecker-agents pods can resolve DNS requests and ping each over within the default namespace. Those pods mimic similar behavior of the rest of the workloads and serve as cluster health indicators.
## Cluster operations
Kubespray provides additional playbooks to manage your cluster: _scale_ and _upgrade_.
### Scale your cluster
You can add worker nodes from your cluster by running the scale playbook. For more information, see "[Adding nodes](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md#adding-nodes)".
You can remove worker nodes from your cluster by running the remove-node playbook. For more information, see "[Remove nodes](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md#remove-nodes)".
### Upgrade your cluster
You can upgrade your cluster by running the upgrade-cluster playbook. For more information, see "[Upgrades](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/upgrades.md)".
## What's next
Check out planned work on Kubespray's [roadmap](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/roadmap.md).
## Cleanup
You can reset your nodes and wipe out all components installed with Kubespray via the [reset playbook](https://github.com/kubernetes-incubator/kubespray/blob/master/reset.yml).
{{< caution >}}
**Caution:** When running the reset playbook, be sure not to accidentally target your production cluster!
{{< /caution >}}
## Feedback
* Slack Channel: [#kubespray](https://kubernetes.slack.com/messages/kubespray/)
* [GitHub Issues](https://github.com/kubernetes-incubator/kubespray/issues)
content/en/docs/getting-started-guides/minikube.md
+363
View File
@@ -0,0 +1,363 @@
---
reviewers:
- dlorenc
- r2d4
- aaron-prindle
title: Running Kubernetes Locally via Minikube
---
Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.
{{< toc >}}
### Minikube Features
* Minikube supports Kubernetes features such as:
* DNS
* NodePorts
* ConfigMaps and Secrets
* Dashboards
* Container Runtime: Docker, [rkt](https://github.com/rkt/rkt) and [CRI-O](https://github.com/kubernetes-incubator/cri-o)
* Enabling CNI (Container Network Interface)
* Ingress
## Installation
See [Installing Minikube](/docs/tasks/tools/install-minikube/).
## Quickstart
Here's a brief demo of minikube usage.
If you want to change the VM driver add the appropriate `--vm-driver=xxx` flag to `minikube start`. Minikube supports
the following drivers:
* virtualbox
* vmwarefusion
* kvm ([driver installation](https://git.k8s.io/minikube/docs/drivers.md#kvm-driver))
* hyperkit ([driver installation](https://git.k8s.io/minikube/docs/drivers.md#hyperkit-driver))
* xhyve ([driver installation](https://git.k8s.io/minikube/docs/drivers.md#xhyve-driver)) (deprecated)
Note that the IP below is dynamic and can change. It can be retrieved with `minikube ip`.
```shell
$ minikube start
Starting local Kubernetes cluster...
Running pre-create checks...
Creating machine...
Starting local Kubernetes cluster...
$ kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
deployment "hello-minikube" created
$ kubectl expose deployment hello-minikube --type=NodePort
service "hello-minikube" exposed
# We have now launched an echoserver pod but we have to wait until the pod is up before curling/accessing it
# via the exposed service.
# To check whether the pod is up and running we can use the following:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
hello-minikube-3383150820-vctvh 0/1 ContainerCreating 0 3s
# We can see that the pod is still being created from the ContainerCreating status
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
hello-minikube-3383150820-vctvh 1/1 Running 0 13s
# We can see that the pod is now Running and we will now be able to curl it:
$ curl $(minikube service hello-minikube --url)
CLIENT VALUES:
client_address=192.168.99.1
command=GET
real path=/
...
$ kubectl delete services hello-minikube
service "hello-minikube" deleted
$ kubectl delete deployment hello-minikube
deployment "hello-minikube" deleted
$ minikube stop
Stopping local Kubernetes cluster...
Stopping "minikube"...
```
### Alternative Container Runtimes
#### CRI-O
To use [CRI-O](https://github.com/kubernetes-incubator/cri-o) as the container runtime, run:
```bash
$ minikube start \
--network-plugin=cni \
--container-runtime=cri-o \
--bootstrapper=kubeadm
```
Or you can use the extended version:
```bash
$ minikube start \
--network-plugin=cni \
--extra-config=kubelet.container-runtime=remote \
--extra-config=kubelet.container-runtime-endpoint=/var/run/crio.sock \
--extra-config=kubelet.image-service-endpoint=/var/run/crio.sock \
--bootstrapper=kubeadm
```
#### rkt container engine
To use [rkt](https://github.com/rkt/rkt) as the container runtime run:
```shell
$ minikube start \
--network-plugin=cni \
--container-runtime=rkt
```
This will use an alternative minikube ISO image containing both rkt, and Docker, and enable CNI networking.
### Driver plugins
See [DRIVERS](https://git.k8s.io/minikube/docs/drivers.md) for details on supported drivers and how to install
plugins, if required.
### Reusing the Docker daemon
When using a single VM of Kubernetes, it's really handy to reuse the minikube's built-in Docker daemon; as this means you don't have to build a docker registry on your host machine and push the image into it - you can just build inside the same docker daemon as minikube which speeds up local experiments. Just make sure you tag your Docker image with something other than 'latest' and use that tag while you pull the image. Otherwise, if you do not specify version of your image, it will be assumed as `:latest`, with pull image policy of `Always` correspondingly, which may eventually result in `ErrImagePull` as you may not have any versions of your Docker image out there in the default docker registry (usually DockerHub) yet.
To be able to work with the docker daemon on your mac/linux host use the `docker-env command` in your shell:
```
eval $(minikube docker-env)
```
You should now be able to use docker on the command line on your host mac/linux machine talking to the docker daemon inside the minikube VM:
```
docker ps
```
On Centos 7, docker may report the following error:
```
Could not read CA certificate "/etc/docker/ca.pem": open /etc/docker/ca.pem: no such file or directory
```
The fix is to update /etc/sysconfig/docker to ensure that minikube's environment changes are respected:
```
< DOCKER_CERT_PATH=/etc/docker
---
> if [ -z "${DOCKER_CERT_PATH}" ]; then
> DOCKER_CERT_PATH=/etc/docker
> fi
```
Remember to turn off the imagePullPolicy:Always, as otherwise Kubernetes won't use images you built locally.
## Managing your Cluster
### Starting a Cluster
The `minikube start` command can be used to start your cluster.
This command creates and configures a virtual machine that runs a single-node Kubernetes cluster.
This command also configures your [kubectl](/docs/user-guide/kubectl-overview/) installation to communicate with this cluster.
If you are behind a web proxy, you will need to pass this information in e.g. via
```
https_proxy=<my proxy> minikube start --docker-env http_proxy=<my proxy> --docker-env https_proxy=<my proxy> --docker-env no_proxy=192.168.99.0/24
```
Unfortunately just setting the environment variables will not work.
Minikube will also create a "minikube" context, and set it to default in kubectl.
To switch back to this context later, run this command: `kubectl config use-context minikube`.
#### Specifying the Kubernetes version
Minikube supports running multiple different versions of Kubernetes. You can
access a list of all available versions via
```
minikube get-k8s-versions
```
You can specify the specific version of Kubernetes for Minikube to use by
adding the `--kubernetes-version` string to the `minikube start` command. For
example, to run version `v1.7.3`, you would run the following:
```
minikube start --kubernetes-version v1.7.3
```
### Configuring Kubernetes
Minikube has a "configurator" feature that allows users to configure the Kubernetes components with arbitrary values.
To use this feature, you can use the `--extra-config` flag on the `minikube start` command.
This flag is repeated, so you can pass it several times with several different values to set multiple options.
This flag takes a string of the form `component.key=value`, where `component` is one of the strings from the below list, `key` is a value on the
configuration struct and `value` is the value to set.
Valid keys can be found by examining the documentation for the Kubernetes `componentconfigs` for each component.
Here is the documentation for each supported configuration:
* [kubelet](https://godoc.org/k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig#KubeletConfiguration)
* [apiserver](https://godoc.org/k8s.io/kubernetes/cmd/kube-apiserver/app/options#ServerRunOptions)
* [proxy](https://godoc.org/k8s.io/kubernetes/pkg/proxy/apis/kubeproxyconfig#KubeProxyConfiguration)
* [controller-manager](https://godoc.org/k8s.io/kubernetes/pkg/apis/componentconfig#KubeControllerManagerConfiguration)
* [etcd](https://godoc.org/github.com/coreos/etcd/etcdserver#ServerConfig)
* [scheduler](https://godoc.org/k8s.io/kubernetes/pkg/apis/componentconfig#KubeSchedulerConfiguration)
#### Examples
To change the `MaxPods` setting to 5 on the Kubelet, pass this flag: `--extra-config=kubelet.MaxPods=5`.
This feature also supports nested structs. To change the `LeaderElection.LeaderElect` setting to `true` on the scheduler, pass this flag: `--extra-config=scheduler.LeaderElection.LeaderElect=true`.
To set the `AuthorizationMode` on the `apiserver` to `RBAC`, you can use: `--extra-config=apiserver.Authorization.Mode=RBAC`.
### Stopping a Cluster
The `minikube stop` command can be used to stop your cluster.
This command shuts down the minikube virtual machine, but preserves all cluster state and data.
Starting the cluster again will restore it to it's previous state.
### Deleting a Cluster
The `minikube delete` command can be used to delete your cluster.
This command shuts down and deletes the minikube virtual machine. No data or state is preserved.
## Interacting With your Cluster
### Kubectl
The `minikube start` command creates a "[kubectl context](/docs/reference/generated/kubectl/kubectl-commands/#-em-set-context-em-)" called "minikube".
This context contains the configuration to communicate with your minikube cluster.
Minikube sets this context to default automatically, but if you need to switch back to it in the future, run:
`kubectl config use-context minikube`,
Or pass the context on each command like this: `kubectl get pods --context=minikube`.
### Dashboard
To access the [Kubernetes Dashboard](/docs/tasks/access-application-cluster/web-ui-dashboard/), run this command in a shell after starting minikube to get the address:
```shell
minikube dashboard
```
### Services
To access a service exposed via a node port, run this command in a shell after starting minikube to get the address:
```shell
minikube service [-n NAMESPACE] [--url] NAME
```
## Networking
The minikube VM is exposed to the host system via a host-only IP address, that can be obtained with the `minikube ip` command.
Any services of type `NodePort` can be accessed over that IP address, on the NodePort.
To determine the NodePort for your service, you can use a `kubectl` command like this:
`kubectl get service $SERVICE --output='jsonpath="{.spec.ports[0].nodePort}"'`
## Persistent Volumes
Minikube supports [PersistentVolumes](/docs/concepts/storage/persistent-volumes/) of type `hostPath`.
These PersistentVolumes are mapped to a directory inside the minikube VM.
The Minikube VM boots into a tmpfs, so most directories will not be persisted across reboots (`minikube stop`).
However, Minikube is configured to persist files stored under the following host directories:
* `/data`
* `/var/lib/localkube`
* `/var/lib/docker`
Here is an example PersistentVolume config to persist data in the `/data` directory:
```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0001
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 5Gi
hostPath:
path: /data/pv0001/
```
## Mounted Host Folders
Some drivers will mount a host folder within the VM so that you can easily share files between the VM and host. These are not configurable at the moment and different for the driver and OS you are using.
**Note:** Host folder sharing is not implemented in the KVM driver yet.
| Driver | OS | HostFolder | VM |
| --- | --- | --- | --- |
| VirtualBox | Linux | /home | /hosthome |
| VirtualBox | OSX | /Users | /Users |
| VirtualBox | Windows | C://Users | /c/Users |
| VMware Fusion | OSX | /Users | /Users |
| Xhyve | OSX | /Users | /Users |
## Private Container Registries
To access a private container registry, follow the steps on [this page](/docs/concepts/containers/images/).
We recommend you use `ImagePullSecrets`, but if you would like to configure access on the minikube VM you can place the `.dockercfg` in the `/home/docker` directory or the `config.json` in the `/home/docker/.docker` directory.
## Add-ons
In order to have minikube properly start or restart custom addons,
place the addons you wish to be launched with minikube in the `~/.minikube/addons`
directory. Addons in this folder will be moved to the minikube VM and
launched each time minikube is started or restarted.
## Using Minikube with an HTTP Proxy
Minikube creates a Virtual Machine that includes Kubernetes and a Docker daemon.
When Kubernetes attempts to schedule containers using Docker, the Docker daemon may require external network access to pull containers.
If you are behind an HTTP proxy, you may need to supply Docker with the proxy settings.
To do this, pass the required environment variables as flags during `minikube start`.
For example:
```shell
$ minikube start --docker-env http_proxy=http://$YOURPROXY:PORT \
--docker-env https_proxy=https://$YOURPROXY:PORT
```
If your Virtual Machine address is 192.168.99.100, then chances are your proxy settings will prevent kubectl from directly reaching it.
To by-pass proxy configuration for this IP address, you should modify your no_proxy settings. You can do so with:
```shell
$ export no_proxy=$no_proxy,$(minikube ip)
```
## Known Issues
* Features that require a Cloud Provider will not work in Minikube. These include:
* LoadBalancers
* Features that require multiple nodes. These include:
* Advanced scheduling policies
## Design
Minikube uses [libmachine](https://github.com/docker/machine/tree/master/libmachine) for provisioning VMs, and [localkube](https://git.k8s.io/minikube/pkg/localkube) (originally written and donated to this project by [RedSpread](https://redspread.com/)) for running the cluster.
For more information about minikube, see the [proposal](https://git.k8s.io/community/contributors/design-proposals/cluster-lifecycle/local-cluster-ux.md).
## Additional Links:
* **Goals and Non-Goals**: For the goals and non-goals of the minikube project, please see our [roadmap](https://git.k8s.io/minikube/docs/contributors/roadmap.md).
* **Development Guide**: See [CONTRIBUTING.md](https://git.k8s.io/minikube/CONTRIBUTING.md) for an overview of how to send pull requests.
* **Building Minikube**: For instructions on how to build/test minikube from source, see the [build guide](https://git.k8s.io/minikube/docs/contributors/build_guide.md)
* **Adding a New Dependency**: For instructions on how to add a new dependency to minikube see the [adding dependencies guide](https://git.k8s.io/minikube/docs/contributors/adding_a_dependency.md)
* **Adding a New Addon**: For instruction on how to add a new addon for minikube see the [adding an addon guide](https://git.k8s.io/minikube/docs/contributors/adding_an_addon.md)
* **Updating Kubernetes**: For instructions on how to update kubernetes see the [updating Kubernetes guide](https://git.k8s.io/minikube/docs/contributors/updating_kubernetes.md)
## Community
Contributions, questions, and comments are all welcomed and encouraged! minikube developers hang out on [Slack](https://kubernetes.slack.com) in the #minikube channel (get an invitation [here](http://slack.kubernetes.io/)). We also have the [kubernetes-dev Google Groups mailing list](https://groups.google.com/forum/#!forum/kubernetes-dev). If you are posting to the list please prefix your subject with "minikube: ".
content/en/docs/getting-started-guides/ovirt.md
+62
View File
@@ -0,0 +1,62 @@
---
reviewers:
- caesarxuchao
- erictune
title: oVirt
---
{{< toc >}}
## What is oVirt
oVirt is a virtual datacenter manager that delivers powerful management of multiple virtual machines on multiple hosts. Using KVM and libvirt, oVirt can be installed on Fedora, CentOS, or Red Hat Enterprise Linux hosts to set up and manage your virtual data center.
## oVirt Cloud Provider Deployment
The oVirt cloud provider allows to easily discover and automatically add new VM instances as nodes to your Kubernetes cluster.
At the moment there are no community-supported or pre-loaded VM images including Kubernetes but it is possible to [import] or [install] Project Atomic (or Fedora) in a VM to [generate a template]. Any other distribution that includes Kubernetes may work as well.
It is mandatory to [install the ovirt-guest-agent] in the guests for the VM ip address and hostname to be reported to ovirt-engine and ultimately to Kubernetes.
Once the Kubernetes template is available it is possible to start instantiating VMs that can be discovered by the cloud provider.
[import]: http://ovedou.blogspot.it/2014/03/importing-glance-images-as-ovirt.html
[install]: https://www.ovirt.org/documentation/quickstart/quickstart-guide/#create-virtual-machines
[generate a template]: https://www.ovirt.org/documentation/quickstart/quickstart-guide/#using-templates
[install the ovirt-guest-agent]: http://www.ovirt.org/documentation/how-to/guest-agent/install-the-guest-agent-in-fedora/
## Using the oVirt Cloud Provider
The oVirt Cloud Provider requires access to the oVirt REST-API to gather the proper information, the required credential should be specified in the `ovirt-cloud.conf` file:
[connection]
uri = https://localhost:8443/ovirt-engine/api
username = admin@internal
password = admin
In the same file it is possible to specify (using the `filters` section) what search query to use to identify the VMs to be reported to Kubernetes:
[filters]
# Search query used to find nodes
vms = tag=kubernetes
In the above example all the VMs tagged with the `kubernetes` label will be reported as nodes to Kubernetes.
The `ovirt-cloud.conf` file then must be specified in kube-controller-manager:
kube-controller-manager ... --cloud-provider=ovirt --cloud-config=/path/to/ovirt-cloud.conf ...
## oVirt Cloud Provider Screencast
This short screencast demonstrates how the oVirt Cloud Provider can be used to dynamically add VMs to your Kubernetes cluster.
[![Screencast](http://img.youtube.com/vi/JyyST4ZKne8/0.jpg)](http://www.youtube.com/watch?v=JyyST4ZKne8)
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
oVirt | | | | [docs](/docs/getting-started-guides/ovirt) | | Community ([@simon3z](https://github.com/simon3z))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
content/en/docs/getting-started-guides/scratch.md
+884
View File
@@ -0,0 +1,884 @@
---
reviewers:
- erictune
- lavalamp
- thockin
title: Creating a Custom Cluster from Scratch
---
This guide is for people who want to craft a custom Kubernetes cluster. If you
can find an existing Getting Started Guide that meets your needs on [this
list](/docs/setup/), then we recommend using it, as you will be able to benefit
from the experience of others. However, if you have specific IaaS, networking,
configuration management, or operating system requirements not met by any of
those guides, then this guide will provide an outline of the steps you need to
take. Note that it requires considerably more effort than using one of the
pre-defined guides.
This guide is also useful for those wanting to understand at a high level some of the
steps that existing cluster setup scripts are making.
{{< toc >}}
## Designing and Preparing
### Learning
1. You should be familiar with using Kubernetes already. We suggest you set
up a temporary cluster by following one of the other Getting Started Guides.
This will help you become familiar with the CLI ([kubectl](/docs/user-guide/kubectl/)) and concepts ([pods](/docs/user-guide/pods/), [services](/docs/concepts/services-networking/service/), etc.) first.
1. You should have `kubectl` installed on your desktop. This will happen as a side
effect of completing one of the other Getting Started Guides. If not, follow the instructions
[here](/docs/tasks/kubectl/install/).
### Cloud Provider
Kubernetes has the concept of a Cloud Provider, which is a module which provides
an interface for managing TCP Load Balancers, Nodes (Instances) and Networking Routes.
The interface is defined in `pkg/cloudprovider/cloud.go`. It is possible to
create a custom cluster without implementing a cloud provider (for example if using
bare-metal), and not all parts of the interface need to be implemented, depending
on how flags are set on various components.
### Nodes
- You can use virtual or physical machines.
- While you can build a cluster with 1 machine, in order to run all the examples and tests you
need at least 4 nodes.
- Many Getting-started-guides make a distinction between the master node and regular nodes. This
is not strictly necessary.
- Nodes will need to run some version of Linux with the x86_64 architecture. It may be possible
to run on other OSes and Architectures, but this guide does not try to assist with that.
- Apiserver and etcd together are fine on a machine with 1 core and 1GB RAM for clusters with 10s of nodes.
Larger or more active clusters may benefit from more cores.
- Other nodes can have any reasonable amount of memory and any number of cores. They need not
have identical configurations.
### Network
#### Network Connectivity
Kubernetes has a distinctive [networking model](/docs/concepts/cluster-administration/networking/).
Kubernetes allocates an IP address to each pod. When creating a cluster, you
need to allocate a block of IPs for Kubernetes to use as Pod IPs. The simplest
approach is to allocate a different block of IPs to each node in the cluster as
the node is added. A process in one pod should be able to communicate with
another pod using the IP of the second pod. This connectivity can be
accomplished in two ways:
- **Using an overlay network**
- An overlay network obscures the underlying network architecture from the
pod network through traffic encapsulation (for example vxlan).
- Encapsulation reduces performance, though exactly how much depends on your solution.
- **Without an overlay network**
- Configure the underlying network fabric (switches, routers, etc.) to be aware of pod IP addresses.
- This does not require the encapsulation provided by an overlay, and so can achieve
better performance.
Which method you choose depends on your environment and requirements. There are various ways
to implement one of the above options:
- **Use a network plugin which is called by Kubernetes**
- Kubernetes supports the [CNI](https://github.com/containernetworking/cni) network plugin interface.
- There are a number of solutions which provide plugins for Kubernetes (listed alphabetically):
- [Calico](http://docs.projectcalico.org/)
- [Flannel](https://github.com/coreos/flannel)
- [Open vSwitch (OVS)](http://openvswitch.org/)
- [Romana](http://romana.io/)
- [Weave](http://weave.works/)
- [More found here](/docs/admin/networking#how-to-achieve-this/)
- You can also write your own.
- **Compile support directly into Kubernetes**
- This can be done by implementing the "Routes" interface of a Cloud Provider module.
- The Google Compute Engine ([GCE](/docs/getting-started-guides/gce/)) and [AWS](/docs/getting-started-guides/aws/) guides use this approach.
- **Configure the network external to Kubernetes**
- This can be done by manually running commands, or through a set of externally maintained scripts.
- You have to implement this yourself, but it can give you an extra degree of flexibility.
You will need to select an address range for the Pod IPs.
- Various approaches:
- GCE: each project has its own `10.0.0.0/8`. Carve off a `/16` for each
Kubernetes cluster from that space, which leaves room for several clusters.
Each node gets a further subdivision of this space.
- AWS: use one VPC for whole organization, carve off a chunk for each
cluster, or use different VPC for different clusters.
- Allocate one CIDR subnet for each node's PodIPs, or a single large CIDR
from which smaller CIDRs are automatically allocated to each node.
- You need max-pods-per-node * max-number-of-nodes IPs in total. A `/24` per
node supports 254 pods per machine and is a common choice. If IPs are
scarce, a `/26` (62 pods per machine) or even a `/27` (30 pods) may be sufficient.
- For example, use `10.10.0.0/16` as the range for the cluster, with up to 256 nodes
using `10.10.0.0/24` through `10.10.255.0/24`, respectively.
- Need to make these routable or connect with overlay.
Kubernetes also allocates an IP to each [service](/docs/concepts/services-networking/service/). However,
service IPs do not necessarily need to be routable. The kube-proxy takes care
of translating Service IPs to Pod IPs before traffic leaves the node. You do
need to allocate a block of IPs for services. Call this
`SERVICE_CLUSTER_IP_RANGE`. For example, you could set
`SERVICE_CLUSTER_IP_RANGE="10.0.0.0/16"`, allowing 65534 distinct services to
be active at once. Note that you can grow the end of this range, but you
cannot move it without disrupting the services and pods that already use it.
Also, you need to pick a static IP for master node.
- Call this `MASTER_IP`.
- Open any firewalls to allow access to the apiserver ports 80 and/or 443.
- Enable ipv4 forwarding sysctl, `net.ipv4.ip_forward = 1`
#### Network Policy
Kubernetes enables the definition of fine-grained network policy between Pods using the [NetworkPolicy](/docs/concepts/services-networking/network-policies/) resource.
Not all networking providers support the Kubernetes NetworkPolicy API, see [Using Network Policy](/docs/tasks/configure-pod-container/declare-network-policy/) for more information.
### Cluster Naming
You should pick a name for your cluster. Pick a short name for each cluster
which is unique from future cluster names. This will be used in several ways:
- by kubectl to distinguish between various clusters you have access to. You will probably want a
second one sometime later, such as for testing new Kubernetes releases, running in a different
region of the world, etc.
- Kubernetes clusters can create cloud provider resources (for example, AWS ELBs) and different clusters
need to distinguish which resources each created. Call this `CLUSTER_NAME`.
### Software Binaries
You will need binaries for:
- etcd
- A container runner, one of:
- docker
- rkt
- Kubernetes
- kubelet
- kube-proxy
- kube-apiserver
- kube-controller-manager
- kube-scheduler
#### Downloading and Extracting Kubernetes Binaries
A Kubernetes binary release includes all the Kubernetes binaries as well as the supported release of etcd.
You can use a Kubernetes binary release (recommended) or build your Kubernetes binaries following the instructions in the
[Developer Documentation](https://git.k8s.io/community/contributors/devel/). Only using a binary release is covered in this guide.
Download the [latest binary release](https://github.com/kubernetes/kubernetes/releases/latest) and unzip it.
Server binary tarballs are no longer included in the Kubernetes final tarball, so you will need to locate and run
`./kubernetes/cluster/get-kube-binaries.sh` to download the client and server binaries.
Then locate `./kubernetes/server/kubernetes-server-linux-amd64.tar.gz` and unzip *that*.
Then, within the second set of unzipped files, locate `./kubernetes/server/bin`, which contains
all the necessary binaries.
#### Selecting Images
You will run docker, kubelet, and kube-proxy outside of a container, the same way you would run any system daemon, so
you just need the bare binaries. For etcd, kube-apiserver, kube-controller-manager, and kube-scheduler,
we recommend that you run these as containers, so you need an image to be built.
You have several choices for Kubernetes images:
- Use images hosted on Google Container Registry (GCR):
- For example `k8s.gcr.io/hyperkube:$TAG`, where `TAG` is the latest
release tag, which can be found on the [latest releases page](https://github.com/kubernetes/kubernetes/releases/latest).
- Ensure $TAG is the same tag as the release tag you are using for kubelet and kube-proxy.
- The [hyperkube](https://releases.k8s.io/{{< param "githubbranch" >}}/cmd/hyperkube) binary is an all in one binary
- `hyperkube kubelet ...` runs the kubelet, `hyperkube apiserver ...` runs an apiserver, etc.
- Build your own images.
- Useful if you are using a private registry.
- The release contains files such as `./kubernetes/server/bin/kube-apiserver.tar` which
can be converted into docker images using a command like
`docker load -i kube-apiserver.tar`
- You can verify if the image is loaded successfully with the right repository and tag using
command like `docker images`
For etcd, you can:
- Use images hosted on Google Container Registry (GCR), such as `k8s.gcr.io/etcd:2.2.1`
- Use images hosted on [Docker Hub](https://hub.docker.com/search/?q=etcd) or [Quay.io](https://quay.io/repository/coreos/etcd), such as `quay.io/coreos/etcd:v2.2.1`
- Use etcd binary included in your OS distro.
- Build your own image
- You can do: `cd kubernetes/cluster/images/etcd; make`
We recommend that you use the etcd version which is provided in the Kubernetes binary distribution. The Kubernetes binaries in the release
were tested extensively with this version of etcd and not with any other version.
The recommended version number can also be found as the value of `TAG` in `kubernetes/cluster/images/etcd/Makefile`.
The remainder of the document assumes that the image identifiers have been chosen and stored in corresponding env vars. Examples (replace with latest tags and appropriate registry):
- `HYPERKUBE_IMAGE=k8s.gcr.io/hyperkube:$TAG`
- `ETCD_IMAGE=k8s.gcr.io/etcd:$ETCD_VERSION`
### Security Models
There are two main options for security:
- Access the apiserver using HTTP.
- Use a firewall for security.
- This is easier to setup.
- Access the apiserver using HTTPS
- Use https with certs, and credentials for user.
- This is the recommended approach.
- Configuring certs can be tricky.
If following the HTTPS approach, you will need to prepare certs and credentials.
#### Preparing Certs
You need to prepare several certs:
- The master needs a cert to act as an HTTPS server.
- The kubelets optionally need certs to identify themselves as clients of the master, and when
serving its own API over HTTPS.
Unless you plan to have a real CA generate your certs, you will need
to generate a root cert and use that to sign the master, kubelet, and
kubectl certs. How to do this is described in the [authentication
documentation](/docs/admin/authentication/#creating-certificates/).
You will end up with the following files (we will use these variables later on)
- `CA_CERT`
- put in on node where apiserver runs, for example in `/srv/kubernetes/ca.crt`.
- `MASTER_CERT`
- signed by CA_CERT
- put in on node where apiserver runs, for example in `/srv/kubernetes/server.crt`
- `MASTER_KEY `
- put in on node where apiserver runs, for example in `/srv/kubernetes/server.key`
- `KUBELET_CERT`
- optional
- `KUBELET_KEY`
- optional
#### Preparing Credentials
The admin user (and any users) need:
- a token or a password to identify them.
- tokens are just long alphanumeric strings, 32 chars for example. See
- `TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)`
Your tokens and passwords need to be stored in a file for the apiserver
to read. This guide uses `/var/lib/kube-apiserver/known_tokens.csv`.
The format for this file is described in the [authentication documentation](/docs/admin/authentication/).
For distributing credentials to clients, the convention in Kubernetes is to put the credentials
into a [kubeconfig file](/docs/concepts/cluster-administration/authenticate-across-clusters-kubeconfig/).
The kubeconfig file for the administrator can be created as follows:
- If you have already used Kubernetes with a non-custom cluster (for example, used a Getting Started
Guide), you will already have a `$HOME/.kube/config` file.
- You need to add certs, keys, and the master IP to the kubeconfig file:
- If using the firewall-only security option, set the apiserver this way:
- `kubectl config set-cluster $CLUSTER_NAME --server=http://$MASTER_IP --insecure-skip-tls-verify=true`
- Otherwise, do this to set the apiserver ip, client certs, and user credentials.
- `kubectl config set-cluster $CLUSTER_NAME --certificate-authority=$CA_CERT --embed-certs=true --server=https://$MASTER_IP`
- `kubectl config set-credentials $USER --client-certificate=$CLI_CERT --client-key=$CLI_KEY --embed-certs=true --token=$TOKEN`
- Set your cluster as the default cluster to use:
- `kubectl config set-context $CONTEXT_NAME --cluster=$CLUSTER_NAME --user=$USER`
- `kubectl config use-context $CONTEXT_NAME`
Next, make a kubeconfig file for the kubelets and kube-proxy. There are a couple of options for how
many distinct files to make:
1. Use the same credential as the admin
- This is simplest to setup.
1. One token and kubeconfig file for all kubelets, one for all kube-proxy, one for admin.
- This mirrors what is done on GCE today
1. Different credentials for every kubelet, etc.
- We are working on this but all the pieces are not ready yet.
You can make the files by copying the `$HOME/.kube/config` or by using the following template:
```yaml
apiVersion: v1
kind: Config
users:
- name: kubelet
user:
token: ${KUBELET_TOKEN}
clusters:
- name: local
cluster:
certificate-authority: /srv/kubernetes/ca.crt
contexts:
- context:
cluster: local
user: kubelet
name: service-account-context
current-context: service-account-context
```
Put the kubeconfig(s) on every node. The examples later in this
guide assume that there are kubeconfigs in `/var/lib/kube-proxy/kubeconfig` and
`/var/lib/kubelet/kubeconfig`.
## Configuring and Installing Base Software on Nodes
This section discusses how to configure machines to be Kubernetes nodes.
You should run three daemons on every node:
- docker or rkt
- kubelet
- kube-proxy
You will also need to do assorted other configuration on top of a
base OS install.
Tip: One possible starting point is to setup a cluster using an existing Getting
Started Guide. After getting a cluster running, you can then copy the init.d scripts or systemd unit files from that
cluster, and then modify them for use on your custom cluster.
### Docker
The minimum required Docker version will vary as the kubelet version changes. The newest stable release is a good choice. Kubelet will log a warning and refuse to start pods if the version is too old, so pick a version and try it.
If you previously had Docker installed on a node without setting Kubernetes-specific
options, you may have a Docker-created bridge and iptables rules. You may want to remove these
as follows before proceeding to configure Docker for Kubernetes.
```shell
iptables -t nat -F
ip link set docker0 down
ip link delete docker0
```
The way you configure docker will depend in whether you have chosen the routable-vip or overlay-network approaches for your network.
Some suggested docker options:
- create your own bridge for the per-node CIDR ranges, call it cbr0, and set `--bridge=cbr0` option on docker.
- set `--iptables=false` so docker will not manipulate iptables for host-ports (too coarse on older docker versions, may be fixed in newer versions)
so that kube-proxy can manage iptables instead of docker.
- `--ip-masq=false`
- if you have setup PodIPs to be routable, then you want this false, otherwise, docker will
rewrite the PodIP source-address to a NodeIP.
- some environments (for example GCE) still need you to masquerade out-bound traffic when it leaves the cloud environment. This is very environment specific.
- if you are using an overlay network, consult those instructions.
- `--mtu=`
- may be required when using Flannel, because of the extra packet size due to udp encapsulation
- `--insecure-registry $CLUSTER_SUBNET`
- to connect to a private registry, if you set one up, without using SSL.
You may want to increase the number of open files for docker:
- `DOCKER_NOFILE=1000000`
Where this config goes depends on your node OS. For example, GCE's Debian-based distro uses `/etc/default/docker`.
Ensure docker is working correctly on your system before proceeding with the rest of the
installation, by following examples given in the Docker documentation.
### rkt
[rkt](https://github.com/coreos/rkt) is an alternative to Docker. You only need to install one of Docker or rkt.
The minimum version required is [v0.5.6](https://github.com/coreos/rkt/releases/tag/v0.5.6).
[systemd](http://www.freedesktop.org/wiki/Software/systemd/) is required on your node to run rkt. The
minimum version required to match rkt v0.5.6 is
[systemd 215](http://lists.freedesktop.org/archives/systemd-devel/2014-July/020903.html).
[rkt metadata service](https://github.com/coreos/rkt/blob/master/Documentation/networking/overview.md) is also required
for rkt networking support. You can start rkt metadata service by using command like
`sudo systemd-run rkt metadata-service`
Then you need to configure your kubelet with flag:
- `--container-runtime=rkt`
### kubelet
All nodes should run kubelet. See [Software Binaries](#software-binaries).
Arguments to consider:
- If following the HTTPS security approach:
- `--kubeconfig=/var/lib/kubelet/kubeconfig`
- Otherwise, if taking the firewall-based security approach
- `--config=/etc/kubernetes/manifests`
- `--cluster-dns=` to the address of the DNS server you will setup (see [Starting Cluster Services](#starting-cluster-services).)
- `--cluster-domain=` to the dns domain prefix to use for cluster DNS addresses.
- `--docker-root=`
- `--root-dir=`
- `--pod-cidr=` The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master.
- `--register-node` (described in [Node](/docs/admin/node/) documentation.)
### kube-proxy
All nodes should run kube-proxy. (Running kube-proxy on a "master" node is not
strictly required, but being consistent is easier.) Obtain a binary as described for
kubelet.
Arguments to consider:
- If following the HTTPS security approach:
- `--master=https://$MASTER_IP`
- `--kubeconfig=/var/lib/kube-proxy/kubeconfig`
- Otherwise, if taking the firewall-based security approach
- `--master=http://$MASTER_IP`
Note that on some Linux platforms, you may need to manually install the
`conntrack` package which is a dependency of kube-proxy, or else kube-proxy
cannot be started successfully.
For more details on debugging kube-proxy problems, please refer to
[Debug Services](/docs/tasks/debug-application-cluster/debug-service/)
### Networking
Each node needs to be allocated its own CIDR range for pod networking.
Call this `NODE_X_POD_CIDR`.
A bridge called `cbr0` needs to be created on each node. The bridge is explained
further in the [networking documentation](/docs/concepts/cluster-administration/networking/). The bridge itself
needs an address from `$NODE_X_POD_CIDR` - by convention the first IP. Call
this `NODE_X_BRIDGE_ADDR`. For example, if `NODE_X_POD_CIDR` is `10.0.0.0/16`,
then `NODE_X_BRIDGE_ADDR` is `10.0.0.1/16`. NOTE: this retains the `/16` suffix
because of how this is used later.
If you have turned off Docker's IP masquerading to allow pods to talk to each
other, then you may need to do masquerading just for destination IPs outside
the cluster network. For example:
```shell
iptables -t nat -A POSTROUTING ! -d ${CLUSTER_SUBNET} -m addrtype ! --dst-type LOCAL -j MASQUERADE
```
This will rewrite the source address from
the PodIP to the Node IP for traffic bound outside the cluster, and kernel
[connection tracking](http://www.iptables.info/en/connection-state.html)
will ensure that responses destined to the node still reach
the pod.
NOTE: This is environment specific. Some environments will not need
any masquerading at all. Others, such as GCE, will not allow pod IPs to send
traffic to the internet, but have no problem with them inside your GCE Project.
### Other
- Enable auto-upgrades for your OS package manager, if desired.
- Configure log rotation for all node components (for example using [logrotate](http://linux.die.net/man/8/logrotate)).
- Setup liveness-monitoring (for example using [supervisord](http://supervisord.org/)).
- Setup volume plugin support (optional)
- Install any client binaries for optional volume types, such as `glusterfs-client` for GlusterFS
volumes.
### Using Configuration Management
The previous steps all involved "conventional" system administration techniques for setting up
machines. You may want to use a Configuration Management system to automate the node configuration
process. There are examples of [Saltstack](/docs/admin/salt/), Ansible, Juju, and CoreOS Cloud Config in the
various Getting Started Guides.
## Bootstrapping the Cluster
While the basic node services (kubelet, kube-proxy, docker) are typically started and managed using
traditional system administration/automation approaches, the remaining *master* components of Kubernetes are
all configured and managed *by Kubernetes*:
- Their options are specified in a Pod spec (yaml or json) rather than an /etc/init.d file or
systemd unit.
- They are kept running by Kubernetes rather than by init.
### etcd
You will need to run one or more instances of etcd.
- Highly available and easy to restore - Run 3 or 5 etcd instances with, their logs written to a directory backed
by durable storage (RAID, GCE PD)
- Not highly available, but easy to restore - Run one etcd instance, with its log written to a directory backed
by durable storage (RAID, GCE PD).
{{< note >}}
**Note:** May result in operations outages in case of instance outage.
{{< /note >}}
- Highly available - Run 3 or 5 etcd instances with non durable storage.
{{< note >}}
**Note:** Log can be written to non-durable storage because storage is replicated.
{{< /note >}}
See [cluster-troubleshooting](/docs/admin/cluster-troubleshooting/) for more discussion on factors affecting cluster
availability.
To run an etcd instance:
1. Copy [`cluster/gce/manifests/etcd.manifest`](https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/manifests/etcd.manifest)
1. Make any modifications needed
1. Start the pod by putting it into the kubelet manifest directory
### Apiserver, Controller Manager, and Scheduler
The apiserver, controller manager, and scheduler will each run as a pod on the master node.
For each of these components, the steps to start them running are similar:
1. Start with a provided template for a pod.
1. Set the `HYPERKUBE_IMAGE` to the values chosen in [Selecting Images](#selecting-images).
1. Determine which flags are needed for your cluster, using the advice below each template.
1. Set the flags to be individual strings in the command array (for example $ARGN below)
1. Start the pod by putting the completed template into the kubelet manifest directory.
1. Verify that the pod is started.
#### Apiserver pod template
```json
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "kube-apiserver"
},
"spec": {
"hostNetwork": true,
"containers": [
{
"name": "kube-apiserver",
"image": "${HYPERKUBE_IMAGE}",
"command": [
"/hyperkube",
"apiserver",
"$ARG1",
"$ARG2",
...
"$ARGN"
],
"ports": [
{
"name": "https",
"hostPort": 443,
"containerPort": 443
},
{
"name": "local",
"hostPort": 8080,
"containerPort": 8080
}
],
"volumeMounts": [
{
"name": "srvkube",
"mountPath": "/srv/kubernetes",
"readOnly": true
},
{
"name": "etcssl",
"mountPath": "/etc/ssl",
"readOnly": true
}
],
"livenessProbe": {
"httpGet": {
"scheme": "HTTP",
"host": "127.0.0.1",
"port": 8080,
"path": "/healthz"
},
"initialDelaySeconds": 15,
"timeoutSeconds": 15
}
}
],
"volumes": [
{
"name": "srvkube",
"hostPath": {
"path": "/srv/kubernetes"
}
},
{
"name": "etcssl",
"hostPath": {
"path": "/etc/ssl"
}
}
]
}
}
```
Here are some apiserver flags you may need to set:
- `--cloud-provider=` see [cloud providers](#cloud-providers)
- `--cloud-config=` see [cloud providers](#cloud-providers)
- `--address=${MASTER_IP}` *or* `--bind-address=127.0.0.1` and `--address=127.0.0.1` if you want to run a proxy on the master node.
- `--service-cluster-ip-range=$SERVICE_CLUSTER_IP_RANGE`
- `--etcd-servers=http://127.0.0.1:4001`
- `--tls-cert-file=/srv/kubernetes/server.cert`
- `--tls-private-key-file=/srv/kubernetes/server.key`
- `--enable-admission-plugins=$RECOMMENDED_LIST`
- See [admission controllers](/docs/admin/admission-controllers/) for recommended arguments.
- `--allow-privileged=true`, only if you trust your cluster user to run pods as root.
If you are following the firewall-only security approach, then use these arguments:
- `--token-auth-file=/dev/null`
- `--insecure-bind-address=$MASTER_IP`
- `--advertise-address=$MASTER_IP`
If you are using the HTTPS approach, then set:
- `--client-ca-file=/srv/kubernetes/ca.crt`
- `--token-auth-file=/srv/kubernetes/known_tokens.csv`
- `--basic-auth-file=/srv/kubernetes/basic_auth.csv`
This pod mounts several node file system directories using the `hostPath` volumes. Their purposes are:
- The `/etc/ssl` mount allows the apiserver to find the SSL root certs so it can
authenticate external services, such as a cloud provider.
- This is not required if you do not use a cloud provider (bare-metal for example).
- The `/srv/kubernetes` mount allows the apiserver to read certs and credentials stored on the
node disk. These could instead be stored on a persistent disk, such as a GCE PD, or baked into the image.
- Optionally, you may want to mount `/var/log` as well and redirect output there (not shown in template).
- Do this if you prefer your logs to be accessible from the root filesystem with tools like journalctl.
*TODO* document proxy-ssh setup.
##### Cloud Providers
Apiserver supports several cloud providers.
- options for `--cloud-provider` flag are `aws`, `azure`, `cloudstack`, `fake`, `gce`, `mesos`, `openstack`, `ovirt`, `photon`, `rackspace`, `vsphere`, or unset.
- unset used for bare metal setups.
- support for new IaaS is added by contributing code [here](https://releases.k8s.io/{{< param "githubbranch" >}}/pkg/cloudprovider/providers)
Some cloud providers require a config file. If so, you need to put config file into apiserver image or mount through hostPath.
- `--cloud-config=` set if cloud provider requires a config file.
- Used by `aws`, `gce`, `mesos`, `openstack`, `ovirt` and `rackspace`.
- You must put config file into apiserver image or mount through hostPath.
- Cloud config file syntax is [Gcfg](https://code.google.com/p/gcfg/).
- AWS format defined by type [AWSCloudConfig](https://releases.k8s.io/{{< param "githubbranch" >}}/pkg/cloudprovider/providers/aws/aws.go)
- There is a similar type in the corresponding file for other cloud providers.
#### Scheduler pod template
Complete this template for the scheduler pod:
```json
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "kube-scheduler"
},
"spec": {
"hostNetwork": true,
"containers": [
{
"name": "kube-scheduler",
"image": "$HYPERKUBE_IMAGE",
"command": [
"/hyperkube",
"scheduler",
"--master=127.0.0.1:8080",
"$SCHEDULER_FLAG1",
...
"$SCHEDULER_FLAGN"
],
"livenessProbe": {
"httpGet": {
"scheme": "HTTP",
"host": "127.0.0.1",
"port": 10251,
"path": "/healthz"
},
"initialDelaySeconds": 15,
"timeoutSeconds": 15
}
}
]
}
}
```
Typically, no additional flags are required for the scheduler.
Optionally, you may want to mount `/var/log` as well and redirect output there.
#### Controller Manager Template
Template for controller manager pod:
```json
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "kube-controller-manager"
},
"spec": {
"hostNetwork": true,
"containers": [
{
"name": "kube-controller-manager",
"image": "$HYPERKUBE_IMAGE",
"command": [
"/hyperkube",
"controller-manager",
"$CNTRLMNGR_FLAG1",
...
"$CNTRLMNGR_FLAGN"
],
"volumeMounts": [
{
"name": "srvkube",
"mountPath": "/srv/kubernetes",
"readOnly": true
},
{
"name": "etcssl",
"mountPath": "/etc/ssl",
"readOnly": true
}
],
"livenessProbe": {
"httpGet": {
"scheme": "HTTP",
"host": "127.0.0.1",
"port": 10252,
"path": "/healthz"
},
"initialDelaySeconds": 15,
"timeoutSeconds": 15
}
}
],
"volumes": [
{
"name": "srvkube",
"hostPath": {
"path": "/srv/kubernetes"
}
},
{
"name": "etcssl",
"hostPath": {
"path": "/etc/ssl"
}
}
]
}
}
```
Flags to consider using with controller manager:
- `--cluster-cidr=`, the CIDR range for pods in cluster.
- `--allocate-node-cidrs=`, if you are using `--cloud-provider=`, allocate and set the CIDRs for pods on the cloud provider.
- `--cloud-provider=` and `--cloud-config` as described in apiserver section.
- `--service-account-private-key-file=/srv/kubernetes/server.key`, used by the [service account](/docs/user-guide/service-accounts) feature.
- `--master=127.0.0.1:8080`
#### Starting and Verifying Apiserver, Scheduler, and Controller Manager
Place each completed pod template into the kubelet config dir
(whatever `--config=` argument of kubelet is set to, typically
`/etc/kubernetes/manifests`). The order does not matter: scheduler and
controller manager will retry reaching the apiserver until it is up.
Use `ps` or `docker ps` to verify that each process has started. For example, verify that kubelet has started a container for the apiserver like this:
```shell
$ sudo docker ps | grep apiserver
5783290746d5 k8s.gcr.io/kube-apiserver:e36bf367342b5a80d7467fd7611ad873 "/bin/sh -c '/usr/lo'" 10 seconds ago Up 9 seconds k8s_kube-apiserver.feb145e7_kube-apiserver-kubernetes-master_default_eaebc600cf80dae59902b44225f2fc0a_225a4695
```
Then try to connect to the apiserver:
```shell
$ echo $(curl -s http://localhost:8080/healthz)
ok
$ curl -s http://localhost:8080/api
{
"versions": [
"v1"
]
}
```
If you have selected the `--register-node=true` option for kubelets, they will now begin self-registering with the apiserver.
You should soon be able to see all your nodes by running the `kubectl get nodes` command.
Otherwise, you will need to manually create node objects.
### Starting Cluster Services
You will want to complete your Kubernetes clusters by adding cluster-wide
services. These are sometimes called *addons*, and [an overview
of their purpose is in the admin guide](/docs/admin/cluster-components/#addons).
Notes for setting up each cluster service are given below:
* Cluster DNS:
* Required for many Kubernetes examples
* [Setup instructions](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/)
* [Admin Guide](/docs/concepts/services-networking/dns-pod-service/)
* Cluster-level Logging
* [Cluster-level Logging Overview](/docs/user-guide/logging/overview/)
* [Cluster-level Logging with Elasticsearch](/docs/user-guide/logging/elasticsearch/)
* [Cluster-level Logging with Stackdriver Logging](/docs/user-guide/logging/stackdriver/)
* Container Resource Monitoring
* [Setup instructions](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/cluster-monitoring/)
* GUI
* [Setup instructions](https://github.com/kubernetes/dashboard)
## Troubleshooting
### Running validate-cluster
`cluster/validate-cluster.sh` is used by `cluster/kube-up.sh` to determine if
the cluster start succeeded.
Example usage and output:
```shell
KUBECTL_PATH=$(which kubectl) NUM_NODES=3 KUBERNETES_PROVIDER=local cluster/validate-cluster.sh
Found 3 node(s).
NAME STATUS AGE VERSION
node1.local Ready 1h v1.6.9+a3d1dfa6f4335
node2.local Ready 1h v1.6.9+a3d1dfa6f4335
node3.local Ready 1h v1.6.9+a3d1dfa6f4335
Validate output:
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
Cluster validation succeeded
```
### Inspect pods and services
Try to run through the "Inspect your cluster" section in one of the other Getting Started Guides, such as [GCE](/docs/getting-started-guides/gce/#inspect-your-cluster).
You should see some services. You should also see "mirror pods" for the apiserver, scheduler and controller-manager, plus any add-ons you started.
### Try Examples
At this point you should be able to run through one of the basic examples, such as the [nginx example](/docs/tutorials/stateless-application/deployment.yaml).
### Running the Conformance Test
You may want to try to run the [Conformance test](http://releases.k8s.io/{{< param "githubbranch" >}}/test/e2e_node/conformance/run_test.sh). Any failures may give a hint as to areas that need more attention.
### Networking
The nodes must be able to connect to each other using their private IP. Verify this by
pinging or SSH-ing from one node to another.
### Getting Help
If you run into trouble, please see the section on [troubleshooting](/docs/getting-started-guides/gce/#troubleshooting), post to the
[kubernetes-users group](https://groups.google.com/forum/#!forum/kubernetes-users), or come ask questions on [Slack](/docs/troubleshooting#slack).
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
any | any | any | any | [docs](/docs/getting-started-guides/scratch/) | | Community ([@erictune](https://github.com/erictune))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions/) chart.
content/en/docs/getting-started-guides/stackpoint.md
+199
View File
@@ -0,0 +1,199 @@
---
reviewers:
- baldwinspc
title: Running Kubernetes on Multiple Clouds with Stackpoint.io
---
{{< toc >}}
## Introduction
StackPointCloud is the universal control plane for Kubernetes Anywhere. StackPointCloud allows you to deploy and manage a Kubernetes cluster to the cloud provider of your choice in 3 steps using a web-based interface.
## AWS
To create a Kubernetes cluster on AWS, you will need an Access Key ID and a Secret Access Key from AWS.
### Choose a Provider
Log in to [stackpoint.io](https://stackpoint.io) with a GitHub, Google, or Twitter account.
Click **+ADD A CLUSTER NOW**.
Click to select Amazon Web Services (AWS).
### Configure Your Provider
Add your Access Key ID and a Secret Access Key from AWS. Select your default StackPointCloud SSH keypair, or click **ADD SSH KEY** to add a new keypair.
Click **SUBMIT** to submit the authorization information.
### Configure Your Cluster
Choose any extra options you may want to include with your cluster, then click **SUBMIT** to create the cluster.
### Running the Cluster
You can monitor the status of your cluster and suspend or delete it from [your stackpoint.io dashboard](https://stackpoint.io/#/clusters).
For information on using and managing a Kubernetes cluster on AWS, [consult the Kubernetes documentation](/docs/getting-started-guides/aws/).
## GCE
To create a Kubernetes cluster on GCE, you will need the Service Account JSON Data from Google.
### Choose a Provider
Log in to [stackpoint.io](https://stackpoint.io) with a GitHub, Google, or Twitter account.
Click **+ADD A CLUSTER NOW**.
Click to select Google Compute Engine (GCE).
### Configure Your Provider
Add your Service Account JSON Data from Google. Select your default StackPointCloud SSH keypair, or click **ADD SSH KEY** to add a new keypair.
Click **SUBMIT** to submit the authorization information.
### Configure Your Cluster
Choose any extra options you may want to include with your cluster, then click **SUBMIT** to create the cluster.
### Running the Cluster
You can monitor the status of your cluster and suspend or delete it from [your stackpoint.io dashboard](https://stackpoint.io/#/clusters).
For information on using and managing a Kubernetes cluster on GCE, [consult the Kubernetes documentation](/docs/getting-started-guides/gce/).
## Google Kubernetes Engine
To create a Kubernetes cluster on Google Kubernetes Engine, you will need the Service Account JSON Data from Google.
### Choose a Provider
Log in to [stackpoint.io](https://stackpoint.io) with a GitHub, Google, or Twitter account.
Click **+ADD A CLUSTER NOW**.
Click to select Google Kubernetes Engine.
### Configure Your Provider
Add your Service Account JSON Data from Google. Select your default StackPointCloud SSH keypair, or click **ADD SSH KEY** to add a new keypair.
Click **SUBMIT** to submit the authorization information.
### Configure Your Cluster
Choose any extra options you may want to include with your cluster, then click **SUBMIT** to create the cluster.
### Running the Cluster
You can monitor the status of your cluster and suspend or delete it from [your stackpoint.io dashboard](https://stackpoint.io/#/clusters).
For information on using and managing a Kubernetes cluster on Google Kubernetes Engine, consult [the official documentation](/docs/home/).
## DigitalOcean
To create a Kubernetes cluster on DigitalOcean, you will need a DigitalOcean API Token.
### Choose a Provider
Log in to [stackpoint.io](https://stackpoint.io) with a GitHub, Google, or Twitter account.
Click **+ADD A CLUSTER NOW**.
Click to select DigitalOcean.
### Configure Your Provider
Add your DigitalOcean API Token. Select your default StackPointCloud SSH keypair, or click **ADD SSH KEY** to add a new keypair.
Click **SUBMIT** to submit the authorization information.
### Configure Your Cluster
Choose any extra options you may want to include with your cluster, then click **SUBMIT** to create the cluster.
### Running the Cluster
You can monitor the status of your cluster and suspend or delete it from [your stackpoint.io dashboard](https://stackpoint.io/#/clusters).
For information on using and managing a Kubernetes cluster on DigitalOcean, consult [the official documentation](/docs/home/).
## Microsoft Azure
To create a Kubernetes cluster on Microsoft Azure, you will need an Azure Subscription ID, Username/Email, and Password.
### Choose a Provider
Log in to [stackpoint.io](https://stackpoint.io) with a GitHub, Google, or Twitter account.
Click **+ADD A CLUSTER NOW**.
Click to select Microsoft Azure.
### Configure Your Provider
Add your Azure Subscription ID, Username/Email, and Password. Select your default StackPointCloud SSH keypair, or click **ADD SSH KEY** to add a new keypair.
Click **SUBMIT** to submit the authorization information.
### Configure Your Cluster
Choose any extra options you may want to include with your cluster, then click **SUBMIT** to create the cluster.
### Running the Cluster
You can monitor the status of your cluster and suspend or delete it from [your stackpoint.io dashboard](https://stackpoint.io/#/clusters).
For information on using and managing a Kubernetes cluster on Azure, [consult the Kubernetes documentation](/docs/getting-started-guides/azure/).
## Packet
To create a Kubernetes cluster on Packet, you will need a Packet API Key.
### Choose a Provider
Log in to [stackpoint.io](https://stackpoint.io) with a GitHub, Google, or Twitter account.
Click **+ADD A CLUSTER NOW**.
Click to select Packet.
### Configure Your Provider
Add your Packet API Key. Select your default StackPointCloud SSH keypair, or click **ADD SSH KEY** to add a new keypair.
Click **SUBMIT** to submit the authorization information.
### Configure Your Cluster
Choose any extra options you may want to include with your cluster, then click **SUBMIT** to create the cluster.
### Running the Cluster
You can monitor the status of your cluster and suspend or delete it from [your stackpoint.io dashboard](https://stackpoint.io/#/clusters).
For information on using and managing a Kubernetes cluster on Packet, consult [the official documentation](/docs/home/).
content/en/docs/getting-started-guides/ubuntu/_index.md
+77
View File
@@ -0,0 +1,77 @@
---
title: Kubernetes on Ubuntu
content_template: templates/concept
---
{{% capture overview %}}
There are multiple ways to run a Kubernetes cluster with Ubuntu. These pages explain how to deploy Kubernetes on Ubuntu on multiple public and private clouds, as well as bare metal.
{{% /capture %}}
{{% capture body %}}
## Official Ubuntu Guides
- [The Canonical Distribution of Kubernetes](https://www.ubuntu.com/cloud/kubernetes)
The latest version of Kubernetes with upstream binaries. Supports AWS, GCE, Azure, Joyent, OpenStack, VMware, Bare Metal and localhost deployments.
### Quick Start
[conjure-up](http://conjure-up.io/) provides the quickest way to deploy Kubernetes on Ubuntu for multiple clouds and bare metal. It provides a user-friendly UI that prompts you for cloud credentials and configuration options
Available for Ubuntu 16.04 and newer:
```
sudo snap install conjure-up --classic
# re-login may be required at that point if you just installed snap utility
conjure-up kubernetes
```
As well as Homebrew for macOS:
```
brew install conjure-up
conjure-up kubernetes
```
### Operational Guides
These are more in-depth guides for users choosing to run Kubernetes in production:
- [Installation](/docs/getting-started-guides/ubuntu/installation/)
- [Validation](/docs/getting-started-guides/ubuntu/validation/)
- [Backups](/docs/getting-started-guides/ubuntu/backups/)
- [Upgrades](/docs/getting-started-guides/ubuntu/upgrades/)
- [Scaling](/docs/getting-started-guides/ubuntu/scaling/)
- [Logging](/docs/getting-started-guides/ubuntu/logging/)
- [Monitoring](/docs/getting-started-guides/ubuntu/monitoring/)
- [Networking](/docs/getting-started-guides/ubuntu/networking/)
- [Security](/docs/getting-started-guides/ubuntu/security/)
- [Storage](/docs/getting-started-guides/ubuntu/storage/)
- [Troubleshooting](/docs/getting-started-guides/ubuntu/troubleshooting/)
- [Decommissioning](/docs/getting-started-guides/ubuntu/decommissioning/)
- [Operational Considerations](/docs/getting-started-guides/ubuntu/operational-considerations/)
- [Glossary](/docs/getting-started-guides/ubuntu/glossary/)
## Third-party Product Integrations
- [Rancher](/docs/getting-started-guides/ubuntu/rancher/)
## Developer Guides
- [Localhost using LXD](/docs/getting-started-guides/ubuntu/local/)
## Where to find us
We're normally following the following Slack channels:
- [kubernetes-users](https://kubernetes.slack.com/messages/kubernetes-users/)
- [kubernetes-novice](https://kubernetes.slack.com/messages/kubernetes-novice/)
- [sig-cluster-lifecycle](https://kubernetes.slack.com/messages/sig-cluster-lifecycle/)
- [sig-cluster-ops](https://kubernetes.slack.com/messages/sig-cluster-ops/)
- [sig-onprem](https://kubernetes.slack.com/messages/sig-onprem/)
and we monitor the Kubernetes mailing lists.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/backups.md
+144
View File
@@ -0,0 +1,144 @@
---
title: Backups
content_template: templates/task
---
{{% capture overview %}}
The state of a Kubernetes cluster is kept in the etcd datastore.
This page shows how to backup and restore the etcd shipped with
the Canonical Distribution of Kubernetes. Backing up application specific data,
normally stored in a persistent volume, is outside the scope of this
document.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{% /capture %}}
{{% capture steps %}}
## Snapshot etcd data
The `snapshot` action of the etcd charm allows the operator to snapshot
a running cluster's data for use in cloning,
backing up, or migrating to a new cluster.
juju run-action etcd/0 snapshot
This will create a snapshot in `/home/ubuntu/etcd-snapshots` by default.
## Restore etcd data
The etcd charm is capable of restoring its data from a cluster-data snapshot
via the `restore` action.
This comes with caveats and a very specific path to restore a cluster:
The cluster must be in a state of only having a single member. So it's best to
deploy a new cluster using the etcd charm, without adding any additional units.
```
juju deploy etcd new-etcd
```
The above code snippet will deploy a single unit of etcd, as 'new-etcd'
```
juju run-action etcd/0 restore target=/mnt/etcd-backups
```
Once the restore action has completed, evaluate the cluster health. If the unit
is healthy, you may resume scaling the application to meet your needs.
- **param** target: destination directory to save the existing data.
- **param** skip-backup: Don't backup any existing data.
## Migrating an etcd cluster
Using the above snapshot and restore operations, migrating etcd is a fairly easy task.
**Step 1:** Snapshot your existing cluster. This is encapsulated in the `snapshot`
action.
```
juju run-action etcd/0 snapshot
```
Results:
```
Action queued with id: b46d5d6f-5625-4320-8cda-b611c6ae580c
```
**Step 2:** Check the status of the action so you can grab the snapshot and verify
the sum. The `copy.cmd` result output is a copy/paste command for you to download
the exact snapshot that you just created.
Download the snapshot archive from the unit that created the snapshot and verify
the sha256 sum
```
juju show-action-output b46d5d6f-5625-4320-8cda-b611c6ae580c
```
Results:
```
results:
copy:
cmd: juju scp etcd/0:/home/ubuntu/etcd-snapshots/etcd-snapshot-2016-11-09-02.41.47.tar.gz
.
snapshot:
path: /home/ubuntu/etcd-snapshots/etcd-snapshot-2016-11-09-02.41.47.tar.gz
sha256: 1dea04627812397c51ee87e313433f3102f617a9cab1d1b79698323f6459953d
size: 68K
status: completed
```
Copy the snapshot to the local disk and then check the sha256sum.
```
juju scp etcd/0:/home/ubuntu/etcd-snapshots/etcd-snapshot-2016-11-09-02.41.47.tar.gz .
sha256sum etcd-snapshot-2016-11-09-02.41.47.tar.gz
```
**Step 3:** Deploy the new cluster leader, and attach the snapshot:
```
juju deploy etcd new-etcd --resource snapshot=./etcd-snapshot-2016-11-09-02.41.47.tar.gz
```
**Step 4:** Reinitialize the master with the data from the resource we just attached
in step 3.
```
juju run-action new-etcd/0 restore
```
{{% /capture %}}
{{% capture discussion %}}
## Known Limitations
#### Loss of PKI warning
If you destroy the leader - identified with the `*` text next to the unit number in status:
all TLS pki will be lost. No PKI migration occurs outside
of the units requesting and registering the certificates.
{{< caution >}}
**Caution:** Mismanaging this configuration will result in locking yourself
out of the cluster, and can potentially break existing deployments in very
strange ways relating to x509 validation of certificates, which affects both
servers and clients.
{{< /caution >}}
#### Restoring from snapshot on a scaled cluster
Restoring from a snapshot on a scaled cluster will result in a broken cluster.
Etcd performs clustering during unit turn-up, and state is stored in Etcd itself.
During the snapshot restore phase, a new cluster ID is initialized, and peers
are dropped from the snapshot state to enable snapshot restoration. Please
follow the migration instructions above in the restore action description.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/decommissioning.md
+80
View File
@@ -0,0 +1,80 @@
---
title: Decommissioning
content_template: templates/task
---
{{% capture overview %}}
This page shows you how to properly decommission a cluster.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{< warning >}}
**Warning:** By the time you've reached this step you should have backed up your workloads and pertinent data; this section is for the complete destruction of a cluster.
{{< /warning >}}
{{% /capture %}}
{{% capture steps %}}
## Destroy the Juju model
It is recommended to deploy individual Kubernetes clusters in their own models, so that there is a clean separation between environments. To remove a cluster first find out which model it's in with `juju list-models`. The controller reserves an `admin` model for itself. If you have chosen to not name your model it might show up as `default`.
```
$ juju list-models
Controller: aws-us-east-2
Model Cloud/Region Status Machines Cores Access Last connection
controller aws/us-east-2 available 1 2 admin just now
my-kubernetes-cluster* aws/us-east-2 available 12 22 admin 2 minutes ago
```
You can then destroy the model, which will in turn destroy the cluster inside of it:
juju destroy-model my-kubernetes-cluster
```
$ juju destroy-model my-kubernetes-cluster
WARNING! This command will destroy the "my-kubernetes-cluster" model.
This includes all machines, applications, data and other resources.
Continue [y/N]? y
Destroying model
Waiting on model to be removed, 12 machine(s), 10 application(s)...
Waiting on model to be removed, 12 machine(s), 9 application(s)...
Waiting on model to be removed, 12 machine(s), 8 application(s)...
Waiting on model to be removed, 12 machine(s), 7 application(s)...
Waiting on model to be removed, 12 machine(s)...
Waiting on model to be removed...
$
```
This will destroy and decommission all nodes. You can confirm all nodes are destroyed by running `juju status`.
If you're using a public cloud this will terminate the instances. If you're on bare metal using MAAS this will release the nodes, optionally wipe the disk, power off the machines, and return them to available pool of machines to deploy from.
## Cleaning up the Controller
If you're not using the controller for anything else, you will also need to remove the controller instance:
```
$ juju list-controllers
Use --refresh flag with this command to see the latest information.
Controller Model User Access Cloud/Region Models Machines HA Version
aws-us-east-2* - admin superuser aws/us-east-2 2 1 none 2.0.1
$ juju destroy-controller aws-us-east-2
WARNING! This command will destroy the "aws-us-east-2" controller.
This includes all machines, applications, data and other resources.
Continue? (y/N):y
Destroying controller
Waiting for hosted model resources to be reclaimed
All hosted models reclaimed, cleaning up controller machines
$
```
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/glossary.md
+24
View File
@@ -0,0 +1,24 @@
---
title: Glossary and Terminology
content_template: templates/concept
---
{{% capture overview %}}
This page explains some of the terminology used in deploying Kubernetes with Juju.
{{% /capture %}}
{{% capture body %}}
**controller** - The management node of a cloud environment. Typically you have one controller per cloud region, or more in HA environments. The controller is responsible for managing all subsequent models in a given environment. It contains the Juju API server and its underlying database.
**model** - A collection of charms and their relationships that define a deployment. This includes machines and units. A controller can host multiple models. It is recommended to separate Kubernetes clusters into individual models for management and isolation reasons.
**charm** - The definition of a service, including its metadata, dependencies with other services, required packages, and application management logic. It contains all the operational knowledge of deploying a Kubernetes cluster. Included charm examples are `kubernetes-core`, `easyrsa`, `flannel`, and `etcd`.
**unit** - A given instance of a service. These may or may not use up a whole machine, and may be colocated on the same machine. So for example you might have a `kubernetes-worker`, and `etcd`, and `easyrsa` units running on a single machine, but they are three distinct units of different services.
**machine** - A physical node, these can either be bare metal nodes, or virtual machines provided by a cloud.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/installation.md
+301
View File
@@ -0,0 +1,301 @@
---
reviewers:
- caesarxuchao
- erictune
title: Setting up Kubernetes with Juju
content_template: templates/task
---
{{% capture overview %}}
Ubuntu 16.04 introduced the [Canonical Distribution of Kubernetes](https://www.ubuntu.com/cloud/kubernetes), a pure upstream distribution of Kubernetes designed for production usage. This page shows you how to deploy a cluster.
{{% /capture %}}
{{% capture prerequisites %}}
- A working [Juju client](https://jujucharms.com/docs/2.3/reference-install); this does not have to be a Linux machine, it can also be Windows or OSX.
- A [supported cloud](#cloud-compatibility).
- Bare Metal deployments are supported via [MAAS](http://maas.io). Refer to the [MAAS documentation](http://maas.io/docs/) for configuration instructions.
- OpenStack deployments are currently only tested on Icehouse and newer.
- One of the following:
- Network access to the following domains
- *.jujucharms.com
- gcr.io
- github.com
- Access to an Ubuntu mirror (public or private)
- Offline deployment prepared with [these](https://github.com/juju-solutions/bundle-canonical-kubernetes/wiki/Running-CDK-in-a-restricted-environment) instructions.
{{% /capture %}}
{{% capture steps %}}
## Deployment overview
Out of the box the deployment comes with the following components on 9 machines:
- Kubernetes (automated deployment, operations, and scaling)
- Four node Kubernetes cluster with one master and three worker nodes.
- TLS used for communication between units for security.
- Flannel Software Defined Network (SDN) plugin
- A load balancer for HA kubernetes-master (Experimental)
- Optional Ingress Controller (on worker)
- Optional Dashboard addon (on master) including Heapster for cluster monitoring
- EasyRSA
- Performs the role of a certificate authority serving self signed certificates
to the requesting units of the cluster.
- ETCD (distributed key value store)
- Three unit cluster for reliability.
The Juju Kubernetes work is curated by the Big Software team at [Canonical Ltd](https://www.canonical.com/),
let us know how we are doing. If you find any problems please open an
[issue on our tracker](https://github.com/juju-solutions/bundle-canonical-kubernetes)
so we can find them.
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Amazon Web Services (AWS) | Juju | Ubuntu | flannel, calico* | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
OpenStack | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Microsoft Azure | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Google Compute Engine (GCE) | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Joyent | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Rackspace | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
VMware vSphere | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Bare Metal (MAAS) | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
## Installation options
You can launch a cluster in one of two ways: [conjure-up](#conjure-up) or [juju deploy](#juju-deploy). Conjure-up is just a convenience wrapper over juju and simplifies the installation. As such, it is the preferred method of install.
Deployment of the cluster is [supported on a wide variety of public clouds](#cloud-compatibility), private OpenStack clouds, or raw bare metal clusters. Bare metal deployments are supported via [MAAS](http://maas.io/).
## Conjure-up
To install Kubernetes with conjure-up, you need only to run the following commands and then follow the prompts:
```
sudo snap install conjure-up --classic
conjure-up kubernetes
```
## Juju deploy
### Configure Juju to use your cloud provider
After deciding which cloud to deploy to, follow the [cloud setup page](https://jujucharms.com/docs/devel/getting-started) to configure deploying to that cloud.
Load your [cloud credentials](https://jujucharms.com/docs/2.3/credentials) for each
cloud provider you would like to use.
In this example
```
juju add-credential aws
credential name: my_credentials
select auth-type [userpass, oauth, etc]: userpass
enter username: jorge
enter password: *******
```
You can also just auto load credentials for popular clouds with the `juju autoload-credentials` command, which will auto import your credentials from the default files and environment variables for each cloud.
Next we need to bootstrap a controller to manage the cluster. You need to define the cloud you want to bootstrap on, the region, and then any name for your controller node:
```
juju update-clouds # This command ensures all the latest regions are up to date on your client
juju bootstrap aws/us-east-2
```
or, another example, this time on Azure:
```
juju bootstrap azure/westus2
```
If you receive this error, it is likely that the default Azure VM size (Standard D1 v2 [1 vcpu, 3.5 GB memory]) is not available in the Azure location:
```
ERROR failed to bootstrap model: instance provisioning failed (Failed)
```
You will need a controller node for each cloud or region you are deploying to. See the [controller documentation](https://jujucharms.com/docs/2.3/controllers) for more information.
Note that each controller can host multiple Kubernetes clusters in a given cloud or region.
### Launch a Kubernetes cluster
The following command will deploy the initial 9-node starter cluster. The speed of execution is very dependent of the performance of the cloud you're deploying to:
```
juju deploy canonical-kubernetes
```
After this command executes the cloud will then launch instances and begin the deployment process.
## Monitor deployment
The `juju status` command provides information about each unit in the cluster. Use the `watch -c juju status --color` command to get a real-time view of the cluster as it deploys. When all the states are green and "Idle", the cluster is ready to be used:
juju status
Output:
```
Model Controller Cloud/Region Version SLA
conjure-canonical-kubern-f48 conjure-up-aws-650 aws/us-east-2 2.3.2 unsupported
App Version Status Scale Charm Store Rev OS Notes
easyrsa 3.0.1 active 1 easyrsa jujucharms 27 ubuntu
etcd 2.3.8 active 3 etcd jujucharms 63 ubuntu
flannel 0.9.1 active 4 flannel jujucharms 40 ubuntu
kubeapi-load-balancer 1.10.3 active 1 kubeapi-load-balancer jujucharms 43 ubuntu exposed
kubernetes-master 1.9.3 active 1 kubernetes-master jujucharms 13 ubuntu
kubernetes-worker 1.9.3 active 3 kubernetes-worker jujucharms 81 ubuntu exposed
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 3 18.219.190.99 Certificate Authority connected.
etcd/0 active idle 5 18.219.56.23 2379/tcp Healthy with 3 known peers
etcd/1* active idle 0 18.219.212.151 2379/tcp Healthy with 3 known peers
etcd/2 active idle 6 13.59.240.210 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 1 18.222.61.65 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 4 18.219.105.220 6443/tcp Kubernetes master running.
flannel/3 active idle 18.219.105.220 Flannel subnet 10.1.78.1/24
kubernetes-worker/0 active idle 2 18.219.221.98 80/tcp,443/tcp Kubernetes worker running.
flannel/1 active idle 18.219.221.98 Flannel subnet 10.1.38.1/24
kubernetes-worker/1* active idle 7 18.219.249.103 80/tcp,443/tcp Kubernetes worker running.
flannel/2 active idle 18.219.249.103 Flannel subnet 10.1.68.1/24
kubernetes-worker/2 active idle 8 52.15.89.16 80/tcp,443/tcp Kubernetes worker running.
flannel/0* active idle 52.15.89.16 Flannel subnet 10.1.73.1/24
Machine State DNS Inst id Series AZ Message
0 started 18.219.212.151 i-065eab4eabc691b25 xenial us-east-2a running
1 started 18.222.61.65 i-0b332955f028d6281 xenial us-east-2b running
2 started 18.219.221.98 i-0879ef1ed95b569bc xenial us-east-2a running
3 started 18.219.190.99 i-08a7b364fc008fc85 xenial us-east-2c running
4 started 18.219.105.220 i-0f92d3420b01085af xenial us-east-2a running
5 started 18.219.56.23 i-0271f6448cebae352 xenial us-east-2c running
6 started 13.59.240.210 i-0789ef5837e0669b3 xenial us-east-2b running
7 started 18.219.249.103 i-02f110b0ab042f7ac xenial us-east-2b running
8 started 52.15.89.16 i-086852bf1bee63d4e xenial us-east-2c running
Relation provider Requirer Interface Type Message
easyrsa:client etcd:certificates tls-certificates regular
easyrsa:client kubeapi-load-balancer:certificates tls-certificates regular
easyrsa:client kubernetes-master:certificates tls-certificates regular
easyrsa:client kubernetes-worker:certificates tls-certificates regular
etcd:cluster etcd:cluster etcd peer
etcd:db flannel:etcd etcd regular
etcd:db kubernetes-master:etcd etcd regular
kubeapi-load-balancer:loadbalancer kubernetes-master:loadbalancer public-address regular
kubeapi-load-balancer:website kubernetes-worker:kube-api-endpoint http regular
kubernetes-master:cni flannel:cni kubernetes-cni subordinate
kubernetes-master:kube-api-endpoint kubeapi-load-balancer:apiserver http regular
kubernetes-master:kube-control kubernetes-worker:kube-control kube-control regular
kubernetes-worker:cni flannel:cni kubernetes-cni subordinate
```
## Interacting with the cluster
After the cluster is deployed you may assume control over the cluster from any kubernetes-master, or kubernetes-worker node.
If you didn't use conjure-up, you will first need to download the credentials and client application to your local workstation:
Create the kubectl config directory.
```
mkdir -p ~/.kube
```
Copy the kubeconfig file to the default location.
```
juju scp kubernetes-master/0:/home/ubuntu/config ~/.kube/config
```
The next step is to install the kubectl client on your local machine. The recommended way to do this on Ubuntu is using the kubectl snap ([https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-snap-on-ubuntu](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-snap-on-ubuntu)).
The following command should be run on the machine you wish to use to control the kubernetes cluster:
```
sudo snap install kubectl --classic
```
This will install and deploy the kubectl binary. You may need to restart your terminal as your $PATH may have been updated.
Query the cluster:
kubectl cluster-info
Output:
```
Kubernetes master is running at https://52.15.104.227:443
Heapster is running at https://52.15.104.227:443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://52.15.104.227:443/api/v1/namespaces/kube-system/services/kube-dns/proxy
Grafana is running at https://52.15.104.227:443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
InfluxDB is running at https://52.15.104.227:443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
```
Congratulations, you've now set up a Kubernetes cluster!
## Scale up cluster
Want larger Kubernetes nodes? It is easy to request different sizes of cloud
resources from Juju by using **constraints**. You can increase the amount of
CPU or memory (RAM) in any of the systems requested by Juju. This allows you
to fine tune the Kubernetes cluster to fit your workload. Use flags on the
bootstrap command or as a separate `juju constraints` command. Look to the
[Juju documentation for machine](https://jujucharms.com/docs/2.3/charms-constraints)
details.
## Scale out cluster
Need more workers? We just add more units:
```shell
juju add-unit kubernetes-worker
```
Or multiple units at one time:
```shell
juju add-unit -n3 kubernetes-worker
```
You can also ask for specific instance types or other machine-specific constraints. See the [constraints documentation](https://jujucharms.com/docs/stable/reference-constraints) for more information. Here are some examples, note that generic constraints such as `cores` and `mem` are more portable between clouds. In this case we'll ask for a specific instance type from AWS:
```shell
juju set-constraints kubernetes-worker instance-type=c4.large
juju add-unit kubernetes-worker
```
You can also scale the etcd charm for more fault tolerant key/value storage:
```shell
juju add-unit -n3 etcd
```
It is strongly recommended to run an odd number of units for quorum.
## Tear down cluster
If you used conjure-up to create your cluster, you can tear it down with `conjure-down`. If you used juju directly, you can tear it down by destroying the Juju model or the controller. Use the `juju switch` command to get the current controller name:
```shell
juju switch
juju destroy-controller $controllername --destroy-all-models
```
This will shutdown and terminate all running instances on that cloud.
{{% /capture %}}
{{% capture discussion %}}
## More Info
The Ubuntu Kubernetes deployment uses open-source operations, or operations as code, known as charms. These charms are assembled from layers which keeps the code smaller and more focused on the operations of just Kubernetes and its components.
The Kubernetes layer and bundles can be found in the `kubernetes`
project on github.com:
- [Bundle location](https://git.k8s.io/kubernetes/cluster/juju/bundles)
- [Kubernetes charm layer location](https://git.k8s.io/kubernetes/cluster/juju/layers)
- [Canonical Kubernetes home](https://jujucharms.com/kubernetes)
- [Main issue tracker](https://github.com/juju-solutions/bundle-canonical-kubernetes)
Feature requests, bug reports, pull requests and feedback are appreciated.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/local.md
+71
View File
@@ -0,0 +1,71 @@
---
title: Local Kubernetes development with LXD
content_template: templates/task
---
{{% capture overview %}}
Running Kubernetes locally has obvious development advantages, such as lower cost and faster iteration than constantly deploying and tearing down clusters on a public cloud. Ideally, a Kubernetes developer can spawn all necessary nodes inside local containers and test new configurations as they are committed. This page will show you how to deploy a cluster to LXD containers on a local machine.
{{% /capture %}}
The purpose of using [LXD](https://linuxcontainers.org/lxd/) on a local machine is to emulate the same deployment that a user would use in a cloud or bare metal. Each node is treated as a machine, with the same characteristics as production. Each node is a separate container, which runs Docker containers and `kubectl` inside (see [Cluster Intro](/docs/tutorials/kubernetes-basics/cluster-intro/) for more info).
{{% capture prerequisites %}}
Install [conjure-up](http://conjure-up.io/), a tool for deploying big software.
Add the current user to the `lxd` user group.
```
sudo snap install conjure-up --classic
sudo usermod -a -G lxd $(whoami)
```
Note: If conjure-up asks you to "Setup an ipv6 subnet" with LXD, answer NO. ipv6 with Juju/LXD is currently unsupported.
{{% /capture %}}
{{% capture steps %}}
## Deploying Kubernetes
Start the deployment with:
conjure-up kubernetes
For this walkthrough we are going to create a new controller - select the `localhost` Cloud type:
![Select Cloud](/images/docs/ubuntu/00-select-cloud.png)
Deploy the applications:
![Deploy Applications](/images/docs/ubuntu/01-deploy.png)
Wait for Juju bootstrap to finish:
![Bootstrap](/images/docs/ubuntu/02-bootstrap.png)
Wait for our Applications to be fully deployed:
![Waiting](/images/docs/ubuntu/03-waiting.png)
Run the final post-processing steps to automatically configure your Kubernetes environment:
![Postprocessing](/images/docs/ubuntu/04-postprocessing.png)
Review the final summary screen:
![Final Summary](/images/docs/ubuntu/05-final-summary.png)
## Accessing the Cluster
You can access your Kubernetes cluster by running the following:
kubectl --kubeconfig=~/.kube/config
Or if you've already run this once it'll create a new config file as shown in the summary screen.
kubectl --kubeconfig=~/.kube/config.conjure-up
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/logging.md
+63
View File
@@ -0,0 +1,63 @@
---
title: Logging
content_template: templates/task
---
{{% capture overview %}}
This page will explain how logging works within a Juju deployed cluster.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{% /capture %}}
{{% capture steps %}}
## Agent Logging
The `juju debug-log` will show all of the consolidated logs of all the Juju agents running on each node of the cluster. This can be useful for finding out why a specific node hasn't deployed or is in an error state. These agent logs are located in `/var/lib/juju/agents` on every node.
See the [Juju documentation](https://jujucharms.com/docs/stable/troubleshooting-logs) for more information.
## Managing log verbosity
Log verbosity in Juju is set at the model level. You can adjust it at any time:
```
juju add-model k8s-development --config logging-config='<root>=DEBUG;unit=DEBUG'
```
and later on your k8s-production model
```
juju model-config -m k8s-production logging-config='<root>=ERROR;unit=ERROR'
```
In addition, the jujud daemon is started in debug mode by default on all controllers. To remove that behavior edit ```/var/lib/juju/init/jujud-machine-0/exec-start.sh``` on the controller node and comment the ```--debug``` section.
It then contains:
```
#!/usr/bin/env bash
# Set up logging.
touch '/var/log/juju/machine-0.log'
chown syslog:syslog '/var/log/juju/machine-0.log'
chmod 0600 '/var/log/juju/machine-0.log'
exec >> '/var/log/juju/machine-0.log'
exec 2>&1
# Run the script.
'/var/lib/juju/tools/machine-0/jujud' machine --data-dir '/var/lib/juju' --machine-id 0 # --debug
```
Then restart the service with:
```
sudo systemctl restart jujud-machine-0.service
```
See the [official documentation](https://jujucharms.com/docs/stable/models-config) for more information about logging and other model settings in Juju.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/monitoring.md
+145
View File
@@ -0,0 +1,145 @@
---
title: Monitoring
content_template: templates/task
---
{{% capture overview %}}
This page shows how to connect various logging solutions to a Juju deployed cluster.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{% /capture %}}
{{% capture steps %}}
## Connecting Datadog
Datadog is a SaaS offering which includes support for a range of integrations, including Kubernetes and ETCD. While the solution is SAAS/Commercial, they include a Free tier which is supported with the following method. To deploy a full Kubernetes stack with Datadog out of the box, do:
```
juju deploy canonical-kubernetes-datadog
```
### Installation of Datadog
To start, deploy the latest version Datadog from the Charm Store:
```
juju deploy datadog
```
Configure Datadog with your api-key, found in the [Datadog dashboard](). Replace `XXXX` with your API key.
```
juju configure datadog api-key=XXXX
```
Finally, attach `datadog` to all applications you wish to monitor. For example, kubernetes-master, kubernetes-worker, and etcd:
```
juju add-relation datadog kubernetes-worker
juju add-relation datadog kubernetes-master
juju add-relation datadog etcd
```
## Connecting Elastic stack
The Elastic stack, formally "ELK" stack, refers to Elastic Search and the suite of tools to facilitate log aggregation, monitoring, and dashboarding. To deploy a full Kubernetes stack with elastic out of the box, do:
```
juju deploy canonical-kubernetes-elastic
```
### New install of ElasticSearch
To start, deploy the latest version of ElasticSearch, Kibana, Filebeat, and Topbeat from the Charm Store:
This can be done in one command as:
```
juju deploy beats-core
```
However, if you wish to customize the deployment, or proceed manually, the following commands can be issued:
```
juju deploy elasticsearch
juju deploy kibana
juju deploy filebeat
juju deploy topbeat
juju add-relation elasticsearch kibana
juju add-relation elasticsearch topbeat
juju add-relation elasticsearch filebeat
```
Finally, connect filebeat and topbeat to all applications you wish to monitor. For example, kubernetes-master and kubernetes-worker:
```
juju add-relation kubernetes-master topbeat
juju add-relation kubernetes-master filebeat
juju add-relation kubernetes-worker topbeat
juju add-relation kubernetes-worker filebeat
```
### Existing ElasticSearch cluster
In the event an ElasticSearch cluster already exists, the following can be used to connect and leverage it instead of creating a new, separate, cluster. First deploy the two beats, filebeat and topbeat
```
juju deploy filebeat
juju deploy topbeat
```
Configure both filebeat and topbeat to connect to your ElasticSearch cluster, replacing `255.255.255.255` with the IP address in your setup.
```
juju configure filebeat elasticsearch=255.255.255.255
juju configure topbeat elasticsearch=255.255.255.255
```
Follow the above instructions on connect topbeat and filebeat to the applications you wish to monitor.
## Connecting Nagios
Nagios utilizes the Nagios Remote Plugin Executor protocol (NRPE protocol) as an agent on each node to derive machine level details of the health and applications.
### New install of Nagios
To start, deploy the latest version of the Nagios and NRPE charms from the store:
```
juju deploy nagios
juju deploy nrpe
```
Connect Nagios to NRPE
```
juju add-relation nagios nrpe
```
Finally, add NRPE to all applications deployed that you wish to monitor, for example `kubernetes-master`, `kubernetes-worker`, `etcd`, `easyrsa`, and `kubeapi-load-balancer`.
```
juju add-relation nrpe kubernetes-master
juju add-relation nrpe kubernetes-worker
juju add-relation nrpe etcd
juju add-relation nrpe easyrsa
juju add-relation nrpe kubeapi-load-balancer
```
### Existing install of Nagios
If you already have an existing Nagios installation, the `nrpe-external-master` charm can be used instead. This will allow you to supply configuration options that map your existing external Nagios installation to NRPE. Replace `255.255.255.255` with the IP address of the nagios instance.
```
juju deploy nrpe-external-master
juju configure nrpe-external-master nagios_master=255.255.255.255
```
Once configured, connect nrpe-external-master as outlined above.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/networking.md
+63
View File
@@ -0,0 +1,63 @@
---
title: Networking
content_template: templates/task
---
{{% capture overview %}}
Kubernetes supports the [Container Network Interface (CNI)](https://github.com/containernetworking/cni).
This is a network plugin architecture that allows you to use whatever
Kubernetes-friendly SDN you want. Currently this means support for Flannel and Canal.
This page shows how the various network portions of a cluster work and how to configure them.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{< note >}}
**Note:** Note that if you deploy a cluster via conjure-up or the CDK bundles, manually deploying CNI plugins is unnecessary.
{{< /note >}}
{{% /capture %}}
{{% capture steps %}}
The CNI charms are [subordinates](https://jujucharms.com/docs/stable/authors-subordinate-applications).
These charms will require a principal charm that implements the `kubernetes-cni` interface in order to properly deploy.
## Flannel
```
juju deploy flannel
juju add-relation flannel kubernetes-master
juju add-relation flannel kubernetes-worker
juju add-relation flannel etcd
```
## Canal
```
juju deploy canal
juju add-relation canal kubernetes-master
juju add-relation canal kubernetes-worker
juju add-relation canal etcd
```
### Configuration
**iface** The interface to configure the flannel or canal SDN binding. If this value is
empty string or undefined the code will attempt to find the default network
adapter similar to the following command:
```bash
$ route | grep default | head -n 1 | awk {'print $8'}
```
**cidr** The network range to configure the flannel or canal SDN to declare when
establishing networking setup with etcd. Ensure this network range is not active
on layers 2/3 you're deploying to, as it will cause collisions and odd behavior
if care is not taken when selecting a good CIDR range to assign to flannel. It's
also good practice to ensure you allot yourself a large enough IP range to support
how large your cluster will potentially scale. Class A IP ranges with /24 are
a good option.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/operational-considerations.md
+166
View File
@@ -0,0 +1,166 @@
---
title: Operational Considerations
content_template: templates/task
---
{{% capture overview %}}
This page gives recommendations and hints for people managing long lived clusters
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you understand the basics of Juju and Kubernetes.
{{% /capture %}}
{{% capture steps %}}
## Managing Juju
### Sizing your controller node
The Juju Controller:
* requires about 2 to 2.5GB RAM to operate.
* uses a MongoDB database as a storage backend for the configuration and state of the cluster. This database can grow significantly, and can also be the biggest consumer of CPU cycles on the instance
* aggregates and stores the log data of all services and units. Therefore, significant storage is needed for long lived models. If your intention is to keep the cluster running, make sure to provision at least 64GB for the logs.
To bootstrap a controller with constraints run the following command:
```
juju bootstrap --contraints "mem=8GB cpu-cores=4 root-disk=128G"
```
Juju will select the cheapest instance type matching your constraints on your target cloud. You can also use the ```instance-type``` constraint in conjunction with ```root-disk``` for strict control. For more information about the constraints available, refer to the [official documentation](https://jujucharms.com/docs/stable/reference-constraints)
Additional information about logging can be found in the [logging section](/docs/getting-started-guides/ubuntu/logging)
### SSHing into the Controller Node
By default, Juju will create a pair of SSH keys that it will use to automate the connection to units. They are stored on the client node in ```~/.local/share/juju/ssh/```
After deployment, Juju Controller is a "silent unit" that acts as a proxy between the client and the deployed applications. Nevertheless it can be useful to SSH into it.
First you need to understand your environment, especially if you run several Juju models and controllers. Run
```
juju list-models --all
$ juju models --all
Controller: k8s
Model Cloud/Region Status Machines Cores Access Last connection
admin/controller lxd/localhost available 1 - admin just now
admin/default lxd/localhost available 0 - admin 2017-01-23
admin/whale* lxd/localhost available 6 - admin 3 minutes ago
```
The first line ```Controller: k8s``` refers to how you bootstrapped.
Then you will see 2, 3 or more models listed below.
* admin/controller is the default model that hosts all controller units of juju
* admin/default is created by default as the primary model to host the user application, such as the Kubernetes cluster
* admin/whale is an additional model created if you use conjure-up as an overlay on top of Juju.
Now to ssh into a controller node, you first ask Juju to switch context, then ssh as you would with a normal unit:
```
juju switch controller
```
At this stage, you can query the controller model as well:
```
juju status
Model Controller Cloud/Region Version
controller k8s lxd/localhost 2.0.2
App Version Status Scale Charm Store Rev OS Notes
Unit Workload Agent Machine Public address Ports Message
Machine State DNS Inst id Series AZ
0 started 10.191.22.15 juju-2a5ed8-0 xenial
```
Note that if you had bootstrapped in HA mode, you would see several machines listed.
Now ssh-ing into the controller follows the same semantic as classic Juju commands:
```
$ juju ssh 0
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.8.0-34-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
0 packages can be updated.
0 updates are security updates.
Last login: Tue Jan 24 16:38:13 2017 from 10.191.22.1
ubuntu@juju-2a5ed8-0:~$
```
When you are done and want to come back to your initial model, exit the controller and
Then if you need to switch back to your cluster and ssh into the units, run
```
juju switch default
```
## Managing your Kubernetes cluster
### Running privileged containers
By default, juju-deployed clusters only allow running privileged containers on nodes with GPUs.
If you need privileged containers on other nodes, you have to enable the ```allow-privileged``` config on both
kubernetes-master and kubernetes-worker:
```
juju config kubernetes-master allow-privileged=true
juju config kubernetes-worker allow-privileged=true
```
### Private registry
With the registry action, you can easily create a private docker registry that
uses TLS authentication. However, note that a registry deployed with that action
is not HA; it uses storage tied to the kubernetes node where the pod is running.
Consequently, if the registry pod is migrated from one node to another, you will
need to re-publish the images.
#### Example usage
Create the relevant authentication files. Let's say you want user ```userA```
to authenticate with the password ```passwordA```. Then you'll do:
```
echo "userA:passwordA" > htpasswd-plain
htpasswd -c -b -B htpasswd userA passwordA
```
(the `htpasswd` program comes with the ```apache2-utils``` package)
Assuming that your registry will be reachable at ```myregistry.company.com```,
you already have your TLS key in the ```registry.key``` file, and your TLS
certificate (with ```myregistry.company.com``` as Common Name) in the
```registry.crt``` file, you would then run:
```
juju run-action kubernetes-worker/0 registry domain=myregistry.company.com htpasswd="$(base64 -w0 htpasswd)" htpasswd-plain="$(base64 -w0 htpasswd-plain)" tlscert="$(base64 -w0 registry.crt)" tlskey="$(base64 -w0 registry.key)" ingress=true
```
If you then decide that you want to delete the registry, just run:
```
juju run-action kubernetes-worker/0 registry delete=true ingress=true
```
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/rancher.md
+362
View File
@@ -0,0 +1,362 @@
---
title: Rancher Integration with Ubuntu Kubernetes
content_template: templates/task
---
{{% capture overview %}}
This repository explains how to deploy Rancher 2.0alpha on Canonical Kubernetes.
These steps are currently in alpha/testing phase and will most likely change.
The original documentation for this integration can be found at [https://github.com/CalvinHartwell/canonical-kubernetes-rancher/](https://github.com/CalvinHartwell/canonical-kubernetes-rancher/).
{{% /capture %}}
{{% capture prerequisites %}}
To use this guide, you must have a working kubernetes cluster that was deployed using Canonical's juju.
The full instructions for deploying Kubernetes with juju can be found at [https://kubernetes.io/docs/getting-started-guides/ubuntu/installation/](https://kubernetes.io/docs/getting-started-guides/ubuntu/installation/).
{{% /capture %}}
{{% capture steps %}}
## Deploying Rancher
To deploy Rancher, we just need to run the Rancher container workload on-top of Kubernetes. Rancher provides their containers through dockerhub ([https://hub.docker.com/r/rancher/server/tags/](https://hub.docker.com/r/rancher/server/tags/)) and can be downloaded freely from the internet.
If you're running your own registry or have an offline deployment, the container should be downloaded and pushed to a private registry before proceeding.
### Deploying Rancher with a nodeport
First create a yaml file which defines how to deploy Rancher on kubernetes. Save the file as cdk-rancher-nodeport.yaml:
```
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-admin
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: rancher
name: rancher
spec:
replicas: 1
selector:
matchLabels:
app: rancher
ima: pod
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: rancher
ima: pod
spec:
containers:
- image: rancher/server:preview
imagePullPolicy: Always
name: rancher
ports:
- containerPort: 80
- containerPort: 443
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: rancher
labels:
app: rancher
spec:
ports:
- port: 443
protocol: TCP
targetPort: 443
selector:
app: rancher
---
apiVersion: v1
kind: Service
metadata:
name: rancher-nodeport
spec:
type: NodePort
selector:
app: rancher
ports:
- name: rancher-api
protocol: TCP
nodePort: 30443
port: 443
targetPort: 443
```
Once kubectl is running and working, run the following command to deploy Rancher:
```
kubectl apply -f cdk-rancher-nodeport.yaml
```
Now we need to open this nodeport so we can access it. For that, we can use juju. We need to run the open-port command for each of the worker nodes in our cluster. Inside the cdk-rancher-nodeport.yaml file, the nodeport has been set to 30443. Below shows how to open the port on each of the worker nodes:
```
# repeat this for each kubernetes worker in the cluster.
juju run --unit kubernetes-worker/0 "open-port 30443"
juju run --unit kubernetes-worker/1 "open-port 30443"
juju run --unit kubernetes-worker/2 "open-port 30443"
```
Rancher can now be accessed on this port through a worker IP or DNS entries if you have created them. It is generally recommended that you create a DNS entry for each of the worker nodes in your cluster. For example, if you have three worker nodes and you own the domain example.com, you could create three A records, one for each worker in the cluster.
As creating DNS entries is outside of the scope of this document, we will use the freely available xip.io service which can return A records for an IP address which is part of the domain name. For example, if you have the domain rancher.35.178.130.245.xip.io, the xip.io service will automatically return the IP address 35.178.130.245 as an A record which is useful for testing purposes. For your deployment, the IP address 35.178.130.245 should be replaced with one of your worker IP address, which can be found using Juju or AWS:
```
calvinh@ubuntu-ws:~/Source/cdk-rancher$ juju status
# ... output omitted.
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 35.178.118.232 Certificate Authority connected.
etcd/0* active idle 1 35.178.49.31 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 35.177.99.171 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 35.178.125.161 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 4 35.178.37.87 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 5 35.177.239.237 6443/tcp Kubernetes master running.
flannel/0* active idle 35.177.239.237 Flannel subnet 10.1.27.1/24
kubernetes-worker/0* active idle 6 35.178.130.245 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/2 active idle 35.178.130.245 Flannel subnet 10.1.82.1/24
kubernetes-worker/1 active idle 7 35.178.121.29 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/3 active idle 35.178.121.29 Flannel subnet 10.1.66.1/24
kubernetes-worker/2 active idle 8 35.177.144.76 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/1 active idle 35.177.144.76
# Note the IP addresses for the kubernetes-workers in the example above. You should pick one of the public addresses.
```
Try opening up Rancher in your browser using the nodeport and the domain name or ip address:
```
# replace the IP address with one of your Kubernetes worker, find this from juju status command.
wget https://35.178.130.245.xip.io:30443 --no-check-certificate
# this should also work
wget https://35.178.130.245:30443 --no-check-certificate
```
If you need to make any changes to the kubernetes configuration file, edit the yaml file and then just use apply again:
```
kubectl apply -f cdk-rancher-nodeport.yaml
```
### Deploying Rancher with an ingress rule
It is also possible to deploy Rancher using an ingress rule. This has the added benefit of not requiring additional ports to be opened up on the Kubernetes cluster. First create a yaml file to describe the deployment called cdk-rancher-ingress.yaml which should contain the following:
```
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-admin
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: rancher
name: rancher
spec:
replicas: 1
selector:
matchLabels:
app: rancher
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: rancher
spec:
containers:
- image: rancher/server:preview
imagePullPolicy: Always
name: rancher
ports:
- containerPort: 443
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: rancher
labels:
app: rancher
spec:
ports:
- port: 443
targetPort: 443
protocol: TCP
selector:
app: rancher
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: rancher
annotations:
kubernetes.io/tls-acme: "true"
ingress.kubernetes.io/secure-backends: "true"
spec:
tls:
- hosts:
- rancher.34.244.118.135.xip.io
rules:
- host: rancher.34.244.118.135.xip.io
http:
paths:
- path: /
backend:
serviceName: rancher
servicePort: 443
```
It is generally recommended that you create a DNS entry for each of the worker nodes in your cluster. For example, if you have three worker nodes and you own the domain example.com, you could create three A records, one for each worker in the cluster.
As creating DNS entries is outside of the scope of this tutorial, we will use the freely available xip.io service which can return A records for an IP address which is part of the domain name. For example, if you have the domain rancher.35.178.130.245.xip.io, the xip.io service will automatically return the IP address 35.178.130.245 as an A record which is useful for testing purposes.
For your deployment, the IP address 35.178.130.245 should be replaced with one of your worker IP address, which can be found using Juju or AWS:
```
calvinh@ubuntu-ws:~/Source/cdk-rancher$ juju status
# ... output omitted.
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 35.178.118.232 Certificate Authority connected.
etcd/0* active idle 1 35.178.49.31 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 35.177.99.171 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 35.178.125.161 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 4 35.178.37.87 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 5 35.177.239.237 6443/tcp Kubernetes master running.
flannel/0* active idle 35.177.239.237 Flannel subnet 10.1.27.1/24
kubernetes-worker/0* active idle 6 35.178.130.245 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/2 active idle 35.178.130.245 Flannel subnet 10.1.82.1/24
kubernetes-worker/1 active idle 7 35.178.121.29 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/3 active idle 35.178.121.29 Flannel subnet 10.1.66.1/24
kubernetes-worker/2 active idle 8 35.177.144.76 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/1 active idle 35.177.144.76
# Note the IP addresses for the kubernetes-workers in the example above. You should pick one of the public addresses.
```
Looking at the output from the juju status above, the Public Address (35.178.130.245) can be used to create a xip.io DNS entry (rancher.35.178.130.245.xip.io) which should be placed into the cdk-rancher-ingress.yaml file. You could also create your own DNS entry as long as it resolves to each of the worker nodes or one of them it will work fine:
```
# The xip.io domain should appear in two places in the file, change both entries.
cat cdk-rancher-ingress.yaml | grep xip.io
- host: rancher.35.178.130.245.xip.io
```
Once you've edited the ingress rule to reflect your DNS entries, run the kubectl apply -f cdk-rancher-ingress.yaml to deploy Kubernetes:
```
kubectl apply -f cdk-rancher-ingress.yaml
```
Rancher can now be accessed on the regular 443 through a worker IP or DNS entries if you have created them. Try opening it up in your browser:
```
# replace the IP address with one of your Kubernetes worker, find this from juju status command.
wget https://35.178.130.245.xip.io:443 --no-check-certificate
```
If you need to make any changes to the kubernetes configuration file, edit the yaml file and then just use apply again:
```
kubectl apply -f cdk-rancher-ingress.yaml
```
### Removing Rancher
You can remove Rancher from your cluster using kubectl. Deleting constructs in Kubernetes is as simple as creating them:
```
# If you used the nodeport example change the yaml filename if you used the ingress example.
kubectl delete -f cdk-rancher-nodeport.yaml
```
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/scaling.md
+88
View File
@@ -0,0 +1,88 @@
---
title: Scaling
content_template: templates/task
---
{{% capture overview %}}
This page shows how to horizontally scale master and worker nodes on a cluster.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
Any of the applications can be scaled out post-deployment. The charms
update the status messages with progress, so it is recommended to run.
```
watch -c juju status --color
```
{{% /capture %}}
{{% capture steps %}}
## Kubernetes masters
The provided Kubernetes master nodes act as a control plane for the cluster.
The deployment has been designed so that these nodes can be scaled independently
of worker nodes to allow for more operational flexibility.
To scale a master node up, simply execute:
juju add-unit kubernetes-master
This will add another master node to the control plane.
See the [building high-availability clusters](/docs/admin/high-availability)
section of the documentation for more information.
## Kubernetes workers
The kubernetes-worker nodes are the load-bearing units of a Kubernetes cluster.
By default pods are automatically spread throughout the kubernetes-worker units
that you have deployed.
To add more kubernetes-worker units to the cluster:
```
juju add-unit kubernetes-worker
```
or specify machine constraints to create larger nodes:
```
juju set-constraints kubernetes-worker "cpu-cores=8 mem=32G"
juju add-unit kubernetes-worker
```
Refer to the
[machine constraints documentation](https://jujucharms.com/docs/stable/charms-constraints)
for other machine constraints that might be useful for the kubernetes-worker units.
## etcd
Etcd is used as a key-value store for the Kubernetes cluster. The bundle
defaults to one instance in this cluster.
For quorum reasons it is recommended to keep an odd number of etcd nodes. 3, 5, 7, and 9 nodes are the recommended amount of nodes, depending on your cluster size. The CoreOS etcd documentation has a chart for the
[optimal cluster size](https://coreos.com/etcd/docs/latest/admin_guide.html#optimal-cluster-size)
to determine fault tolerance.
To add an etcd unit:
```
juju add-unit etcd
```
Shrinking of an etcd cluster after growth is not recommended.
## Juju controller
A single node is responsible for coordinating with all the Juju agents
on each machine that manage Kubernetes; it is called the controller node.
For production deployments it is recommended to enable HA of the controller node:
juju enable-ha
Enabling HA results in 3 controller nodes, this should be sufficient for most use cases. 5 and 7 controller nodes are also supported for extra large deployments.
Refer to the [Juju HA controller documentation](https://jujucharms.com/docs/2.2/controllers-ha) for more information.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/security.md
+36
View File
@@ -0,0 +1,36 @@
---
title: Security Considerations
content_template: templates/task
---
{{% capture overview %}}
By default all connections between every provided node are secured via TLS by easyrsa, including the etcd cluster.
This page explains the security considerations of a deployed cluster and production recommendations.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{% /capture %}}
{{% capture steps %}}
## Implementation
The TLS and easyrsa implementations use the following [layers](https://jujucharms.com/docs/2.2/developer-layers).
[layer-tls-client](https://github.com/juju-solutions/layer-tls-client)
[layer-easyrsa](https://github.com/juju-solutions/layer-easyrsa)
## Limiting ssh access
By default the administrator can ssh to any deployed node in a cluster. You can mass disable ssh access to the cluster nodes by issuing the following command.
juju model-config proxy-ssh=true
Note: The Juju controller node will still have open ssh access in your cloud, and will be used as a jump host in this case.
Refer to the [model management](https://jujucharms.com/docs/2.2/models) page in the Juju documentation for instructions on how to manage ssh keys.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/storage.md
+92
View File
@@ -0,0 +1,92 @@
---
title: Storage
content_template: templates/task
---
{{% capture overview %}}
This page explains how to install and configure persistent storage on a cluster.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{% /capture %}}
{{% capture steps %}}
## Ceph Persistent Volumes
The Canonical Distribution of Kubernetes allows you to connect with durable
storage devices such as [Ceph](http://ceph.com). When paired with the
[Juju Storage](https://jujucharms.com/docs/2.0/charms-storage) feature you
can add durable storage easily and across clouds.
Deploy a minimum of three ceph-mon and three ceph-osd units.
```
juju deploy cs:ceph-mon -n 3
juju deploy cs:ceph-osd -n 3
```
Relate the units together:
```
juju add-relation ceph-mon ceph-osd
```
List the storage pools available to Juju for your cloud:
juju storage-pools
Output:
```
Name Provider Attrs
ebs ebs
ebs-ssd ebs volume-type=ssd
loop loop
rootfs rootfs
tmpfs tmpfs
```
> **Note**: This listing is for the Amazon Web Services public cloud.
> Different clouds may have different pool names.
Add a storage pool to the ceph-osd charm by NAME,SIZE,COUNT:
```
juju add-storage ceph-osd/0 osd-devices=ebs,10G,1
juju add-storage ceph-osd/1 osd-devices=ebs,10G,1
juju add-storage ceph-osd/2 osd-devices=ebs,10G,1
```
Next relate the storage cluster with the Kubernetes cluster:
```
juju add-relation kubernetes-master ceph-mon
```
We are now ready to enlist
[Persistent Volumes](/docs/concepts/storage/persistent-volumes/)
in Kubernetes which our workloads can consume via Persistent Volume (PV) claims.
```
juju run-action kubernetes-master/0 create-rbd-pv name=test size=50
```
This example created a "test" Rados Block Device (rbd) in the size of 50 MB.
Use watch on your Kubernetes cluster like the following, you should see the PV
become enlisted and be marked as available:
watch kubectl get pv
Output:
```
NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
test 50M RWO Available 10s
```
To consume these Persistent Volumes, your pods will need an associated
Persistent Volume Claim with them, and is outside the scope of this README. See the
[Persistent Volumes](/docs/concepts/storage/persistent-volumes/)
documentation for more information.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/troubleshooting.md
+181
View File
@@ -0,0 +1,181 @@
---
title: Troubleshooting
content_template: templates/task
---
{{% capture overview %}}
This document with highlighting how to troubleshoot the deployment of a Kubernetes cluster, it will not cover debugging of workloads inside Kubernetes.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{% /capture %}}
{{% capture steps %}}
## Understanding Cluster Status
Using `juju status` can give you some insight as to what's happening in a cluster:
```
Model Controller Cloud/Region Version
kubes work-multi aws/us-east-2 2.0.2.1
App Version Status Scale Charm Store Rev OS Notes
easyrsa 3.0.1 active 1 easyrsa jujucharms 3 ubuntu
etcd 2.2.5 active 1 etcd jujucharms 17 ubuntu
flannel 0.6.1 active 2 flannel jujucharms 6 ubuntu
kubernetes-master 1.4.5 active 1 kubernetes-master jujucharms 8 ubuntu exposed
kubernetes-worker 1.4.5 active 1 kubernetes-worker jujucharms 11 ubuntu exposed
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0/lxd/0 10.0.0.55 Certificate Authority connected.
etcd/0* active idle 0 52.15.47.228 2379/tcp Healthy with 1 known peers.
kubernetes-master/0* active idle 0 52.15.47.228 6443/tcp Kubernetes master services ready.
flannel/1 active idle 52.15.47.228 Flannel subnet 10.1.75.1/24
kubernetes-worker/0* active idle 1 52.15.177.233 80/tcp,443/tcp Kubernetes worker running.
flannel/0* active idle 52.15.177.233 Flannel subnet 10.1.63.1/24
Machine State DNS Inst id Series AZ
0 started 52.15.47.228 i-0bb211a18be691473 xenial us-east-2a
0/lxd/0 started 10.0.0.55 juju-153b74-0-lxd-0 xenial
1 started 52.15.177.233 i-0502d7de733be31bb xenial us-east-2b
```
In this example we can glean some information. The `Workload` column will show the status of a given service. The `Message` section will show you the health of a given service in the cluster. During deployment and maintenance these workload statuses will update to reflect what a given node is doing. For example the workload my say `maintenance` while message will describe this maintenance as `Installing docker`.
During normal operation the Workload should read `active`, the Agent column (which reflects what the Juju agent is doing) should read `idle`, and the messages will either say `Ready` or another descriptive term. `juju status --color` will also return all green results when a cluster's deployment is healthy.
Status can become unwieldy for large clusters, it is then recommended to check status on individual services, for example to check the status on the workers only:
juju status kubernetes-worker
or just on the etcd cluster:
juju status etcd
Errors will have an obvious message, and will return a red result when used with `juju status --color`. Nodes that come up in this manner should be investigated.
## SSHing to units
You can ssh to individual units easily with the following convention, `juju ssh <servicename>/<unit#>`:
juju ssh kubernetes-worker/3
Will automatically ssh you to the 3rd worker unit.
juju ssh easyrsa/0
This will automatically ssh you to the easyrsa unit.
## Collecting debug information
Sometimes it is useful to collect all the information from a cluster to share with a developer to identify problems. This is best accomplished with [CDK Field Agent](https://github.com/juju-solutions/cdk-field-agent).
Download and execute the collect.py script from [CDK Field Agent](https://github.com/juju-solutions/cdk-field-agent) on a box that has a Juju client configured with the current controller and model pointing at the CDK deployment of interest.
Running the script will generate a tarball of system information and includes basic information such as systemctl status, Juju logs, charm unit data, etc. Additional application-specific information may be included as well.
## Common Problems
### Load Balancer interfering with Helm
This section assumes you have a working deployment of Kubernetes via Juju using a Load Balancer for the API, and that you are using Helm to deploy charts.
To deploy Helm you will have run:
```
helm init
$HELM_HOME has been configured at /home/ubuntu/.helm
Tiller (the helm server side component) has been installed into your Kubernetes Cluster.
Happy Helming!
```
Then when using helm you may see one of the following errors:
* Helm doesn't get the version from the Tiller server
```
helm version
Client: &version.Version{SemVer:"v2.1.3", GitCommit:"5cbc48fb305ca4bf68c26eb8d2a7eb363227e973", GitTreeState:"clean"}
Error: cannot connect to Tiller
```
* Helm cannot install your chart
```
helm install <chart> --debug
Error: forwarding ports: error upgrading connection: Upgrade request required
```
This is caused by the API load balancer not forwarding ports in the context of the helm client-server relationship. To deploy using helm, you will need to follow these steps:
1. Expose the Kubernetes Master service
```
juju expose kubernetes-master
```
1. Identify the public IP address of one of your masters
```
juju status kubernetes-master
Model Controller Cloud/Region Version
production k8s-admin aws/us-east-1 2.0.0
App Version Status Scale Charm Store Rev OS Notes
flannel 0.6.1 active 1 flannel jujucharms 7 ubuntu
kubernetes-master 1.5.1 active 1 kubernetes-master jujucharms 10 ubuntu exposed
Unit Workload Agent Machine Public address Ports Message
kubernetes-master/0* active idle 5 54.210.100.102 6443/tcp Kubernetes master running.
flannel/0 active idle 54.210.100.102 Flannel subnet 10.1.50.1/24
Machine State DNS Inst id Series AZ
5 started 54.210.100.102 i-002b7150639eb183b xenial us-east-1a
Relation Provides Consumes Type
certificates easyrsa kubernetes-master regular
etcd etcd flannel regular
etcd etcd kubernetes-master regular
cni flannel kubernetes-master regular
loadbalancer kubeapi-load-balancer kubernetes-master regular
cni kubernetes-master flannel subordinate
cluster-dns kubernetes-master kubernetes-worker regular
cni kubernetes-worker flannel subordinate
```
In this context the public IP address is 54.210.100.102.
If you want to access this data programmatically you can use the JSON output:
```
juju show-status kubernetes-master --format json | jq --raw-output '.applications."kubernetes-master".units | keys[]'
54.210.100.102
```
1. Update the kubeconfig file
Identify the kubeconfig file or section used for this cluster, and edit the server configuration.
By default, it will look like ```https://54.213.123.123:443```. Replace it with the Kubernetes Master endpoint ```https://54.210.100.102:6443``` and save.
Note that the default port used by CDK for the Kubernetes Master API is 6443 while the port exposed by the load balancer is 443.
1. Start helm again!
```
helm install <chart> --debug
Created tunnel using local port: '36749'
SERVER: "localhost:36749"
CHART PATH: /home/ubuntu/.helm/<chart>
NAME: <chart>
...
...
```
## Logging and monitoring
By default there is no log aggregation of the Kubernetes nodes, each node logs locally. Please read over the [logging](https://kubernetes.io/docs/getting-started-guides/ubuntu/logging/) page for more information.
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/upgrades.md
+164
View File
@@ -0,0 +1,164 @@
---
title: Upgrades
content_template: templates/task
---
{{% capture overview %}}
This page will outline how to manage and execute a Kubernetes upgrade.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working deployed cluster.
{{< warning >}}
**Warning:** You should always back up all your data before attempting an upgrade.
Don't forget to include the workload inside your cluster!
Refer to the [backup documentation](/docs/getting-started-guides/ubuntu/backups).
{{< /warning >}}
{{% /capture %}}
{{% capture steps %}}
## Patch kubernetes upgrades for example 1.9.0 -> 1.9.1
Clusters are transparently upgraded to the latest Kubernetes patch release.
To be clear, a cluster deployed using the 1.9/stable channel
will transparently receive unattended upgrades for the 1.9.X Kubernetes
releases.
The upgrade causes no disruption to the operation of the cluster and requires
no intervention from a cluster administrator.
Each patch release is evaluated by the
Canonical Kubernetes Distribution team.
Once a patch release passes internal testing and is deemed safe for upgrade,
it is packaged in snap format and pushed to the stable channel.
## Upgrading a minor Kubernetes release for example 1.8.1 -> 1.9.0
The Kubernetes charms follow the Kubernetes releases. Please consult
your support plan on the upgrade frequency. Important operational considerations
and changes in behaviour will always be documented in the release notes.
### Upgrade etcd
Backing up etcd requires an export and snapshot, refer to the
[backup documentation](/docs/getting-started-guides/ubuntu/backups) to create a snapshot.
After the snapshot, upgrade the etcd service with:
juju upgrade-charm etcd
This will handle upgrades between minor versions of etcd. Instructions on how to upgrade from 2.x to 3.x can be found [here](https://github.com/juju-solutions/bundle-canonical-kubernetes/wiki/Etcd-2.3-to-3.x-upgrade) in the juju-solutions wiki.
### Upgrade kubeapi-load-balancer
The Kubernetes Charms are generally all updated and released at the same time. A core part of a cluster on Ubuntu is the kubeapi-load-balancer component. Incorrect or missing changes there can have an effect on API availability and access controls. To ensure API service continuity for the master and workers when they are updated, this upgrade needs to precede them.
To upgrade the charm run:
juju upgrade-charm kubeapi-load-balancer
### Upgrade Kubernetes
The Kubernetes Charms use snap channels to drive payloads.
The channels are defined by `X.Y/channel` where `X.Y` is the `major.minor` release
of Kubernetes (for example 1.9) and `channel` is one of the four following channels:
| Channel name | Description |
| ------------------- | ------------ |
| stable | The latest stable released patch version of Kubernetes |
| candidate | Release candidate releases of Kubernetes |
| beta | Latest alpha or beta of Kubernetes for that minor release |
| edge | Nightly builds of that minor release of Kubernetes |
If a release isn't available, the next highest channel is used.
For example, 1.9/beta will load `/candidate` or `/stable` depending on availability of release.
Development versions of Kubernetes are available in the edge channel for each minor release.
There is no guarantee that edge snaps will work with the current charms.
### Master Upgrades
First you need to upgrade the masters:
juju upgrade-charm kubernetes-master
{{< note >}}
**Note:** Always upgrade the masters before the workers.
{{< /note >}}
Once the latest charm is deployed, the channel for Kubernetes can be selected by issuing the following:
juju config kubernetes-master channel=1.x/stable
Where `x` is the minor version of Kubernetes. For example, `1.9/stable`. See above for Channel definitions.
Once you've configured kubernetes-master with the appropriate channel, run the upgrade action on each master:
juju run-action kubernetes-master/0 upgrade
juju run-action kubernetes-master/1 upgrade
...
### Worker Upgrades
Two methods of upgrading workers are supported.
[Blue/Green Deployment](http://martinfowler.com/bliki/BlueGreenDeployment.html)
and upgrade-in-place. Both methods are provided for operational flexibility and both
are supported and tested. Blue/Green will require more hardware up front than in-place,
but is a safer upgrade route.
#### Blue/green worker upgrade
Given a deployment where the workers are named kubernetes-alpha.
Deploy new workers:
juju deploy kubernetes-beta
Pause the old workers so your workload migrates:
juju run-action kubernetes-alpha/# pause
Verify old workloads have migrated with:
kubectl get pod -o wide
Tear down old workers with:
juju remove-application kubernetes-alpha
#### In place worker upgrade
juju upgrade-charm kubernetes-worker
juju config kubernetes-worker channel=1.x/stable
Where `x` is the minor version of Kubernetes. For example, `1.9/stable`.
See above for Channel definitions. Once you've configured kubernetes-worker with the appropriate channel,
run the upgrade action on each worker:
juju run-action kubernetes-worker/0 upgrade
juju run-action kubernetes-worker/1 upgrade
...
### Verify upgrade
`kubectl version` should return the newer version.
It is recommended to rerun a [cluster validation](/docs/getting-started-guides/ubuntu/validation)
to ensure that the cluster upgrade has successfully completed.
### Upgrade Flannel
Upgrading flannel can be done at any time, it is independent of Kubernetes upgrades.
Be advised that networking is interrupted during the upgrade. You can initiate a flannel upgrade with:
juju upgrade-charm flannel
### Upgrade easyrsa
Upgrading easyrsa can be done at any time, it is independent of Kubernetes upgrades.
Upgrading easyrsa should result in zero downtime as it is not a running service:
juju upgrade-charm easyrsa
{{% /capture %}}
content/en/docs/getting-started-guides/ubuntu/validation.md
+169
View File
@@ -0,0 +1,169 @@
---
title: Validation - End-to-end Testing
content_template: templates/task
---
{{% capture overview %}}
This page will outline how to ensure that a Juju-deployed Kubernetes
cluster has stood up correctly and is ready to accept workloads.
{{% /capture %}}
{{% capture prerequisites %}}
This page assumes you have a working Juju deployed cluster.
{{% /capture %}}
{{% capture steps %}}
## End-to-end testing
End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end
behavior of the system, and is the last signal to ensure end user operations
match developer specifications. Although unit and integration tests provide a
good signal, in a distributed system like Kubernetes it is not uncommon that a
minor change may pass all unit and integration tests, but cause unforeseen
changes at the system level.
The primary objectives of the e2e tests are to ensure a consistent and reliable
behavior of the kubernetes code base, and to catch hard-to-test bugs before
users do, when unit and integration tests are insufficient.
End-to-end tests will pass on a properly running CDK cluster outside of bugs in the tests.
### Deploy kubernetes-e2e charm
To deploy the end-to-end test suite, you need to relate the `kubernetes-e2e` charm
to your existing kubernetes-master nodes and easyrsa:
```
juju deploy cs:~containers/kubernetes-e2e
juju add-relation kubernetes-e2e easyrsa
juju add-relation kubernetes-e2e:kubernetes-master kubernetes-master:kube-api-endpoint
juju add-relation kubernetes-e2e:kube-control kubernetes-master:kube-control
```
Once the relations have settled, you can do `juju status` until the workload status results in
`Ready to test.` - you may then kick off an end to end validation test.
### Running the e2e test
The e2e test is encapsulated as an action to ensure consistent runs of the
end to end test. The defaults are sensible for most deployments.
juju run-action kubernetes-e2e/0 test
### Tuning the e2e test
The e2e test is configurable. By default it will focus on or skip the declared
conformance tests in a cloud agnostic way. Default behaviors are configurable.
This allows the operator to test only a subset of the conformance tests, or to
test more behaviors not enabled by default. You can see all tunable options on
the charm by inspecting the schema output of the actions:
juju actions kubernetes-e2e --format=yaml --schema
Output:
```
test:
description: Run end-to-end validation test suite
properties:
focus:
default: \[Conformance\]
description: Regex focus for executing the test
type: string
skip:
default: \[Flaky\]
description: Regex of tests to skip
type: string
timeout:
default: 30000
description: Timeout in nanoseconds
type: integer
title: test
type: object
```
As an example, you can run a more limited set of tests for rapid validation of
a deployed cluster. The following example will skip the `Flaky`, `Slow`, and
`Feature` labeled tests:
juju run-action kubernetes-e2e/0 test skip='\[(Flaky|Slow|Feature:.*)\]'
{{< note >}}
**Note:** the escaping of the regex due to how bash handles brackets.
{{< /note >}}
To see the different types of tests the Kubernetes end-to-end charm has access
to, we encourage you to see the [upstream documentation on the different types
of tests](https://git.k8s.io/community/contributors/devel/e2e-tests.md#kinds-of-tests),
and to thoroughly understand what subsets of the tests you are running.
### More information on end-to-end testing
Along with the above descriptions, end-to-end testing is a much larger subject
than this readme can encapsulate. There is far more information in the
[end-to-end testing guide](https://git.k8s.io/community/contributors/devel/e2e-tests.md).
### Evaluating end-to-end results
It is not enough to just simply run the test. Result output is stored in two
places. The raw output of the e2e run is available in the `juju show-action-output`
command, as well as a flat file on disk on the `kubernetes-e2e` unit that
executed the test.
{{< note >}}
**Note:** The results will only be available once the action has
completed the test run. End-to-end testing can be quite time consuming, often
taking more than 1 hour, depending on configuration.
{{< /note >}}
##### Accessing the results in a flat file
Here's how to copy the output out as a file:
juju run-action kubernetes-e2e/0 test
Output:
Action queued with id: 4ceed33a-d96d-465a-8f31-20d63442e51b
Copy output to your local machine:
juju scp kubernetes-e2e/0:4ceed33a-d96d-465a-8f31-20d63442e51b.log .
##### Action result output
Or you can just show the output inline:
juju run-action kubernetes-e2e/0 test
Output:
Action queued with id: 4ceed33a-d96d-465a-8f31-20d63442e51b
Show the results in your terminal:
juju show-action-output 4ceed33a-d96d-465a-8f31-20d63442e51b
### Known issues
The e2e test suite assumes egress network access. It will pull container
images from `gcr.io`. You will need to have this registry unblocked in your
firewall to successfully run e2e test results. Or you may use the exposed
proxy settings [properly configured](https://github.com/juju-solutions/bundle-canonical-kubernetes#proxy-configuration)
on the kubernetes-worker units.
## Upgrading the e2e tests
The e2e tests are always expanding; you can see if there's an upgrade
available by running `juju status kubernetes-e2e`.
When an upgrade is available, upgrade your deployment:
juju upgrade-charm kubernetes-e2e
{{% /capture %}}
content/en/docs/getting-started-guides/windows/OVN_OVS_Windows_Installer.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

content/en/docs/getting-started-guides/windows/UpstreamRouting.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 87 KiB

content/en/docs/getting-started-guides/windows/_index.md
+509
View File
@@ -0,0 +1,509 @@
---
title: Using Windows Server Containers in Kubernetes
toc_hide: true
---
{{< note >}}
**Note:** These instructions were recently updated based on Windows Server platform enhancements and the Kubernetes v1.9 release
{{< /note >}}
Kubernetes version 1.5 introduced Alpha support for Windows Server Containers based on the Windows Server 2016 operating system. With the release of Windows Server version 1709 and using Kubernetes v1.9 users are able to deploy a Kubernetes cluster either on-premises or in a private/public cloud using a number of different network topologies and CNI plugins. Some key feature improvements for Windows Server Containers on Kubernetes include:
- Improved support for pods! Shared network namespace (compartment) with multiple Windows Server containers (shared kernel)
- Reduced network complexity by using a single network endpoint per pod
- Kernel-Based load-balancing using the Virtual Filtering Platform (VFP) Hyper-v Switch Extension (analogous to Linux iptables)
- Container Runtime Interface (CRI) pod and node level statistics
- Support for kubeadm commands to add Windows Server nodes to a Kubernetes environment
The Kubernetes control plane (API Server, Scheduler, Controller Manager, etc) continue to run on Linux, while the kubelet and kube-proxy can be run on Windows Server 2016 or later
{{< note >}}
**Note:** Windows Server Containers on Kubernetes is a Beta feature in Kubernetes v1.9
{{< /note >}}
## Get Windows Binaries
We recommend using the release binaries that can be found at [https://github.com/kubernetes/kubernetes/releases/latest](https://github.com/kubernetes/kubernetes/releases/latest). Under the CHANGELOG you can find the Node Binaries link for Windows-amd64, which will include kubeadm, kubectl, kubelet and kube-proxy.
If you wish to build the code yourself, please refer to detailed build instructions [here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/compiling-kubernetes-binaries).
## Prerequisites
In Kubernetes version 1.9 or later, Windows Server Containers for Kubernetes are supported using the following:
1. Kubernetes control plane running on existing Linux infrastructure (version 1.9 or later).
2. Kubenet network plugin setup on the Linux nodes.
3. Windows Server 2016 RTM or later. Windows Server version 1709 or later is preferred; it unlocks key capabilities like shared network namespace.
4. Docker Version 17.06.1-ee-2 or later for Windows Server nodes (Linux nodes and Kubernetes control plane can run any Kubernetes supported Docker Version).
## Networking
There are several supported network configurations with Kubernetes v1.9 on Windows, including both Layer-3 routed and overlay topologies using third-party network plugins.
1. [Upstream L3 Routing](#upstream-l3-routing-topology) - IP routes configured in upstream ToR
2. [Host-Gateway](#host-gateway-topology) - IP routes configured on each host
3. [Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay](#using-ovn-with-ovs) - overlay networks (supports STT and Geneve tunneling types)
4. [Future - In Review] Overlay - VXLAN or IP-in-IP encapsulation using Flannel
5. [Future] Layer-3 Routing with BGP (Calico)
The selection of which network configuration and topology to deploy depends on the physical network topology and a user's ability to configure routes, performance concerns with encapsulation, and requirement to integrate with third-party network plugins.
### Future CNI Plugins
An additional two CNI plugins [win-l2bridge (host-gateway) and win-overlay (vxlan)] are in [PR review](https://github.com/containernetworking/plugins/pull/85). These two CNI plugins, when ready, can either be used directly or with Flannel.
### Linux
The above networking approaches are already supported on Linux using a bridge interface, which essentially creates a private network local to the node. Similar to the Windows side, routes to all other pod CIDRs must be created in order to send packets via the "public" NIC.
### Windows
Windows supports the CNI network model and uses plugins to interface with the Windows Host Networking Service (HNS) to configure host networking and policy. At the time of this writing, the only publicly available CNI plugin from Microsoft is built from a private repo and available here [wincni.exe](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/cni/wincni.exe). It uses an l2bridge network created through the Windows Host Networking Service (HNS) by an administrator using HNS PowerShell commands on each node as documented in the [Windows Host Setup](#windows-host-setup) section below. Source code for the future CNI plugins will be made available publicly.
#### Upstream L3 Routing Topology
In this topology, networking is achieved using L3 routing with static IP routes configured in an upstream Top of Rack (ToR) switch/router. Each cluster node is connected to the management network with a host IP. Additionally, each node uses a local 'l2bridge' network with a pod CIDR assigned. All pods on a given worker node will be connected to the pod CIDR subnet ('l2bridge' network). In order to enable network communication between pods running on different nodes, the upstream router has static routes configured with pod CIDR prefix => Host IP.
The following example diagram illustrates the Windows Server networking setup for Kubernetes using Upstream L3 Routing Setup:
![K8s Cluster using L3 Routing with ToR](UpstreamRouting.png)
#### Host-Gateway Topology
This topology is similar to the Upstream L3 Routing topology with the only difference being that static IP routes are configured directly on each cluster node and not in the upstream ToR. Each node uses a local 'l2bridge' network with a pod CIDR assigned as before and has routing table entries for all other pod CIDR subnets assigned to the remote cluster nodes.
#### Using OVN with OVS
The following diagram gives a general overview of the architecture and interaction between components:
![Overlay using OVN controller and OVS Switch Extension](ovn_kubernetes.png)
(The above image is from [https://github.com/openvswitch/ovn-kubernetes#overlay-mode-architecture-diagram](https://github.com/openvswitch/ovn-kubernetes#overlay-mode-architecture-diagram))
Due to its architecture, OVN has a central component which stores your networking intent in a database. Other components i.e. kube-apiserver, kube-controller-manager, kube-scheduler etc. can be deployed on that central node as well.
## Setting up Windows Server Containers on Kubernetes
To run Windows Server Containers on Kubernetes, you'll need to set up both your host machines and the Kubernetes node components for Windows. Depending on your network topology, routes may need to be set up for pod communication on different nodes.
### Host Setup
#### For 1. Upstream L3 Routing Topology and 2. Host-Gateway Topology
##### Linux Host Setup
1. Linux hosts should be setup according to their respective distro documentation and the requirements of the Kubernetes version you will be using.
2. Configure Linux Master node using steps [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/creating-a-linux-master.md)
3. [Optional] CNI network plugin installed.
##### Windows Host Setup
1. Windows Server container host running the required Windows Server and Docker versions. Follow the setup instructions outlined by this help topic: https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server.
2. [Get Windows Binaries](#get-windows-binaries) kubelet.exe, kube-proxy.exe, and kubectl.exe using instructions
3. Copy Node spec file (kube config) from Linux master node with X.509 keys
4. Create the HNS Network, ensure the correct CNI network config, and start kubelet.exe using this script [start-kubelet.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubelet.ps1)
5. Start kube-proxy using this script [start-kubeproxy.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubeproxy.ps1)
6. [Only required for #2 Host-Gateway mode] Add static routes on Windows host using this script [AddRoutes.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/AddRoutes.ps1)
More detailed instructions can be found [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows.md).
**Windows CNI Config Example**
Today, Windows CNI plugin is based on wincni.exe code with the following example, configuration file. This is based on the ToR example diagram shown above, specifying the configuration to apply to Windows node-1. Of special interest is Windows node-1 pod CIDR (10.10.187.64/26) and the associated gateway of cbr0 (10.10.187.66). The exception list is specifying the Service CIDR (11.0.0.0/8), Cluster CIDR (10.10.0.0/16), and Management (or Host) CIDR (10.127.132.128/25).
Note: this file assumes that a user previous created 'l2bridge' host networks on each Windows node using `<Verb>-HNSNetwork` cmdlets as shown in the `start-kubelet.ps1` and `start-kubeproxy.ps1` scripts linked above
```json
{
"cniVersion": "0.2.0",
"name": "l2bridge",
"type": "wincni.exe",
"master": "Ethernet",
"ipam": {
"environment": "azure",
"subnet": "10.10.187.64/26",
"routes": [{
"GW": "10.10.187.66"
}]
},
"dns": {
"Nameservers": [
"11.0.0.10"
]
},
"AdditionalArgs": [{
"Name": "EndpointPolicy",
"Value": {
"Type": "OutBoundNAT",
"ExceptionList": [
"11.0.0.0/8",
"10.10.0.0/16",
"10.127.132.128/25"
]
}
},
{
"Name": "EndpointPolicy",
"Value": {
"Type": "ROUTE",
"DestinationPrefix": "11.0.0.0/8",
"NeedEncap": true
}
},
{
"Name": "EndpointPolicy",
"Value": {
"Type": "ROUTE",
"DestinationPrefix": "10.127.132.213/32",
"NeedEncap": true
}
}
]
}
```
#### For 3. Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay
##### Linux Host Setup
Setting up the central node and the components needed is out of scope of this document. You can read [these instructions](https://github.com/openvswitch/ovn-kubernetes#k8s-master-node-initialization) for that.
Adding a Linux minion is also out of scope and you can read it here: [Linux minion](https://github.com/openvswitch/ovn-kubernetes#k8s-minion-node-initializations).
##### Windows Host Setup
Adding a Windows minion requires you to install OVS and OVN binaries. Windows Server container host running the required Windows Server and Docker versions. Follow the setup instructions outlined by [this help topic](https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server). This type of deployment is supported starting with Windows Server 2016 RTM.
Compiling OVS and generating the installer will not be treated in this document. For a step by step instruction please visit [this link](http://docs.openvswitch.org/en/latest/intro/install/windows/#open-vswitch-on-windows).
For a prebuilt certified installer please visit [this link](https://cloudbase.it/openvswitch/#download) and download the latest version of it.
The following guide uses the prebuilt certified installer.
Installing OVS can be done either via the GUI dialogs or unattended. Adding a Windows host to your setup requires you to have `OVN Host` together with the default installation features. Below is the dialog image on what needs to be installed:
![OVN OVS Windows Installer](OVN_OVS_Windows_Installer.png)
For an unattended installation please use the following command:
```
cmd /c 'msiexec /i openvswitch.msi ADDLOCAL="OpenvSwitchCLI,OpenvSwitchDriver,OVNHost" /qn'
```
The installer propagates new environment variables. Please open a new command shell or logoff/logon to ensure the environment variables are refreshed.
For overlay, OVS on Windows requires a transparent docker network to function properly. Please use the following to create a transparent docker network which will be used by OVS. From powershell:
```
docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET `
-o com.docker.network.windowsshim.interface="$INTERFACE_ALIAS" external
```
Where $SUBNET is the minion subnet which will be used to spawn pods on (the one which will be used by kubernetes), $GATEWAY_IP is the first IP of the $SUBNET and $INTERFACE_ALIAS is the interface used for creating the overlay tunnels (must have connectivity with the rests of the OVN hosts).
Example:
```
docker network create -d transparent --gateway 10.0.1.1 --subnet 10.0.1.0/24 `
-o com.docker.network.windowsshim.interface="Ethernet0" external
```
After creating the docker network please run the next commands from powershell. (creates an OVS bridge, adds the interface under the bridge and enables the OVS forwarding switch extension)
```
$a = Get-NetAdapter | where Name -Match HNSTransparent
Rename-NetAdapter $a[0].Name -NewName HNSTransparent
Stop-Service ovs-vswitchd -force; Disable-VMSwitchExtension "Cloudbase Open vSwitch Extension";
ovs-vsctl --no-wait del-br br-ex
ovs-vsctl --no-wait --may-exist add-br br-ex
ovs-vsctl --no-wait add-port br-ex HNSTransparent -- set interface HNSTransparent type=internal
ovs-vsctl --no-wait add-port br-ex $INTERFACE_ALIAS
Enable-VMSwitchExtension "Cloudbase Open vSwitch Extension"; sleep 2; Restart-Service ovs-vswitchd
```
Besides of the above, setting up a Windows host is the same as the Linux host. Follow the steps from [here](https://github.com/openvswitch/ovn-kubernetes#k8s-minion-node-initializations).
**Windows CNI Setup**
Today, Windows OVN&OVS CNI plugin is based on ovn_cni.exe which can be downloaded from [here](https://cloudbase.it/downloads/ovn_cni.exe). A sample of CNI config file is the following:
```
{
"name": "net",
"type": "ovn_cni.exe",
"bridge": "br-int",
"isGateway": "true",
"ipMasq": "false",
"ipam": {
"type": "host-local",
"subnet": "$SUBNET"
}
}
```
Where $SUBNET is the subnet that was used in the previous ```docker network create``` command.
For a complete guide on Google Cloud Platform (GCP), namely Google Compute Engine (GCE) visit [this](https://github.com/apprenda/kubernetes-ovn-heterogeneous-cluster#heterogeneous-kubernetes-cluster-on-top-of-ovn).
For a complete guide on Amazon Web Services (AWS) visit [this](https://github.com/justeat/kubernetes-windows-aws-ovs#kubernetes-on-windows-in-aws-using-ovn).
## Starting the Cluster
To start your cluster, you'll need to start both the Linux-based Kubernetes control plane, and the Windows Server-based Kubernetes node components (kubelet and kube-proxy). For the OVS & OVN only the kubelet is required.
## Starting the Linux-based Control Plane
Use your preferred method to start Kubernetes cluster on Linux. Please note that Cluster CIDR might need to be updated.
## Support for kubeadm join
If your cluster has been created by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/),
and your networking is setup correctly using one of the methods listed above (networking is setup outside of kubeadm), you can use kubeadm to add a Windows node to your cluster. At a high level, you first have to initialize the master with kubeadm (Linux), then set up the CNI based networking (outside of kubeadm), and finally start joining Windows or Linux worker nodes to the cluster. For additional documentation and reference material, visit the kubeadm link above.
The kubeadm binary can be found at [Kubernetes Releases](https://github.com/kubernetes/kubernetes/releases), inside the node binaries archive. Adding a Windows node is not any different than adding a Linux node:
`kubeadm.exe join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>`
See [joining-your-nodes](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#44-joining-your-nodes) for more details.
## Supported Features
The examples listed below assume running Windows nodes on Windows Server 1709. If you are running Windows Server 2016, the examples will need the image updated to specify `image: microsoft/windowsservercore:ltsc2016`. This is due to the requirement for container images to match the host operating system version when using process isolation. Not specifying a tag will implicitly use the `:latest` tag which can lead to surprising behaviors. Please consult with [https://hub.docker.com/r/microsoft/windowsservercore/](https://hub.docker.com/r/microsoft/windowsservercore/) for additional information on Windows Server Core image tagging.
### Scheduling Pods on Windows
Because your cluster has both Linux and Windows nodes, you must explicitly set the `nodeSelector` constraint to be able to schedule pods to Windows nodes. You must set nodeSelector with the label `beta.kubernetes.io/os` to the value `windows`; see the following example:
```yaml
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "iis",
"labels": {
"name": "iis"
}
},
"spec": {
"containers": [
{
"name": "iis",
"image": "microsoft/iis:windowsservercore-1709",
"ports": [
{
"containerPort": 80
}
]
}
],
"nodeSelector": {
"beta.kubernetes.io/os": "windows"
}
}
}
```
**Note:** this example assumes you are running on Windows Server 1709, so uses the image tag to support that. If you are on a different version, you will need to update the tag. For example, if on Windows Server 2016, update to use `"image": "microsoft/iis"` which will default to that OS version.
### Secrets and ConfigMaps
Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be used as environment variables. See limitations section below for additional details.
**Examples:**
Windows pod with secrets mapped to environment variables
```yaml
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm
---
apiVersion: v1
kind: Pod
metadata:
name: my-secret-pod
spec:
containers:
- name: my-secret-pod
image: microsoft/windowsservercore:1709
env:
- name: USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
nodeSelector:
beta.kubernetes.io/os: windows
```
Windows pod with configMap values mapped to environment variables
```yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: example-config
data:
example.property.1: hello
example.property.2: world
---
apiVersion: v1
kind: Pod
metadata:
name: my-configmap-pod
spec:
containers:
- name: my-configmap-pod
image: microsoft/windowsservercore:1709
env:
- name: EXAMPLE_PROPERTY_1
valueFrom:
configMapKeyRef:
name: example-config
key: example.property.1
- name: EXAMPLE_PROPERTY_2
valueFrom:
configMapKeyRef:
name: example-config
key: example.property.2
nodeSelector:
beta.kubernetes.io/os: windows
```
### Volumes
Some supported Volume Mounts are local, emptyDir, hostPath. One thing to remember is that paths must either be escaped, or use forward slashes, for example `mountPath: "C:\\etc\\foo"` or `mountPath: "C:/etc/foo"`.
Persistent Volume Claims are supported for supported volume types.
**Examples:**
Windows pod with a hostPath volume
```yaml
apiVersion: v1
kind: Pod
metadata:
name: my-hostpath-volume-pod
spec:
containers:
- name: my-hostpath-volume-pod
image: microsoft/windowsservercore:1709
volumeMounts:
- name: foo
mountPath: "C:\\etc\\foo"
readOnly: true
nodeSelector:
beta.kubernetes.io/os: windows
volumes:
- name: foo
hostPath:
path: "C:\\etc\\foo"
```
Windows pod with multiple emptyDir volumes
```yaml
apiVersion: v1
kind: Pod
metadata:
name: my-empty-dir-pod
spec:
containers:
- image: microsoft/windowsservercore:1709
name: my-empty-dir-pod
volumeMounts:
- mountPath: /cache
name: cache-volume
- mountPath: C:/scratch
name: scratch-volume
volumes:
- name: cache-volume
emptyDir: {}
- name: scratch-volume
emptyDir: {}
nodeSelector:
beta.kubernetes.io/os: windows
```
### DaemonSets
DaemonSets are supported
```yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: my-DaemonSet
labels:
app: foo
spec:
template:
metadata:
labels:
app: foo
spec:
containers:
- name: foo
image: microsoft/windowsservercore:1709
nodeSelector:
beta.kubernetes.io/os: windows
```
### Metrics
Windows Stats use a hybrid model: pod and container level stats come from CRI (via dockershim), while node level stats come from the "winstats" package that exports cadvisor like data structures using windows specific perf counters from the node.
### Container Resources
Container resources (CPU and memory) could be set now for windows containers in v1.10.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: iis
spec:
replicas: 3
template:
metadata:
labels:
app: iis
spec:
containers:
- name: iis
image: microsoft/iis
resources:
limits:
memory: "128Mi"
cpu: 2
ports:
- containerPort: 80
```
### Hyper-V Containers
Hyper-V containers are supported as experimental in v1.10. To create a Hyper-V container, kubelet should be started with feature gates `HyperVContainer=true` and Pod should include annotation `experimental.windows.kubernetes.io/isolation-type=hyperv`.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: iis
spec:
replicas: 3
template:
metadata:
labels:
app: iis
annotations:
experimental.windows.kubernetes.io/isolation-type: hyperv
spec:
containers:
- name: iis
image: microsoft/iis
ports:
- containerPort: 80
```
## Known Limitations for Windows Server Containers with v1.9
Some of these limitations will be addressed by the community in future releases of Kubernetes
- Shared network namespace (compartment) with multiple Windows Server containers (shared kernel) per pod is only supported on Windows Server 1709 or later
- Using Secrets and ConfigMaps as volume mounts is not supported
- Mount propagation is not supported on Windows
- The StatefulSet functionality for stateful applications is not supported
- Horizontal Pod Autoscaling for Windows Server Container pods has not been verified to work end-to-end
- Hyper-V isolated containers are not supported.
- Windows container OS must match the Host OS. If it does not, the pod will get stuck in a crash loop.
- Under the networking models of L3 or Host GW, Kubernetes Services are inaccessible to Windows nodes due to a Windows issue. This is not an issue if using OVN/OVS for networking.
- Windows kubelet.exe may fail to start when running on Windows Server under VMware Fusion [issue 57110](https://github.com/kubernetes/kubernetes/pull/57124)
- Flannel and Weavenet are not yet supported
- Some .Net Core applications expect environment variables with a colon (`:`) in the name. Kubernetes currently does not allow this. Replace colon (`:`) with double underscore (`__`) as documented [here](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?tabs=basicconfiguration#configuration-by-environment).
- As cgroups are not supported on windows, kubelet.exe should be started with the following additional arguments `--cgroups-per-qos=false --enforce-node-allocatable=""` [issue 61716](https://github.com/kubernetes/kubernetes/issues/61716)
## Next steps and resources
- Support for Windows is in Beta as of v1.9 and your feedback is welcome. For information on getting involved, please head to [SIG-Windows](https://github.com/kubernetes/community/blob/master/sig-windows/README.md)
- Troubleshooting and Common Problems: [Link](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems)
content/en/docs/getting-started-guides/windows/configmap-pod.yaml
+31
View File
@@ -0,0 +1,31 @@
kind: ConfigMap
apiVersion: v1
metadata:
name: example-config
data:
example.property.1: hello
example.property.2: world
---
apiVersion: v1
kind: Pod
metadata:
name: configmap-pod
spec:
containers:
- name: configmap-redis
image: redis:3.0-nanoserver
env:
- name: EXAMPLE_PROPERTY_1
valueFrom:
configMapKeyRef:
name: example-config
key: example.property.1
- name: EXAMPLE_PROPERTY_2
valueFrom:
configMapKeyRef:
name: example-config
key: example.property.2
nodeSelector:
beta.kubernetes.io/os: windows
content/en/docs/getting-started-guides/windows/emptydir-pod.yaml
+20
View File
@@ -0,0 +1,20 @@
apiVersion: v1
kind: Pod
metadata:
name: empty-dir-pod
spec:
containers:
- image: redis:3.0-nanoserver
name: empty-dir-redis
volumeMounts:
- mountPath: /cache
name: cache-volume
- mountPath: C:/scratch
name: scratch-volume
volumes:
- name: cache-volume
emptyDir: {}
- name: scratch-volume
emptyDir: {}
nodeSelector:
beta.kubernetes.io/os: windows
content/en/docs/getting-started-guides/windows/hostpath-volume-pod.yaml
+18
View File
@@ -0,0 +1,18 @@
apiVersion: v1
kind: Pod
metadata:
name: hostpath-volume-pod
spec:
containers:
- name: hostpath-redis
image: redis:3.0-nanoserver
volumeMounts:
- name: blah
mountPath: "C:\\etc\\foo"
readOnly: true
nodeSelector:
beta.kubernetes.io/os: windows
volumes:
- name: blah
hostPath:
path: "C:\\etc\\foo"
content/en/docs/getting-started-guides/windows/ovn_kubernetes.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 104 KiB

content/en/docs/getting-started-guides/windows/sample-l2bridge-wincni-config.json
+49
View File
@@ -0,0 +1,49 @@
{
"cniVersion": "0.2.0",
"name": "l2bridge",
"type": "wincni.exe",
"master": "Ethernet",
"ipam": {
"environment": "azure",
"subnet": "10.10.187.64/26",
"routes": [
{
"GW": "10.10.187.66"
}
]
},
"dns": {
"Nameservers": [
"11.0.0.10"
]
},
"AdditionalArgs": [
{
"Name": "EndpointPolicy",
"Value": {
"Type": "OutBoundNAT",
"ExceptionList": [
"11.0.0.0/8",
"10.10.0.0/16",
"10.127.132.128/25"
]
}
},
{
"Name": "EndpointPolicy",
"Value": {
"Type": "ROUTE",
"DestinationPrefix": "11.0.0.0/8",
"NeedEncap": true
}
},
{
"Name": "EndpointPolicy",
"Value": {
"Type": "ROUTE",
"DestinationPrefix": "10.127.132.213/32",
"NeedEncap": true
}
}
]
}
content/en/docs/getting-started-guides/windows/secret-pod.yaml
+32
View File
@@ -0,0 +1,32 @@
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm
---
apiVersion: v1
kind: Pod
metadata:
name: mypod-secret
spec:
containers:
- name: mypod-secret
image: redis:3.0-nanoserver
env:
- name: USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
nodeSelector:
beta.kubernetes.io/os: windows
content/en/docs/getting-started-guides/windows/windows-setup.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB