whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated Getting Started Guide for vSphere Cloud Provider

Browse Source
This commit is contained in:
divyenpatel
2017-07-19 17:53:20 -07:00
committed by Andrew Chen
parent f9ac865c5f
commit 3d05d95e71
1 changed files with 144 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/getting-started-guides/vsphere.md
+144 -185
View File
@@ -10,7 +10,7 @@ This page covers how to get started with deploying Kubernetes on vSphere and det
* TOC
{:toc}
### Getting started with vSphere
### Getting started with the vSphere Cloud Provider
Kubernetes comes with a cloud provider for vSphere. A quick and easy way to try out the cloud provider is to deploy Kubernetes using [Kubernetes-Anywhere](https://github.com/kubernetes/kubernetes-anywhere).
@@ -20,7 +20,7 @@ This page also describes how to configure and get started with the cloud provide
To start using Kubernetes on top of vSphere and use the vSphere Cloud Provider use Kubernetes-Anywhere. Kubernetes-Anywhere will deploy and configure a cluster from scratch.
Detailed steps can be found at the [getting started with Kubernetes-Anywhere on vSphere page](https://git.k8s.io/kubernetes-anywhere/phase1/vsphere/README.md)
Detailed steps can be found at the [getting started with Kubernetes-Anywhere on vSphere](https://git.k8s.io/kubernetes-anywhere/phase1/vsphere/README.md) page
### vSphere Cloud Provider
@@ -31,219 +31,178 @@ vSphere Cloud Provider allows using vSphere managed storage within Kubernetes. I
3. Storage Classes and provisioning of volumes.
4. vSphere Storage Policy Based Management for Containers orchestrated by Kubernetes.
Documentation for how to use vSphere managed storage can be found in the
[persistent volumes user
guide](/docs/concepts/storage/persistent-volumes/#vsphere) and the
[volumes user
guide](/docs/concepts/storage/volumes/#vspherevolume)
Documentation for how to use vSphere managed storage can be found in the [persistent volumes user guide](/docs/concepts/storage/persistent-volumes/#vsphere) and the [volumes user guide](/docs/concepts/storage/volumes/#vspherevolume)
Examples can be found
[here](https://git.k8s.io/kubernetes/examples/volumes/vsphere)
Examples can be found [here](https://git.k8s.io/kubernetes/examples/volumes/vsphere)
#### Configuring vSphere Cloud Provider
#### Enable vSphere Cloud Provider
If a Kubernetes cluster has not been deployed using Kubernetes-Anywhere, follow the instructions below to use the vSphere Cloud Provider. These steps are not needed when using Kubernetes-Anywhere, they will be done as part of the deployment.
If a Kubernetes cluster has not been deployed using Kubernetes-Anywhere, follow the instructions below to enable the vSphere Cloud Provider. These steps are not needed when using Kubernetes-Anywhere, they will be done as part of the deployment.
* Enable UUID for a VM
**Step-1** [Create a VM folder](https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.vcenterhost.doc/GUID-031BDB12-D3B2-4E2D-80E6-604F304B4D0C.html) and move Kubernetes Node VMs to this folder.
This can be done via [govc tool](https://github.com/vmware/govmomi/tree/master/govc)
**Step-2** Make sure Node VM names must comply with the regex `[a-z](([-0-9a-z]+)?[0-9a-z])?(\.[a-z0-9](([-0-9a-z]+)?[0-9a-z])?)*` If Node VMs does not comply with this regex, rename them and make it compliant to this regex.
```
export GOVC_URL=<IP/URL>
export GOVC_USERNAME=<vCenter User>
export GOVC_PASSWORD=<vCenter Password>
export GOVC_INSECURE=1
govc vm.change -e="disk.enableUUID=1" -vm=<VMNAME>
```
Node VM names constraints:
* Create Role and User with Required Privileges for vSphere Cloud Provider
* VM names can not begin with numbers.
* VM names can not have capital letters, any special charaters except `.` and `-`.
* VM names can not be shorter than 3 chars and longer than 63
vSphere Cloud Provider requires the following minimal set of privileges to interact with vCenter:
**Step-3** Enable disk UUID on Node virtual machines
Please refer [vSphere Documentation Center](http://pubs.vmware.com/vsphere-65/index.jsp?topic=%2Fcom.vmware.vsphere.security.doc%2FGUID-18071E9A-EED1-4968-8D51-E0B4F526FDA3.html&resultof=%22%43%72%65%61%74%65%22%20%22%63%72%65%61%74%22%20%22%43%75%73%74%6f%6d%22%20%22%63%75%73%74%6f%6d%22%20%22%52%6f%6c%65%22%20%22%72%6f%6c%65%22%20) to know about steps for creating a Custom Role, User and Role Assignment.
The disk.EnableUUID parameter must be set to "TRUE" for each Node VM. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly.
Note: Assign Permissions at the vCenter Level and make sure to check Propagate.
For each of the virtual machine nodes that will be participating in the cluster, follow the steps below using [GOVC tool](https://github.com/vmware/govmomi/tree/master/govc)
```
Datastore > Allocate space
Datastore > Low level file Operations
Virtual Machine > Configuration > Add existing disk
Virtual Machine > Configuration > Add or remove device
Virtual Machine > Configuration > Remove disk
```
* Set up GOVC environment
For the VSAN policy based volume provisioning feature, the following additional privileges are required.
export GOVC_URL='vCenter IP OR FQDN'
export GOVC_USERNAME='vCenter User'
export GOVC_PASSWORD='vCenter Password'
export GOVC_INSECURE=1
```
Network > Assign network
Virtual machine > Configuration > Add new disk
Virtual Machine > Inventory > Create new
Virtual machine > Configuration > Add new disk
Resource > Assign virtual machine to resource pool
```
* Find Node VM Paths
* Provide the cloud config file to each instance of kubelet, apiserver and controller manager via ```--cloud-config=<path to file>``` flag. Cloud config [template can be found at Kubernetes-Anywhere](https://git.k8s.io/kubernetes-anywhere/phase1/vsphere/vsphere.conf)
govc ls /datacenter/vm/<vm-folder-name>
Sample Config:
* Set disk.EnableUUID to true for all VMs
govc vm.change -e="disk.enableUUID=1" -vm='VM Path'
Note: If Kubernetes Node VMs are created from template VM then `disk.EnableUUID=1` can be set on the template VM. VMs cloned from this template, will automatically inherit this property.
**Step-4** Create and assign Roles to the vSphere Cloud Provider user and vSphere entities.
Note: if you want to use Administrator account then this step can be skipped.
vSphere Cloud Provider requires the following minimal set of privileges to interact with vCenter. Please refer [vSphere Documentation Center](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-18071E9A-EED1-4968-8D51-E0B4F526FDA3.html) to know about steps for creating a Custom Role, User and Role Assignment.
<table>
<thead>
<tr>
<th>Roles</th>
<th>Privileges</th>
<th>Entities</th>
<th>Propagate to Children</th>
</tr>
</thead>
<tbody><tr>
<td>manage-k8s-node-vms</td>
<td>Resource.AssignVMToPool<br> System.Anonymous<br> System.Read<br> System.View<br> VirtualMachine.Config.AddExistingDisk<br> VirtualMachine.Config.AddNewDisk<br> VirtualMachine.Config.AddRemoveDevice<br> VirtualMachine.Config.RemoveDisk<br> VirtualMachine.Inventory.Create<br> VirtualMachine.Inventory.Delete</td>
<td>Cluster,<br> Hosts,<br> VM Folder</td>
<td>Yes</td>
</tr>
<tr>
<td>manage-k8s-volumes</td>
<td>Datastore.AllocateSpace<br> Datastore.FileManagement<br> System.Anonymous<br> System.Read<br> System.View</td>
<td>Datastore</td>
<td>No</td>
</tr>
<tr>
<td>k8s-system-read-and-spbm-profile-view</td>
<td>StorageProfile.View<br> System.Anonymous<br> System.Read<br> System.View</td>
<td>vCenter</td>
<td>No</td>
</tr>
<tr>
<td>ReadOnly</td>
<td>System.Anonymous<br>System.Read<br>System.View</td>
<td>Datacenter,<br> Datastore Cluster,<br> Datastore Storage Folder</td>
<td>No</td>
</tr>
</tbody>
</table>
**Step-5** Create the vSphere cloud config file (`vsphere.conf`). Cloud config template can be found [here](https://github.com/kubernetes/kubernetes-anywhere/blob/master/phase1/vsphere/vsphere.conf)
This config file needs to be placed in the shared directory which should be accessible from kubelet container, controller-manager pod, and API server pod.
**```vsphere.conf``` for Master Node:**
```
[Global]
user = <User name for vCenter>
password = <Password for vCenter>
server = <IP/URL for vCenter>
port = <Default 443 for vCenter>
insecure-flag = <set to 1 if the host above uses a self-signed cert>
datacenter = <Datacenter to be used>
datastore = <Datastore to use for provisioning volumes using storage classes/dynamic provisioning>
working-dir = <Folder in which VMs are provisioned, can be null>
vm-uuid = <VM Instance UUID of virtual machine which can be retrieved from instanceUuid property in VmConfigInfo, or also set as vc.uuid in VMX file. If empty, will be retrieved from sysfs (requires root)>
user = "vCenter username for cloud provider"
password = "password"
server = "IP/FQDN for vCenter"
port = "443" #Optional
insecure-flag = "1" #set to 1 if the vCenter uses a self-signed cert
datacenter = "Datacenter name"
datastore = "Datastore name" #Datastore to use for provisioning volumes using storage classes/dynamic provisioning
working-dir = "vCenter VM folder path in which node VMs are located"
vm-name = "VM name of the Master Node" #Optional
vm-uuid = "UUID of the Node VM" # Optional
[Disk]
scsicontrollertype = pvscsi
```
* Set the cloud provider via ```--cloud-provider=vsphere``` flag for each instance of kubelet, apiserver and controller manager.
Note: **```vm-name``` parameter is introduced in 1.6.4 release.** Both ```vm-uuid``` and ```vm-name``` are optional parameters. if ```vm-name``` is specified then ```vm-uuid``` is not used. if both are not specified then kubelet will get vm-uuid from `/sys/class/dmi/id/product_serial` and query vCenter to find the Node VM's name.
* When upgrading to 1.6 install the default storage class addons, [click here for more details](https://github.com/kubernetes/kubernetes/issues/40070)
**```vsphere.conf``` for Worker Nodes:** (Only Applicable to 1.6.4 release and above. For older releases this file should have all the parameters specified in Master node's ```vSphere.conf``` file)
```
[Global]
vm-name = "VM name of the Worker Node"
```
Below is summary of supported parameters in the `vsphere.conf` file
* ```user``` is the vCenter username for vSphere Cloud Provider.
* ```password``` is the password for vCenter user specified with `user`.
* ```server``` is the vCenter Server IP or FQDN
* ```port``` is the vCenter Server Port. Default is 443 if not specified.
* ```insecure-flag``` is set to 1 if vCenter used a self-signed certificate.
* ```datacenter``` is the name of the datacenter on which Node VMs are deployed.
* ```datastore``` is the default datastore to use for provisioning volumes using storage classes/dynamic provisioning.
* ```vm-name``` is recently added configuration parameter. This is optional parameter. When this parameter is present, ```vsphere.conf``` file on the worker node does not need vCenter credentials.
**Note:** ```vm-name``` is added in the release 1.6.4. Prior releases does not support this parameter.
* ```working-dir``` can be set to empty ( working-dir = ""), if Node VMs are located in the root VM folder.
* ```vm-uuid``` is the VM Instance UUID of virtual machine. ```vm-uuid``` can be set to empty (```vm-uuid = ""```). if set to empty, this will be retrieved from /sys/class/dmi/id/product_serial file on virtual machine (requires root access).
* ```vm-uuid``` needs to be set in this format - ```423D7ADC-F7A9-F629-8454-CE9615C810F1```
* ```vm-uuid``` can be retrieved from Node Virtual machines using following command. This will be different on each node VM.
cat /sys/class/dmi/id/product_serial | sed -e 's/^VMware-//' -e 's/-/ /' | awk '{ print toupper($1$2$3$4 "-" $5$6 "-" $7$8 "-" $9$10 "-" $11$12$13$14$15$16) }'
* `datastore` is the default datastore used for provisioning volumes using storage classes. If datastore is located in storage folder or datastore is member of datastore cluster, make sure to specify full datastore path. Make sure vSphere Cloud Provider user has Read Privilege set on the datastore cluster or storage folder to be able to find datastore.
* For datastore located in the datastore cluster, specify datastore as mentioned below
datastore = "DatastoreCluster/datastore1"
* For datastore located in the datastore cluster, specify datastore as mentioned below
datastore = "DatastoreFolder/datastore1"
**Step-6** Add flags to controller-manager, API server and Kubelet to enable vSphere Cloud Provider.
* Add following flags to kubelet running on every node and to the controller-manager and API server pods manifest files.
```
--cloud-provider=vsphere
--cloud-config=<Path of the vsphere.conf file>
```
Manifest files for API server and controller-manager are generally located at `/etc/kubernetes`
**Step-7** Restart Kubelet on all nodes.
* Reload kubelet systemd unit file using ```systemctl daemon-reload```
* Restart kubelet service using ```systemctl restart kubelet.service```
Note: After enabling the vSphere Cloud Provider, Node names will be set to the VM names from the vCenter Inventory.
#### Known issues
### Kube-up (Deprecated)
Kube-up.sh is no longer supported and is deprecated. The steps for kube-up are included but going forward [kube-anywhere](https://github.com/kubernetes/kubernetes-anywhere) is preferred.
The recommended version for kube-up is [v1.4.7](https://github.com/kubernetes/kubernetes/releases/tag/v1.4.7)
The example below creates a Kubernetes cluster with 4 worker node Virtual.
Machines and a master Virtual Machine (i.e. 5 VMs in your cluster). This cluster is set up and controlled from your workstation (or wherever you find convenient).
#### Prerequisites
* You need administrator credentials to an ESXi machine or vCenter instance with write mode api access enabled (not available on the free ESXi license).
* You must have Go (see [here](https://git.k8s.io/community/contributors/devel/development.md#go-versions) for supported versions) installed: [www.golang.org](http://www.golang.org).
* You must have your `GOPATH` set up and include `$GOPATH/bin` in your `PATH`.
```shell
export GOPATH=$HOME/src/go
mkdir -p $GOPATH
export PATH=$PATH:$GOPATH/bin
```
* Install the govc tool to interact with ESXi/vCenter. Head to [govc Releases](https://github.com/vmware/govmomi/releases) to download the latest.
```shell
# Sample commands for v0.8.0 for 64 bit Linux.
curl -OL https://github.com/vmware/govmomi/releases/download/v0.8.0/govc_linux_amd64.gz
gzip -d govc_linux_amd64.gz
chmod +x govc_linux_amd64
mv govc_linux_amd64 /usr/local/bin/govc
```
* Get or build a [binary release](/docs/getting-started-guides/binary_release)
#### Setup
Download a prebuilt Debian 8.2 VMDK that we'll use as a base image:
```shell
curl --remote-name-all https://storage.googleapis.com/govmomi/vmdk/2016-01-08/kube.vmdk.gz{,.md5}
md5sum -c kube.vmdk.gz.md5
gzip -d kube.vmdk.gz
```
Configure the environment for govc
```shell
export GOVC_URL='hostname' # hostname of the vc
export GOVC_USERNAME='username' # username for logging into the vsphere.
export GOVC_PASSWORD='password' # password for the above username
export GOVC_NETWORK='Network Name' # Name of the network the vms should join. Many times it could be "VM Network"
export GOVC_INSECURE=1 # If the host above uses a self-signed cert
export GOVC_DATASTORE='target datastore'
# To get resource pool via govc: govc ls -l 'host/*' | grep ResourcePool | awk '{print $1}' | xargs -n1 -t govc pool.info
export GOVC_RESOURCE_POOL='resource pool or cluster with access to datastore'
export GOVC_GUEST_LOGIN='kube:kube' # Used for logging into kube.vmdk during deployment.
export GOVC_PORT=443 # The port to be used by vSphere cloud provider plugin
# To get datacente via govc: govc datacenter.info
export GOVC_DATACENTER='ha-datacenter' # The datacenter to be used by vSphere cloud provider plugin
```
Sample environment
```shell
export GOVC_URL='10.161.236.217'
export GOVC_USERNAME='administrator'
export GOVC_PASSWORD='MyPassword1'
export GOVC_NETWORK='VM Network'
export GOVC_INSECURE=1
export GOVC_DATASTORE='datastore1'
export GOVC_RESOURCE_POOL='/Datacenter/host/10.20.104.24/Resources'
export GOVC_GUEST_LOGIN='kube:kube'
export GOVC_PORT='443'
export GOVC_DATACENTER='Datacenter'
```
Import this VMDK into your vSphere datastore:
```shell
govc import.vmdk kube.vmdk ./kube/
```
Verify that the VMDK was correctly uploaded and expanded to ~3GiB:
```shell
govc datastore.ls ./kube/
```
If you need to debug any part of the deployment, the guest login for
the image that you imported is `kube:kube`. It is normally specified
in the GOVC_GUEST_LOGIN parameter above.
Also take a look at the file `cluster/vsphere/config-default.sh` and
make any needed changes. You can configure the number of nodes
as well as the IP subnets you have made available to Kubernetes, pods,
and services.
#### Starting a cluster
Now, let's continue with deploying Kubernetes.
This process takes about ~20-30 minutes depending on your network.
##### From extracted binary release
```shell
cd kubernetes
KUBERNETES_PROVIDER=vsphere cluster/kube-up.sh
```
##### Build from source
```shell
cd kubernetes
make release
KUBERNETES_PROVIDER=vsphere cluster/kube-up.sh
```
Refer to the top level README and the getting started guide for Google Compute
Engine. Once you have successfully reached this point, your vSphere Kubernetes
deployment works just as any other one!
**Enjoy!**
#### Extra: debugging deployment failure
The output of `kube-up.sh` displays the IP addresses of the VMs it deploys. You
can log into any VM as the `kube` user to poke around and figure out what is
going on (find yourself authorized with your SSH key, or use the password
`kube` otherwise).
[vmware#220](https://github.com/vmware/kubernetes/issues/220) :
vSphere Cloud Provider can not be used on the Kubernetes Cluster when vCenter port is configured other than the default port 443. Fix for this issue is already out (Kubernetes PR# [49689](https://github.com/kubernetes/kubernetes/pull/49689)). We will make sure that, PR 49689 is cherry picked to 1.7, 1.6 and 1.5 branches.
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Vmware vSphere | Kube-anywhere | Photon OS | Flannel | [docs](/docs/getting-started-guides/vsphere) | | Community ([@abrarshivani](https://github.com/abrarshivani)), ([@kerneltime](https://github.com/kerneltime)), ([@BaluDontu](https://github.com/BaluDontu)), ([@luomiao](https://github.com/luomiao)), ([@divyenpatel](https://github.com/divyenpatel))
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | --------- | ----------------------------
Vmware vSphere | Kube-anywhere | Photon OS | Flannel | [docs](/docs/getting-started-guides/vsphere) | | Community ([@abrarshivani](https://github.com/abrarshivani)), ([@kerneltime](https://github.com/kerneltime)), ([@BaluDontu](https://github.com/BaluDontu)), ([@luomiao](https://github.com/luomiao)), ([@divyenpatel](https://github.com/divyenpatel))
If you identify any issues/problems using the vSphere cloud provider, you can create an issue in our repo - [VMware Kubernetes](https://github.com/vmware/kubernetes).
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.