whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge master to 1.10, with fixes (#7682)

Browse Source
This commit is contained in:
Jennifer Rondeau
2018-03-08 14:03:55 -05:00
committed by k8s-ci-robot
parent bb8c59a640
commit 44b51d6056
548 changed files with 11634 additions and 318622 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/getting-started-guides/OWNERS
+1 -1
View File
@@ -1,3 +1,3 @@
approvers:
reviewers:
- errordeveloper
docs/getting-started-guides/alibaba-cloud.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- colemickens
- brendandburns
title: Running Kubernetes on Alibaba Cloud
docs/getting-started-guides/alternatives.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- pwittrock
title: Deprecated Alternatives
---
docs/getting-started-guides/aws.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- justinsb
- clove
title: Running Kubernetes on AWS EC2
docs/getting-started-guides/azure.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- colemickens
- brendandburns
title: Running Kubernetes on Azure
docs/getting-started-guides/centos/OWNERS
-4
View File
@@ -1,4 +0,0 @@
approvers:
- lavalamp
- thockin
docs/getting-started-guides/centos/centos_manual_config.md
-238
View File
@@ -1,238 +0,0 @@
---
approvers:
- thockin
title: CentOS
---
* TOC
{:toc}
**Caution:** This guide was originally written for Kubernetes 1.1.0 and [is deprecated](https://github.com/kubernetes/website/issues/1613) and is replaced by [kubeadm](/docs/admin/kubeadm/).
{: .caution}
## Prerequisites
To configure Kubernetes with CentOS, you'll need a machine to act as a master, and one or more CentOS 7 hosts to act as cluster nodes.
## Starting a cluster
This is a getting started guide for CentOS. It is a manual configuration so you understand all the underlying packages / services / ports, etc...
The Kubernetes package provides a few services: kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, kube-proxy. These services are managed by systemd and the configuration resides in a central location: /etc/kubernetes. We will break the services up between the hosts. The first host, centos-master, will be the Kubernetes master. This host will run the kube-apiserver, kube-controller-manager and kube-scheduler. In addition, the master will also run _etcd_. The remaining hosts, centos-minion-n will be the nodes and run kubelet, proxy, cadvisor and docker.
All of them run flanneld as networking overlay.
**System Information:**
Hosts:
Please replace host IP with your environment.
```conf
centos-master = 192.168.121.9
centos-minion-1 = 192.168.121.65
centos-minion-2 = 192.168.121.66
centos-minion-3 = 192.168.121.67
```
**Prepare the hosts:**
* Create a /etc/yum.repos.d/virt7-docker-common-release.repo on all hosts - centos-{master,minion-n} with following information.
```conf
[virt7-docker-common-release]
name=virt7-docker-common-release
baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/
gpgcheck=0
```
* Install Kubernetes, etcd and flannel on all hosts - centos-{master,minion-n}. This will also pull in docker and cadvisor.
```shell
yum -y install --enablerepo=virt7-docker-common-release kubernetes etcd flannel
```
* Add master and node to /etc/hosts on all machines (not needed if hostnames already in DNS)
```shell
echo "192.168.121.9 centos-master
192.168.121.65 centos-minion-1
192.168.121.66 centos-minion-2
192.168.121.67 centos-minion-3" >> /etc/hosts
```
* Edit /etc/kubernetes/config which will be the same on all hosts to contain:
```shell
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"
# How the replication controller and scheduler find the kube-apiserver
KUBE_MASTER="--master=http://centos-master:8080"
```
* Disable the firewall on the master and all the nodes, as docker does not play well with other firewall rule managers. CentOS won't let you disable the firewall as long as SELinux is enforcing, so that needs to be disabled first.
* If you disable SELinux, make sure you reboot your machine before continuing to more steps.
```shell
setenforce 0
systemctl disable iptables-services firewalld
systemctl stop iptables-services firewalld
```
**Configure the Kubernetes services on the master.**
* Edit /etc/etcd/etcd.conf to appear as such:
```shell
# [member]
ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
#[cluster]
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
```
* Edit /etc/kubernetes/apiserver to appear as such:
```shell
# The address on the local server to listen to.
KUBE_API_ADDRESS="--address=0.0.0.0"
# The port on the local server to listen on.
KUBE_API_PORT="--port=8080"
# Port kubelets listen on
KUBELET_PORT="--kubelet-port=10250"
# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://centos-master:2379"
# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
# Add your own!
KUBE_API_ARGS=""
```
* Start ETCD and configure it to hold the network overlay configuration on master:
**Warning** This network must be unused in your network infrastructure! `172.30.0.0/16` is free in our network.
```shell
systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"
```
* Configure flannel to overlay Docker network in /etc/sysconfig/flanneld on the master (also in the nodes as we'll see):
```shell
# Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://centos-master:2379"
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/kube-centos/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""
```
* Start the appropriate services on master:
```shell
for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
```
**Configure the Kubernetes services on the nodes.**
***We need to configure the kubelet and start the kubelet and proxy***
* Edit /etc/kubernetes/kubelet to appear as such:
```shell
# The address for the info server to serve on
KUBELET_ADDRESS="--address=0.0.0.0"
# The port for the info server to serve on
KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
# Check the node number!
KUBELET_HOSTNAME="--hostname-override=centos-minion-n"
# Location of the api-server
KUBELET_API_SERVER="--api-servers=http://centos-master:8080"
# Add your own!
KUBELET_ARGS=""
```
* Configure flannel to overlay Docker network in /etc/sysconfig/flanneld (in all the nodes)
```shell
# Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://centos-master:2379"
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/kube-centos/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""
```
* Start the appropriate services on node (centos-minion-n).
```shell
for SERVICES in kube-proxy kubelet flanneld docker; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
```
* Configure kubectl
```shell
kubectl config set-cluster default-cluster --server=http://centos-master:8080
kubectl config set-context default-context --cluster=default-cluster --user=default-admin
kubectl config use-context default-context
```
*You should be finished!*
* Check to make sure the cluster can see the node (on centos-master)
```shell
$ kubectl get nodes
NAME STATUS AGE VERSION
centos-minion-1 Ready 3d v1.6.0+fff5156
centos-minion-2 Ready 3d v1.6.0+fff5156
centos-minion-3 Ready 3d v1.6.0+fff5156
```
**The cluster should be running! Launch a test pod.**
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Bare-metal | custom | CentOS | flannel | [docs](/docs/getting-started-guides/centos/centos_manual_config) | | Community ([@coolsvap](https://github.com/coolsvap))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
docs/getting-started-guides/cloudstack.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- thockin
title: Cloudstack
---
docs/getting-started-guides/coreos/OWNERS
+1 -1
View File
@@ -1,3 +1,3 @@
approvers:
reviewers:
- errordeveloper
docs/getting-started-guides/coreos/bare_metal_offline.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- erictune
- thockin
title: Offline
docs/getting-started-guides/dcos.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- smugcloud
title: Kubernetes on DCOS
---
docs/getting-started-guides/docker-multinode/OWNERS
-4
View File
@@ -1,4 +0,0 @@
approvers:
- dchen1107
- resouer
docs/getting-started-guides/fedora/OWNERS
+1 -1
View File
@@ -1,4 +1,4 @@
approvers:
reviewers:
- aveshagarwal
- eparis
- thockin
docs/getting-started-guides/fedora/fedora_ansible_config.md
-240
View File
@@ -1,240 +0,0 @@
---
approvers:
- aveshagarwal
- erictune
title: Fedora via Ansible
---
Configuring Kubernetes on Fedora via Ansible offers a simple way to quickly create a clustered environment with little effort.
* TOC
{:toc}
## Prerequisites
1. Host able to run ansible and able to clone the following repo: [Kubernetes](https://github.com/kubernetes/kubernetes.git)
2. A Fedora 21+ host to act as cluster master
3. As many Fedora 21+ hosts as you would like, that act as cluster nodes
The hosts can be virtual or bare metal. Ansible will take care of the rest of the configuration for you - configuring networking, installing packages, handling the firewall, etc. This example will use one master and two nodes.
## Architecture of the cluster
A Kubernetes cluster requires etcd, a master, and n nodes, so we will create a cluster with three hosts, for example:
```shell
master,etcd = kube-master.example.com
node1 = kube-node-01.example.com
node2 = kube-node-02.example.com
```
**Make sure your local machine has**
- ansible (must be 1.9.0+)
- git
- python-netaddr
If not
```shell
dnf install -y ansible git python-netaddr
```
**Now clone down the Kubernetes repository**
```shell
git clone https://github.com/kubernetes/contrib.git
cd contrib/ansible
```
**Tell ansible about each machine and its role in your cluster**
Get the IP addresses from the master and nodes. Add those to the `~/contrib/ansible/inventory/localhost.ini` file on the host running Ansible.
```shell
[masters]
kube-master.example.com
[etcd]
kube-master.example.com
[nodes]
kube-node-01.example.com
kube-node-02.example.com
```
## Setting up ansible access to your nodes
If you already are running on a machine which has passwordless ssh access to the kube-master and kube-node-{01,02} nodes, and 'sudo' privileges, simply set the value of `ansible_ssh_user` in `~/contrib/ansible/inventory/group_vars/all.yml` to the username which you use to ssh to the nodes (i.e. `fedora`), and proceed to the next step...
*Otherwise* setup ssh on the machines like so (you will need to know the root password to all machines in the cluster).
edit: `~/contrib/ansible/inventory/group_vars/all.yml`
```yaml
ansible_ssh_user: root
```
**Configuring ssh access to the cluster**
If you already have ssh access to every machine using ssh public keys you may skip to [setting up the cluster](#setting-up-the-cluster)
Make sure your local machine (root) has an ssh key pair if not
```shell
ssh-keygen
```
Copy the ssh public key to **all** nodes in the cluster
```shell
for node in kube-master.example.com kube-node-01.example.com kube-node-02.example.com; do
ssh-copy-id ${node}
done
```
## Setting up the cluster
Although the default value of variables in `~/contrib/ansible/inventory/group_vars/all.yml` should be good enough, if not, change them as needed.
```conf
edit: ~/contrib/ansible/inventory/group_vars/all.yml
```
**Configure access to Kubernetes packages**
Modify `source_type` as below to access Kubernetes packages through the package manager.
```yaml
source_type: packageManager
```
**Configure the IP addresses used for services**
Each Kubernetes service gets its own IP address. These are not real IPs. You need to only select a range of IPs which are not in use elsewhere in your environment.
```yaml
kube_service_addresses: 10.254.0.0/16
```
**Managing flannel**
Modify `flannel_subnet`, `flannel_prefix` and `flannel_host_prefix` only if defaults are not appropriate for your cluster.
**Managing add on services in your cluster**
Set `cluster_logging` to false or true (default) to disable or enable logging with elasticsearch.
```yaml
cluster_logging: true
```
Turn `cluster_monitoring` to true (default) or false to enable or disable cluster monitoring with heapster and influxdb.
```yaml
cluster_monitoring: true
```
Turn `dns_setup` to true (recommended) or false to enable or disable whole DNS configuration.
```yaml
dns_setup: true
```
**Tell ansible to get to work!**
This will finally setup your whole Kubernetes cluster for you.
```shell
cd ~/contrib/ansible/scripts/
./deploy-cluster.sh
```
## Testing and using your new cluster
That's all there is to it. It's really that easy. At this point you should have a functioning Kubernetes cluster.
**Show Kubernetes nodes**
Run the following on the kube-master:
```shell
kubectl get nodes
```
**Show services running on masters and nodes**
```shell
systemctl | grep -i kube
```
**Show firewall rules on the masters and nodes**
```shell
iptables -nvL
```
**Create /tmp/apache.json on the master with the following contents and deploy pod**
```json
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "fedoraapache",
"labels": {
"name": "fedoraapache"
}
},
"spec": {
"containers": [
{
"name": "fedoraapache",
"image": "fedora/apache",
"ports": [
{
"hostPort": 80,
"containerPort": 80
}
]
}
]
}
}
```
```shell
kubectl create -f /tmp/apache.json
```
**Check where the pod was created**
```shell
kubectl get pods
```
**Check Docker status on nodes**
```shell
docker ps
docker images
```
**After the pod is 'Running' Check web server access on the node**
```shell
curl http://localhost
```
That's it!
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Bare-metal | Ansible | Fedora | flannel | [docs](/docs/getting-started-guides/fedora/fedora_ansible_config) | | Project
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
docs/getting-started-guides/fedora/fedora_manual_config.md
+26 -6
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- aveshagarwal
- eparis
- thockin
@@ -60,11 +60,14 @@ the name of the master server:
KUBE_MASTER="--master=http://fed-master:8080"
```
* Disable the firewall on both the master and node, as docker does not play well with other firewall rule managers. Please note that iptables-services does not exist on default fedora server install.
* Disable the firewall on both the master and node, as Docker does not play well with other firewall rule managers. Please note that iptables.service does not exist on the default Fedora Server install.
```shell
systemctl disable iptables-services firewalld
systemctl stop iptables-services firewalld
systemctl mask firewalld.service
systemctl stop firewalld.service
systemctl disable iptables.service
systemctl stop iptables.service
```
**Configure the Kubernetes services on the master.**
@@ -118,10 +121,27 @@ KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_HOSTNAME="--hostname-override=fed-node"
# location of the api-server
KUBELET_API_SERVER="--api-servers=http://fed-master:8080"
KUBELET_ARGS="--cgroup-driver=systemd --kubeconfig=/etc/kubernetes/master-kubeconfig.yaml --require-kubeconfig"
# Add your own!
#KUBELET_ARGS=""
KUBELET_ARGS=""
```
```yaml
kind: Config
clusters:
- name: local
cluster:
server: http://fed-master:8080
users:
- name: kubelet
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
```
* Start the appropriate services on the node (fed-node).
docs/getting-started-guides/fedora/flannel_multi_node_cluster.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- dchen1107
- erictune
- thockin
docs/getting-started-guides/fluentd-gcp.yaml
-479
View File
@@ -1,479 +0,0 @@
apiVersion: v1
kind: Pod
metadata:
name: fluentd-cloud-logging
namespace: kube-system
labels:
k8s-app: fluentd-logging
# This annotation ensures that fluentd does not get evicted if the node
# supports critical pod annotation based priority scheme.
# Note that this does not guarantee admission on the nodes (#40573).
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
dnsPolicy: Default
containers:
- name: fluentd-cloud-logging
image: k8s.gcr.io/fluentd-gcp:2.0.2
# If fluentd consumes its own logs, the following situation may happen:
# fluentd fails to send a chunk to the server => writes it to the log =>
# tries to send this message to the server => fails to send a chunk and so on.
# Writing to a file, which is not exported to the back-end prevents it.
# It also allows to increase the fluentd verbosity by default.
command:
- '/bin/sh'
- '-c'
- |-
mkdir /etc/fluent/config.d &&
echo "$FLUENTD_CONFIG" > /etc/fluent/config.d/main.conf &&
/run.sh $FLUENTD_ARGS 2>&1 >>/var/log/fluentd.log
env:
- name: FLUENTD_ARGS
value: --no-supervisor
# Keep this config as close as possible to cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml
# Note that backslashes should be doubled, because this is interpreted as shell variable
# TODO(crassirostris): Refactor this
- name: FLUENTD_CONFIG
value: |-
# This configuration file for Fluentd is used
# to watch changes to Docker log files that live in the
# directory /var/lib/docker/containers/ and are symbolically
# linked to from the /var/log/containers directory using names that capture the
# pod name and container name. These logs are then submitted to
# Google Cloud Logging which assumes the installation of the cloud-logging plug-in.
#
# Example
# =======
# A line in the Docker log file might look like this JSON:
#
# {"log":"2014/09/25 21:15:03 Got request with path wombat\\n",
# "stream":"stderr",
# "time":"2014-09-25T21:15:03.499185026Z"}
#
# The record reformer is used to write the tag to focus on the pod name
# and the Kubernetes container name. For example a Docker container's logs
# might be in the directory:
# /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b
# and in the file:
# 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
# where 997599971ee6... is the Docker ID of the running container.
# The Kubernetes kubelet makes a symbolic link to this file on the host machine
# in the /var/log/containers directory which includes the pod name and the Kubernetes
# container name:
# synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
# ->
# /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
# The /var/log directory on the host is mapped to the /var/log directory in the container
# running this instance of Fluentd and we end up collecting the file:
# /var/log/containers/synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
# This results in the tag:
# var.log.containers.synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
# The record reformer is used is discard the var.log.containers prefix and
# the Docker container ID suffix and "kubernetes." is pre-pended giving the tag:
# kubernetes.synthetic-logger-0.25lps-pod_default-synth-lgr
# Tag is then parsed by google_cloud plugin and translated to the metadata,
# visible in the log viewer
# Example:
# {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}
<source>
type tail
format json
time_key time
path /var/log/containers/*.log
pos_file /var/log/gcp-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%N%Z
tag reform.*
read_from_head true
</source>
<filter reform.**>
type parser
format /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<log>.*)/
reserve_data true
suppress_parse_error_log true
key_name log
</filter>
<match reform.**>
type record_reformer
enable_ruby true
tag raw.kubernetes.${tag_suffix[4].split('-')[0..-2].join('-')}
</match>
# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
@type copy
<store>
@type prometheus
<metric>
type counter
name logging_line_count
desc Total number of lines generated by application containers
<labels>
tag ${tag}
</labels>
</metric>
</store>
<store>
@type detect_exceptions
remove_tag_prefix raw
message log
stream stream
multiline_flush_interval 5
max_bytes 500000
max_lines 1000
</store>
</match>
# Example:
# 2015-12-21 23:17:22,066 [salt.state ][INFO ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081
<source>
type tail
format /^(?<time>[^ ]* [^ ,]*)[^\\[]*\\[[^\\]]*\\]\\[(?<severity>[^ \\]]*) *\\] (?<message>.*)$/
time_format %Y-%m-%d %H:%M:%S
path /var/log/salt/minion
pos_file /var/log/gcp-salt.pos
tag salt
</source>
# Example:
# Dec 21 23:17:22 gke-foo-1-1-4b5cbd14-node-4eoj startupscript: Finished running startup script /var/run/google.startup.script
<source>
type tail
format syslog
path /var/log/startupscript.log
pos_file /var/log/gcp-startupscript.log.pos
tag startupscript
</source>
# Examples:
# time="2016-02-04T06:51:03.053580605Z" level=info msg="GET /containers/json"
# time="2016-02-04T07:53:57.505612354Z" level=error msg="HTTP Error" err="No such image: -f" statusCode=404
<source>
type tail
format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\\d+))?/
path /var/log/docker.log
pos_file /var/log/gcp-docker.log.pos
tag docker
</source>
# Example:
# 2016/02/04 06:52:38 filePurge: successfully removed file /var/etcd/data/member/wal/00000000000006d0-00000000010a23d1.wal
<source>
type tail
# Not parsing this, because it doesn't have anything particularly useful to
# parse out of it (like severities).
format none
path /var/log/etcd.log
pos_file /var/log/gcp-etcd.log.pos
tag etcd
</source>
# Multi-line parsing is required for all the kube logs because very large log
# statements, such as those that include entire object bodies, get split into
# multiple lines by glog.
# Example:
# I0204 07:32:30.020537 3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537]
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kubelet.log
pos_file /var/log/gcp-kubelet.log.pos
tag kubelet
</source>
# Example:
# I1118 21:26:53.975789 6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-proxy.log
pos_file /var/log/gcp-kube-proxy.log.pos
tag kube-proxy
</source>
# Example:
# I0204 07:00:19.604280 5 handlers.go:131] GET /api/v1/nodes: (1.624207ms) 200 [[kube-controller-manager/v1.1.3 (linux/amd64) kubernetes/6a81b50] 127.0.0.1:38266]
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-apiserver.log
pos_file /var/log/gcp-kube-apiserver.log.pos
tag kube-apiserver
</source>
# Example:
# 2017-02-09T00:15:57.992775796Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" ip="104.132.1.72" method="GET" user="kubecfg" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
# 2017-02-09T00:15:57.993528822Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" response="200"
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\S+\\s+AUDIT:/
# Fields must be explicitly captured by name to be parsed into the record.
# Fields may not always be present, and order may change, so this just looks
# for a list of key="\\"quoted\\" value" pairs separated by spaces.
# Unknown fields are ignored.
# Note: We can't separate query/response lines as format1/format2 because
# they don't always come one after the other for a given query.
# TODO: Maybe add a JSON output mode to audit log so we can get rid of this?
format1 /^(?<time>\\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\\\]|\\\\.)*)"|ip="(?<ip>(?:[^"\\\\]|\\\\.)*)"|method="(?<method>(?:[^"\\\\]|\\\\.)*)"|user="(?<user>(?:[^"\\\\]|\\\\.)*)"|groups="(?<groups>(?:[^"\\\\]|\\\\.)*)"|as="(?<as>(?:[^"\\\\]|\\\\.)*)"|asgroups="(?<asgroups>(?:[^"\\\\]|\\\\.)*)"|namespace="(?<namespace>(?:[^"\\\\]|\\\\.)*)"|uri="(?<uri>(?:[^"\\\\]|\\\\.)*)"|response="(?<response>(?:[^"\\\\]|\\\\.)*)"|\\w+="(?:[^"\\\\]|\\\\.)*"))*/
time_format %FT%T.%L%Z
path /var/log/kube-apiserver-audit.log
pos_file /var/log/gcp-kube-apiserver-audit.log.pos
tag kube-apiserver-audit
</source>
# Example:
# I0204 06:55:31.872680 5 servicecontroller.go:277] LB already exists and doesn't need update for service kube-system/kube-ui
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-controller-manager.log
pos_file /var/log/gcp-kube-controller-manager.log.pos
tag kube-controller-manager
</source>
# Example:
# W0204 06:49:18.239674 7 reflector.go:245] pkg/scheduler/factory/factory.go:193: watch of *api.Service ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [2578313/2577886]) [2579312]
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-scheduler.log
pos_file /var/log/gcp-kube-scheduler.log.pos
tag kube-scheduler
</source>
# Example:
# I1104 10:36:20.242766 5 rescheduler.go:73] Running Rescheduler
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/rescheduler.log
pos_file /var/log/gcp-rescheduler.log.pos
tag rescheduler
</source>
# Example:
# I0603 15:31:05.793605 6 cluster_manager.go:230] Reading config from path /etc/gce.conf
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/glbc.log
pos_file /var/log/gcp-glbc.log.pos
tag glbc
</source>
# Example:
# I0603 15:31:05.793605 6 cluster_manager.go:230] Reading config from path /etc/gce.conf
<source>
type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\\w\\d{4}/
format1 /^(?<severity>\\w)(?<time>\\d{4} [^\\s]*)\\s+(?<pid>\\d+)\\s+(?<source>[^ \\]]+)\\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/cluster-autoscaler.log
pos_file /var/log/gcp-cluster-autoscaler.log.pos
tag cluster-autoscaler
</source>
# Logs from systemd-journal for interesting services.
<source>
type systemd
filters [{ "_SYSTEMD_UNIT": "docker.service" }]
pos_file /var/log/gcp-journald-docker.pos
read_from_head true
tag docker
</source>
<source>
type systemd
filters [{ "_SYSTEMD_UNIT": "kubelet.service" }]
pos_file /var/log/gcp-journald-kubelet.pos
read_from_head true
tag kubelet
</source>
# Prometheus monitoring
<source>
@type prometheus
port 80
</source>
<source>
@type prometheus_monitor
</source>
# We use 2 output stanzas - one to handle the container logs and one to handle
# the node daemon logs, the latter of which explicitly sends its logs to the
# compute.googleapis.com service rather than container.googleapis.com to keep
# them separate since most users don't care about the node logs.
<match kubernetes.**>
@type copy
<store>
@type google_cloud
# Set the buffer type to file to improve the reliability and reduce the memory consumption
buffer_type file
buffer_path /var/log/fluentd-buffers/kubernetes.containers.buffer
# Set queue_full action to block because we want to pause gracefully
# in case of the off-the-limits load instead of throwing an exception
buffer_queue_full_action block
# Set the chunk limit conservatively to avoid exceeding the GCL limit
# of 10MiB per write request.
buffer_chunk_limit 2M
# Cap the combined memory usage of this buffer and the one below to
# 2MiB/chunk * (6 + 2) chunks = 16 MiB
buffer_queue_limit 6
# Never wait more than 5 seconds before flushing logs in the non-error case.
flush_interval 5s
# Never wait longer than 30 seconds between retries.
max_retry_wait 30
# Disable the limit on the number of retries (retry forever).
disable_retry_limit
# Use multiple threads for processing.
num_threads 2
</store>
<store>
@type prometheus
<metric>
type counter
name logging_entry_count
desc Total number of log entries generated by either an application container or a system component
<labels>
tag ${tag}
component container
</labels>
</metric>
</store>
</match>
# Keep a smaller buffer here since these logs are less important than the user's
# container logs.
<match **>
@type copy
<store>
@type google_cloud
detect_subservice false
buffer_type file
buffer_path /var/log/fluentd-buffers/kubernetes.system.buffer
buffer_queue_full_action block
buffer_chunk_limit 2M
buffer_queue_limit 2
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 2
</store>
<store>
@type prometheus
<metric>
type counter
name logging_entry_count
desc Total number of log entries generated by either an application container or a system component
<labels>
tag ${tag}
component system
</labels>
</metric>
</store>
</match>
resources:
limits:
memory: 200Mi
requests:
# Any change here should be accompanied by a proportional change in CPU
# requests of other per-node add-ons (e.g. kube-proxy).
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: libsystemddir
mountPath: /host/lib
readOnly: true
# Liveness probe is aimed to help in situarions where fluentd
# silently hangs for no apparent reasons until manual restart.
# The idea of this probe is that if fluentd is not queueing or
# flushing chunks for 5 minutes, something is not right. If
# you want to change the fluentd configuration, reducing amount of
# logs fluentd collects, consider changing the threshold or turning
# liveness probe off completely.
livenessProbe:
initialDelaySeconds: 600
periodSeconds: 60
exec:
command:
- '/bin/sh'
- '-c'
- >
LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300};
STUCK_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-900};
if [ ! -e /var/log/fluentd-buffers ];
then
exit 1;
fi;
LAST_MODIFIED_DATE=`stat /var/log/fluentd-buffers | grep Modify | sed -r "s/Modify: (.*)/\1/"`;
LAST_MODIFIED_TIMESTAMP=`date -d "$LAST_MODIFIED_DATE" +%s`;
if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $STUCK_THRESHOLD_SECONDS` ];
then
rm -rf /var/log/fluentd-buffers;
exit 1;
fi;
if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $LIVENESS_THRESHOLD_SECONDS` ];
then
exit 1;
fi;
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: libsystemddir
hostPath:
path: /usr/lib64
docs/getting-started-guides/gce.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- brendandburns
- jbeda
- mikedanese
docs/getting-started-guides/kops.md
+2 -2
View File
@@ -34,7 +34,7 @@ Download kops from the [releases page](https://github.com/kubernetes/kops/releas
On MacOS:
```
wget https://github.com/kubernetes/kops/releases/download/1.8.0/kops-darwin-amd64
curl -OL https://github.com/kubernetes/kops/releases/download/1.8.0/kops-darwin-amd64
chmod +x kops-darwin-amd64
mv kops-darwin-amd64 /usr/local/bin/kops
# you can also install using Homebrew
@@ -156,7 +156,7 @@ See the [list of add-ons](/docs/concepts/cluster-administration/addons/) to expl
## Cleanup
* To delete you cluster: `kops delete cluster useast1.dev.example.com --yes`
* To delete your cluster: `kops delete cluster useast1.dev.example.com --yes`
## Feedback
docs/getting-started-guides/kubespray.md
+8 -13
View File
@@ -10,7 +10,7 @@ Kubespray is a composition of [Ansible](http://docs.ansible.com/) playbooks, [in
* a highly available cluster
* composable attributes
* support for most popular Linux distributions
* support for most popular Linux distributions (CoreOS, Debian Jessie, Ubuntu 16.04, CentOS/RHEL 7)
* continuous integration tests
To choose a tool which best fits your use case, read [this comparison](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/comparisons.md) to [kubeadm](/docs/admin/kubeadm/) and [kops](../kops).
@@ -21,7 +21,7 @@ To choose a tool which best fits your use case, read [this comparison](https://g
Provision servers with the following requirements:
* `Ansible v2.3` (or newer)
* `Ansible v2.4` (or newer)
* `Jinja 2.9` (or newer)
* `python-netaddr` installed on the machine that running Ansible commands
* Target servers must have access to the Internet in order to pull docker images
@@ -37,10 +37,6 @@ Kubespray provides the following utilities to help provision your environment:
* [Terraform](https://www.terraform.io/) scripts for the following cloud providers:
* [AWS](https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/terraform/aws)
* [OpenStack](https://github.com/kubernetes-incubator/kubespray/tree/master/contrib/terraform/openstack)
* [kubespray-cli](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md)
**Note:** kubespray-cli is no longer actively maintained.
{. :note}
### (2/5) Compose an inventory file
@@ -62,15 +58,14 @@ Kubespray customizations can be made to a [variable file](http://docs.ansible.co
### (4/5) Deploy a Cluster
Next, deploy your cluster with one of two methods:
Next, deploy your cluster:
* [ansible-playbook](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md#starting-custom-deployment).
* [kubespray-cli tool](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md)
Cluster deployment using [ansible-playbook](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/getting-started.md#starting-custom-deployment).
```console
ansible-playbook -i your/inventory/hosts.ini cluster.yml -b -v \
--private-key=~/.ssh/private_key
```
**Note:** kubespray-cli is no longer actively maintained.
{: .note}
Both methods run the default [cluster definition file](https://github.com/kubernetes-incubator/kubespray/blob/master/cluster.yml).
Large deployments (100+ nodes) may require [specific adjustments](https://github.com/kubernetes-incubator/kubespray/blob/master/docs/large-deployments.md) for best results.
docs/getting-started-guides/mesos-docker.md
-320
View File
@@ -1,320 +0,0 @@
---
approvers:
- jdef
- karlkfi
title: Kubernetes on Mesos on Docker
---
The mesos/docker provider uses docker-compose to launch Kubernetes as a Mesos framework, running in docker with its
dependencies (etcd & mesos).
* TOC
{:toc}
## Cluster Goals
- kubernetes development
- pod/service development
- demoing
- fast deployment
- minimal hardware requirements
- minimal configuration
- entry point for exploration
- simplified networking
- fast end-to-end tests
- local deployment
Non-Goals:
- high availability
- fault tolerance
- remote deployment
- production usage
- monitoring
- long running
- state persistence across restarts
## Cluster Topology
The cluster consists of several docker containers linked together by docker-managed hostnames:
| Component | Hostname | Description |
|-------------------------------|-----------------------------|-----------------------------------------------------------------------------------------|
| docker-grand-ambassador | | Proxy to allow circular hostname linking in docker |
| etcd | etcd | Key/Value store used by Mesos |
| Mesos Master | mesosmaster1 | REST endpoint for interacting with Mesos |
| Mesos Slave (x2) | mesosslave1, mesosslave2 | Mesos agents that offer resources and run framework executors (e.g. Kubernetes Kublets) |
| Kubernetes API Server | apiserver | REST endpoint for interacting with Kubernetes |
| Kubernetes Controller Manager | controller | |
| Kubernetes Scheduler | scheduler | Schedules container deployment by accepting Mesos offers |
## Prerequisites
Required:
- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) - version control system
- [Docker CLI](https://docs.docker.com/) - container management command line client
- [Docker Engine](https://docs.docker.com/) - container management daemon
- On Mac, use [Docker Machine](https://docs.docker.com/machine/install-machine/)
- [Docker Compose](https://docs.docker.com/compose/install/) - multi-container application orchestration
Optional:
- [Virtual Box](https://www.virtualbox.org/wiki/Downloads)
- Free x86 virtualization engine with a Docker Machine driver
- [Golang](https://golang.org/doc/install) - Go programming language
- Required to build Kubernetes locally
- [Make](https://en.wikipedia.org/wiki/Make_(software)) - Utility for building executables from source
- Required to build Kubernetes locally with make
### Install on Mac (Homebrew)
It's possible to install all of the above via [Homebrew](http://brew.sh/) on a Mac.
Some steps print instructions for configuring or launching. Make sure each is properly set up before continuing to the next step.
```shell
brew install git
brew install caskroom/cask/brew-cask
brew cask install virtualbox
brew install docker
brew install docker-machine
brew install docker-compose
```
### Install on Linux
Most of the above are available via apt and yum, but depending on your distribution, you may have to install via other
means to get the latest versions.
It is recommended to use Ubuntu, simply because it best supports AUFS, used by docker to mount volumes. Alternate file
systems may not fully support docker-in-docker.
In order to build Kubernetes, the current user must be in a docker group with sudo privileges.
See the docker docs for [instructions](https://docs.docker.com/installation/ubuntulinux/#create-a-docker-group).
#### Docker Machine Config (Mac)
If on a Mac using docker-machine, the following steps will make the docker IPs (in the virtualbox VM) reachable from the
host machine (Mac).
1. Create VM
oracle-virtualbox
```shell
docker-machine create --driver virtualbox kube-dev
eval "$(docker-machine env kube-dev)"
```
2. Set the VM's host-only network to "promiscuous mode":
oracle-virtualbox
```conf
docker-machine stop kube-dev
VBoxManage modifyvm kube-dev --nicpromisc2 allow-all
docker-machine start kube-dev
```
This allows the VM to accept packets that were sent to a different IP.
Since the host-only network routes traffic between VMs and the host, other VMs will also be able to access the docker
IPs, if they have the following route.
1. Route traffic to docker through the docker-machine IP:
```shell
sudo route -n add -net 172.17.0.0 $(docker-machine ip kube-dev)
```
Since the docker-machine IP can change when the VM is restarted, this route may need to be updated over time.
To delete the route later: `sudo route delete 172.17.0.0`
## Walkthrough
1. Checkout source
```shell
git clone https://github.com/kubernetes/kubernetes
cd kubernetes
```
By default, that will get you the bleeding edge of master branch.
You may want a [release branch](https://github.com/kubernetes/kubernetes/releases) instead,
if you have trouble with master.
1. Build binaries
You'll need to build kubectl (CLI) for your local architecture and operating system and the rest of the server binaries for linux/amd64.
Building a new release covers both cases:
```shell
KUBERNETES_CONTRIB=mesos build/release.sh
```
For developers, it may be faster to [build locally](#build-locally).
1. [Optional] Build docker images
The following docker images are built as part of `./cluster/kube-up.sh`, but it may make sense to build them manually the first time because it may take a while.
1. Test image includes all the dependencies required for running e2e tests.
```shell
./cluster/mesos/docker/test/build.sh
```
In the future, this image may be available to download. It doesn't contain anything specific to the current release, except its build dependencies.
1. Kubernetes-Mesos image includes the compiled linux binaries.
```shell
./cluster/mesos/docker/km/build.sh
```
This image needs to be built every time you recompile the server binaries.
1. [Optional] Configure Mesos resources
By default, the mesos-slaves are configured to offer a fixed amount of resources (cpus, memory, disk, ports).
If you want to customize these values, update the `MESOS_RESOURCES` environment variables in `./cluster/mesos/docker/docker-compose.yml`.
If you delete the `MESOS_RESOURCES` environment variables, the resource amounts will be auto-detected based on the host resources, which will over-provision by > 2x.
If the configured resources are not available on the host, you may want to increase the resources available to Docker Engine.
You may have to increase you VM disk, memory, or cpu allocation. See the Docker Machine docs for details
([Virtualbox](https://docs.docker.com/machine/drivers/virtualbox))
1. Configure provider
```shell
export KUBERNETES_PROVIDER=mesos/docker
```
This tells cluster scripts to use the code within `cluster/mesos/docker`.
1. Create cluster
```shell
./cluster/kube-up.sh
```
If you manually built all the above docker images, you can skip that step during kube-up:
```shell
MESOS_DOCKER_SKIP_BUILD=true ./cluster/kube-up.sh
```
After deploying the cluster, `~/.kube/config` will be created or updated to configure kubectl to target the new cluster.
1. Explore tutorials
To learn more about Pods, Volumes, Labels, Services, and Replication Controllers, start with the
[Kubernetes Tutorials](/docs/tutorials/).
To skip to a more advanced example, see the [Guestbook Example](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/guestbook/)
1. Destroy cluster
```shell
./cluster/kube-down.sh
```
## Addons
The `kube-up` for the mesos/docker provider will automatically deploy KubeDNS and KubeUI addons as pods/services.
Check their status with:
```shell
./cluster/kubectl.sh get pods --namespace=kube-system
```
### KubeUI
The web-based Kubernetes UI is accessible in a browser through the API Server proxy: `https://<apiserver>:6443/ui/`.
By default, basic-auth is configured with user `admin` and password `admin`.
The IP of the API Server can be found using `./cluster/kubectl.sh cluster-info`.
## End To End Testing
Warning: e2e tests can take a long time to run. You may not want to run them immediately if you're just getting started.
While your cluster is up, you can run the end-to-end tests:
```shell
./cluster/test-e2e.sh
```
Notable parameters:
- Increase the logging verbosity: `-v=2`
- Run only a subset of the tests (regex matching): `-ginkgo.focus=<pattern>`
To build, deploy, test, and destroy, all in one command (plus unit & integration tests):
```shell
make test_e2e
```
## Kubernetes CLI
When compiling from source, it's simpler to use the `./cluster/kubectl.sh` script, which detects your platform &
architecture and proxies commands to the appropriate `kubectl` binary.
ex: `./cluster/kubectl.sh get pods`
## Helpful scripts
- Kill all docker containers
```shell
docker ps -q -a | xargs docker rm -f
```
- Clean up unused docker volumes
```shell
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes
```
## Build Locally
The steps above tell you how to build in a container, for minimal local dependencies. But if you have Go and Make installed you can build locally much faster:
```shell
KUBERNETES_CONTRIB=mesos make
```
However, if you're not on linux, you'll still need to compile the linux/amd64 server binaries:
```shell
KUBERNETES_CONTRIB=mesos build/run.sh hack/build-go.sh
```
The above two steps should be significantly faster than cross-compiling a whole new release for every supported platform (which is what `./build/release.sh` does).
Breakdown:
- `KUBERNETES_CONTRIB=mesos` - enables building of the contrib/mesos binaries
- `hack/build-go.sh` - builds the Go binaries for the current architecture (linux/amd64 when in a docker container)
- `make` - delegates to `hack/build-go.sh`
- `build/run.sh` - executes a command in the build container
- `build/release.sh` - cross compiles Kubernetes for all supported architectures and operating systems (slow)
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Mesos/Docker | custom | Ubuntu | Docker | [docs](/docs/getting-started-guides/mesos-docker) | | Community ([Kubernetes-Mesos Authors](https://github.com/mesosphere/kubernetes-mesos/blob/master/AUTHORS.md))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
docs/getting-started-guides/mesos/OWNERS
-5
View File
@@ -1,5 +0,0 @@
approvers:
- jdef
- sttts
- thockin
docs/getting-started-guides/mesos/index.md
-341
View File
@@ -1,341 +0,0 @@
---
approvers:
- jdef
title: Kubernetes on Mesos
---
* TOC
{:toc}
## About Kubernetes on Mesos
<!-- TODO: Update, clean up. -->
Mesos allows dynamic sharing of cluster resources between Kubernetes and other first-class Mesos frameworks such as [HDFS][1], [Spark][2], and [Chronos][3].
Mesos also ensures applications from different frameworks running on your cluster are isolated and that resources are allocated fairly among them.
Mesos clusters can be deployed on nearly every IaaS cloud provider infrastructure or in your own physical datacenter. Kubernetes on Mesos runs on-top of that and therefore allows you to easily move Kubernetes workloads from one of these environments to the other.
This tutorial will walk you through setting up Kubernetes on a Mesos cluster.
It provides a step by step walk through of adding Kubernetes to a Mesos cluster and starting your first pod with an nginx webserver.
**NOTE:** There are [known issues with the current implementation][7] and support for centralized logging and monitoring is not yet available.
Please [file an issue against the kubernetes-mesos project][8] if you have problems completing the steps below.
Further information is available in the Kubernetes on Mesos [contrib directory][13].
### Prerequisites
- Understanding of [Apache Mesos][6]
- A running [Mesos cluster on Google Compute Engine][5]
- A [VPN connection][10] to the cluster
- A machine in the cluster which should become the Kubernetes *master node* with:
- Go (see [here](https://git.k8s.io/community/contributors/devel/development.md) for required versions)
- make (i.e. build-essential)
- Docker
**Note**: You *can*, but you *don't have to* deploy Kubernetes-Mesos on the same machine the Mesos master is running on.
### Deploy Kubernetes-Mesos
Log into the future Kubernetes *master node* over SSH, replacing the placeholder below with the correct IP address.
```shell
ssh jclouds@${ip_address_of_master_node}
```
Build Kubernetes-Mesos.
```shell
git clone https://github.com/kubernetes-incubator/kube-mesos-framework
cd kube-mesos-framework
make
```
Set some environment variables.
The internal IP address of the master may be obtained via `hostname -i`.
```shell
export KUBERNETES_MASTER_IP=$(hostname -i)
export KUBERNETES_MASTER=http://${KUBERNETES_MASTER_IP}:8888
```
Note that KUBERNETES_MASTER is used as the api endpoint. If you have existing `~/.kube/config` and point to another endpoint, you need to add option `--server=${KUBERNETES_MASTER}` to kubectl in later steps.
### Deploy etcd
Start etcd and verify that it is running:
```shell
sudo docker run -d --hostname $(uname -n) --name etcd \
-p 4001:4001 -p 7001:7001 quay.io/coreos/etcd:v2.2.1 \
--listen-client-urls http://0.0.0.0:4001 \
--advertise-client-urls http://${KUBERNETES_MASTER_IP}:4001
```
```shell
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fd7bac9e2301 quay.io/coreos/etcd:v2.2.1 "/etcd" 5s ago Up 3s 2379/tcp, 2380/... etcd
```
It's also a good idea to ensure your etcd instance is reachable by testing it
```shell
curl -L http://${KUBERNETES_MASTER_IP}:4001/v2/keys/
```
If connectivity is OK, you will see an output of the available keys in etcd (if any).
### Start Kubernetes-Mesos Services
Update your PATH to more easily run the Kubernetes-Mesos binaries:
```shell
export PATH="$(pwd)/_output/local/go/bin:$PATH"
```
Identify your Mesos master: depending on your Mesos installation this is either a `host:port` like `mesos-master:5050` or a ZooKeeper URL like `zk://zookeeper:2181/mesos`.
In order to let Kubernetes survive Mesos master changes, the ZooKeeper URL is recommended for production environments.
```shell
export MESOS_MASTER=<host:port or zk:// url>
```
Create a cloud config file `mesos-cloud.conf` in the current directory with the following contents:
```shell
$ cat <<EOF >mesos-cloud.conf
[mesos-cloud]
mesos-master = ${MESOS_MASTER}
EOF
```
Now start the kubernetes-mesos API server, controller manager, and scheduler on the master node:
```shell
$ km apiserver \
--address=${KUBERNETES_MASTER_IP} \
--etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \
--service-cluster-ip-range=10.10.10.0/24 \
--port=8888 \
--cloud-provider=mesos \
--cloud-config=mesos-cloud.conf \
--secure-port=0 \
--v=1 >apiserver.log 2>&1 &
$ km controller-manager \
--master=${KUBERNETES_MASTER_IP}:8888 \
--cloud-provider=mesos \
--cloud-config=./mesos-cloud.conf \
--v=1 >controller.log 2>&1 &
$ km scheduler \
--address=${KUBERNETES_MASTER_IP} \
--mesos-master=${MESOS_MASTER} \
--etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \
--mesos-user=root \
--api-servers=${KUBERNETES_MASTER_IP}:8888 \
--cluster-dns=10.10.10.10 \
--cluster-domain=cluster.local \
--v=2 >scheduler.log 2>&1 &
```
Disown your background jobs so that they'll stay running if you log out.
```shell
disown -a
```
#### Validate KM Services
Interact with the kubernetes-mesos framework via `kubectl`:
```shell
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
```
```shell
# NOTE: your service IPs will likely differ
$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8sm-scheduler 10.10.10.113 <none> 10251/TCP 1d
kubernetes 10.10.10.1 <none> 443/TCP 1d
```
Lastly, look for Kubernetes in the Mesos web GUI by pointing your browser to
`http://<mesos-master-ip:port>`. Make sure you have an active VPN connection.
Go to the Frameworks tab, and look for an active framework named "Kubernetes".
## Spin up a pod
Write a JSON pod description to a local file:
```shell
$ cat <<EOPOD >nginx.yaml
```
```yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
EOPOD
```
Send the pod description to Kubernetes using the `kubectl` CLI:
```shell
$ kubectl create -f ./nginx.yaml
pod "nginx" created
```
Wait a minute or two while `dockerd` downloads the image layers from the internet.
We can use the `kubectl` interface to monitor the status of our pod:
```shell
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 14s
```
Verify that the pod task is running in the Mesos web GUI. Click on the
Kubernetes framework. The next screen should show the running Mesos task that
started the Kubernetes pod.
## Launching kube-dns
Kube-dns is an addon for Kubernetes which adds DNS-based service discovery to the cluster. For a detailed explanation see [DNS in Kubernetes][4].
The kube-dns addon runs as a pod inside the cluster. The pod consists of three co-located containers:
- a local etcd instance
- the [kube-dns][11] DNS server
We assume that kube-dns will use
- the service IP `10.10.10.10`
- and the `cluster.local` domain.
Note that we have passed these two values already as parameter to the apiserver above.
A template for a replication controller spinning up the pod with the 3 containers can be found at [cluster/addons/dns/kubedns-controller.yaml.in][12] in the repository. The following steps are necessary in order to get a valid replication controller yaml file:
{% assign dns_replicas = "{{ pillar['dns_replicas'] }}" %}
{% assign dns_domain = "{{ pillar['dns_domain'] }}" %}
- replace `{{ dns_replicas }}` with `1`
- replace `{{ dns_domain }}` with `cluster.local.`
- add `--kube_master_url=${KUBERNETES_MASTER}` parameter to the kube2sky container command.
In addition the service template at [cluster/addons/dns/kubedns-controller.yaml.in][12] needs the following replacement:
{% assign dns_server = "{{ pillar['dns_server'] }}" %}
- `{{ dns_server }}` with `10.10.10.10`.
To do this automatically:
```shell{% raw %}
sed -e "s/{{ pillar\['dns_replicas'\] }}/1/g;"\
"s,\(command = \"/kube2sky\"\),\\1\\"$'\n'" - --kube_master_url=${KUBERNETES_MASTER},;"\
"s/{{ pillar\['dns_domain'\] }}/cluster.local/g" \
cluster/addons/dns/kubedns-controller.yaml.in > kubedns-controller.yaml
sed -e "s/{{ pillar\['dns_server'\] }}/10.10.10.10/g" \
cluster/addons/dns/kubedns-svc.yaml.in > kubedns-svc.yaml{% endraw %}
```
Now the kube-dns pod and service are ready to be launched:
```shell
kubectl create -f ./kubedns-controller.yaml
kubectl create -f ./kubedns-svc.yaml
```
Check with `kubectl get pods --namespace=kube-system` that 3/3 containers of the pods are eventually up and running. Note that the kube-dns pods run in the `kube-system` namespace, not in `default`.
To check that the new DNS service in the cluster works, we start a busybox pod and use that to do a DNS lookup. First create the `busybox.yaml` pod spec:
```shell
cat <<EOF >busybox.yaml
```
```yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
restartPolicy: Always
EOF
```
Then start the pod:
```shell
kubectl create -f ./busybox.yaml
```
When the pod is up and running, start a lookup for the Kubernetes master service, made available on 10.10.10.1 by default:
```shell
kubectl exec busybox -- nslookup kubernetes
```
If everything works fine, you will get this output:
```shell
Server: 10.10.10.10
Address 1: 10.10.10.10
Name: kubernetes
Address 1: 10.10.10.1
```
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | ---------| ----------------------------
Mesos/GCE | | | | [docs](/docs/getting-started-guides/mesos/) | | Community ([Kubernetes-Mesos Authors](https://github.com/mesosphere/kubernetes-mesos/blob/master/AUTHORS.md))
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions/) chart.
## What next?
Try out some of the standard [Kubernetes examples][9].
Read about Kubernetes on Mesos' architecture in the [contrib directory][13].
**NOTE:** Some examples require Kubernetes DNS to be installed on the cluster.
Future work will add instructions to this guide to enable support for Kubernetes DNS.
**NOTE:** Please be aware that there are [known issues with the current Kubernetes-Mesos implementation][7].
[1]: https://docs.mesosphere.com/latest/usage/service-guides/hdfs/
[2]: https://docs.mesosphere.com/latest/usage/service-guides/spark/
[3]: https://mesos.github.io/chronos/docs/getting-started.html
[4]: https://releases.k8s.io/{{page.githubbranch}}/cluster/addons/dns/README.md
[5]: https://dcos.io/docs/latest/administration/installing/cloud/gce/
[6]: http://mesos.apache.org/
[7]: https://github.com/kubernetes-incubator/kube-mesos-framework/blob/master/docs/issues.md
[8]: https://github.com/mesosphere/kubernetes-mesos/issues
[9]: https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/
[10]: http://open.mesosphere.com/getting-started/cloud/google/mesosphere/#vpn-setup
[11]: https://git.k8s.io/kubernetes/cluster/addons/dns/README.md#kube-dns
[12]: https://git.k8s.io/kubernetes/cluster/addons/dns/kubedns-controller.yaml.in
[13]: https://github.com/kubernetes-incubator/kube-mesos-framework/blob/master/README.md
docs/getting-started-guides/mesos/k8s-firewall.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 87 KiB

docs/getting-started-guides/mesos/k8s-guestbook.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 43 KiB

docs/getting-started-guides/minikube.md
+3 -3
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- dlorenc
- r2d4
- aaron-prindle
@@ -57,7 +57,7 @@ service "hello-minikube" exposed
# To check whether the pod is up and running we can use the following:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
hello-minikube-3383150820-vctvh 1/1 ContainerCreating 0 3s
hello-minikube-3383150820-vctvh 0/1 ContainerCreating 0 3s
# We can see that the pod is still being created from the ContainerCreating status
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
@@ -298,7 +298,7 @@ Some drivers will mount a host folder within the VM so that you can easily share
| VirtualBox | Linux | /home | /hosthome |
| VirtualBox | OSX | /Users | /Users |
| VirtualBox | Windows | C://Users | /c/Users |
| VMWare Fusion | OSX | /Users | /Users |
| VMware Fusion | OSX | /Users | /Users |
| Xhyve | OSX | /Users | /Users |
docs/getting-started-guides/ovirt.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- caesarxuchao
- erictune
title: oVirt
docs/getting-started-guides/rkt/OWNERS
-4
View File
@@ -1,4 +0,0 @@
approvers:
- lavalamp
- yifan-gu
docs/getting-started-guides/rkt/index.md
-229
View File
@@ -1,229 +0,0 @@
---
approvers:
- yifan-gu
title: Running Kubernetes with rkt
---
This document describes how to run Kubernetes using [rkt](https://github.com/coreos/rkt) as the container runtime.
*Note*: This document describes how to use what is known as "rktnetes". In future, Kubernetes will support the rkt runtime through the Container Runtime Interface (CRI). At present the [rkt shim for the CRI](https://github.com/kubernetes-incubator/rktlet) is considered "experimental", but if you wish to use it you will find instructions in the [kubeadm reference](/docs/admin/kubeadm/#use-kubeadm-with-other-cri-runtimes).
* TOC
{:toc}
## Prerequisites
* [Systemd](http://www.freedesktop.org/wiki/Software/systemd/) must be installed and enabled. The minimum systemd version required for Kubernetes v1.3 is `219`. Systemd is used to monitor and manage the pods on each node.
* [Install the latest rkt release](https://coreos.com/rkt/docs/latest/trying-out-rkt.html). The minimum rkt version required is [v1.13.0](https://github.com/coreos/rkt/releases/tag/v1.13.0). The [CoreOS Linux alpha channel](https://coreos.com/releases/) ships with a recent rkt release, and you can easily [upgrade rkt on CoreOS](https://coreos.com/rkt/docs/latest/install-rkt-in-coreos.html), if necessary.
* The [rkt API service](https://coreos.com/rkt/docs/latest/subcommands/api-service.html) must be running on the node.
* You will need [kubelet](/docs/getting-started-guides/scratch/#kubelet) installed on the node, and it's recommended that you run [kube-proxy](/docs/getting-started-guides/scratch/#kube-proxy) on all nodes. This document describes how to set the parameters for kubelet so that it uses rkt as the runtime.
## Pod networking in rktnetes
### Kubernetes CNI networking
You can configure Kubernetes pod networking with the usual Container Network Interface (CNI) [network plugins](/docs/concepts/cluster-administration/network-plugins/) by setting the kubelet's `--network-plugin` and `--network-plugin-dir` options appropriately. Configured in this fashion, the rkt container engine will be unaware of network details, and expects to connect pods to the provided subnet.
#### kubenet: Google Compute Engine (GCE) network
The `kubenet` plugin can be selected with the kubelet option `--network-plugin=kubenet`. This plugin is currently only supported on GCE. When using kubenet, Kubernetes CNI creates and manages the network, and rkt is provided with a subnet from a bridge device connected to the GCE network.
### rkt contained network
Rather than delegating pod networking to Kubernetes, rkt can configure connectivity directly with its own [*contained network*](https://coreos.com/rkt/docs/latest/networking/overview.html#contained-mode) on a subnet provided by a bridge device, the flannel SDN, or another CNI plugin. Configured this way, rkt looks in its [config directories](https://coreos.com/rkt/docs/latest/configuration.html#command-line-flags), usually `/etc/rkt/net.d`, to discover the CNI configuration and invoke the appropriate plugins to create the pod network.
#### rkt contained network with bridge
The *contained network* is rkt's default, so you can leave the kubelet's `--network-plugin` option empty to select this network. The contained network can be backed by any CNI plugin. With the *contained network*, rkt will attempt to join pods to a network named `rkt.kubernetes.io`, so this network name must be used for whatever desired CNI configuration.
When using the contained network, create a network configuration file beneath the rkt network config directory that defines how to create this `rkt.kubernetes.io` network in your environment. This example sets up a bridge device with the `bridge` CNI plugin:
```shell
$ cat <<EOF >/etc/rkt/net.d/k8s_network_example.conf
{
"name": "rkt.kubernetes.io",
"type": "bridge",
"bridge": "mybridge",
"mtu": 1460,
"addIf": "true",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.22.0.0/16",
"gateway": "10.22.0.1",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
EOF
```
#### rkt contained network with flannel
While it is recommended to operate flannel through the Kubernetes CNI support, you can alternatively configure the flannel plugin directly to provide the subnet for rkt's contained network. An example CNI/flannel config file looks like this:
```shell
$ cat <<EOF >/etc/rkt/net.d/k8s_flannel_example.conf
{
"name": "rkt.kubernetes.io",
"type": "flannel",
"delegate": {
"isDefaultGateway": true
}
}
EOF
```
For more information on flannel configuration, see the [CNI/flannel README](https://github.com/containernetworking/plugins/blob/master/plugins/meta/flannel/README.md).
#### Contained network caveats:
* You must create an appropriate CNI configuration file with a network name of `rkt.kubernetes.io`.
* The downwards API and environment variable substitution will not contain the pod IP address.
* The `/etc/hosts` file will not contain the pod's own hostname, although `/etc/hostname` is populated.
## Running rktnetes
### Spin up a local Kubernetes cluster with the rkt runtime
To use rkt as the container runtime in a local Kubernetes cluster, supply the following flags to the kubelet:
* `--container-runtime=rkt` Set the node's container runtime to rkt.
* `--rkt-api-endpoint=HOST:PORT` Set the endpoint of the rkt API service. Default: `localhost:15441`.
* `--rkt-path=PATH_TO_RKT_BINARY` Set the path of the rkt binary. Optional. If empty, look for `rkt` in `$PATH`.
* `--rkt-stage1-image=STAGE1` Set the name of the stage1 image, e.g. `coreos.com/rkt/stage1-coreos`. Optional. If not set, the default Linux kernel software isolation stage1 is used.
If you are using the [hack/local-up-cluster.sh](https://github.com/kubernetes/kubernetes/tree/{{page.githubbranch}}/hack/local-up-cluster.sh) script to launch the cluster, you can edit the environment variables `CONTAINER_RUNTIME`, `RKT_PATH`, and `RKT_STAGE1_IMAGE` to set these flags. `RKT_PATH` and `RKT_STAGE1_IMAGE` are optional if `rkt` is in your $PATH` with appropriate configuration.
```shell
$ export CONTAINER_RUNTIME=rkt
$ export RKT_PATH=<rkt_binary_path>
$ export RKT_STAGE1_IMAGE=<stage1-name>
```
Now you can launch the cluster using the `local-up-cluster.sh` script:
```shell
$ hack/local-up-cluster.sh
```
We are also working on getting rkt working as the container runtime in [minikube](https://github.com/kubernetes/minikube/issues/168).
### Launch a rktnetes cluster on Google Compute Engine (GCE)
This section outlines using the `kube-up` script to launch a CoreOS/rkt cluster on GCE.
Specify the OS distribution, the GCE distributor's master project, and the instance images for the Kubernetes master and nodes. Set the `KUBE_CONTAINER_RUNTIME` to `rkt`:
```shell
$ export KUBE_OS_DISTRIBUTION=coreos
$ export KUBE_GCE_MASTER_PROJECT=coreos-cloud
$ export KUBE_GCE_MASTER_IMAGE=<image_id>
$ export KUBE_GCE_NODE_PROJECT=coreos-cloud
$ export KUBE_GCE_NODE_IMAGE=<image_id>
$ export KUBE_CONTAINER_RUNTIME=rkt
```
Optionally, set the version of rkt by setting `KUBE_RKT_VERSION`:
```shell
$ export KUBE_RKT_VERSION=1.13.0
```
Optionally, select an alternative [stage1 isolator](#modular-isolation-with-interchangeable-stage1-images) for the container runtime by setting `KUBE_RKT_STAGE1_IMAGE`:
```shell
$ export KUBE_RKT_STAGE1_IMAGE=<stage1-name>
```
Then you can launch the cluster with:
```shell
$ cluster/kube-up.sh
```
### Launch a rktnetes cluster on AWS
The `kube-up` script is not yet supported on AWS. Instead, we recommend following the [Kubernetes on AWS guide](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html) to launch a CoreOS Kubernetes cluster on AWS, then setting kubelet options as above.
### Deploy apps to the cluster
After creating the cluster, you can start deploying applications. For an introductory example, [deploy a simple nginx web server](/docs/user-guide/simple-nginx). Note that this example did not have to be modified for use with a "rktnetes" cluster. More examples can be found in the [Kubernetes examples directory](https://github.com/kubernetes/examples/tree/{{page.githubbranch}}/).
## Modular isolation with interchangeable stage1 images
rkt executes containers in an interchangeable isolation environment. This facility is called the [*stage1* image](https://coreos.com/rkt/docs/latest/devel/architecture.html#stage-1). There are currently three supported rkt stage1 images:
* `systemd-nspawn` stage1, the default. Isolates running containers with Linux kernel namespaces and cgroups in a manner similar to the default container runtime.
* [`KVM` stage1](https://coreos.com/rkt/docs/latest/running-lkvm-stage1.html), runs containers inside a KVM hypervisor-managed virtual machine. Experimental in the Kubernetes v1.3 release.
* [`fly stage1`](https://coreos.com/rkt/docs/latest/running-fly-stage1.html), which isolates containers with only a `chroot`, giving host-level access to mount and network namespaces for specially-privileged utilities.
In addition to the three provided stage1 images, you can [create your own](https://coreos.com/rkt/docs/latest/devel/stage1-implementors-guide.html) for specific isolation requirements. If no configuration is set, the [default stage1](https://coreos.com/rkt/docs/latest/build-configure.html#parameters-for-setting-up-default-stage1-image) is used. There are two ways to select a different stage1; either per-node, or per-pod:
* Set the kubelet's `--rkt-stage1-image` flag, which tells the kubelet the stage1 image to use for every pod on the node. For example, `--rkt-stage1-image=coreos/rkt/stage1-coreos` selects the default systemd-nspawn stage1.
* Set the annotation `rkt.alpha.kubernetes.io/stage1-name-override` to override the stage1 used to execute a given pod. This allows for mixing different container isolation mechanisms on the same cluster or on the same node. For example, the following (shortened) pod manifest will run its pod with the `fly stage1` to give the application -- the `kubelet` in this case -- access to the host's namespace:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: kubelet
namespace: kube-system
labels:
k8s-app: kubelet
annotations:
rkt.alpha.kubernetes.io/stage1-name-override: coreos.com/rkt/stage1-fly
spec:
containers:
- name: kubelet
image: quay.io/coreos/hyperkube:v1.3.0-beta.2_coreos.0
command:
- kubelet
- --api-servers=127.0.0.1:8080
- --config=/etc/kubernetes/manifests
- --allow-privileged
- --kubeconfig=/etc/kubernetes/kubeconfig
securityContext:
privileged: true
[...]
```
### Notes on using different stage1 images
Setting the stage1 annotation could potentially give the pod root privileges. Because of this, the `privileged` boolean in the pod's `securityContext` must be set to `true`.
Use rkt's [*contained network*](#rkt-contained-network) with the KVM stage1, because the CNI plugin driver does not yet fully support the hypervisor-based runtime.
## Known issues and differences between rkt and Docker
rkt and the default node container engine have very different designs, as do rkt's native ACI and the Docker container image format. Users may experience different behaviors when switching from one container engine to the other. More information can be found [in the Kubernetes rkt notes](/docs/getting-started-guides/rkt/notes/).
## Troubleshooting
Here are a few tips for troubleshooting Kubernetes with the rkt container engine:
### Check rkt pod status
To check the status of running pods, use the rkt subcommands [`rkt list`](https://coreos.com/rkt/docs/latest/subcommands/list.html), [`rkt status`](https://coreos.com/rkt/docs/latest/subcommands/status.html), and [`rkt image list`](https://coreos.com/rkt/docs/latest/subcommands/image.html#rkt-image-list). See the [rkt commands documentation](https://coreos.com/rkt/docs/latest/commands.html) for more information about rkt subcommands.
### Check journal logs
Check a pod's log using `journalctl` on the node. Pods are managed and named as systemd units. The pod's unit name is formed by concatenating a `k8s_` prefix with the pod UUID, in a format like `k8s_${RKT_UUID}`. Find the pod's UUID with `rkt list` to assemble its service name, then ask journalctl for the logs:
```shell
$ sudo journalctl -u k8s_ad623346
```
#### Log verbosity
By default, the log verbosity level is 2. In order to see more log messages related to rkt, set this level to 4 or above. For a local cluster, set the environment variable: `LOG_LEVEL=4`.
### Check Kubernetes events and logs.
Kubernetes provides various tools for troubleshooting and examination. More information can be found [in the app troubleshooting guide](/docs/tasks/debug-application-cluster/debug-application/).
docs/getting-started-guides/rkt/notes.md
-106
View File
@@ -1,106 +0,0 @@
---
approvers:
- dchen1107
- yifan-gu
title: Known Issues when Using rkt
---
The following features either are not supported or have large caveats when using the rkt container runtime. Increasing support for these items and others, including reasonable feature parity with the default container engine, is planned through future releases.
## Non-existent host volume paths
When mounting a host volume path that does not exist, rkt will error out. Under the Docker runtime, an empty directory will be created at the referenced path.
An example of a pod which will error out:
```yaml
apiVersion: v1
kind: Pod
metadata:
labels:
name: mount-dne
name: mount-dne
spec:
volumes:
- name: does-not-exist
hostPath:
path: /does/not/exist
containers:
- name: exit
image: busybox
command: ["sh", "-c", "ls /test; sleep 60"]
volumeMounts:
- mountPath: /test
name: does-not-exist
```
Also note that if `subPath` is specified in the container's volumeMounts and the `subPath` doesn't exist in the corresponding volume, the pod execution will fail as well.
## Kubectl attach
The `kubectl attach` command does not work under the rkt container runtime.
Because of this, some flags in `kubectl run` are not supported, including:
* `--attach=true`
* `--leave-stdin-open=true`
* `--rm=true`
## Port forwarding for kvm and fly stage1s
`kubectl port-forward` is not supported for pods that are executed with `stage1-kvm` or `stage1-fly`.
## Volume relabeling
Currently rkt supports only *per-pod* volume relabeling. After relabeling, the mounted volume is shared by all Containers in the pod. There is not yet a way to make the relabeled volume accessible to only one, or some subset, of Containers in the pod. [Kubernetes issue # 28187](https://github.com/kubernetes/kubernetes/issues/28187) has the details.
## kubectl get logs
Under rktnetes, `kubectl get logs` currently cannot get logs from applications that write them to directly to `/dev/stdout`. Currently such log messages are printed on the node's console.
## Init Containers
[Init Containers](/docs/concepts/workloads/pods/init-containers) are currently not supported.
## Container restart back-off
Exponential restart back-off for a failing container is currently not supported.
## Experimental NVIDIA GPU support
The `--feature-gates="Accelerators=true"` flag, and related [GPU features](https://git.k8s.io/community/contributors/design-proposals/resource-management/gpu-support.md) are not supported.
## QoS Classes
Under rkt, QoS classes do not adjust the `OOM Score` of Containers as occurs under Docker.
## HostPID and HostIPC namespaces
Setting the hostPID or hostIPC flags on a pod is not supported.
For example, the following pod will not run correctly:
```yaml
apiVersion: v1
kind: Pod
metadata:
labels:
name: host-ipc-pid
name: host-ipc-pid
spec:
hostIPC: true
hostPID: true
containers:
...
```
On the other hand, when running the pod with [stage1-fly](https://coreos.com/rkt/docs/latest/running-fly-stage1.html), the pod will be run in the host namespace.
## Container image updates (patch)
Patching a pod to change the image will result in the entire pod restarting, not just the container that was changed.
## ImagePullPolicy 'Always'
When the container's image pull policy is `Always`, rkt will always pull the image from remote even if the image has not changed at all.
This can add significant latency for large images.
The issue is tracked by rkt upstream at [#2937](https://github.com/coreos/rkt/issues/2937).
docs/getting-started-guides/scratch.md
+4 -37
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- erictune
- lavalamp
- thockin
@@ -91,12 +91,12 @@ to implement one of the above options:
- You can also write your own.
- **Compile support directly into Kubernetes**
- This can be done by implementing the "Routes" interface of a Cloud Provider module.
- The Google Compute Engine ([GCE](/docs/getting-started-guides/gce/)/) and [AWS](/docs/getting-started-guides/aws/) guides use this approach.
- The Google Compute Engine ([GCE](/docs/getting-started-guides/gce/)) and [AWS](/docs/getting-started-guides/aws/) guides use this approach.
- **Configure the network external to Kubernetes**
- This can be done by manually running commands, or through a set of externally maintained scripts.
- You have to implement this yourself, but it can give you an extra degree of flexibility.
You will need to select an address range for the Pod IPs. Note that IPv6 is not yet supported for Pod IPs.
You will need to select an address range for the Pod IPs.
- Various approaches:
- GCE: each project has its own `10.0.0.0/8`. Carve off a `/16` for each
@@ -116,7 +116,7 @@ You will need to select an address range for the Pod IPs. Note that IPv6 is not
Kubernetes also allocates an IP to each [service](/docs/concepts/services-networking/service/). However,
service IPs do not necessarily need to be routable. The kube-proxy takes care
of translating Service IPs to Pod IPs before traffic leaves the node. You do
need to Allocate a block of IPs for services. Call this
need to allocate a block of IPs for services. Call this
`SERVICE_CLUSTER_IP_RANGE`. For example, you could set
`SERVICE_CLUSTER_IP_RANGE="10.0.0.0/16"`, allowing 65534 distinct services to
be active at once. Note that you can grow the end of this range, but you
@@ -405,7 +405,6 @@ Arguments to consider:
- `--docker-root=`
- `--root-dir=`
- `--pod-cidr=` The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master.
- `--configure-cbr0=` (described below)
- `--register-node` (described in [Node](/docs/admin/node/) documentation.)
### kube-proxy
@@ -441,38 +440,6 @@ this `NODE_X_BRIDGE_ADDR`. For example, if `NODE_X_POD_CIDR` is `10.0.0.0/16`,
then `NODE_X_BRIDGE_ADDR` is `10.0.0.1/16`. NOTE: this retains the `/16` suffix
because of how this is used later.
- Recommended, automatic approach:
1. Set `--configure-cbr0=true` option in kubelet init script and restart kubelet service. Kubelet will configure cbr0 automatically.
It will wait to do this until the node controller has set Node.Spec.PodCIDR. Since you have not setup apiserver and node controller
yet, the bridge will not be setup immediately.
- Alternate, manual approach:
1. Set `--configure-cbr0=false` on kubelet and restart.
1. Create a bridge.
```
ip link add name cbr0 type bridge
```
1. Set appropriate MTU. NOTE: the actual value of MTU will depend on your network environment
```
ip link set dev cbr0 mtu 1460
```
1. Add the node's network to the bridge (docker will go on other side of bridge).
```
ip addr add $NODE_X_BRIDGE_ADDR dev cbr0
```
1. Turn it on
```
ip link set dev cbr0 up
```
If you have turned off Docker's IP masquerading to allow pods to talk to each
other, then you may need to do masquerading just for destination IPs outside
the cluster network. For example:
docs/getting-started-guides/stackpoint.md
+1 -1
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- baldwinspc
title: Running Kubernetes on Multiple Clouds with Stackpoint.io
---
docs/getting-started-guides/ubuntu/backups.md
+2 -3
View File
@@ -21,10 +21,9 @@ The `snapshot` action of the etcd charm allows the operator to snapshot
a running cluster's data for use in cloning,
backing up, or migrating to a new cluster.
juju run-action etcd/0 snapshot target=/mnt/etcd-backups
- **param** target: destination directory to save the resulting snapshot archive.
juju run-action etcd/0 snapshot
This will create a snapshot in `/home/ubuntu/etcd-snapshots` by default.
## Restore etcd data
docs/getting-started-guides/ubuntu/glossary.md
+2 -2
View File
@@ -13,9 +13,9 @@ This page explains some of the terminology used in deploying Kubernetes with Juj
**model** - A collection of charms and their relationships that define a deployment. This includes machines and units. A controller can host multiple models. It is recommended to separate Kubernetes clusters into individual models for management and isolation reasons.
**charm** - The definition of a service, including its metadata, dependencies with other services, required packages, and application management logic. It contains all the operational knowledge of deploying a Kubernetes cluster. Included charm examples are `kubernetes-core`, `easy-rsa`, `kibana`, and `etcd`.
**charm** - The definition of a service, including its metadata, dependencies with other services, required packages, and application management logic. It contains all the operational knowledge of deploying a Kubernetes cluster. Included charm examples are `kubernetes-core`, `easyrsa`, `flannel`, and `etcd`.
**unit** - A given instance of a service. These may or may not use up a whole machine, and may be colocated on the same machine. So for example you might have a `kubernetes-worker`, and `filebeat`, and `topbeat` units running on a single machine, but they are three distinct units of different services.
**unit** - A given instance of a service. These may or may not use up a whole machine, and may be colocated on the same machine. So for example you might have a `kubernetes-worker`, and `etcd`, and `easyrsa` units running on a single machine, but they are three distinct units of different services.
**machine** - A physical node, these can either be bare metal nodes, or virtual machines provided by a cloud.
{% endcapture %}
docs/getting-started-guides/ubuntu/index.md
+8 -1
View File
@@ -11,7 +11,7 @@ There are multiple ways to run a Kubernetes cluster with Ubuntu. These pages exp
- [The Canonical Distribution of Kubernetes](https://www.ubuntu.com/cloud/kubernetes)
Supports AWS, GCE, Azure, Joyent, OpenStack, VMWare, Bare Metal and localhost deployments.
The latest version of Kubernetes with upstream binaries. Supports AWS, GCE, Azure, Joyent, OpenStack, VMware, Bare Metal and localhost deployments.
### Quick Start
@@ -51,6 +51,11 @@ These are more in-depth guides for users choosing to run Kubernetes in productio
- [Operational Considerations](/docs/getting-started-guides/ubuntu/operational-considerations/)
- [Glossary](/docs/getting-started-guides/ubuntu/glossary/)
## Third-party Product Integrations
- [Rancher](/docs/getting-started-guides/ubuntu/rancher/)
## Developer Guides
- [Localhost using LXD](/docs/getting-started-guides/ubuntu/local/)
@@ -59,6 +64,8 @@ These are more in-depth guides for users choosing to run Kubernetes in productio
We're normally following the following Slack channels:
- [kubernetes-users](https://kubernetes.slack.com/messages/kubernetes-users/)
- [kubernetes-novice](https://kubernetes.slack.com/messages/kubernetes-novice/)
- [sig-cluster-lifecycle](https://kubernetes.slack.com/messages/sig-cluster-lifecycle/)
- [sig-cluster-ops](https://kubernetes.slack.com/messages/sig-cluster-ops/)
- [sig-onprem](https://kubernetes.slack.com/messages/sig-onprem/)
docs/getting-started-guides/ubuntu/installation.md
+9 -3
View File
@@ -1,5 +1,5 @@
---
approvers:
reviewers:
- caesarxuchao
- erictune
title: Setting up Kubernetes with Juju
@@ -54,7 +54,7 @@ Microsoft Azure | Juju | Ubuntu | flannel | [docs](/docs
Google Compute Engine (GCE) | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Joyent | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Rackspace | Juju | Ubuntu | flannel | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
VMWare vSphere | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
VMware vSphere | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
Bare Metal (MAAS) | Juju | Ubuntu | flannel, calico | [docs](/docs/getting-started-guides/ubuntu) | | [Commercial](https://ubuntu.com/cloud/kubernetes), [Community](https://github.com/juju-solutions/bundle-kubernetes-core)
@@ -90,9 +90,15 @@ juju bootstrap aws/us-east-2
or, another example, this time on Azure:
```
juju bootstrap azure/centralus
juju bootstrap azure/westus2
```
If you receive this error, it is likely that the default Azure VM size (Standard D1 v2 [1 vcpu, 3.5 GB memory]) is not available in the Azure location:
```
ERROR failed to bootstrap model: instance provisioning failed (Failed)
```
You will need a controller node for each cloud or region you are deploying to. See the [controller documentation](https://jujucharms.com/docs/2.2/controllers) for more information.
Note that each controller can host multiple Kubernetes clusters in a given cloud or region.
docs/getting-started-guides/ubuntu/logging.md
+2 -2
View File
@@ -26,10 +26,10 @@ Log verbosity in Juju is set at the model level. You can adjust it at any time:
juju add-model k8s-development --config logging-config='<root>=DEBUG;unit=DEBUG'
```
and later
and later on your k8s-production model
```
juju config-model k8s-production --config logging-config='<root>=ERROR;unit=ERROR'
juju model-config -m k8s-production logging-config='<root>=ERROR;unit=ERROR'
```
In addition, the jujud daemon is started in debug mode by default on all controllers. To remove that behavior edit ```/var/lib/juju/init/jujud-machine-0/exec-start.sh``` on the controller node and comment the ```--debug``` section.
docs/getting-started-guides/ubuntu/networking.md
+20 -11
View File
@@ -5,34 +5,43 @@ title: Networking
{% capture overview %}
Kubernetes supports the [Container Network Interface (CNI)](https://github.com/containernetworking/cni).
This is a network plugin architecture that allows you to use whatever
Kubernetes-friendly SDN you want. Currently this means support for Flannel.
Kubernetes-friendly SDN you want. Currently this means support for Flannel and Canal.
This page shows how to the various network portions of a cluster work, and how to configure them.
This page shows how the various network portions of a cluster work and how to configure them.
{% endcapture %}
{% capture prerequisites %}
This page assumes you have a working Juju deployed cluster.
**Note:** Note that if you deploy a cluster via conjure-up or the CDK bundles, manually deploying CNI plugins is unnecessary.
{: .note}
{% endcapture %}
{% capture steps %}
## Flannel
The CNI charms are [subordinates](https://jujucharms.com/docs/stable/authors-subordinate-applications).
These charms will require a principal charm that implements the `kubernetes-cni` interface in order to properly deploy.
The flannel charm is a
[subordinate](https://jujucharms.com/docs/stable/authors-subordinate-applications).
This charm will require a principal charm that implements the `kubernetes-cni`
interface in order to properly deploy.
## Flannel
```
juju deploy flannel
juju deploy etcd
juju deploy kubernetes-master
juju add-relation flannel kubernetes-master
juju add-relation flannel kubernetes-worker
juju add-relation flannel etcd
```
## Canal
```
juju deploy canal
juju add-relation canal kubernetes-master
juju add-relation canal kubernetes-worker
juju add-relation canal etcd
```
### Configuration
**iface** The interface to configure the flannel SDN binding. If this value is
**iface** The interface to configure the flannel or canal SDN binding. If this value is
empty string or undefined the code will attempt to find the default network
adapter similar to the following command:
@@ -40,7 +49,7 @@ adapter similar to the following command:
$ route | grep default | head -n 1 | awk {'print $8'}
```
**cidr** The network range to configure the flannel SDN to declare when
**cidr** The network range to configure the flannel or canal SDN to declare when
establishing networking setup with etcd. Ensure this network range is not active
on layers 2/3 you're deploying to, as it will cause collisions and odd behavior
if care is not taken when selecting a good CIDR range to assign to flannel. It's
docs/getting-started-guides/ubuntu/operational-considerations.md
+2 -2
View File
@@ -115,8 +115,8 @@ juju switch default
### Running privileged containers
By default, juju-deployed clusters do not support running privileged containers.
If you need them, you have to enable the ```allow-privileged``` config on both
By default, juju-deployed clusters only allow running privileged containers on nodes with GPUs.
If you need privileged containers on other nodes, you have to enable the ```allow-privileged``` config on both
kubernetes-master and kubernetes-worker:
```
docs/getting-started-guides/ubuntu/rancher.md
+360
View File
@@ -0,0 +1,360 @@
---
title: Rancher Integration with Ubuntu Kubernetes
---
{% capture overview %}
This repository explains how to deploy Rancher 2.0alpha on Canonical Kubernetes.
These steps are currently in alpha/testing phase and will most likely change.
The original documentation for this integration can be found at [https://github.com/CalvinHartwell/canonical-kubernetes-rancher/](https://github.com/CalvinHartwell/canonical-kubernetes-rancher/).
{% endcapture %}
{% capture prerequisites %}
To use this guide, you must have a working kubernetes cluster that was deployed using Canonical's juju.
The full instructions for deploying Kubernetes with juju can be found at [https://kubernetes.io/docs/getting-started-guides/ubuntu/installation/](https://kubernetes.io/docs/getting-started-guides/ubuntu/installation/).
{% endcapture %}
{% capture steps %}
## Deploying Rancher
To deploy Rancher, we just need to run the Rancher container workload on-top of Kubernetes. Rancher provides their containers through dockerhub ([https://hub.docker.com/r/rancher/server/tags/](https://hub.docker.com/r/rancher/server/tags/)) and can be downloaded freely from the internet.
If you're running your own registry or have an offline deployment, the container should be downloaded and pushed to a private registry before proceeding.
### Deploying Rancher with a nodeport
First create a yaml file which defines how to deploy Rancher on kubernetes. Save the file as cdk-rancher-nodeport.yaml:
```
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-admin
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: rancher
name: rancher
spec:
replicas: 1
selector:
matchLabels:
app: rancher
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: rancher
ima: pod
spec:
containers:
- image: rancher/server:preview
imagePullPolicy: Always
name: rancher
ports:
- containerPort: 80
- containerPort: 443
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: rancher
labels:
app: rancher
spec:
ports:
- port: 443
protocol: TCP
targetPort: 443
selector:
app: rancher
---
apiVersion: v1
kind: Service
metadata:
name: rancher-nodeport
spec:
type: NodePort
selector:
app: rancher
ports:
- name: rancher-api
protocol: TCP
nodePort: 30443
port: 443
targetPort: 443
```
Once kubectl is running and working, run the following command to deploy Rancher:
```
kubectl apply -f cdk-rancher-nodeport.yaml
```
Now we need to open this nodeport so we can access it. For that, we can use juju. We need to run the open-port command for each of the worker nodes in our cluster. Inside the cdk-rancher-nodeport.yaml file, the nodeport has been set to 30443. Below shows how to open the port on each of the worker nodes:
```
# repeat this for each kubernetes worker in the cluster.
juju run --unit kubernetes-worker/0 "open-port 30443"
juju run --unit kubernetes-worker/1 "open-port 30443"
juju run --unit kubernetes-worker/2 "open-port 30443"
```
Rancher can now be accessed on this port through a worker IP or DNS entries if you have created them. It is generally recommended that you create a DNS entry for each of the worker nodes in your cluster. For example, if you have three worker nodes and you own the domain example.com, you could create three A records, one for each worker in the cluster.
As creating DNS entries is outside of the scope of this document, we will use the freely available xip.io service which can return A records for an IP address which is part of the domain name. For example, if you have the domain rancher.35.178.130.245.xip.io, the xip.io service will automatically return the IP address 35.178.130.245 as an A record which is useful for testing purposes. For your deployment, the IP address 35.178.130.245 should be replaced with one of your worker IP address, which can be found using Juju or AWS:
```
calvinh@ubuntu-ws:~/Source/cdk-rancher$ juju status
# ... output omitted.
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 35.178.118.232 Certificate Authority connected.
etcd/0* active idle 1 35.178.49.31 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 35.177.99.171 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 35.178.125.161 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 4 35.178.37.87 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 5 35.177.239.237 6443/tcp Kubernetes master running.
flannel/0* active idle 35.177.239.237 Flannel subnet 10.1.27.1/24
kubernetes-worker/0* active idle 6 35.178.130.245 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/2 active idle 35.178.130.245 Flannel subnet 10.1.82.1/24
kubernetes-worker/1 active idle 7 35.178.121.29 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/3 active idle 35.178.121.29 Flannel subnet 10.1.66.1/24
kubernetes-worker/2 active idle 8 35.177.144.76 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/1 active idle 35.177.144.76
# Note the IP addresses for the kubernetes-workers in the example above. You should pick one of the public addresses.
```
Try opening up Rancher in your browser using the nodeport and the domain name or ip address:
```
# replace the IP address with one of your Kubernetes worker, find this from juju status command.
wget https://35.178.130.245.xip.io:30443 --no-check-certificate
# this should also work
wget https://35.178.130.245:30443 --no-check-certificate
```
If you need to make any changes to the kubernetes configuration file, edit the yaml file and then just use apply again:
```
kubectl apply -f cdk-rancher-nodeport.yaml
```
### Deploying Rancher with an ingress rule
It is also possible to deploy Rancher using an ingress rule. This has the added benefit of not requiring additional ports to be opened up on the Kubernetes cluster. First create a yaml file to describe the deployment called cdk-rancher-ingress.yaml which should contain the following:
```
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-admin
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-admin
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: rancher
name: rancher
spec:
replicas: 1
selector:
matchLabels:
app: rancher
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: rancher
spec:
containers:
- image: rancher/server:preview
imagePullPolicy: Always
name: rancher
ports:
- containerPort: 443
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: rancher
labels:
app: rancher
spec:
ports:
- port: 443
targetPort: 443
protocol: TCP
selector:
app: rancher
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: rancher
annotations:
kubernetes.io/tls-acme: "true"
ingress.kubernetes.io/secure-backends: "true"
spec:
tls:
- hosts:
- rancher.34.244.118.135.xip.io
rules:
- host: rancher.34.244.118.135.xip.io
http:
paths:
- path: /
backend:
serviceName: rancher
servicePort: 443
```
It is generally recommended that you create a DNS entry for each of the worker nodes in your cluster. For example, if you have three worker nodes and you own the domain example.com, you could create three A records, one for each worker in the cluster.
As creating DNS entries is outside of the scope of this tutorial, we will use the freely available xip.io service which can return A records for an IP address which is part of the domain name. For example, if you have the domain rancher.35.178.130.245.xip.io, the xip.io service will automatically return the IP address 35.178.130.245 as an A record which is useful for testing purposes.
For your deployment, the IP address 35.178.130.245 should be replaced with one of your worker IP address, which can be found using Juju or AWS:
```
calvinh@ubuntu-ws:~/Source/cdk-rancher$ juju status
# ... output omitted.
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 35.178.118.232 Certificate Authority connected.
etcd/0* active idle 1 35.178.49.31 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 35.177.99.171 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 35.178.125.161 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 4 35.178.37.87 443/tcp Loadbalancer ready.
kubernetes-master/0* active idle 5 35.177.239.237 6443/tcp Kubernetes master running.
flannel/0* active idle 35.177.239.237 Flannel subnet 10.1.27.1/24
kubernetes-worker/0* active idle 6 35.178.130.245 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/2 active idle 35.178.130.245 Flannel subnet 10.1.82.1/24
kubernetes-worker/1 active idle 7 35.178.121.29 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/3 active idle 35.178.121.29 Flannel subnet 10.1.66.1/24
kubernetes-worker/2 active idle 8 35.177.144.76 80/tcp,443/tcp,30443/tcp Kubernetes worker running.
flannel/1 active idle 35.177.144.76
# Note the IP addresses for the kubernetes-workers in the example above. You should pick one of the public addresses.
```
Looking at the output from the juju status above, the Public Address (35.178.130.245) can be used to create a xip.io DNS entry (rancher.35.178.130.245.xip.io) which should be placed into the cdk-rancher-ingress.yaml file. You could also create your own DNS entry as long as it resolves to each of the worker nodes or one of them it will work fine:
```
# The xip.io domain should appear in two places in the file, change both entries.
cat cdk-rancher-ingress.yaml | grep xip.io
- host: rancher.35.178.130.245.xip.io
```
Once you've edited the ingress rule to reflect your DNS entries, run the kubectl apply -f cdk-rancher-ingress.yaml to deploy Kubernetes:
```
kubectl apply -f cdk-rancher-ingress.yaml
```
Rancher can now be accessed on the regular 443 through a worker IP or DNS entries if you have created them. Try opening it up in your browser:
```
# replace the IP address with one of your Kubernetes worker, find this from juju status command.
wget https://35.178.130.245.xip.io:443 --no-check-certificate
```
If you need to make any changes to the kubernetes configuration file, edit the yaml file and then just use apply again:
```
kubectl apply -f cdk-rancher-ingress.yaml
```
### Removing Rancher
You can remove Rancher from your cluster using kubectl. Deleting constructs in Kubernetes is as simple as creating them:
```
# If you used the nodeport example change the yaml filename if you used the ingress example.
kubectl delete -f cdk-rancher-nodeport.yaml
```
{% endcapture %}
{% include templates/task.md %}
docs/getting-started-guides/ubuntu/troubleshooting.md
+6 -36
View File
@@ -46,7 +46,7 @@ During normal operation the Workload should read `active`, the Agent column (whi
Status can become unwieldy for large clusters, it is then recommended to check status on individual services, for example to check the status on the workers only:
juju status kubernetes-workers
juju status kubernetes-worker
or just on the etcd cluster:
@@ -68,41 +68,11 @@ This will automatically ssh you to the easyrsa unit.
## Collecting debug information
Sometimes it is useful to collect all the information from a node to share with a developer so problems can be identifying. This section will deal on how to use the debug action to collect this information. The debug action is only supported on `kubernetes-worker` nodes.
Sometimes it is useful to collect all the information from a cluster to share with a developer to identify problems. This is best accomplished with [CDK Field Agent](https://github.com/juju-solutions/cdk-field-agent).
juju run-action kubernetes-worker/0 debug
Download and execute the collect.py script from [CDK Field Agent](https://github.com/juju-solutions/cdk-field-agent) on a box that has a Juju client configured with the current controller and model pointing at the CDK deployment of interest.
Which returns:
```
Action queued with id: 4b26e339-7366-4dc7-80ed-255ac0377020`
```
This produces a .tar.gz file which you can retrieve:
juju show-action-output 4b26e339-7366-4dc7-80ed-255ac0377020
This will give you the path for the debug results:
```
results:
command: juju scp debug-test/0:/home/ubuntu/debug-20161110151539.tar.gz .
path: /home/ubuntu/debug-20161110151539.tar.gz
status: completed
timing:
completed: 2016-11-10 15:15:41 +0000 UTC
enqueued: 2016-11-10 15:15:38 +0000 UTC
started: 2016-11-10 15:15:40 +0000 UTC
```
You can now copy the results to your local machine:
juju scp kubernetes-worker/0:/home/ubuntu/debug-20161110151539.tar.gz .
The archive includes basic information such as systemctl status, Juju logs,
charm unit data, etc. Additional application-specific information may be
included as well.
Running the script will generate a tarball of system information and includes basic information such as systemctl status, Juju logs, charm unit data, etc. Additional application-specific information may be included as well.
## Common Problems
@@ -190,7 +160,7 @@ This is caused by the API load balancer not forwarding ports in the context of t
Note that the default port used by CDK for the Kubernetes Master API is 6443 while the port exposed by the load balancer is 443.
1. Start helming again!
1. Start helm again!
```
helm install <chart> --debug
@@ -204,7 +174,7 @@ This is caused by the API load balancer not forwarding ports in the context of t
## Logging and monitoring
By default there is no log aggregation of the Kubernetes nodes, each node logs locally. It is recommended to deploy the Elastic Stack for log aggregation if you desire centralized logging.
By default there is no log aggregation of the Kubernetes nodes, each node logs locally. Please read over the [logging](https://kubernetes.io/docs/getting-started-guides/ubuntu/logging/) page for more information.
{% endcapture %}
{% include templates/task.md %}
docs/getting-started-guides/ubuntu/upgrades.md
+19 -23
View File
@@ -3,7 +3,7 @@ title: Upgrades
---
{% capture overview %}
This page will outline how to manage and execute a Kubernetes upgrade.
This page will outline how to manage and execute a Kubernetes upgrade.
{% endcapture %}
{% capture prerequisites %}
@@ -17,11 +17,11 @@ Refer to the [backup documentation](/docs/getting-started-guides/ubuntu/backups)
{% endcapture %}
{% capture steps %}
## Patch kubernetes upgrades eg 1.7.0 -> 1.7.1
## Patch kubernetes upgrades for example 1.9.0 -> 1.9.1
Clusters are transparently upgraded to the latest Kubernetes patch release.
To be clear, a cluster deployed using the 1.7/stable channel
will transparently receive unattended upgrades for the 1.7.X Kubernetes
To be clear, a cluster deployed using the 1.9/stable channel
will transparently receive unattended upgrades for the 1.9.X Kubernetes
releases.
The upgrade causes no disruption to the operation of the cluster and requires
no intervention from a cluster administrator.
@@ -31,32 +31,28 @@ Once a patch release passes internal testing and is deemed safe for upgrade,
it is packaged in snap format and pushed to the stable channel.
## Upgrading a minor Kubernetes release eg 1.7.1 -> 1.8.0
## Upgrading a minor Kubernetes release for example 1.8.1 -> 1.9.0
The Kubernetes charms follow the Kubernetes releases. Please consult
your support plan on the upgrade frequency. Important operational considerations
and changes in behaviour will always be documented in the release notes.
You can use `juju status` to see if an upgrade is available.
There may be an upgrade available for kubernetes, ectd, or both.
### Upgrade etcd
Backing up etcd requires an export and snapshot, refer to the
[backup documentation](/docs/getting-started-guides/ubuntu/backups) to create a snapshot.
After the snapshot upgrade the etcd service with:
After the snapshot, upgrade the etcd service with:
juju upgrade-charm etcd
This will handle upgrades between minor versions of etcd. Major upgrades from
etcd 2.x to 3.x are currently unsupported. Instead, data will be run in etcdv2 stores over the etcdv3 api.
This will handle upgrades between minor versions of etcd. Instructions on how to upgrade from 2.x to 3.x can be found [here](https://github.com/juju-solutions/bundle-canonical-kubernetes/wiki/Etcd-2.3-to-3.x-upgrade) in the juju-solutions wiki.
### Upgrade Kubernetes
The Kubernetes Charms use snap channels to drive payloads.
The channels are defined by `X.Y/channel` where `X.Y` is the `major.minor` release
of Kubernetes (e.g. 1.6) and `channel` is one of the four following channels:
of Kubernetes (for example 1.9) and `channel` is one of the four following channels:
| Channel name | Description |
| ------------------- | ------------ |
@@ -66,24 +62,24 @@ of Kubernetes (e.g. 1.6) and `channel` is one of the four following channels:
| edge | Nightly builds of that minor release of Kubernetes |
If a release isn't available, the next highest channel is used.
For example, 1.6/beta will load `/candidate` or `/stable` depending on availability of release.
For example, 1.9/beta will load `/candidate` or `/stable` depending on availability of release.
Development versions of Kubernetes are available in the edge channel for each minor release.
There is no guarantee that edge snaps will work with the current charms.
### Master Upgrades
First you need to upgrade the masters:
First you need to upgrade the masters:
juju upgrade-charm kubernetes-master
**Node:** Always upgrade the masters before the workers.
**Note:** Always upgrade the masters before the workers.
{: .note}
Once the latest charm is deployed, the channel for Kubernetes can be selected by issuing the following:
juju config kubernetes-master channel=1.x/stable
Where `x` is the minor version of Kubernetes. For example, `1.6/stable`. See above for Channel definitions.
Where `x` is the minor version of Kubernetes. For example, `1.9/stable`. See above for Channel definitions.
Once you've configured kubernetes-master with the appropriate channel, run the upgrade action on each master:
juju run-action kubernetes-master/0 upgrade
@@ -102,19 +98,19 @@ but is a safer upgrade route.
Given a deployment where the workers are named kubernetes-alpha.
Deploy new worker(s):
Deploy new workers:
juju deploy kubernetes-beta
Pause the old workers so your workload migrates:
Pause the old workers so your workload migrates:
juju run-action kubernetes-alpha/# pause
Verify old workloads have migrated with:
Verify old workloads have migrated with:
kubectl get pod -o wide
Tear down old workers with:
Tear down old workers with:
juju remove-application kubernetes-alpha
@@ -123,7 +119,7 @@ Tear down old workers with:
juju upgrade-charm kubernetes-worker
juju config kubernetes-worker channel=1.x/stable
Where `x` is the minor version of Kubernetes. For example, `1.6/stable`.
Where `x` is the minor version of Kubernetes. For example, `1.9/stable`.
See above for Channel definitions. Once you've configured kubernetes-worker with the appropriate channel,
run the upgrade action on each worker:
@@ -133,7 +129,7 @@ run the upgrade action on each worker:
### Verify upgrade
`kubectl version` should return the newer version.
`kubectl version` should return the newer version.
It is recommended to rerun a [cluster validation](/docs/getting-started-guides/ubuntu/validation)
to ensure that the cluster upgrade has successfully completed.
@@ -141,7 +137,7 @@ to ensure that the cluster upgrade has successfully completed.
### Upgrade Flannel
Upgrading flannel can be done at any time, it is independent of Kubernetes upgrades.
Be advised that networking is interrupted during the upgrade. You can initiate a flannel upgrade:
Be advised that networking is interrupted during the upgrade. You can initiate a flannel upgrade with:
juju upgrade-charm flannel
docs/getting-started-guides/ubuntu/validation.md
+2
View File
@@ -25,6 +25,8 @@ The primary objectives of the e2e tests are to ensure a consistent and reliable
behavior of the kubernetes code base, and to catch hard-to-test bugs before
users do, when unit and integration tests are insufficient.
End-to-end tests will pass on a properly running CDK cluster outside of bugs in the tests.
### Deploy kubernetes-e2e charm
To deploy the end-to-end test suite, you need to relate the `kubernetes-e2e` charm
docs/getting-started-guides/vsphere.md
-211
View File
@@ -1,211 +0,0 @@
---
approvers:
- erictune
- jbeda
title: VMware vSphere
---
This page covers how to get started with deploying Kubernetes on vSphere and details for how to configure the vSphere Cloud Provider.
* TOC
{:toc}
### Getting started with the vSphere Cloud Provider
Kubernetes comes with *vSphere Cloud Provider*, a cloud provider for vSphere that allows Kubernetes Pods to use vSphere Storage.
### Deploy Kubernetes on vSphere
To deploy Kubernetes on vSphere and use the vSphere Cloud Provider, see [Kubernetes-Anywhere](https://github.com/kubernetes/kubernetes-anywhere).
Detailed steps can be found at the [getting started with Kubernetes-Anywhere on vSphere](https://git.k8s.io/kubernetes-anywhere/phase1/vsphere/README.md) page.
### vSphere Cloud Provider
vSphere Cloud Provider allows Kubernetes to use vSphere-managed storage. It supports:
- Services such as de-duplication and encryption with vSAN, QoS, high availability and data reliability.
- Policy based management at granularity of container volumes.
- Volumes, Persistent Volumes, Storage Classes, dynamic provisioning of volumes, and scalable deployment of Stateful Apps with StatefulSets.
For more detail visit [vSphere Storage for Kubernetes Documentation](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/index.html).
Documentation for how to use vSphere managed storage can be found in the [persistent volumes user guide](/docs/concepts/storage/persistent-volumes/#vsphere) and the [volumes user guide](/docs/concepts/storage/volumes/#vspherevolume).
Examples can be found [here](https://github.com/kubernetes/examples/tree/master/staging/volumes/vsphere).
#### Enable vSphere Cloud Provider
If a Kubernetes cluster has not been deployed using Kubernetes-Anywhere, follow the instructions below to enable the vSphere Cloud Provider. These steps are not needed when using Kubernetes-Anywhere, they will be done as part of the deployment.
**Step-1** [Create a VM folder](https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.vcenterhost.doc/GUID-031BDB12-D3B2-4E2D-80E6-604F304B4D0C.html) and move Kubernetes Node VMs to this folder.
**Step-2** Make sure Node VM names must comply with the regex `[a-z](([-0-9a-z]+)?[0-9a-z])?(\.[a-z0-9](([-0-9a-z]+)?[0-9a-z])?)*`. If Node VMs do not comply with this regex, rename them and make it compliant to this regex.
Node VM names constraints:
* VM names can not begin with numbers.
* VM names can not have capital letters, any special characters except `.` and `-`.
* VM names can not be shorter than 3 chars and longer than 63.
**Step-3** Enable disk UUID on Node virtual machines.
The disk.EnableUUID parameter must be set to "TRUE" for each Node VM. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly.
For each of the virtual machine nodes that will be participating in the cluster, follow the steps below using [govc tool](https://github.com/vmware/govmomi/tree/master/govc)
* Set up the **govc** environment
export GOVC_URL='vCenter IP OR FQDN'
export GOVC_USERNAME='vCenter User'
export GOVC_PASSWORD='vCenter Password'
export GOVC_INSECURE=1
* Find Node VM Paths
govc ls /datacenter/vm/<vm-folder-name>
* Set disk.EnableUUID to true for all VMs
govc vm.change -e="disk.enableUUID=1" -vm='VM Path'
Note: If Kubernetes Node VMs are created from template VM then `disk.EnableUUID=1` can be set on the template VM. VMs cloned from this template, will automatically inherit this property.
**Step-4** Create and assign Roles to the vSphere Cloud Provider user and vSphere entities.
Note: if you want to use Administrator account then this step can be skipped.
vSphere Cloud Provider requires the following minimal set of privileges to interact with vCenter. Please refer [vSphere Documentation Center](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-18071E9A-EED1-4968-8D51-E0B4F526FDA3.html) to know about steps for creating a Custom Role, User and Role Assignment.
<table>
<thead>
<tr>
<th>Roles</th>
<th>Privileges</th>
<th>Entities</th>
<th>Propagate to Children</th>
</tr>
</thead>
<tbody><tr>
<td>manage-k8s-node-vms</td>
<td>Resource.AssignVMToPool<br> System.Anonymous<br> System.Read<br> System.View<br> VirtualMachine.Config.AddExistingDisk<br> VirtualMachine.Config.AddNewDisk<br> VirtualMachine.Config.AddRemoveDevice<br> VirtualMachine.Config.RemoveDisk<br> VirtualMachine.Inventory.Create<br> VirtualMachine.Inventory.Delete</td>
<td>Cluster,<br> Hosts,<br> VM Folder</td>
<td>Yes</td>
</tr>
<tr>
<td>manage-k8s-volumes</td>
<td>Datastore.AllocateSpace<br> Datastore.FileManagement<br> System.Anonymous<br> System.Read<br> System.View</td>
<td>Datastore</td>
<td>No</td>
</tr>
<tr>
<td>k8s-system-read-and-spbm-profile-view</td>
<td>StorageProfile.View<br> System.Anonymous<br> System.Read<br> System.View</td>
<td>vCenter</td>
<td>No</td>
</tr>
<tr>
<td>ReadOnly</td>
<td>System.Anonymous<br>System.Read<br>System.View</td>
<td>Datacenter,<br> Datastore Cluster,<br> Datastore Storage Folder</td>
<td>No</td>
</tr>
</tbody>
</table>
**Step-5** Create the vSphere cloud config file (`vsphere.conf`). Cloud config template can be found [here](https://github.com/kubernetes/kubernetes-anywhere/blob/master/phase1/vsphere/vsphere.conf).
This config file needs to be placed in the shared directory which should be accessible from kubelet container, controller-manager pod, and API server pod.
**`vsphere.conf` for master node:**
```
[Global]
user = "vCenter username for cloud provider"
password = "password"
server = "IP/FQDN for vCenter"
port = "443" #Optional
insecure-flag = "1" #set to 1 if the vCenter uses a self-signed cert
datacenter = "Datacenter name"
datastore = "Datastore name" #Datastore to use for provisioning volumes using storage classes/dynamic provisioning
working-dir = "vCenter VM folder path in which node VMs are located"
vm-name = "VM name of the Master Node" #Optional
vm-uuid = "UUID of the Node VM" # Optional
[Disk]
scsicontrollertype = pvscsi
```
Note: **```vm-name``` parameter is introduced in 1.6.4 release.** Both ```vm-uuid``` and ```vm-name``` are optional parameters. If ```vm-name``` is specified then ```vm-uuid``` is not used. If both are not specified then kubelet will get vm-uuid from `/sys/class/dmi/id/product_serial` and query vCenter to find the Node VM's name.
**`vsphere.conf` for worker nodes:**
Applicable only to versions 1.6.4 to 1.8.x. For versions earlier than 1.6.4, this file should have all the parameters specified in the master node's `vsphere.conf` file. In version 1.9.0 and later, the worker nodes do not need a cloud config file.
```
[Global]
vm-name = "VM name of the Worker Node"
```
Below is summary of supported parameters in the `vsphere.conf` file
* ```user``` is the vCenter username for vSphere Cloud Provider.
* ```password``` is the password for vCenter user specified with `user`.
* ```server``` is the vCenter Server IP or FQDN
* ```port``` is the vCenter Server Port. Default is 443 if not specified.
* ```insecure-flag``` is set to 1 if vCenter used a self-signed certificate.
* ```datacenter``` is the name of the datacenter on which Node VMs are deployed.
* ```datastore``` is the default datastore to use for provisioning volumes using storage classes/dynamic provisioning.
* ```vm-name``` is recently added configuration parameter. This is optional parameter. When this parameter is present, ```vsphere.conf``` file on the worker node does not need vCenter credentials.
**Note:** ```vm-name``` is added in the release 1.6.4. Prior releases does not support this parameter.
* ```working-dir``` can be set to empty ( working-dir = ""), if Node VMs are located in the root VM folder.
* ```vm-uuid``` is the VM Instance UUID of virtual machine. ```vm-uuid``` can be set to empty (```vm-uuid = ""```). If set to empty, this will be retrieved from /sys/class/dmi/id/product_serial file on virtual machine (requires root access).
* ```vm-uuid``` needs to be set in this format - ```423D7ADC-F7A9-F629-8454-CE9615C810F1```
* ```vm-uuid``` can be retrieved from Node Virtual machines using following command. This will be different on each node VM.
cat /sys/class/dmi/id/product_serial | sed -e 's/^VMware-//' -e 's/-/ /' | awk '{ print toupper($1$2$3$4 "-" $5$6 "-" $7$8 "-" $9$10 "-" $11$12$13$14$15$16) }'
* `datastore` is the default datastore used for provisioning volumes using storage classes. If datastore is located in storage folder or datastore is member of datastore cluster, make sure to specify full datastore path. Make sure vSphere Cloud Provider user has Read Privilege set on the datastore cluster or storage folder to be able to find datastore.
* For datastore located in the datastore cluster, specify datastore as mentioned below
datastore = "DatastoreCluster/datastore1"
* For datastore located in the storage folder, specify datastore as mentioned below
datastore = "DatastoreStorageFolder/datastore1"
**Step-6** Add flags to controller-manager, API server and Kubelet to enable vSphere Cloud Provider.
* Add following flags to kubelet running on every node and to the controller-manager and API server pods manifest files.
```
--cloud-provider=vsphere
--cloud-config=<Path of the vsphere.conf file>
```
Manifest files for API server and controller-manager are generally located at `/etc/kubernetes/manifests`.
**Step-7** Restart Kubelet on all nodes.
* Reload kubelet systemd unit file using ```systemctl daemon-reload```
* Restart kubelet service using ```systemctl restart kubelet.service```
Note: After enabling the vSphere Cloud Provider, Node names will be set to the VM names from the vCenter Inventory.
#### Known issues
Please visit [known issues](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/known-issues.html) for the list of major known issues with Kubernetes vSphere Cloud Provider.
## Support Level
For quick support please join VMware Code Slack ([kubernetes](https://vmwarecode.slack.com/messages/kubernetes/)) and post your question.
IaaS Provider | Config. Mgmt | OS | Networking | Docs | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | --------------------------------------------- | --------- | ----------------------------
Vmware vSphere | Kube-anywhere | Photon OS | Flannel | [docs](/docs/getting-started-guides/vsphere/) | | Community ([@abrarshivani](https://github.com/abrarshivani)), ([@kerneltime](https://github.com/kerneltime)), ([@BaluDontu](https://github.com/BaluDontu)), ([@luomiao](https://github.com/luomiao)), ([@divyenpatel](https://github.com/divyenpatel))
If you identify any issues/problems using the vSphere cloud provider, you can create an issue in our repo - [VMware Kubernetes](https://github.com/vmware/kubernetes).
For support level information on all solutions, see the [Table of solutions](/docs/getting-started-guides/#table-of-solutions) chart.
docs/getting-started-guides/windows/index.md
+74 -66
View File
@@ -16,35 +16,10 @@ The Kubernetes control plane (API Server, Scheduler, Controller Manager, etc) co
**Note:** Windows Server Containers on Kubernetes is a Beta feature in Kubernetes v1.9
{: .note}
## Build
We recommend using the release binaries that can be found at [https://github.com/kubernetes/kubernetes/releases](https://github.com/kubernetes/kubernetes/releases). Look for the Node Binaries section by visiting the binary downloads link.
## Get Windows Binaries
We recommend using the release binaries that can be found at [https://github.com/kubernetes/kubernetes/releases/latest](https://github.com/kubernetes/kubernetes/releases/latest). Under the CHANGELOG you can find the Node Binaries link for Windows-amd64, which will include kubeadm, kubectl, kubelet and kube-proxy.
If you wish to build the code yourself, please follow the next instructions:
1. Install the pre-requisites on a Linux host:
```
sudo apt-get install curl git build-essential docker.io conntrack
```
2. Run the following commands to build kubelet and kube-proxy:
```bash
K8SREPO="github.com/kubernetes/kubernetes"
go get -d $K8SREPO
# Note: the above command may spit out a message about
# "no Go files in...", but it can be safely ignored!
cd $GOPATH/src/k8s.io/kubernetes
# Build the kubelet
KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kubelet
# Build the kube-proxy
KUBE_BUILD_PLATFORMS=windows/amd64 make WHAT=cmd/kube-proxy
# You will find the output binaries under the folder _output/local/bin/windows/
```
More detailed build instructions will be maintained and kept up to date [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/compiling-kubernetes-binaries.md).
If you wish to build the code yourself, please refer to detailed build instructions [here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/compiling-kubernetes-binaries).
## Prerequisites
In Kubernetes version 1.9 or later, Windows Server Containers for Kubernetes are supported using the following:
@@ -77,9 +52,7 @@ Windows supports the CNI network model and uses plugins to interface with the Wi
#### Upstream L3 Routing Topology
In this topology, networking is achieved using L3 routing with static IP routes configured in an upstream Top of Rack (ToR) switch/router. Each cluster node is connected to the management network with a host IP. Additionally, each node uses a local 'l2bridge' network with a pod CIDR assigned. All pods on a given worker node will be connected to the pod CIDR subnet ('l2bridge' network). In order to enable network communication between pods running on different nodes, the upstream router has static routes configured with pod CIDR prefix => Host IP.
Each Window Server node should have the following configuration:
The following diagram illustrates the Windows Server networking setup for Kubernetes using Upstream L3 Routing Setup:
The following example diagram illustrates the Windows Server networking setup for Kubernetes using Upstream L3 Routing Setup:
![K8s Cluster using L3 Routing with ToR](UpstreamRouting.png)
#### Host-Gateway Topology
@@ -111,7 +84,7 @@ To run Windows Server Containers on Kubernetes, you'll need to set up both your
1. Windows Server container host running the required Windows Server and Docker versions. Follow the setup instructions outlined by this help topic: https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server.
2. [Build](#Build) or download kubelet.exe, kube-proxy.exe, and kubectl.exe using instructions
2. [Get Windows Binaries](#get-windows-binaries) kubelet.exe, kube-proxy.exe, and kubectl.exe using instructions
3. Copy Node spec file (kube config) from Linux master node with X.509 keys
4. Create the HNS Network, ensure the correct CNI network config, and start kubelet.exe using this script [start-kubelet.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubelet.ps1)
5. Start kube-proxy using this script [start-kubeproxy.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/start-kubeproxy.ps1)
@@ -120,7 +93,7 @@ To run Windows Server Containers on Kubernetes, you'll need to set up both your
More detailed instructions can be found [here](https://github.com/MicrosoftDocs/Virtualization-Documentation/blob/live/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows.md).
**Windows CNI Config Example**
Today, Windows CNI plugin is based on wincni.exe code with the following example, configuration file.
Today, Windows CNI plugin is based on wincni.exe code with the following example, configuration file. This is based on the ToR example diagram shown above, specifying the configuration to apply to Windows node-1. Of special interest is Windows node-1 pod CIDR (10.10.187.64/26) and the associated gateway of cbr0 (10.10.187.66). The exception list is specifying the Service CIDR (11.0.0.0/8), Cluster CIDR (10.10.0.0/16), and Management (or Host) CIDR (10.127.132.128/25).
Note: this file assumes that a user previous created 'l2bridge' host networks on each Windows node using `<Verb>-HNSNetwork` cmdlets as shown in the `start-kubelet.ps1` and `start-kubeproxy.ps1` scripts linked above
@@ -254,8 +227,23 @@ To start your cluster, you'll need to start both the Linux-based Kubernetes cont
## Starting the Linux-based Control Plane
Use your preferred method to start Kubernetes cluster on Linux. Please note that Cluster CIDR might need to be updated.
## Scheduling Pods on Windows
Because your cluster has both Linux and Windows nodes, you must explicitly set the nodeSelector constraint to be able to schedule pods to Windows nodes. You must set nodeSelector with the label beta.kubernetes.io/os to the value windows; see the following example:
## Support for kubeadm join
If your cluster has been created by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/),
and your networking is setup correctly using one of the methods listed above (networking is setup outside of kubeadm), you can use kubeadm to add a Windows node to your cluster. At a high level, you first have to initialize the master with kubeadm (Linux), then set up the CNI based networking (outside of kubeadm), and finally start joining Windows or Linux worker nodes to the cluster. For additional documentation and reference material, visit the kubeadm link above.
The kubeadm binary can be found at [Kubernetes Releases](https://github.com/kubernetes/kubernetes/releases), inside the node binaries archive. Adding a Windows node is not any different than adding a Linux node:
`kubeadm.exe join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>`
See [joining-your-nodes](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#44-joining-your-nodes) for more details.
## Supported Features
The examples listed below assume running Windows nodes on Windows Server 1709. If you are running Windows Server 2016, the examples will need the image updated to specify `image: microsoft/windowsservercore:ltsc2016`. This is due to the requirement for container images to match the host operating system version when using process isolation. Not specifying a tag will implicitly use the `:latest` tag which can lead to surprising behaviors. Please consult with [https://hub.docker.com/r/microsoft/windowsservercore/](https://hub.docker.com/r/microsoft/windowsservercore/) for additional information on Windows Server Core image tagging.
### Scheduling Pods on Windows
Because your cluster has both Linux and Windows nodes, you must explicitly set the `nodeSelector` constraint to be able to schedule pods to Windows nodes. You must set nodeSelector with the label `beta.kubernetes.io/os` to the value `windows`; see the following example:
```yaml
{
@@ -271,7 +259,7 @@ Because your cluster has both Linux and Windows nodes, you must explicitly set t
"containers": [
{
"name": "iis",
"image": "microsoft/iis",
"image": "microsoft/iis:windowsservercore-1709",
"ports": [
{
"containerPort": 80
@@ -285,18 +273,7 @@ Because your cluster has both Linux and Windows nodes, you must explicitly set t
}
}
```
## Support for kubeadm join
If your cluster has been created by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/),
and your networking is setup correctly using one of the methods listed above (networking is setup outside of kubeadm), you can use kubeadm to add a Windows node to your cluster. At a high level, you first have to initialize the master with kubeadm (Linux), then set up the CNI based networking (outside of kubeadm), and finally start joining Windows or Linux worker nodes to the cluster. For additional documentation and reference material, visit the kubeadm link above.
The kubeadm binary can be found at [Kubernetes Releases](https://github.com/kubernetes/kubernetes/releases), inside the node binaries archive. Adding a Windows node is not any different than adding a Linux node:
`kubeadm.exe join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>`
See [joining-your-nodes](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#44-joining-your-nodes) for more details.
## Supported Features
**Note:** this example assumes you are running on Windows Server 1709, so uses the image tag to support that. If you are on a different version, you will need to update the tag. For example, if on Windows Server 2016, update to use `"image": "microsoft/iis"` which will default to that OS version.
### Secrets and ConfigMaps
Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be used as environment variables. See limitations section below for additional details.
@@ -319,11 +296,11 @@ Secrets and ConfigMaps can be utilized in Windows Server Containers, but must be
apiVersion: v1
kind: Pod
metadata:
name: mypod-secret
name: my-secret-pod
spec:
containers:
- name: mypod-secret
image: redis:3.0-nanoserver
- name: my-secret-pod
image: microsoft/windowsservercore:1709
env:
- name: USERNAME
valueFrom:
@@ -355,11 +332,11 @@ data:
apiVersion: v1
kind: Pod
metadata:
name: configmap-pod
name: my-configmap-pod
spec:
containers:
- name: configmap-redis
image: redis:3.0-nanoserver
- name: my-configmap-pod
image: microsoft/windowsservercore:1709
env:
- name: EXAMPLE_PROPERTY_1
valueFrom:
@@ -387,19 +364,19 @@ Persistent Volume Claims are supported for supported volume types.
apiVersion: v1
kind: Pod
metadata:
name: hostpath-volume-pod
name: my-hostpath-volume-pod
spec:
containers:
- name: hostpath-redis
image: redis:3.0-nanoserver
- name: my-hostpath-volume-pod
image: microsoft/windowsservercore:1709
volumeMounts:
- name: blah
- name: foo
mountPath: "C:\\etc\\foo"
readOnly: true
nodeSelector:
beta.kubernetes.io/os: windows
volumes:
- name: blah
- name: foo
hostPath:
path: "C:\\etc\\foo"
```
@@ -410,11 +387,11 @@ Persistent Volume Claims are supported for supported volume types.
apiVersion: v1
kind: Pod
metadata:
name: empty-dir-pod
name: my-empty-dir-pod
spec:
containers:
- image: redis:3.0-nanoserver
name: empty-dir-redis
- image: microsoft/windowsservercore:1709
name: my-empty-dir-pod
volumeMounts:
- mountPath: /cache
name: cache-volume
@@ -428,19 +405,50 @@ Persistent Volume Claims are supported for supported volume types.
nodeSelector:
beta.kubernetes.io/os: windows
```
### DaemonSets
DaemonSets are supported
```yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: my-DaemonSet
labels:
app: foo
spec:
template:
metadata:
labels:
app: foo
spec:
containers:
- name: foo
image: microsoft/windowsservercore:1709
nodeSelector:
beta.kubernetes.io/os: windows
```
### Metrics
Windows Stats use a hybrid model: pod and container level stats come from CRI (via dockershim), while node level stats come from the "winstats" package that exports cadvisor like datastructures using windows specific perf counters from the node.
Windows Stats use a hybrid model: pod and container level stats come from CRI (via dockershim), while node level stats come from the "winstats" package that exports cadvisor like data structures using windows specific perf counters from the node.
## Known Limitations for Windows Server Containers with v1.9
Some of these limitations will be addressed by the community in future releases of Kubernetes
- Shared network namespace (compartment) with multiple Windows Server containers (shared kernel) per pod is only supported on Windows Server 1709 or later
- Using Secrets and ConfigMaps as volume mounts is not supported
- Mount propagation is not supported on Windows
- The StatefulSet functionality for stateful applications is not supported
- Horizontal Pod Autoscaling for Windows Server Container pods has not been verified to work end-to-end
- Hyper-V Containers are not supported
- Hyper-V isolated containers are not supported.
- Windows container OS must match the Host OS. If it does not, the pod will get stuck in a crash loop.
- Under the networking models of L3 or Host GW, Kubernetes Services are inaccessible to Windows nodes due to a Windows issue. This is not an issue if using OVN/OVS for networking.
- Windows kubelet.exe may fail to start when running on Windows Server under VMware Fusion [issue 57110](https://github.com/kubernetes/kubernetes/pull/57124)
- Flannel and Weavenet are not yet supported
- Some .Net Core applications expect environment variables with a colon (`:`) in the name. Kubernetes currently does not allow this. Replace colon (`:`) with double underscore (`__`) as documented [here](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?tabs=basicconfiguration#configuration-by-environment).
## Next steps and resources
> As of this writing, the Kube-proxy binary requires a pending Kubernetes [pull request](https://github.com/kubernetes/kubernetes/pull/56529) to work properly. You may need to [build](#build) the binaries manually to work around this.
- Support for Windows is in Beta as of v1.9 and your feedback is welcome. For information on getting involved, please head to [SIG-Windows](https://github.com/kubernetes/community/blob/master/sig-windows/README.md)
- Troubleshooting and Common Problems: [Link](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems)