reorganize kubeadm files, part 1 (#9439)
* reorganize kubeadm files, part 1 * edit troubleshooting kubeadm for layout and content
This commit is contained in:
committed by
k8s-ci-robot
parent
2ee2e9d5bb
commit
1e554ef725
@@ -3,12 +3,12 @@ reviewers:
|
||||
- sig-cluster-lifecycle
|
||||
title: Customizing control plane configuration with kubeadm
|
||||
content_template: templates/concept
|
||||
weight: 50
|
||||
weight: 40
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
kubeadm’s configuration exposes the following fields that can be used to override the default flags passed to control plane components such as the APIServer, ControllerManager and Scheduler:
|
||||
The kubeadm configuration exposes the following fields that can override the default flags passed to control plane components such as the APIServer, ControllerManager and Scheduler:
|
||||
|
||||
- `APIServerExtraArgs`
|
||||
- `ControllerManagerExtraArgs`
|
||||
|
||||
@@ -3,6 +3,7 @@ reviewers:
|
||||
- sig-cluster-lifecycle
|
||||
title: Creating a single master cluster with kubeadm
|
||||
content_template: templates/task
|
||||
weight: 30
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
@@ -3,6 +3,7 @@ reviewers:
|
||||
- sig-cluster-lifecycle
|
||||
title: Creating Highly Available Clusters with kubeadm
|
||||
content_template: templates/task
|
||||
weight: 50
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
---
|
||||
title: Installing kubeadm
|
||||
content_template: templates/task
|
||||
weight: 20
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
<img src="https://raw.githubusercontent.com/cncf/artwork/master/kubernetes/certified-kubernetes/versionless/color/certified-kubernetes-color.png" align="right" width="150px">This page shows how to install the `kubeadm` toolbox.
|
||||
For information how to create a cluster with kubeadm once you have performed this installation process,
|
||||
see the [Using kubeadm to Create a Cluster](/docs/setup/independent/create-cluster-kubeadm/) page.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
* One or more machines running one of:
|
||||
- Ubuntu 16.04+
|
||||
- Debian 9
|
||||
- CentOS 7
|
||||
- RHEL 7
|
||||
- Fedora 25/26 (best-effort)
|
||||
- HypriotOS v1.0.1+
|
||||
- Container Linux (tested with 1576.4.0)
|
||||
* 2 GB or more of RAM per machine (any less will leave little room for your apps)
|
||||
* 2 CPUs or more
|
||||
* Full network connectivity between all machines in the cluster (public or private network is fine)
|
||||
* Unique hostname, MAC address, and product_uuid for every node. See [here](#verify-the-mac-address-and-product-uuid-are-unique-for-every-node) for more details.
|
||||
* Certain ports are open on your machines. See [here](#check-required-ports) for more details.
|
||||
* Swap disabled. You **MUST** disable swap in order for the kubelet to work properly.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Verify the MAC address and product_uuid are unique for every node
|
||||
|
||||
* You can get the MAC address of the network interfaces using the command `ip link` or `ifconfig -a`
|
||||
* The product_uuid can be checked by using the command `sudo cat /sys/class/dmi/id/product_uuid`
|
||||
|
||||
It is very likely that hardware devices will have unique addresses, although some virtual machines may have
|
||||
identical values. Kubernetes uses these values to uniquely identify the nodes in the cluster.
|
||||
If these values are not unique to each node, the installation process
|
||||
may [fail](https://github.com/kubernetes/kubeadm/issues/31).
|
||||
|
||||
## Check network adapters
|
||||
|
||||
If you have more than one network adapter, and your Kubernetes components are not reachable on the default
|
||||
route, we recommend you add IP route(s) so Kubernetes cluster addresses go via the appropriate adapter.
|
||||
|
||||
## Check required ports
|
||||
|
||||
### Master node(s)
|
||||
|
||||
| Protocol | Direction | Port Range | Purpose | Used By |
|
||||
|----------|-----------|------------|-------------------------|---------------------------|
|
||||
| TCP | Inbound | 6443* | Kubernetes API server | All |
|
||||
| TCP | Inbound | 2379-2380 | etcd server client API | kube-apiserver, etcd |
|
||||
| TCP | Inbound | 10250 | Kubelet API | Self, Control plane |
|
||||
| TCP | Inbound | 10251 | kube-scheduler | Self |
|
||||
| TCP | Inbound | 10252 | kube-controller-manager | Self |
|
||||
|
||||
### Worker node(s)
|
||||
|
||||
| Protocol | Direction | Port Range | Purpose | Used By |
|
||||
|----------|-----------|-------------|-----------------------|-------------------------|
|
||||
| TCP | Inbound | 10250 | Kubelet API | Self, Control plane |
|
||||
| TCP | Inbound | 30000-32767 | NodePort Services** | All |
|
||||
|
||||
** Default port range for [NodePort Services](/docs/concepts/services-networking/service/).
|
||||
|
||||
Any port numbers marked with * are overridable, so you will need to ensure any
|
||||
custom ports you provide are also open.
|
||||
|
||||
Although etcd ports are included in master nodes, you can also host your own
|
||||
etcd cluster externally or on custom ports.
|
||||
|
||||
The pod network plugin you use (see below) may also require certain ports to be
|
||||
open. Since this differs with each pod network plugin, please see the
|
||||
documentation for the plugins about what port(s) those need.
|
||||
|
||||
## Installing Docker
|
||||
|
||||
On each of your machines, install Docker.
|
||||
Version 17.03 is recommended, but 1.11, 1.12 and 1.13 are known to work as well.
|
||||
Versions 17.06+ _might work_, but have not yet been tested and verified by the Kubernetes node team.
|
||||
Keep track of the latest verified Docker version in the Kubernetes release notes.
|
||||
|
||||
Please proceed with executing the following commands based on your OS as root. You may become the root user by executing `sudo -i` after SSH-ing to each host.
|
||||
|
||||
If you already have the required versions of the Docker installed, you can move on to next section.
|
||||
If not, you can use the following commands to install Docker on your system:
|
||||
|
||||
{{< tabs name="docker_install" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
Install Docker from Ubuntu's repositories:
|
||||
|
||||
```bash
|
||||
apt-get update
|
||||
apt-get install -y docker.io
|
||||
```
|
||||
|
||||
or install Docker CE 17.03 from Docker's repositories for Ubuntu or Debian:
|
||||
|
||||
```bash
|
||||
apt-get update
|
||||
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
|
||||
add-apt-repository "deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") $(lsb_release -cs) stable"
|
||||
apt-get update && apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
Install Docker using your operating system's bundled package:
|
||||
|
||||
```bash
|
||||
yum install -y docker
|
||||
systemctl enable docker && systemctl start docker
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{% tab name="Container Linux" %}}
|
||||
Enable and start Docker:
|
||||
|
||||
```bash
|
||||
systemctl enable docker && systemctl start docker
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
|
||||
Refer to the [official Docker installation guides](https://docs.docker.com/engine/installation/)
|
||||
for more information.
|
||||
|
||||
## Installing kubeadm, kubelet and kubectl
|
||||
|
||||
You will install these packages on all of your machines:
|
||||
|
||||
* `kubeadm`: the command to bootstrap the cluster.
|
||||
|
||||
* `kubelet`: the component that runs on all of the machines in your cluster
|
||||
and does things like starting pods and containers.
|
||||
|
||||
* `kubectl`: the command line util to talk to your cluster.
|
||||
|
||||
kubeadm **will not** install or manage `kubelet` or `kubectl` for you, so you will
|
||||
need to ensure they match the version of the Kubernetes control panel you want
|
||||
kubeadm to install for you. If you do not, there is a risk of a version skew occurring that
|
||||
can lead to unexpected, buggy behaviour. However, _one_ minor version skew between the
|
||||
kubelet and the control plane is supported, but the kubelet version may never exceed the API
|
||||
server version. For example, kubelets running 1.7.0 should be fully compatible with a 1.8.0 API server,
|
||||
but not vice versa.
|
||||
|
||||
For more information on version skews, please read our
|
||||
[version skew policy](/docs/setup/independent/create-cluster-kubeadm/#version-skew-policy).
|
||||
|
||||
{{< tabs name="k8s_install" >}}
|
||||
{{% tab name="Ubuntu, Debian or HypriotOS" %}}
|
||||
```bash
|
||||
apt-get update && apt-get install -y apt-transport-https curl
|
||||
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
|
||||
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
|
||||
deb http://apt.kubernetes.io/ kubernetes-xenial main
|
||||
EOF
|
||||
apt-get update
|
||||
apt-get install -y kubelet kubeadm kubectl
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{% tab name="CentOS, RHEL or Fedora" %}}
|
||||
```bash
|
||||
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
|
||||
[kubernetes]
|
||||
name=Kubernetes
|
||||
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
|
||||
enabled=1
|
||||
gpgcheck=1
|
||||
repo_gpgcheck=1
|
||||
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
|
||||
EOF
|
||||
setenforce 0
|
||||
yum install -y kubelet kubeadm kubectl
|
||||
systemctl enable kubelet && systemctl start kubelet
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- Disabling SELinux by running `setenforce 0` is required to allow containers to access the host filesystem, which is required by pod networks for example.
|
||||
You have to do this until SELinux support is improved in the kubelet.
|
||||
- Some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed. You should ensure
|
||||
`net.bridge.bridge-nf-call-iptables` is set to 1 in your `sysctl` config, e.g.
|
||||
|
||||
```bash
|
||||
cat <<EOF > /etc/sysctl.d/k8s.conf
|
||||
net.bridge.bridge-nf-call-ip6tables = 1
|
||||
net.bridge.bridge-nf-call-iptables = 1
|
||||
EOF
|
||||
sysctl --system
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{% tab name="Container Linux" %}}
|
||||
Install CNI plugins (required for most pod network):
|
||||
|
||||
```bash
|
||||
CNI_VERSION="v0.6.0"
|
||||
mkdir -p /opt/cni/bin
|
||||
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz
|
||||
```
|
||||
|
||||
Install `kubeadm`, `kubelet`, `kubectl` and add a `kubelet` systemd service:
|
||||
|
||||
```bash
|
||||
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
|
||||
|
||||
mkdir -p /opt/bin
|
||||
cd /opt/bin
|
||||
curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
|
||||
chmod +x {kubeadm,kubelet,kubectl}
|
||||
|
||||
curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service
|
||||
mkdir -p /etc/systemd/system/kubelet.service.d
|
||||
curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
|
||||
```
|
||||
|
||||
Enable and start `kubelet`:
|
||||
|
||||
```bash
|
||||
systemctl enable kubelet && systemctl start kubelet
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
|
||||
The kubelet is now restarting every few seconds, as it waits in a crashloop for
|
||||
kubeadm to tell it what to do.
|
||||
|
||||
## Configure cgroup driver used by kubelet on Master Node
|
||||
|
||||
Make sure that the cgroup driver used by kubelet is the same as the one used by Docker. Verify that your Docker cgroup driver matches the kubelet config:
|
||||
|
||||
```bash
|
||||
docker info | grep -i cgroup
|
||||
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
|
||||
```
|
||||
|
||||
If the Docker cgroup driver and the kubelet config don't match, change the kubelet config to match the Docker cgroup driver. The
|
||||
flag you need to change is `--cgroup-driver`. If it's already set, you can update like so:
|
||||
|
||||
```bash
|
||||
sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
|
||||
```
|
||||
|
||||
Otherwise, you will need to open the systemd file and add the flag to an existing environment line.
|
||||
|
||||
Then restart kubelet:
|
||||
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl restart kubelet
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If you are running into difficulties with kubeadm, please consult our [troubleshooting docs](/docs/setup/independent/troubleshooting-kubeadm/).
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
* [Using kubeadm to Create a Cluster](/docs/setup/independent/create-cluster-kubeadm/)
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,265 @@
|
||||
---
|
||||
reviewers:
|
||||
- sig-cluster-lifecycle
|
||||
title: Set up a Highly Availabile etcd Cluster With kubeadm
|
||||
content_template: templates/task
|
||||
weight: 60
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
Kubeadm defaults to running a single member etcd cluster in a static pod managed
|
||||
by the kubelet on the control plane node. This is not a highly available setup
|
||||
as the etcd cluster contains only one member and cannot sustain any members
|
||||
becoming unavailable. This task walks through the process of creating a highly
|
||||
available etcd cluster of three members that can be used as an external etcd
|
||||
when using kubeadm to set up a kubernetes cluster.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture prerequisites %}}
|
||||
|
||||
* Three hosts that can talk to each other over ports 2379 and 2380. This
|
||||
document assumes these default ports. However, they are configurable through
|
||||
the kubeadm config file.
|
||||
* Each host must [have docker, kubelet, and kubeadm installed][toolbox].
|
||||
* Some infrastructure to copy files between hosts. For example `ssh` and `scp`
|
||||
can satisfy this requirement.
|
||||
|
||||
[toolbox]: /docs/tasks/tools/install-kubeadm/
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture steps %}}
|
||||
|
||||
## Setting up the cluster
|
||||
|
||||
The general approach is to generate all certs on one node and only distribute
|
||||
the *necessary* files to the other nodes.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** kubeadm contains all the necessary crytographic machinery to generate
|
||||
the certificates described below; no other cryptographic tooling is required for
|
||||
this example.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
1. Configure the kubelet to be a service manager for etcd.
|
||||
|
||||
Running etcd is simpler than running kubernetes so you must override the
|
||||
kubeadm-provided kubelet unit file by creating a new one with a higher
|
||||
precedence.
|
||||
|
||||
```sh
|
||||
cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/bin/kubelet --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true
|
||||
Restart=always
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl restart kubelet
|
||||
```
|
||||
|
||||
1. Create configuration files for kubeadm.
|
||||
|
||||
Generate one kubeadm configuration file for each host that will have an etcd
|
||||
member running on it using the following script.
|
||||
|
||||
```sh
|
||||
# Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
|
||||
export HOST0=10.0.0.6
|
||||
export HOST1=10.0.0.7
|
||||
export HOST2=10.0.0.8
|
||||
|
||||
# Create temp directories to store files that will end up on other hosts.
|
||||
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
|
||||
|
||||
ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
|
||||
NAMES=("infra0" "infra1" "infra2")
|
||||
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
NAME=${NAMES[$i]}
|
||||
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
|
||||
apiVersion: "kubeadm.k8s.io/v1alpha2"
|
||||
kind: MasterConfiguration
|
||||
etcd:
|
||||
localEtcd:
|
||||
serverCertSANs:
|
||||
- "${HOST}"
|
||||
peerCertSANs:
|
||||
- "${HOST}"
|
||||
extraArgs:
|
||||
initial-cluster: infra0=https://${ETCDHOSTS[0]}:2380,infra1=https://${ETCDHOSTS[1]}:2380,infra2=https://${ETCDHOSTS[2]}:2380
|
||||
initial-cluster-state: new
|
||||
name: ${NAME}
|
||||
listen-peer-urls: https://${HOST}:2380
|
||||
listen-client-urls: https://${HOST}:2379
|
||||
advertise-client-urls: https://${HOST}:2379
|
||||
initial-advertise-peer-urls: https://${HOST}:2380
|
||||
EOF
|
||||
done
|
||||
```
|
||||
|
||||
1. Generate the certificate authority
|
||||
|
||||
If you already have a CA then the only action that is copying the CA's `crt` and
|
||||
`key` file to `/etc/kubernetes/pki/etcd/ca.crt` and
|
||||
`/etc/kubernetes/pki/etcd/ca.key`. After those files have been copied, please
|
||||
skip this step.
|
||||
|
||||
If you do not already have a CA then run this command on `$HOST0` (where you
|
||||
generated the configuration files for kubeadm).
|
||||
|
||||
```
|
||||
kubeadm alpha phase certs etcd-ca
|
||||
```
|
||||
|
||||
This creates two files
|
||||
|
||||
- `/etc/kubernetes/pki/etcd/ca.crt`
|
||||
- `/etc/kubernetes/pki/etcd/ca.key`
|
||||
|
||||
1. Create certificates for each member
|
||||
|
||||
```sh
|
||||
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
|
||||
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
|
||||
# cleanup non-reusable certificates
|
||||
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
|
||||
|
||||
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
|
||||
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
|
||||
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
|
||||
|
||||
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
|
||||
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
|
||||
# No need to move the certs because they are for HOST0
|
||||
|
||||
# clean up certs that should not be copied off this host
|
||||
find /tmp/${HOST2} -name ca.key -type f -delete
|
||||
find /tmp/${HOST1} -name ca.key -type f -delete
|
||||
```
|
||||
|
||||
1. Copy certificates and kubeadm configs
|
||||
|
||||
The certificates have been generated and now they must be moved to their
|
||||
respective hosts.
|
||||
|
||||
```sh
|
||||
USER=ubuntu
|
||||
HOST=${HOST1}
|
||||
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
|
||||
ssh ${USER}@${HOST}
|
||||
USER@HOST $ sudo -Es
|
||||
root@HOST $ chown -R root:root pki
|
||||
root@HOST $ mv pki /etc/kubernetes/
|
||||
```
|
||||
|
||||
1. Ensure all expected files exist
|
||||
|
||||
The complete list of required files on `$HOST0` is:
|
||||
|
||||
```
|
||||
/tmp/${HOST0}
|
||||
└── kubeadmcfg.yaml
|
||||
---
|
||||
/etc/kubernetes/pki
|
||||
├── apiserver-etcd-client.crt
|
||||
├── apiserver-etcd-client.key
|
||||
└── etcd
|
||||
├── ca.crt
|
||||
├── ca.key
|
||||
├── healthcheck-client.crt
|
||||
├── healthcheck-client.key
|
||||
├── peer.crt
|
||||
├── peer.key
|
||||
├── server.crt
|
||||
└── server.key
|
||||
```
|
||||
|
||||
On `$HOST1`:
|
||||
|
||||
```
|
||||
$HOME
|
||||
└── kubeadmcfg.yaml
|
||||
---
|
||||
/etc/kubernetes/pki
|
||||
├── apiserver-etcd-client.crt
|
||||
├── apiserver-etcd-client.key
|
||||
└── etcd
|
||||
├── ca.crt
|
||||
├── healthcheck-client.crt
|
||||
├── healthcheck-client.key
|
||||
├── peer.crt
|
||||
├── peer.key
|
||||
├── server.crt
|
||||
└── server.key
|
||||
```
|
||||
|
||||
On `$HOST2`
|
||||
|
||||
```
|
||||
$HOME
|
||||
└── kubeadmcfg.yaml
|
||||
---
|
||||
/etc/kubernetes/pki
|
||||
├── apiserver-etcd-client.crt
|
||||
├── apiserver-etcd-client.key
|
||||
└── etcd
|
||||
├── ca.crt
|
||||
├── healthcheck-client.crt
|
||||
├── healthcheck-client.key
|
||||
├── peer.crt
|
||||
├── peer.key
|
||||
├── server.crt
|
||||
└── server.key
|
||||
```
|
||||
|
||||
1. Create the static pod manifests
|
||||
|
||||
Now that the certificates and configs are in place it's time to create the
|
||||
manifests. On each host run the `kubeadm` command to generate a static manifest
|
||||
for etcd.
|
||||
|
||||
```sh
|
||||
root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
|
||||
root@HOST1 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
|
||||
root@HOST2 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
|
||||
```
|
||||
|
||||
1. Optional: Check the cluster health
|
||||
|
||||
```sh
|
||||
docker run --rm -it \
|
||||
--net host \
|
||||
-v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.18 etcdctl \
|
||||
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
|
||||
--key-file /etc/kubernetes/pki/etcd/peer.key \
|
||||
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
|
||||
--endpoints https://${HOST0}:2379 cluster-health
|
||||
...
|
||||
cluster is healthy
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
Once your have a working 3 member etcd cluster, you can continue setting up a
|
||||
highly available control plane using the [external etcd method with
|
||||
kubeadm](/docs/setup/independent/high-availability/).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
---
|
||||
title: Troubleshooting kubeadm
|
||||
content_template: templates/concept
|
||||
weight: 70
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
As with any program, you might run into an error using or operating it. Below we have listed
|
||||
common failure scenarios and have provided steps that will help you to understand and hopefully
|
||||
fix the problem.
|
||||
As with any program, you might run into an error installing or running kubeadm.
|
||||
This page lists some common failure scenarios and have provided steps that can help you understand and fix the problem.
|
||||
|
||||
If your problem is not listed below, please follow the following steps:
|
||||
|
||||
@@ -14,16 +15,14 @@ If your problem is not listed below, please follow the following steps:
|
||||
- Go to [github.com/kubernetes/kubeadm](https://github.com/kubernetes/kubeadm/issues) and search for existing issues.
|
||||
- If no issue exists, please [open one](https://github.com/kubernetes/kubeadm/issues/new) and follow the issue template.
|
||||
|
||||
- If you are unsure about how kubeadm or kubernetes works, and would like to receive
|
||||
support about your question, please ask on Slack in #kubeadm, or open a question on StackOverflow. Please include
|
||||
- If you are unsure about how kubeadm works, you can ask on Slack in #kubeadm, or open a question on StackOverflow. Please include
|
||||
relevant tags like `#kubernetes` and `#kubeadm` so folks can help you.
|
||||
|
||||
If your cluster is in an error state, you may have trouble in the configuration if you see Pod statuses like `RunContainerError`,
|
||||
`CrashLoopBackOff` or `Error`. If this is the case, please read below.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
#### `ebtables` or some similar executable not found during installation
|
||||
{{% capture body %}}
|
||||
|
||||
## `ebtables` or some similar executable not found during installation
|
||||
|
||||
If you see the following warnings while running `kubeadm init`
|
||||
|
||||
@@ -37,7 +36,7 @@ Then you may be missing `ebtables`, `ethtool` or a similar executable on your no
|
||||
- For Ubuntu/Debian users, run `apt install ebtables ethtool`.
|
||||
- For CentOS/Fedora users, run `yum install ebtables ethtool`.
|
||||
|
||||
#### kubeadm blocks waiting for control plane during installation
|
||||
## kubeadm blocks waiting for control plane during installation
|
||||
|
||||
If you notice that `kubeadm init` hangs after printing out the following line:
|
||||
|
||||
@@ -66,7 +65,7 @@ This may be caused by a number of problems. The most common are:
|
||||
|
||||
- control plane Docker containers are crashlooping or hanging. You can check this by running `docker ps` and investigating each container by running `docker logs`.
|
||||
|
||||
#### kubeadm blocks when removing managed containers
|
||||
## kubeadm blocks when removing managed containers
|
||||
|
||||
The following could happen if Docker halts and does not remove any Kubernetes-managed containers:
|
||||
|
||||
@@ -92,7 +91,7 @@ Inspecting the logs for docker may also be useful:
|
||||
journalctl -ul docker
|
||||
```
|
||||
|
||||
#### Pods in `RunContainerError`, `CrashLoopBackOff` or `Error` state
|
||||
## Pods in `RunContainerError`, `CrashLoopBackOff` or `Error` state
|
||||
|
||||
Right after `kubeadm init` there should not be any pods in these states.
|
||||
|
||||
@@ -106,14 +105,14 @@ Right after `kubeadm init` there should not be any pods in these states.
|
||||
might have to grant it more RBAC privileges or use a newer version. Please file
|
||||
an issue in the Pod Network providers' issue tracker and get the issue triaged there.
|
||||
|
||||
#### `coredns` (or `kube-dns`) is stuck in the `Pending` state
|
||||
## `coredns` (or `kube-dns`) is stuck in the `Pending` state
|
||||
|
||||
This is **expected** and part of the design. kubeadm is network provider-agnostic, so the admin
|
||||
should [install the pod network solution](/docs/concepts/cluster-administration/addons/)
|
||||
of choice. You have to install a Pod Network
|
||||
before CoreDNS may deployed fully. Hence the `Pending` state before the network is set up.
|
||||
|
||||
#### `HostPort` services do not work
|
||||
## `HostPort` services do not work
|
||||
|
||||
The `HostPort` and `HostIP` functionality is available depending on your Pod Network
|
||||
provider. Please contact the author of the Pod Network solution to find out whether
|
||||
@@ -126,7 +125,7 @@ For more information, see the [CNI portmap documentation](https://github.com/con
|
||||
If your network provider does not support the portmap CNI plugin, you may need to use the [NodePort feature of
|
||||
services](/docs/concepts/services-networking/service/#type-nodeport) or use `HostNetwork=true`.
|
||||
|
||||
#### Pods are not accessible via their Service IP
|
||||
## Pods are not accessible via their Service IP
|
||||
|
||||
- Many network add-ons do not yet enable [hairpin mode](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#a-pod-cannot-reach-itself-via-service-ip)
|
||||
which allows pods to access themselves via their Service IP. This is an issue related to
|
||||
@@ -139,7 +138,7 @@ services](/docs/concepts/services-networking/service/#type-nodeport) or use `Hos
|
||||
is to modify `/etc/hosts`, see this [Vagrantfile](https://github.com/errordeveloper/k8s-playground/blob/22dd39dfc06111235620e6c4404a96ae146f26fd/Vagrantfile#L11)
|
||||
for an example.
|
||||
|
||||
#### TLS certificate errors
|
||||
## TLS certificate errors
|
||||
|
||||
The following error indicates a possible certificate mismatch.
|
||||
|
||||
@@ -160,7 +159,7 @@ Unable to connect to the server: x509: certificate signed by unknown authority (
|
||||
sudo chown $(id -u):$(id -g) $HOME/.kube/config
|
||||
```
|
||||
|
||||
#### Default NIC When using flannel as the pod network in Vagrant
|
||||
## Default NIC When using flannel as the pod network in Vagrant
|
||||
|
||||
The following error might indicate that something was wrong in the pod network:
|
||||
|
||||
@@ -174,7 +173,7 @@ Error from server (NotFound): the server could not find the requested resource
|
||||
|
||||
This may lead to problems with flannel, which defaults to the first interface on a host. This leads to all hosts thinking they have the same public IP address. To prevent this, pass the `--iface eth1` flag to flannel so that the second interface is chosen.
|
||||
|
||||
#### Non-public IP used for containers
|
||||
## Non-public IP used for containers
|
||||
|
||||
In some situations `kubectl logs` and `kubectl run` commands may return with the following errors in an otherwise functional cluster:
|
||||
|
||||
@@ -200,7 +199,7 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6
|
||||
systemctl restart kubelet
|
||||
```
|
||||
|
||||
#### Services with externalTrafficPolicy=Local are not reachable
|
||||
## Services with externalTrafficPolicy=Local are not reachable
|
||||
|
||||
On nodes where the hostname for the kubelet is overridden using the `--hostname-override` option, kube-proxy will default to treating 127.0.0.1 as the node IP, which results in rejecting connections for Services configured for `externalTrafficPolicy=Local`. This situation can be verified by checking the output of `kubectl -n kube-system logs <kube-proxy pod name>`:
|
||||
|
||||
@@ -239,3 +238,4 @@ EOF
|
||||
)"
|
||||
|
||||
```
|
||||
{{% /capture %}}
|
||||
Reference in New Issue
Block a user