Merge branch 'master' into fix-typo
This commit is contained in:
Peter Lee
@@ -4,6 +4,7 @@ assignees:
|
||||
- lavalamp
|
||||
- ericchiang
|
||||
- deads2k
|
||||
- liggitt
|
||||
|
||||
---
|
||||
* TOC
|
||||
@@ -382,6 +383,13 @@ option to the API server during startup. The plugin is implemented in
|
||||
`plugin/pkg/auth/authenticator/password/keystone/keystone.go` and currently uses
|
||||
basic auth to verify used by username and password.
|
||||
|
||||
If you have configured self-signed certificates for the Keystone server,
|
||||
you may need to set the `--experimental-keystone-ca-file=SOMEFILE` option when
|
||||
starting the Kubernetes API server. If you set the option, the Keystone
|
||||
server's certificate is verified by one of the authorities in the
|
||||
`experimental-keystone-ca-file`. Otherwise, the certificate is verified by
|
||||
the host's root Certificate Authority.
|
||||
|
||||
For details on how to use keystone to manage projects and users, refer to the
|
||||
[Keystone documentation](http://docs.openstack.org/developer/keystone/). Please
|
||||
note that this plugin is still experimental, under active development, and likely
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
assignees:
|
||||
- erictune
|
||||
- lavalamp
|
||||
- deads2k
|
||||
- liggitt
|
||||
|
||||
---
|
||||
|
||||
@@ -565,10 +567,10 @@ Access to non-resource paths are sent as:
|
||||
|
||||
Non-resource paths include: `/api`, `/apis`, `/metrics`, `/resetMetrics`,
|
||||
`/logs`, `/debug`, `/healthz`, `/swagger-ui/`, `/swaggerapi/`, `/ui`, and
|
||||
`/version.` Clients require access to `/api`, `/api/*/`, `/apis/`, `/apis/*`,
|
||||
`/apis/*/*`, and `/version` to discover what resources and versions are present
|
||||
on the server. Access to other non-resource paths can be disallowed without
|
||||
restricting access to the REST api.
|
||||
`/version.` Clients require access to `/api`, `/api/*`, `/apis`, `/apis/*`,
|
||||
and `/version` to discover what resources and versions are present on the server.
|
||||
Access to other non-resource paths can be disallowed without restricting access
|
||||
to the REST api.
|
||||
|
||||
For further documentation refer to the authorization.v1beta1 API objects and
|
||||
plugin/pkg/auth/authorizer/webhook/webhook.go.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
assignees:
|
||||
- mml
|
||||
- davidopp
|
||||
|
||||
---
|
||||
This guide is for anyone wishing to specify safety constraints on pods or anyone
|
||||
@@ -59,7 +59,7 @@ itself. To attempt an eviction (perhaps more REST-precisely, to attempt to
|
||||
|
||||
```json
|
||||
{
|
||||
"apiVersion": "policy/v1alpha1",
|
||||
"apiVersion": "policy/v1beta1",
|
||||
"kind": "Eviction",
|
||||
"metadata": {
|
||||
"name": "quux",
|
||||
|
||||
@@ -356,3 +356,5 @@ for more information.
|
||||
|
||||
- [Docs for the DNS cluster addon](http://releases.k8s.io/{{page.githubbranch}}/build-tools/kube-dns/README.md)
|
||||
|
||||
## What's next
|
||||
- [Autoscaling the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/).
|
||||
|
||||
@@ -14,11 +14,11 @@ This guide explains how to set up cluster federation that lets us control multip
|
||||
## Prerequisites
|
||||
|
||||
This guide assumes that you have a running Kubernetes cluster.
|
||||
If not, then head over to the [getting started guides](/docs/getting-started-guides/) to bring up a cluster.
|
||||
If you need to start a new cluster, see the [getting started guides](/docs/getting-started-guides/) for instructions on bringing a cluster up.
|
||||
|
||||
This guide also assumes that you have a Kubernetes release
|
||||
[downloaded from here](/docs/getting-started-guides/binary_release/),
|
||||
extracted into a directory and all the commands in this guide are run from
|
||||
To use the commands in this guide, you must download a Kubernetes release from the
|
||||
[getting started binary releases](/docs/getting-started-guides/binary_release/) and
|
||||
extract into a directory; all the commands in this guide are run from
|
||||
that directory.
|
||||
|
||||
```shell
|
||||
@@ -26,8 +26,8 @@ $ curl -L https://github.com/kubernetes/kubernetes/releases/download/v1.4.0/kube
|
||||
$ cd kubernetes
|
||||
```
|
||||
|
||||
This guide also assumes that you have an installation of Docker running
|
||||
locally, i.e. on the machine where you run the commands described in this
|
||||
You must also have a Docker installation running
|
||||
locally--meaning on the machine where you run the commands described in this
|
||||
guide.
|
||||
|
||||
## Setting up a federation control plane
|
||||
@@ -212,47 +212,81 @@ cluster1 Ready 3m
|
||||
|
||||
## Updating KubeDNS
|
||||
|
||||
Once the cluster is registered with the federation, you are all set to use it.
|
||||
But for the cluster to be able to route federation service requests, you need to restart
|
||||
KubeDNS and pass it a `--federations` flag which tells it about valid federation DNS hostnames.
|
||||
Format of the flag is like this:
|
||||
Once you've registered your cluster with the federation, you'll need to update KubeDNS so that your cluster can route federation service requests. The update method varies depending on your Kubernetes version; on Kubernetes 1.5 or later, you must pass the
|
||||
`--federations` flag to kube-dns via the kube-dns config map. In version 1.4 or earlier, you must set the `--federations` flag directly on kube-dns-rc on other clusters.
|
||||
|
||||
### Kubernetes 1.5+: Passing federations flag via config map to kube-dns
|
||||
|
||||
For kubernetes clusters of version 1.5+, you can pass the
|
||||
`--federations` flag to kube-dns via the kube-dns config map.
|
||||
The flag uses the following format:
|
||||
|
||||
```
|
||||
--federations=${FEDERATION_NAME}=${DNS_DOMAIN_NAME}
|
||||
```
|
||||
|
||||
To update KubeDNS with federations flag, you can edit the existing kubedns replication controller to
|
||||
include that flag in pod template spec and then delete the existing pod. Replication controller will
|
||||
recreate the pod with updated template.
|
||||
To pass this flag to KubeDNS, create a config-map with name `kube-dns` in
|
||||
namespace `kube-system`. The configmap should look like the following:
|
||||
|
||||
To find the name of existing kubedns replication controller, run
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: kube-dns
|
||||
namespace: kube-system
|
||||
data:
|
||||
federations: <federation-name>=<dns-domain-name>
|
||||
```
|
||||
|
||||
where `<federation-name>` should be replaced by the name you want to give to your
|
||||
federation, and
|
||||
`federation-domain-name` should be replaced by the domain name you want to use
|
||||
in your federation DNS.
|
||||
|
||||
You can find more details about config maps in general at
|
||||
http://kubernetes.io/docs/user-guide/configmap/.
|
||||
|
||||
### Kubernetes 1.4 and earlier: Setting federations flag on kube-dns-rc
|
||||
|
||||
If your cluster is running Kubernetes version 1.4 or earlier, you must to restart
|
||||
KubeDNS and pass it a `--federations` flag, which tells it about valid federation DNS hostnames.
|
||||
The flag uses the following format:
|
||||
|
||||
```
|
||||
--federations=${FEDERATION_NAME}=${DNS_DOMAIN_NAME}
|
||||
```
|
||||
|
||||
To update KubeDNS with the `--federations` flag, you can edit the existing kubedns replication controller to
|
||||
include that flag in pod template spec, and then delete the existing pod. The replication controller then
|
||||
recreates the pod with updated template.
|
||||
|
||||
To find the name of existing kubedns replication controller, run the following command:
|
||||
|
||||
```shell
|
||||
$ kubectl get rc --namespace=kube-system
|
||||
```
|
||||
|
||||
This will list all the replication controllers. Name of the kube-dns replication
|
||||
controller will look like `kube-dns-v18`. You can then edit it by running:
|
||||
You should see a list of all the replication controllers on the cluster. The kube-dns replication
|
||||
controller should have a name similar to `kube-dns-v18`. To edit the replication controller, specify it by name as follows:
|
||||
|
||||
```shell
|
||||
$ kubectl edit rc <rc-name> --namespace=kube-system
|
||||
```
|
||||
Add the `--federations` flag as args to kube-dns container in the YAML file that
|
||||
pops up after running the above command.
|
||||
In the resulting YAML file for the kube-dns replication controller, add the `--federations` flag as an argument to kube-dns container.
|
||||
|
||||
To delete the existing kube dns pod, you can first find it by running:
|
||||
Then, you must delete the existing kube dns pod. You can find the pod by running:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods --namespace=kube-system
|
||||
```
|
||||
|
||||
And then delete it by running:
|
||||
And then delete the appropriate pod by running:
|
||||
|
||||
```shell
|
||||
$ kubectl delete pods <pod-name> --namespace=kube-system
|
||||
```
|
||||
|
||||
You are now all set to start using federation.
|
||||
Once you've completed the kube-dns configuration, your federation is ready for use.
|
||||
|
||||
## Turn down
|
||||
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
---
|
||||
assignees:
|
||||
- madhusudancs
|
||||
|
||||
---
|
||||
Kubernetes version 1.5 includes a new command line tool called
|
||||
`kubefed` to help you administrate your federated clusters.
|
||||
`kubefed` helps you to deploy a new Kubernetes cluster federation
|
||||
control plane, and to add clusters to or remove clusters from an
|
||||
existing federation control plane.
|
||||
|
||||
This guide explains how to administer a Kubernetes Cluster Federation
|
||||
using `kubefed`.
|
||||
|
||||
> Note: `kubefed` is an alpha feature in Kubernetes 1.5.
|
||||
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This guide assumes that you have a running Kubernetes cluster. Please
|
||||
see one of the [getting started](/docs/getting-started-guides/) guides
|
||||
for installation instructions for your platform.
|
||||
|
||||
|
||||
## Getting `kubefed`
|
||||
|
||||
Download the client tarball corresponding to Kubernetes version 1.5
|
||||
or later
|
||||
[from the release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md),
|
||||
extract the binaries in the tarball to one of the directories
|
||||
in your `$PATH` and set the executable permission on those binaries.
|
||||
|
||||
```shell
|
||||
curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.0/kubernetes-client-linux-amd64.tar.gz
|
||||
tar -xzvf kubernetes-client-linux-amd64.tar.gz
|
||||
sudo cp kubernetes/client/bin/kubefed /usr/local/bin
|
||||
sudo chmod +x /usr/local/bin/kubefed
|
||||
sudo cp kubernetes/client/bin/kubectl /usr/local/bin
|
||||
sudo chmod +x /usr/local/bin/kubectl
|
||||
```
|
||||
|
||||
|
||||
## Choosing a host cluster.
|
||||
|
||||
You'll need to choose one of your Kubernetes clusters to be the
|
||||
*host cluster*. The host cluster hosts the components that make up
|
||||
your federation control plane. Ensure that you have a `kubeconfig`
|
||||
entry in your local `kubeconfig` that corresponds to the host cluster.
|
||||
You can verify that you have the required `kubeconfig` entry by
|
||||
running:
|
||||
|
||||
```shell
|
||||
kubectl config get-contexts
|
||||
```
|
||||
|
||||
The output should contain an entry corresponding to your host cluster,
|
||||
similar to the following:
|
||||
|
||||
```
|
||||
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
|
||||
gke_myproject_asia-east1-b_gce-asia-east1 gke_myproject_asia-east1-b_gce-asia-east1 gke_myproject_asia-east1-b_gce-asia-east1
|
||||
```
|
||||
|
||||
|
||||
You'll need to provide the `kubeconfig` context (called name in the
|
||||
entry above) for your host cluster when you deploy your federation
|
||||
control plane.
|
||||
|
||||
|
||||
## Deploying a federation control plane.
|
||||
|
||||
"To deploy a federation control plane on your host cluster, run
|
||||
`kubefed init` command. When you use `kubefed init`, you must provide
|
||||
the following:
|
||||
|
||||
* Federation name
|
||||
* `--host-cluster-context`, the `kubeconfig` context for the host cluster
|
||||
* `--dns-zone-name`, a domain name suffix for your federated services
|
||||
|
||||
The following example command deploys a federation control plane with
|
||||
the name `fellowship`, a host cluster context `rivendell`, and the
|
||||
domain suffix `example.com`:
|
||||
|
||||
```shell
|
||||
kubefed init fellowship --host-cluster-context=rivendell --dns-zone-name="example.com"
|
||||
```
|
||||
|
||||
The domain suffix you specify in `--dns-zone-name` must be an existing
|
||||
domain that you control, and that is programmable by your DNS provider.
|
||||
|
||||
`kubefed init` sets up the federation control plane in the host
|
||||
cluster and also adds an entry for the federation API server in your
|
||||
local kubeconfig. Note that in the alpha release in Kubernetes 1.5,
|
||||
`kubefed init` does not automatically set the current context to the
|
||||
newly deployed federation. You can set the current context manually by
|
||||
running:
|
||||
|
||||
```shell
|
||||
kubectl config use-context fellowship
|
||||
```
|
||||
|
||||
where `fellowship` is the name of your federation.
|
||||
|
||||
|
||||
## Adding a cluster to a federation
|
||||
|
||||
Once you've deployed a federation control plane, you'll need to make
|
||||
that control plane aware of the clusters it should manage. You can add
|
||||
a cluster to your federation by using the `kubefed join` command.
|
||||
|
||||
To use `kubefed join`, you'll need to provide the name of the cluster
|
||||
you want to add to the federation, and the `--host-cluster-context`
|
||||
for the federation control plane's host cluster.
|
||||
|
||||
The following example command adds the cluster `gondor` to the
|
||||
federation with host cluster `rivendell`:
|
||||
|
||||
```
|
||||
kubefed join gondor --host-cluster-context=rivendell
|
||||
```
|
||||
|
||||
> Note: Kubernetes requires that you manually join clusters to a
|
||||
federation because the federation control plane manages only those
|
||||
clusters that it is responsible for managing. Adding a cluster tells
|
||||
the federation control plane that it is responsible for managing that
|
||||
cluster.
|
||||
|
||||
### Naming rules and customization
|
||||
|
||||
The cluster name you supply to `kubefed join` must be a valid RFC 1035
|
||||
label.
|
||||
|
||||
Furthermore, federation control plane requires credentials of the
|
||||
joined clusters to operate on them. These credentials are obtained
|
||||
from the local kubeconfig. `kubefed join` uses the cluster name
|
||||
specified as the argument to look for the cluster's context in the
|
||||
local kubeconfig. If it fails to find a matching context, it exits
|
||||
with an error.
|
||||
|
||||
This might cause issues in cases where context names for each cluster
|
||||
in the federation don't follow RFC 1035 label naming rules. In such
|
||||
cases, you can specify a cluster name that conforms to the RFC 1035
|
||||
label naming rules and specify the cluster context using the
|
||||
`--cluster-context` flag. For example, if context of the cluster your
|
||||
are joining is `gondor_needs-no_king`, then you can
|
||||
join the cluster by running:
|
||||
|
||||
```shell
|
||||
kubefed join gondor --host-cluster-context=rivendell --cluster-context=gondor_needs-no_king
|
||||
```
|
||||
|
||||
#### Secret name
|
||||
|
||||
Cluster credentials required by the federation control plane as
|
||||
described above are stored as a secret in the host cluster. The name
|
||||
of the secret is also derived from the cluster name.
|
||||
|
||||
However, the name of a secret object in Kubernetes should conform
|
||||
to the subdomain name specification described in RFC 1123. If this
|
||||
isn't case, you can pass the secret name to `kubefed join` using the
|
||||
`--secret-name` flag. For example, if the cluster name is `noldor` and
|
||||
the secret name is `11kingdom`, you can join the cluster by
|
||||
running:
|
||||
|
||||
```shell
|
||||
kubefed join noldor --host-cluster-context=rivendell --secret-name=11kingdom
|
||||
```
|
||||
|
||||
## Removing a cluster from a federation
|
||||
|
||||
To remove a cluster from a federation, run the `kubefed unjoin`
|
||||
command with the cluster name and the federation's
|
||||
`--host-cluster-context`:
|
||||
|
||||
```
|
||||
kubefed unjoin gondor --host-cluster-context=rivendell
|
||||
```
|
||||
|
||||
|
||||
## Turning down the federation control plane:
|
||||
|
||||
Proper cleanup of federation control plane is not fully implemented in
|
||||
this alpha release of `kubefed`. However, for the time being, deleting
|
||||
the federation system namespace should remove all the resources except
|
||||
the persistent storage volume dynamically provisioned for the
|
||||
federation control plane's etcd. You can delete the federation
|
||||
namespace by running the following command:
|
||||
|
||||
```
|
||||
$ kubectl delete ns federation-system
|
||||
```
|
||||
@@ -0,0 +1,160 @@
|
||||
---
|
||||
assignees:
|
||||
- jszczepkowski
|
||||
|
||||
---
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Introduction
|
||||
|
||||
Kubernetes version 1.5 adds alpha support for replicating Kubernetes masters in `kube-up` or `kube-down` scripts for Google Compute Engine.
|
||||
This document describes how to use kube-up/down scripts to manage highly available (HA) masters and how HA masters are implemented for use with GCE.
|
||||
|
||||
## Starting an HA-compatible cluster
|
||||
|
||||
To create a new HA-compatible cluster, you must set the following flags in your `kube-up` script:
|
||||
|
||||
* `MULTIZONE=true` - to prevent removal of master replicas kubelets from zones different than server's default zone.
|
||||
Required if you want to run master replicas in different zones, which is recommended.
|
||||
|
||||
* `ENABLE_ETCD_QUORUM_READS=true` - to ensure that reads from all API servers will return most up-to-date data.
|
||||
If true, reads will be directed to leader etcd replica.
|
||||
Setting this value to true is optional: reads will be more reliable but will also be slower.
|
||||
|
||||
Optionally, you can specify a GCE zone where the first master replica is to be created.
|
||||
Set the the following flag:
|
||||
|
||||
* `KUBE_GCE_ZONE=zone` - zone where the first master replica will run.
|
||||
|
||||
The following sample command sets up a HA-compatible cluster in the GCE zone europe-west1-b:
|
||||
|
||||
```shell
|
||||
$ MULTIZONE=true KUBE_GCE_ZONE=europe-west1-b ENABLE_ETCD_QUORUM_READS=true ./cluster/kube-up.sh
|
||||
```
|
||||
|
||||
Note that the commands above create a cluster with one master;
|
||||
however, you can add new master replicas to the cluster with subsequent commands.
|
||||
|
||||
## Adding a new master replica
|
||||
|
||||
After you have created an HA-compatible cluster, you can add master replicas to it.
|
||||
You add master replicas by using a `kube-up` script with the following flags:
|
||||
|
||||
* `KUBE_REPLICATE_EXISTING_MASTER=true` - to create a replica of an existing
|
||||
master.
|
||||
|
||||
* `KUBE_GCE_ZONE=zone` - zone where the master replica will run.
|
||||
Must be in the same region as other replicas' zones.
|
||||
|
||||
You don't need to set the `MULTIZONE` or `ENABLE_ETCD_QUORUM_READS` flags,
|
||||
as those are inherited from when you started your HA-compatible cluster.
|
||||
|
||||
The following sample command replicates the master on an existing HA-compatible cluster:
|
||||
|
||||
```shell
|
||||
$ KUBE_GCE_ZONE=europe-west1-c KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
|
||||
```
|
||||
|
||||
## Removing a master replica
|
||||
|
||||
You can remove a master replica from an HA cluster by using a `kube-down` script with the following flags:
|
||||
|
||||
* `KUBE_DELETE_NODES=false` - to restrain deletion of kubelets.
|
||||
|
||||
* `KUBE_GCE_ZONE=zone` - the zone from where master replica will be removed.
|
||||
|
||||
* `KUBE_REPLICA_NAME=replica_name` - (optional) the name of master replica to remove.
|
||||
If empty: any replica from the given zone will be removed.
|
||||
|
||||
The following sample command removes a master replica from an existing HA cluster:
|
||||
|
||||
```shell
|
||||
$ KUBE_DELETE_NODES=false KUBE_GCE_ZONE=europe-west1-c ./cluster/kube-down.sh
|
||||
```
|
||||
|
||||
## Handling master replica failures
|
||||
|
||||
If one of the master replicas in your HA cluster fails,
|
||||
the best practice is to remove the replica from your cluster and add a new replica in the same zone.
|
||||
The following sample commands demonstrate this process:
|
||||
|
||||
1. Remove the broken replica:
|
||||
|
||||
```shell
|
||||
$ KUBE_DELETE_NODES=false KUBE_GCE_ZONE=replica_zone KUBE_REPLICA_NAME=replica_name ./cluster/kube-down.sh
|
||||
```
|
||||
|
||||
2. Add a new replica in place of the old one:
|
||||
|
||||
```shell
|
||||
$ KUBE_GCE_ZONE=replica-zone KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh
|
||||
```
|
||||
|
||||
## Best practices for replicating masters for HA clusters
|
||||
|
||||
* Try to place masters replicas in different zones. During a zone failure, all master placed inside the zone will fail.
|
||||
To survive zone failure, also place nodes in multiple zones
|
||||
(see [multiple-zones](http://kubernetes.io/docs/admin/multiple-zones/) for details).
|
||||
|
||||
* Do not use a cluster with two master replicas. Consensus on a two replica cluster requires both replicas running when changing persistent state.
|
||||
As a result, both replicas are needed and a failure of any replica turns cluster into majority failure state.
|
||||
A two-replica cluster is thus inferior, in terms of HA, to a single replica cluster.
|
||||
|
||||
* When you add a master replica, cluster state (etcd) is copied to a new instance.
|
||||
If the cluster is large, it may take a long time to duplicate its state.
|
||||
This operation may be speed up by migrating etcd data directory, as described [here](https://coreos.com/etcd/docs/latest/admin_guide.html#member-migration) here
|
||||
(we are considering adding support for etcd data dir migration in future).
|
||||
|
||||
## Implementation notes
|
||||
|
||||
|
||||
|
||||
### Overview
|
||||
|
||||
Each of master replicas will run the following components in the following mode:
|
||||
|
||||
* etcd instance: all instances will be clustered together using consensus;
|
||||
|
||||
* API server: each server will talk to local etcd - all API servers in the cluster will be available;
|
||||
|
||||
* controllers, scheduler, and cluster auto-scaler: will use lease mechanism - only one instance of each of them will be active in the cluster;
|
||||
|
||||
* add-on manager: each manager will work independently trying to keep add-ons in sync.
|
||||
|
||||
In addition, there will be a load balancer in front of API servers that will route external and internal traffic to them.
|
||||
|
||||
### Load balancing
|
||||
|
||||
When starting the second master replica, a load balancer containing the two replicas will be created
|
||||
and the IP address of the first replica will be promoted to IP address of load balancer.
|
||||
Similarly, after removal of the penultimate master replica, the load balancer will be removed and its IP address will be assigned to the last remaining replica.
|
||||
Please note that creation and removal of load balancer are complex operations and it may take some time (~20 minutes) for them to propagate.
|
||||
|
||||
### Master service & kubelets
|
||||
|
||||
Instead of trying to keep an up-to-date list of Kubernetes apiserver in the Kubernetes service,
|
||||
the system directs all traffic to the external IP:
|
||||
|
||||
* in one master cluster the IP points to the single master,
|
||||
|
||||
* in multi-master cluster the IP points to the load balancer in-front of the masters.
|
||||
|
||||
Similarly, the external IP will be used by kubelets to communicate with master.
|
||||
|
||||
### Master certificates
|
||||
|
||||
Kubernetes generates Master TLS certificates for the external public IP and local IP for each replica.
|
||||
There are no certificates for the ephemeral public IP for replicas;
|
||||
to access a replica via its ephemeral public IP, you must skip TLS verification.
|
||||
|
||||
### Clustering etcd
|
||||
|
||||
To allow etcd clustering, ports needed to communicate between etcd instances will be opened (for inside cluster communication).
|
||||
To make such deployment secure, communication between etcd instances is authorized using SSL.
|
||||
|
||||
## Additional reading
|
||||
|
||||
[Automated HA master deployment - design doc](https://github.com/kubernetes/kubernetes/blob/master/docs/design/ha_master.md)
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
@@ -84,3 +84,8 @@ project](/docs/admin/salt).
|
||||
* **Sysctls** [sysctls](/docs/admin/sysctls.md)
|
||||
|
||||
* **Audit** [audit](/docs/admin/audit)
|
||||
|
||||
* **Securing the kubelet**
|
||||
* [Master-Node communication](/docs/admin/master-node-communication/)
|
||||
* [TLS bootstrapping](/docs/admin/kubelet-tls-bootstrapping/)
|
||||
* [Kubelet authentication/authorization](/docs/admin/kubelet-authentication-authorization/)
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
assignees:
|
||||
- liggitt
|
||||
|
||||
---
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Overview
|
||||
|
||||
A kubelet's HTTPS endpoint exposes APIs which give access to data of varying sensitivity,
|
||||
and allow you to perform operations with varying levels of power on the node and within containers.
|
||||
|
||||
This document describes how to authenticate and authorize access to the kubelet's HTTPS endpoint.
|
||||
|
||||
## Kubelet authentication
|
||||
|
||||
By default, requests to the kubelet's HTTPS endpoint that are not rejected by other configured
|
||||
authentication methods are treated as anonymous requests, and given a username of `system:anonymous`
|
||||
and a group of `system:unauthenticated`.
|
||||
|
||||
To disable anonymous access and send `401 Unauthorized` responses to unauthenticated requests:
|
||||
* start the kubelet with the `--anonymous-auth=false` flag
|
||||
|
||||
To enable X509 client certificate authentication to the kubelet's HTTPS endpoint:
|
||||
* start the kubelet with the `--client-ca-file` flag, providing a CA bundle to verify client certificates with
|
||||
* start the apiserver with `--kubelet-client-certificate` and `--kubelet-client-key` flags
|
||||
* see the [apiserver authentication documentation](/docs/admin/authentication/#x509-client-certs) for more details
|
||||
|
||||
To enable API bearer tokens (including service account tokens) to be used to authenticate to the kubelet's HTTPS endpoint:
|
||||
* ensure the `authentication.k8s.io/v1beta1` API group is enabled in the API server
|
||||
* start the kubelet with the `--authentication-token-webhook`, `--kubeconfig`, and `--require-kubeconfig` flags
|
||||
* the kubelet calls the `TokenReview` API on the configured API server to determine user information from bearer tokens
|
||||
|
||||
## Kubelet authorization
|
||||
|
||||
Any request that is successfully authenticated (including an anonymous request) is then authorized. The default authorization mode is `AlwaysAllow`, which allows all requests.
|
||||
|
||||
There are many possible reasons to subdivide access to the kubelet API:
|
||||
* anonymous auth is enabled, but anonymous users' ability to call the kubelet API should be limited
|
||||
* bearer token auth is enabled, but arbitrary API users' (like service accounts) ability to call the kubelet API should be limited
|
||||
* client certificate auth is enabled, but only some of the client certificates signed by the configured CA should be allowed to use the kubelet API
|
||||
|
||||
To subdivide access to the kubelet API, delegate authorization to the API server:
|
||||
* ensure the `authorization.k8s.io/v1beta1` API group is enabled in the API server
|
||||
* start the kubelet with the `--authorization-mode=Webhook`, `--kubeconfig`, and `--require-kubeconfig` flags
|
||||
* the kubelet calls the `SubjectAccessReview` API on the configured API server to determine whether each request is authorized
|
||||
|
||||
The kubelet authorizes API requests using the same [request attributes](/docs/admin/authorization/#request-attributes) approach as the apiserver.
|
||||
|
||||
The verb is determined from the incoming request's HTTP verb:
|
||||
|
||||
HTTP verb | request verb
|
||||
----------|---------------
|
||||
POST | create
|
||||
GET, HEAD | get
|
||||
PUT | update
|
||||
PATCH | patch
|
||||
DELETE | delete
|
||||
|
||||
The resource and subresource is determined from the incoming request's path:
|
||||
|
||||
Kubelet API | resource | subresource
|
||||
-------------|----------|------------
|
||||
/stats/* | nodes | stats
|
||||
/metrics/* | nodes | metrics
|
||||
/logs/* | nodes | log
|
||||
/spec/* | nodes | spec
|
||||
*all others* | nodes | proxy
|
||||
|
||||
The namespace and API group attributes are always an empty string, and
|
||||
the resource name is always the name of the kubelet's `Node` API object.
|
||||
|
||||
When running in this mode, ensure the user identified by the `--kubelet-client-certificate` and `--kubelet-client-key`
|
||||
flags passed to the apiserver is authorized for the following attributes:
|
||||
* verb=*, resource=nodes, subresource=proxy
|
||||
* verb=*, resource=nodes, subresource=stats
|
||||
* verb=*, resource=nodes, subresource=log
|
||||
* verb=*, resource=nodes, subresource=spec
|
||||
* verb=*, resource=nodes, subresource=metrics
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
assignees:
|
||||
- mikedanese
|
||||
|
||||
---
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes how to set up TLS client certificate boostrapping for kubelets.
|
||||
Kubernetes 1.4 introduces an experimental API for requesting certificates from a cluster-level
|
||||
Certificate Authority (CA). The first supported use of this API is the provisioning of TLS client
|
||||
certificates for kubelets. The proposal can be found [here](https://github.com/kubernetes/kubernetes/pull/20439)
|
||||
and progress on the feature is being tracked as [feature #43](https://github.com/kubernetes/features/issues/43).
|
||||
|
||||
## apiserver configuration
|
||||
|
||||
You must provide a token file which specifies at least one "bootstrap token" assigned to a kubelet boostrap-specific group.
|
||||
This group will later be used in the controller-manager configuration to scope approvals in the default approval
|
||||
controller. As this feature matures, you should ensure tokens are bound to an RBAC policy which limits requests
|
||||
using the bootstrap token to only be able to make requests related to certificate provisioning. When RBAC policy
|
||||
is in place, scoping the tokens to a group will allow great flexibility (e.g. you could disable a particular
|
||||
bootstrap group's access when you are done provisioning the nodes).
|
||||
|
||||
### Token auth file
|
||||
Tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number
|
||||
generator (such as /dev/urandom on most modern systems). There are multiple ways you can generate a token. For example:
|
||||
|
||||
`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`
|
||||
|
||||
will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8`
|
||||
|
||||
The token file will look like the following example, where the first three values can be anything and the quoted group
|
||||
name should be as depicted:
|
||||
|
||||
```
|
||||
02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
|
||||
```
|
||||
|
||||
Add the `--token-auth-file=FILENAME` flag to the apiserver command to enable the token file.
|
||||
See docs at http://kubernetes.io/docs/admin/authentication/#static-token-file for further details.
|
||||
|
||||
### Client certificate CA bundle
|
||||
|
||||
Add the `--client-ca-file=FILENAME` flag to the apiserver command to enable client certificate authentication,
|
||||
referencing a certificate authority bundle containing the signing certificate.
|
||||
|
||||
## controller-manager configuration
|
||||
The API for requesting certificates adds a certificate-issuing control loop to the KCM. This takes the form of a
|
||||
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk.
|
||||
Currently, all certificates issued have one year validity and a default set of key usages.
|
||||
|
||||
### Signing assets
|
||||
You must provide a Certificate Authority in order to provide the cryptographic materials necessary to issue certificates.
|
||||
This CA should be trusted by the apiserver for authentication with the `--client-ca-file=SOMEFILE` flag. The management
|
||||
of the CA is beyond the scope of this document but it is recommended that you generate a dedicated CA for Kubernetes.
|
||||
Both certificate and key are assumed to be PEM-encoded.
|
||||
|
||||
The new controller-manager flags are:
|
||||
```
|
||||
--cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key"
|
||||
```
|
||||
|
||||
### Auto-approval
|
||||
To ease deployment and testing, the alpha version of the certificate request API includes a flag to approve all certificate
|
||||
requests made by users in a certain group. The intended use of this is to whitelist only the group corresponding to the bootstrap
|
||||
token in the token file above. Use of this flag circumvents makes the "approval" process described below and is not recommended
|
||||
for production use.
|
||||
|
||||
The flag is:
|
||||
```
|
||||
--insecure-experimental-approve-all-kubelet-csrs-for-group="system:kubelet-bootstrap"
|
||||
```
|
||||
|
||||
## kubelet configuration
|
||||
To use request a client cert from the certificate request API, the kubelet needs a path to a kubeconfig file that contains the
|
||||
bootstrap auth token. If the file specified by `--kubeconfig` does not exist, the bootstrap kubeconfig is used to request a
|
||||
client certificate from the API server. On success, a kubeconfig file referencing the generated key and obtained certificate
|
||||
is written to the path specified by `--kubeconfig`. The certificate and key file will be stored in the directory pointed
|
||||
by `--cert-dir`. The new flag is:
|
||||
|
||||
```
|
||||
--experimental-bootstrap-kubeconfig="/path/to/bootstrap/kubeconfig"
|
||||
```
|
||||
|
||||
## kubectl approval
|
||||
The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an
|
||||
"Approved" status by an appropriately-privileged user. This is intended to eventually be an automated process handled by an external
|
||||
approval controller, but for the alpha version of the API it can be done manually by a cluster administrator using kubectl.
|
||||
An administrator can list CSRs with `kubectl get csr`, describe one in detail with `kubectl describe <name>`. There are
|
||||
[currently no direct approve/deny commands](https://github.com/kubernetes/kubernetes/issues/30163) so an approver will need to update
|
||||
the Status field directly. A rough example of how to do this in bash which should only be used until the porcelain merges is available
|
||||
at [https://github.com/gtank/csrctl](https://github.com/gtank/csrctl).
|
||||
|
||||
@@ -2,13 +2,14 @@
|
||||
assignees:
|
||||
- dchen1107
|
||||
- roberthbailey
|
||||
- liggitt
|
||||
|
||||
---
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Summary
|
||||
## Overview
|
||||
|
||||
This document catalogs the communication paths between the master (really the
|
||||
apiserver) and the Kubernetes cluster. The intent is to allow users to
|
||||
@@ -22,14 +23,21 @@ All communication paths from the cluster to the master terminate at the
|
||||
apiserver (none of the other master components are designed to expose remote
|
||||
services). In a typical deployment, the apiserver is configured to listen for
|
||||
remote connections on a secure HTTPS port (443) with one or more forms of
|
||||
client [authentication](/docs/admin/authentication/) enabled.
|
||||
client [authentication](/docs/admin/authentication/) enabled. One or more forms
|
||||
of [authorization](/docs/admin/authorization/) should be enabled, especially
|
||||
if [anonymous requests](/docs/admin/authentication/#anonymous-requests) or
|
||||
[service account tokens](/docs/admin/authentication/#service-account-tokens)
|
||||
are allowed.
|
||||
|
||||
Nodes should be provisioned with the public root certificate for the cluster
|
||||
such that they can connect securely to the apiserver along with valid client
|
||||
credentials. For example, on a default GCE deployment, the client credentials
|
||||
provided to the kubelet are in the form of a client certificate. Pods that
|
||||
wish to connect to the apiserver can do so securely by leveraging a service
|
||||
account so that Kubernetes will automatically inject the public root
|
||||
provided to the kubelet are in the form of a client certificate. See
|
||||
[kubelet TLS bootstrapping](/docs/admin/kubelet-tls-bootstrapping/) for
|
||||
automated provisioning of kubelet client certificates.
|
||||
|
||||
Pods that wish to connect to the apiserver can do so securely by leveraging a
|
||||
service account so that Kubernetes will automatically inject the public root
|
||||
certificate and a valid bearer token into the pod when it is instantiated.
|
||||
The `kubernetes` service (in all namespaces) is configured with a virtual IP
|
||||
address that is redirected (via kube-proxy) to the HTTPS endpoint on the
|
||||
@@ -54,16 +62,29 @@ cluster. The first is from the apiserver to the kubelet process which runs on
|
||||
each node in the cluster. The second is from the apiserver to any node, pod,
|
||||
or service through the apiserver's proxy functionality.
|
||||
|
||||
### apiserver -> kubelet
|
||||
|
||||
The connections from the apiserver to the kubelet are used for fetching logs
|
||||
for pods, attaching (through kubectl) to running pods, and using the kubelet's
|
||||
port-forwarding functionality. These connections terminate at the kubelet's
|
||||
HTTPS endpoint, which is typically using a self-signed certificate, and
|
||||
ignore the certificate presented by the kubelet (although you can override this
|
||||
behavior by specifying the `--kubelet-certificate-authority`,
|
||||
`--kubelet-client-certificate`, and `--kubelet-client-key` flags when starting
|
||||
the cluster apiserver). By default, these connections **are not currently safe**
|
||||
to run over untrusted and/or public networks as they are subject to
|
||||
man-in-the-middle attacks.
|
||||
port-forwarding functionality. These connections terminate at the kubelet's
|
||||
HTTPS endpoint.
|
||||
|
||||
By default, the apiserver does not verify the kubelet's serving certificate,
|
||||
which makes the connection subject to man-in-the-middle attacks, and
|
||||
**unsafe** to run over untrusted and/or public networks.
|
||||
|
||||
To verify this connection, use the `--kubelet-certificate-authority` flag to
|
||||
provide the apiserver with a root certificates bundle to use to verify the
|
||||
kubelet's serving certificate.
|
||||
|
||||
If that is not possible, use [SSH tunneling](/docs/admin/master-node-communication/#ssh-tunnels)
|
||||
between the apiserver and kubelet if required to avoid connecting over an
|
||||
untrusted or public network.
|
||||
|
||||
Finally, [Kubelet authentication and/or authorization](/docs/admin/kubelet-authentication-authorization/)
|
||||
should be enabled to secure the kubelet API.
|
||||
|
||||
### apiserver -> nodes, pods, and services
|
||||
|
||||
The connections from the apiserver to a node, pod, or service default to plain
|
||||
HTTP connections and are therefore neither authenticated nor encrypted. They
|
||||
@@ -83,83 +104,3 @@ cluster (connecting to the ssh server listening on port 22) and passes all
|
||||
traffic destined for a kubelet, node, pod, or service through the tunnel.
|
||||
This tunnel ensures that the traffic is not exposed outside of the private
|
||||
GCE network in which the cluster is running.
|
||||
|
||||
### Kubelet TLS Bootstrap
|
||||
|
||||
Kubernetes 1.4 introduces an experimental API for requesting certificates from a cluster-level
|
||||
Certificate Authority (CA). The first supported use of this API is the provisioning of TLS client
|
||||
certificates for kubelets. The proposal can be found [here](https://github.com/kubernetes/kubernetes/pull/20439)
|
||||
and progress on the feature is being tracked as [feature #43](https://github.com/kubernetes/features/issues/43).
|
||||
|
||||
##### apiserver configuration
|
||||
You must provide a token file which specifies at least one "bootstrap token" assigned to a kubelet boostrap-specific group.
|
||||
This group will later be used in the controller-manager configuration to scope approvals in the default approval
|
||||
controller. As this feature matures, you should ensure tokens are bound to an RBAC policy which limits requests
|
||||
using the bootstrap token to only be able to make requests related to certificate provisioning. When RBAC policy
|
||||
is in place, scoping the tokens to a group will allow great flexibility (e.g. you could disable a particular
|
||||
bootstrap group's access when you are done provisioning the nodes).
|
||||
|
||||
##### Token auth file
|
||||
Tokens are arbitrary but should represent at least 128 bits of entropy derived from a secure random number
|
||||
generator (such as /dev/urandom on most modern systems). There are multiple ways you can generate a token. For example:
|
||||
|
||||
`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`
|
||||
|
||||
will generate tokens that look like `02b50b05283e98dd0fd71db496ef01e8`
|
||||
|
||||
The token file will look like the following example, where the first three values can be anything and the quoted group
|
||||
name should be as depicted:
|
||||
|
||||
```
|
||||
02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
|
||||
```
|
||||
|
||||
Add the `--token-auth-file=FILENAME` flag to the apiserver command to enable the token file.
|
||||
See docs at http://kubernetes.io/docs/admin/authentication/#static-token-file for further details.
|
||||
|
||||
#### controller-manager configuration
|
||||
The API for requesting certificates adds a certificate-issuing control loop to the KCM. This takes the form of a
|
||||
[cfssl](https://blog.cloudflare.com/introducing-cfssl/) local signer using assets on disk.
|
||||
Currently, all certificates issued have one year validity and a default set of key usages.
|
||||
|
||||
##### Signing assets
|
||||
You must provide a Certificate Authority in order to provide the cryptographic materials necessary to issue certificates.
|
||||
This CA should be trusted by the apiserver for authentication with the `--client-ca-file=SOMEFILE` flag. The management
|
||||
of the CA is beyond the scope of this document but it is recommended that you generate a dedicated CA for Kubernetes.
|
||||
Both certificate and key are assumed to be PEM-encoded.
|
||||
|
||||
The new controller-manager flags are:
|
||||
```
|
||||
--cluster-signing-cert-file="/etc/path/to/kubernetes/ca/ca.crt" --cluster-signing-key-file="/etc/path/to/kubernetes/ca/ca.key"
|
||||
```
|
||||
|
||||
##### Auto-approval
|
||||
To ease deployment and testing, the alpha version of the certificate request API includes a flag to approve all certificate
|
||||
requests made by users in a certain group. The intended use of this is to whitelist only the group corresponding to the bootstrap
|
||||
token in the token file above. Use of this flag circumvents makes the "approval" process described below and is not recommended
|
||||
for production use.
|
||||
|
||||
The flag is:
|
||||
```
|
||||
--insecure-experimental-approve-all-kubelet-csrs-for-group="system:kubelet-bootstrap"
|
||||
```
|
||||
|
||||
#### kubelet configuration
|
||||
To use request a client cert from the certificate request API, the kubelet needs a path to a kubeconfig file that contains the
|
||||
bootstrap auth token. If the file specified by `--kubeconfig` does not exist, the bootstrap kubeconfig is used to request a
|
||||
client certificate from the API server. On success, a kubeconfig file referencing the generated key and obtained certificate
|
||||
is written to the path specified by `--kubeconfig`. The certificate and key file will be stored in the directory pointed
|
||||
by `--cert-dir`. The new flag is:
|
||||
|
||||
```
|
||||
--experimental-bootstrap-kubeconfig="/path/to/bootstrap/kubeconfig"
|
||||
```
|
||||
|
||||
#### kubectl approval
|
||||
The signing controller does not immediately sign all certificate requests. Instead, it waits until they have been flagged with an
|
||||
"Approved" status by an appropriately-privileged user. This is intended to eventually be an automated process handled by an external
|
||||
approval controller, but for the alpha version of the API it can be done manually by a cluster administrator using kubectl.
|
||||
An administrator can list CSRs with `kubectl get csr`, describe one in detail with `kubectl describe <name>`. There are
|
||||
[currently no direct approve/deny commands](https://github.com/kubernetes/kubernetes/issues/30163) so an approver will need to update
|
||||
the Status field directly. A rough example of how to do this in bash which should only be used until the porcelain merges is available
|
||||
at https://github.com/gtank/csrctl.
|
||||
|
||||
@@ -9,54 +9,52 @@ assignees:
|
||||
|
||||
## Node Conformance Test
|
||||
|
||||
*Node conformance test* is a test framework validating whether a node meets the
|
||||
minimum requirement of Kubernetes with a set of system verification and
|
||||
functionality test. A node which passes the tests is qualified to join a
|
||||
Kubernetes cluster.
|
||||
*Node conformance test* is a containerized test framework that provides a system
|
||||
verification and functionality test for a node. The test validates whether the
|
||||
node meets the minimum requirements for Kubernetes; a node that passes the test
|
||||
is qualified to join a Kubernetes cluster.
|
||||
|
||||
## Limitations
|
||||
|
||||
There are following limitations in the current implementation of node
|
||||
conformance test. They'll be improved in future version.
|
||||
In Kubernetes version 1.5, node conformance test has the following limitations:
|
||||
|
||||
* Node conformance test only supports Docker as the container runtime.
|
||||
* Node conformance test doesn't validate network related system configurations
|
||||
and functionalities.
|
||||
|
||||
## Prerequisite
|
||||
## Node Prerequisite
|
||||
|
||||
Node conformance test is used to test whether a node is ready to join a
|
||||
Kubernetes cluster, so the prerequisite is the same with a standard Kubernetes
|
||||
node. At least, the node should have properly installed:
|
||||
To run node conformance test, a node must satisfy the same prerequisites as a
|
||||
standard Kubernetes node. At a minimum, the node should have the following
|
||||
daemons installed:
|
||||
|
||||
* Container Runtime (Docker)
|
||||
* Kubelet
|
||||
|
||||
Node conformance test validates kernel configurations. If the kenrel module
|
||||
`configs` is built as module in your environment, it must be loaded before the
|
||||
test. (See [Caveats #3](#caveats) for more information)
|
||||
## Running Node Conformance Test
|
||||
|
||||
## Usage
|
||||
To run the node conformance test, perform the following steps:
|
||||
|
||||
### Run Node Conformance Test
|
||||
1. Point your Kubelet to localhost `--api-servers="http://localhost:8080"`,
|
||||
because the test framework starts a local master to test Kubelet. There are some
|
||||
other Kubelet flags you may care:
|
||||
* `--pod-cidr`: If you are using `kubenet`, you should specify an arbitrary CIDR
|
||||
to Kubelet, for example `--pod-cidr=10.180.0.0/24`.
|
||||
* `--cloud-provider`: If you are using `--cloud-provider=gce`, you should
|
||||
remove the flag to run the test.
|
||||
|
||||
* **Step 1:** Point your Kubelet to localhost `--api-servers="http://localhost:8080"`,
|
||||
because the test framework starts a local master to test Kubelet.
|
||||
|
||||
* **Step 2:** Run the node conformance test with command:
|
||||
2. Run the node conformance test with command:
|
||||
|
||||
```shell
|
||||
# $CONFIG_DIR is the pod manifest path of your kubelet.
|
||||
# $CONFIG_DIR is the pod manifest path of your Kubelet.
|
||||
# $LOG_DIR is the test output path.
|
||||
sudo docker run -it --rm --privileged --net=host \
|
||||
-v /:/rootfs:ro -v /var/run:/var/run \
|
||||
-v $CONFIG_DIR:/etc/manifest -v $LOG_DIR:/var/result \
|
||||
gcr.io/google_containers/node-test-amd64:v0.1
|
||||
-v /:/rootfs -v $CONFIG_DIR:$CONFIG_DIR -v $LOG_DIR:/var/result \
|
||||
gcr.io/google_containers/node-test:0.2
|
||||
```
|
||||
|
||||
### Run Node Conformance Test for Other Architectures
|
||||
## Running Node Conformance Test for Other Architectures
|
||||
|
||||
We also build node conformance test docker images for other architectures:
|
||||
Kubernetes also provides node conformance test docker images for other
|
||||
architectures:
|
||||
|
||||
Arch | Image |
|
||||
--------|:-----------------:|
|
||||
@@ -64,25 +62,16 @@ We also build node conformance test docker images for other architectures:
|
||||
arm | node-test-arm |
|
||||
arm64 | node-test-arm64 |
|
||||
|
||||
### Run Selected Test
|
||||
|
||||
In fact, Node conformance test is a containerized version of [node e2e
|
||||
test](https://github.com/kubernetes/kubernetes/blob/release-1.4/docs/devel/e2e-node-tests.md).
|
||||
By default, it runs all conformance test.
|
||||
|
||||
Theoretically, you can run any node e2e test if you configure the container and
|
||||
mount required volumes properly. But **it is strongly recommended to only run conformance
|
||||
test**, because the non-conformance test needs much more complex framework configuration.
|
||||
## Running Selected Test
|
||||
|
||||
To run specific tests, overwrite the environment variable `FOCUS` with the
|
||||
regular expression of tests you want to run.
|
||||
|
||||
```shell
|
||||
sudo docker run -it --rm --privileged --net=host \
|
||||
-v /:/rootfs:ro -v /var/run:/var/run \
|
||||
-v $CONFIG_DIR:/etc/manifest -v $LOG_DIR:/var/result \
|
||||
-v /:/rootfs:ro -v $CONFIG_DIR:$CONFIG_DIR -v $LOG_DIR:/var/result \
|
||||
-e FOCUS=MirrorPod \ # Only run MirrorPod test
|
||||
gcr.io/google_containers/node-test-amd64:v0.1
|
||||
gcr.io/google_containers/node-test:0.2
|
||||
```
|
||||
|
||||
To skip specific tests, overwrite the environment variable `SKIP` with the
|
||||
@@ -90,25 +79,22 @@ regular expression of tests you want to skip.
|
||||
|
||||
```shell
|
||||
sudo docker run -it --rm --privileged --net=host \
|
||||
-v /:/rootfs:ro -v /var/run:/var/run \
|
||||
-v $CONFIG_DIR:/etc/manifest -v $LOG_DIR:/var/result \
|
||||
-e SKIP=MirrorPod \ # Run all conformance test and skip MirrorPod test
|
||||
gcr.io/google_containers/node-test-amd64:v0.1
|
||||
-v /:/rootfs:ro -v $CONFIG_DIR:$CONFIG_DIR -v $LOG_DIR:/var/result \
|
||||
-e SKIP=MirrorPod \ # Run all conformance tests but skip MirrorPod test
|
||||
gcr.io/google_containers/node-test:0.2
|
||||
```
|
||||
|
||||
### Caveats
|
||||
Node conformance test is a containerized version of [node e2e test](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/devel/e2e-node-tests.md).
|
||||
By default, it runs all conformance tests.
|
||||
|
||||
* The test will leave some docker images on the node, including the node
|
||||
conformance test image and images of containers used in the functionality
|
||||
Theoretically, you can run any node e2e test if you configure the container and
|
||||
mount required volumes properly. But **it is strongly recommended to only run conformance
|
||||
test**, because it requires much more complex configuration to run non-conformance test.
|
||||
|
||||
## Caveats
|
||||
|
||||
* The test leaves some docker images on the node, including the node conformance
|
||||
test image and images of containers used in the functionality
|
||||
test.
|
||||
* The test will leave dead containers on the node, these containers are created
|
||||
* The test leaves dead containers on the node. These containers are created
|
||||
during the functionality test.
|
||||
* Node conformance test validates kernel configuration. However, in some os
|
||||
distro the kernel module `configs` may not be loaded by default, and you will get
|
||||
the error `no config path in [POSSIBLE KERNEL CONFIG FILE PATHS] is
|
||||
available`. In that case please do either of the followings:
|
||||
* Manually load/unload `configs` kernel module: run `sudo modprobe configs` to
|
||||
load the kernel module, and `sudo modprobe -r configs` to unload it after the test.
|
||||
* Mount `modprobe` into the container: Add option `-v /bin/kmod:/bin/kmod
|
||||
-v /sbin/modprobe:/sbin/modprobe -v /lib/modules:/lib/modules` when starting
|
||||
the test container.
|
||||
|
||||
+3
-4
@@ -54,10 +54,9 @@ The node condition is represented as a JSON object. For example, the following r
|
||||
]
|
||||
```
|
||||
|
||||
If the Status of the Ready condition is Unknown or False for more than five
|
||||
minutes, then all of the pods on the node are terminated by the node
|
||||
controller. (The timeout length is configurable by the `--pod-eviction-timeout`
|
||||
parameter on the controller manager.)
|
||||
If the Status of the Ready condition is "Unknown" or "False" for longer than the `pod-eviction-timeout`, an argument passed to the [kube-controller-manager](docs/admin/kube-controller-manager/), all of the Pods on the node are scheduled for deletion by the Node Controller. The default eviction timeout duration is **five minutes**. In some cases when the node is unreachable, the apiserver is unable to communicate with the kubelet on it. The decision to delete the pods cannot be communicated to the kubelet until it re-establishes communication with the apiserver. In the meantime, the pods which are scheduled for deletion may continue to run on the partitioned node.
|
||||
|
||||
In versions of Kubernetes prior to 1.5, the node controller would [force delete](/docs/user-guide/pods/#force-deletion-of-pods) these unreachable pods from the apiserver. However, in 1.5 and higher, the node controller does not force delete pods until it is confirmed that they have stopped running in the cluster. One can see these pods which may be running on an unreachable node as being in the "Terminating" or "Unknown" states. In cases where Kubernetes cannot deduce from the underlying infrastructure if a node has permanently left a cluster, the cluster administrator may need to delete the node object by hand. Deleting the node object from Kubernetes causes all the Pod objects running on it to be deleted from the apiserver, freeing up their names.
|
||||
|
||||
### Capacity
|
||||
|
||||
|
||||
Reference in New Issue
Block a user