Release 1.8 (#5659)

* GC now supports non-core resources * Add two examples about how to analysis audits of kube-apiserver (#4264) * Deprecate system:nodes binding * [1.8] StatefulSet `initialized` annotation is now ignored. * inits the kubeadm upgrade docs addresses kubernetes/kubernetes.github.io/issues/4689 * adds kubeadm upgrade cmd to ToC addresses kubernetes/kubernetes.github.io/issues/4689 * add workload placement docs * ScaleIO - document udpate for 1.8 * Add documentation on storageClass.mountOptions and PV.mountOptions (#5254) * Add documentation on storageClass.mountOptions and PV.mountOptions * convert notes into callouts * Add docs for CustomResource validation add info about supported fields * advanced audit beta features (#5300) * Update job workload doc with backoff failure policy (#5319) Add to the Jobs documentation how to use the new backoffLimit field that limit the number of Pod failure before considering the Job as failed. * Documented additional AWS Service annotations (#4864) * Add device plugin doc under concepts/cluster-administration. (#5261) * Add device plugin doc under concepts/cluster-administration. * Update device-plugins.md * Update device-plugins.md Add meta description. Fix typo. Change bare metal deployment to manual deployment. * Update device-plugins.md Fix typo again. * Update page.version. (#5341) * Add documentation on storageClass.reclaimPolicy (#5171) * [Advanced audit] use new herf for audit-api (#5349) This tag contains all the changes in v1beta1 version. Update it now. * Added documentation around creating the InitializerConfiguration for the persistent volume label controller in the cloud-controller-manager (#5255) * Documentation for kubectl plugins (#5294) * Documentation for kubectl plugins * Update kubectl-plugins.md * Update kubectl-plugins.md * Updated CPU manager docs to match implementation. (#5332) * Noted limitation of alpha static cpumanager. * Updated CPU manager docs to match implementation. - Removed references to CPU pressure node condition and evictions. - Added note about new --cpu-manager-reconcile-period flag. - Added note about node allocatable requirements for static policy. - Noted limitation of alpha static cpumanager. * Move cpu-manager task link to rsc mgmt section. * init containers annotation removed in 1.8 (#5390) * Add documentation for TaintNodesByCondition (#5352) * Add documentation for TaintNodesByCondition * Update nodes.md * Update taint-and-toleration.md * Update daemonset.md * Update nodes.md * Update taint-and-toleration.md * Update daemonset.md * Fix deployments (#5421) * Document extended resources and OIR deprecation. (#5399) * Document extended resources and OIR deprecation. * Updated extended resources doc per reviews. * reverts extra spacing in _data/tasks.yml * addresses `kubeadm upgrade` review comments Feedback from @chenopis, @luxas, and @steveperry-53 addressed with this commit * HugePages documentation (#5419) * Update cpu-management-policies.md (#5407) Fixed the bad link. Modified "cpu" to "CPU". Added more 'yaml' as supplement. * Update RBAC docs for v1 (#5445) * Add user docs for pod priority and preemption (#5328) * Add user docs for pod priority and preemption * Update pod-priority-preemption.md * More updates * Update docs/admin/kubeadm.md for 1.8 (#5440) - Made a couple of minor wording changes (not strictly 1.8 related). - Did some reformatting (not strictly 1.8 related). - Updated references to the default token TTL (was infinite, now 24 hours). - Documented the new `--discovery-token-ca-cert-hash` and `--discovery-token-unsafe-skip-ca-verification` flags for `kubeadm join`. - Added references to the new `--discovery-token-ca-cert-hash` flag in all the default examples. - Added a new _Security model_ section that describes the security tradeoffs of the various discovery modes. - Documented the new `--groups` flag for `kubeadm token create`. - Added a note of caution under _Automating kubeadm_ that references the _Security model_ section. - Updated the component version table to drop 1.6 and add 1.8. - Update `_data/reference.yml` to try to get the sidebar fixed up and more consistent with `kubefed`. * Update StatefulSet Basics for 1.8 release (#5398) * addresses `kubeadm upgrade` review comments 2nd iteration review comments by @luxas * adds kubelet upgrade section to kubeadm upgrade * Fix a bulleted list on docs/admin/kubeadm.md. (#5458) I updated this doc yesterday and I was absolutely sure I fixed this, but I just saw that this commit got lost somehow. This was introduced recently in https://github.com/kubernetes/kubernetes.github.io/pull/5440. * Clarify the API to check for device plugins * Moving Flexvolume to separate out-of-tree section * addresses `kubeadm upgrade` review comments CC: @luxas * fixes kubeadm upgrade index * Update Stackdriver Logging documentation (#5495) * Re-update WordPress and MySQL PV doc to use apps/v1beta2 APIs (#5526) * Update statefulset concepts doc to use apps/v1beta2 APIs (#5420) * add document on kubectl's behavior regarding initializers (#5505) * Update docs/admin/kubeadm.md to cover self-hosting in 1.8. (#5497) This is a new beta feature in 1.8. * Update kubectl patch doc to use apps/v1beta2 APIs (#5422) * [1.8] Update "Run Applications" tasks to apps/v1beta2. (#5525) * Update replicated stateful application task for 1.8. * Update single instance stateful app task for 1.8. * Update stateless app task for 1.8. * Update kubectl patch task for 1.8. * fix the link of persistent storage (#5515) * update the admission-controllers.md index.md what-is-kubernetes.md link * fix the link of persistent storage * Add quota support for local ephemeral storage (#5493) * Add quota support for local ephemeral storage update the doc to this alpha feature * Update resource-quotas.md * Updated Deployments concepts doc (#5491) * Updated Deployments concepts doc * Addressed comments * Addressed more comments * Modify allocatable storage to ephemeral-storage (#5490) Update the doc to use ephemeral-storage instead of storage * Revamped concepts doc for ReplicaSet (#5463) * Revamped concepts doc for ReplicaSet * Minor changes to call out specific versions for selector defaulting and immutability * Addressed doc review comments * Remove petset documentations (#5395) * Update docs to use batch/v1beta1 cronjobs (#5475) * add federation job doc (#5485) * add federation job doc * Update job.md Edits for clarity and consistency * Update job.md Fixed a typo * update DaemonSet concept for 1.8 release (#5397) * update DaemonSet concept for 1.8 release * Update daemonset.md Fix typo. than -> then * Update bootstrap tokens doc for 1.8. (#5479) * Update bootstrap tokens doc for 1.8. This has some changes I missed when I was updating the main kubeadm documention: - Bootstrap tokens are now beta, not alpha (https://github.com/kubernetes/features/issues/130) - The apiserver flag to enable the authenticator changedin 1.8 (https://github.com/kubernetes/kubernetes/pull/51198) - Added `auth-extra-groups` documentaion (https://github.com/kubernetes/kubernetes/pull/50933) - Updated the _Token Management with `kubeadm`_ section to link to the main kubeadm docs, since it was just duplicated information. * Update bootstrap-tokens.md * Updated the Cassandra tutorial to use apps/v1beta2 (#5548) * add docs for AllowPrivilegeEscalation (#5448) Signed-off-by: Jess Frazelle <acidburn@microsoft.com> * Add local ephemeral storage alpha feature in managing compute resource (#5522) * Add local ephemeral storage alpha feature in managing compute resource Since 1.8, we add the local ephemeral storage alpha feature as one resource type to manage. Add this feature into the doc. * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Update manage-compute-resources-container.md * Added documentation for Metrics Server (#5560) * authorization: improve authorization debugging docs (#5549) * Document mount propagation (#5544) * Update /docs/setup/independent/create-cluster-kubeadm.md for 1.8. (#5524) This introduction needed a couple of small tweaks to cover the `--discovery-token-ca-cert-hash` flag added in https://github.com/kubernetes/kubernetes/pull/49520 and some version bumps. * Add task doc for alpha dynamic kubelet configuration (#5523) * Fix input/output of selfsubjectaccess review (#5593) * Add docs for implementing resize (#5528) * Add docs for implementing resize * Update admission-controllers.md * Added link to PVC section * minor typo fixes * Update NetworkPolicy concept guide with egress and CIDR changes (#5529) * update zookeeper tutorial for 1.8 release * add doc for hostpath type (#5503) * Federated Hpa feature doc (#5487) * Federated Hpa feature doc * Federated Hpa feature doc review fixes * Update hpa.md * Update hpa.md * update cloud controller manager docs for v1.8 * Update cronjob with defaults information (#5556) * Kubernetes 1.8 reference docs (#5632) * Kubernetes 1.8 reference docs * Kubectl reference docs for 1.8 * Update side bar with 1.8 kubectl and api ref docs links * remove petset.md * update on state of HostAlias in 1.8 with hostNetwork Pod support (#5644) * Fix cron job deletion section (#5655) * update imported docs (#5656) * Add documentation for certificate rotation. (#5639) * Link to using kubeadm page * fix the command output fix the command output * fix typo in api/resources reference: "Worloads" * Add documentation for certificate rotation. * Create TOC entry for cloud controller manager. (#5662) * Updates for new versions of API types * Followup 5655: fix link to garbage collection (#5666) * Temporarily redirect resources-reference to api-reference. (#5668) * Update config for 1.8 release. (#5661) * Update config for 1.8 release. * Address reviewer comments. * Switch references in HPA docs from alpha to beta (#5671) The HPA docs still referenced the alpha version. This switches them to talk about v2beta1, which is the appropriate version for Kubernetes 1.8 * Deprecate openstack heat (#5670) * Fix typo in pod preset conflict example Move container port definition to the correct line. * Highlight openstack-heat provider deprecation The openstack-heat provider for kube-up is being deprecated and will be removed in a future release. * Temporarily fix broken links by redirecting. (#5672) * Fix broken links. (#5675) * Fix render of code block (#5674) * Fix broken links. (#5677) * Add a small note about auto-bootstrapped CSR ClusterRoles (#5660) * Update kubeadm install doc for v1.8 (#5676) * add draft workloads api content for 1.8 (#5650) * add draft workloads api content for 1.8 * edits per review, add tables, for 1.8 workloads api doc * fix typo * Minor fixes to kubeadm 1.8 upgrade guide. (#5678) - The kubelet upgrade instructions should be done on every host, not just worker nodes. - We should just upgrade all packages, instead of calling out kubelet specifically. This will also upgrade kubectl, kubeadm, and kubernetes-cni, if installed. - Draining nodes should also ignore daemonsets, and master errors can be ignored. - Make sure that the new kubeadm download is chmoded correctly. - Add a step to run `kubeadm version` to verify after downloading. - Manually approve new kubelet CSRs if rotation is enabled (known issue). * Release 1.8 (#5680) * Fix versions for 1.8 API ref docs * Updates for 1.8 kubectl reference docs * Kubeadm /docs/admin/kubeadm.md cleanup, editing. (#5681) * Update docs/admin/kubeadm.md (mostly 1.8 related). This is Fabrizio's work, which I'm committing along with my edits (in a commit on top of this). * A few of my own edits to clarify and clean up some Markdown.
2017-09-28 21:46:51 -07:00
parent 88d3ed821d
commit 799eaf4229
1553 changed files with 318572 additions and 3286 deletions
@@ -0,0 +1,69 @@
+# This is an example of how to setup cloud-controller-manger as a Daemonset in your cluster.
+# It assumes that your masters can run pods and has the role node-role.kubernetes.io/master
+# Note that this Daemonset will not work straight out of the box for your cloud, this is
+# meant to be a guideline.
+
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cloud-controller-manager
+  namespace: kube-system
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: system:cloud-controller-manager
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+- kind: ServiceAccount
+  name: cloud-controller-manager
+  namespace: kube-system
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  labels:
+    k8s-app: cloud-controller-manager
+  name: cloud-controller-manager
+  namespace: kube-system
+spec:
+  selector:
+    matchLabels:
+      k8s-app: cloud-controller-manager
+  template:
+    metadata:
+      labels:
+        k8s-app: cloud-controller-manager
+    spec:
+      serviceAccountName: cloud-controller-manager
+      containers:
+      - name: cloud-controller-manager
+        # for in-tree providers we use gcr.io/google_containers/cloud-controller-manager
+        # this can be replaced with any other image for out-of-tree providers
+        image: gcr.io/google_containers/cloud-controller-manager:v1.8.0
+        command:
+        - /usr/local/bin/cloud-controller-manager
+        - --cloud-provider=<YOUR_CLOUD_PROVIDER>   # Add your own cloud provider here!
+        - --leader-elect=true
+        - --use-service-account-credentials
+        # these flags will vary for every cloud provider
+        - --allocate-node-cidrs=true
+        - --configure-cloud-routes=true
+        - --cluster-cidr=172.17.0.0/16
+      tolerations:
+      # this is required so CCM can bootstrap itself
+      - key: node.cloudprovider.kubernetes.io/uninitialized
+        value: "true"
+        effect: NoSchedule
+      # this is to have the daemonset runnable on master nodes
+      # the taint may vary depending on your cluster setup
+      - key: node-role.kubernetes.io/master
+        effect: NoSchedule
+      # this is to restrict CCM to only run on master nodes
+      # the node selector may vary depending on your cluster setup
+      nodeSelector:
+        node-role.kubernetes.io/master: ""
@@ -100,7 +100,7 @@ First, update the following fields in your YAML file:
 If RBAC is enabled on your cluster, you must update the `system:kube-scheduler` cluster role. Add you scheduler name to the resourceNames of the rule applied for endpoints resources, as in the following example:
 ```
 $ kubectl edit clusterrole system:kube-scheduler
- apiVersion: rbac.authorization.k8s.io/v1beta1
+- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
@@ -0,0 +1,192 @@
+---
+title: Control CPU Management Policies on the Node
+---
+
+* TOC
+{:toc}
+
+Kubernetes keeps many aspects of how pods execute on nodes abstracted
+from the user. This is by design.  However, some workloads require
+stronger guarantees in terms of latency and/or performance in order to operate
+acceptably.  The kubelet provides methods to enable more complex workload
+placement policies while keeping the abstraction free from explicit placement
+directives.
+
+## CPU Management Policies
+
+By default, the kubelet uses [CFS quota](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler)
+to enforce pod CPU limits.  When the node runs many CPU-bound pods,
+the workload can move to different CPU cores depending on
+whether the pod is throttled and which CPU cores are available at
+scheduling time.  Many workloads are not sensitive to this migration and thus
+work fine without any intervention.
+
+However, in workloads where CPU cache affinity and scheduling latency
+significantly affect workload performance, the kubelet allows alternative CPU
+management policies to determine some placement preferences on the node.
+
+### Configuration
+
+The CPU Manager is introduced as an alpha feature in Kubernetes v1.8. It
+must be explicitly enabled in the kubelet feature gates:
+`--feature-gates=CPUManager=true`.
+
+The CPU Manager policy is set with the `--cpu-manager-policy` kubelet
+option. There are two supported policies:
+
+* `none`: the default, which represents the existing scheduling behavior.
+* `static`: allows pods with certain resource characteristics to be
+  granted increased CPU affinity and exclusivity on the node.
+
+The CPU manager periodically writes resource updates through the CRI in
+order to reconcile in-memory CPU assignments with cgroupfs. The reconcile
+frequency is set through a new Kubelet configuration value
+`--cpu-manager-reconcile-period`. If not specified, it defaults to the same
+duration as `--node-status-update-frequency`.
+
+### None policy
+
+The `none` policy explicitly enables the existing default CPU
+affinity scheme, providing no affinity beyond what the OS scheduler does
+automatically.  Limits on CPU usage for
+[Guaranteed pods](/docs/tasks/configure-pod-container/quality-service-pod/)
+are enforced using CFS quota.
+
+### Static policy
+
+The `static` policy allows containers in `Guaranteed` pods with integer CPU
+`requests` access to exclusive CPUs on the node. This exclusivity is enforced
+using the [cpuset cgroup controller](https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt).
+
+**Note:** System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs.  The exclusivity only extends to other pods.
+{: .note}
+
+**Note:** The alpha version of this policy does not guarantee static
+exclusive allocations across Kubelet restarts.
+{: .note}
+
+This policy manages a shared pool of CPUs that initially contains all CPUs in the
+node. The amount of exclusively allocatable CPUs is equal to the total
+number of CPUs in the node minus any CPU reservations by the kubelet `--kube-reserved` or
+`--system-reserved` options. CPUs reserved by these options are taken, in
+integer quantity, from the initial shared pool in ascending order by physical
+core ID.  This shared pool is the set of CPUs on which any containers in
+`BestEffort` and `Burstable` pods run. Containers in `Guaranteed` pods with fractional
+CPU `requests` also run on CPUs in the shared pool. Only containers that are
+both part of a `Guaranteed` pod and have integer CPU `requests` are assigned
+exclusive CPUs.
+
+**Note:** The kubelet requires a CPU reservation greater than zero be made
+using either `--kube-reserved` and/or `--system-reserved` when the static
+policy is enabled. This is because zero CPU reservation would allow the shared
+pool to become empty.
+{: .note}
+
+As `Guaranteed` pods whose containers fit the requirements for being statically
+assigned are scheduled to the node, CPUs are removed from the shared pool and
+placed in the cpuset for the container. CFS quota is not used to bound
+the CPU usage of these containers as their usage is bound by the scheduling domain
+itself. In others words, the number of CPUs in the container cpuset is equal to the integer
+CPU `limit` specified in the pod spec. This static assignment increases CPU
+affinity and decreases context switches due to throttling for the CPU-bound
+workload.
+
+Consider the containers in the following pod specs:
+
+```yaml
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+```
+
+This pod runs in the `BestEffort` QoS class because no resource `requests` or
+`limits` are specified. It runs in the shared pool.
+
+```yaml
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    resources:
+      limits:
+        memory: "200Mi"
+      requests:
+        memory: "100Mi"
+```
+
+This pod runs in the `Burstable` QoS class because resource `requests` do not
+equal `limits` and the `cpu` quantity is not specified. It runs in the shared
+pool.
+
+```yaml
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    resources:
+      limits:
+        memory: "200Mi"
+        cpu: "2"
+      requests:
+        memory: "100Mi"
+        cpu: "1"
+```
+
+This pod runs in the `Burstable` QoS class because resource `requests` do not
+equal `limits`. It runs in the shared pool.
+
+```yaml
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    resources:
+      limits:
+        memory: "200Mi"
+        cpu: "2"
+      requests:
+        memory: "200Mi"
+        cpu: "2"
+```
+
+This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
+And the container's resource limit for the CPU resource is an integer greater than 
+or equal to one. The `nginx` container is granted 2 exclusive CPUs.
+
+
+```yaml
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    resources:
+      limits:
+        memory: "200Mi"
+        cpu: "1.5"
+      requests:
+        memory: "200Mi"
+        cpu: "1.5"
+```
+
+This pod runs in the `Guaranteed` QoS class because `requests` are equal to `limits`.
+But the container's resource limit for the CPU resource is a fraction. It runs in
+the shared pool.
+
+
+```yaml
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    resources:
+      limits:
+        memory: "200Mi"
+        cpu: "2"
+```
+
+This pod runs in the `Guaranteed` QoS class because only `limits` are specified
+and `requests` are set equal to `limits` when not explicitly specified. And the 
+container's resource limit for the CPU resource is an integer greater than or 
+equal to one.The `nginx` container is granted 2 exclusive CPUs.
+
@@ -0,0 +1,36 @@
+---
+approvers:
+- luxas
+- thockin
+- wlan0
+title: Developing Cloud Controller Manager
+---
+
+**Cloud Controller Manager is an alpha feature in 1.8. In upcoming releases it will
+be the preferred way to integrate Kubernetes with any cloud. This will ensure cloud providers
+can develop their features independantly from the core Kubernetes release cycles.**
+
+* TOC
+{:toc}
+
+## Background
+
+Before going into how to build your own cloud controller manager, some background on how it works under the hood is helpful. The cloud controller manager is code from `kube-controller-manager` utilizing Go interfaces to allow implementations from any cloud to be plugged in. Most of the scaffolding and generic controller implementations will be in core, but it will always exec out to the cloud interfaces it is provided, so long as the [cloud provider interface](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go#L29-L50) is satisifed.
+
+To dive a little deeper into implementation details, all cloud controller managers will import packages from Kubernetes core, the only difference being each project will register their own cloud providers by calling [cloudprovider.RegisterCloudProvier](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52) where a global variable of available cloud providers is updated.
+
+## Developing
+
+### Out of Tree
+
+To build an out-of-tree cloud-controller-manager for your cloud, follow these steps:
+
+1. Create a go package with an implementation that satisfies [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go).
+2. Use [main.go in cloud-controller-manager](https://github.com/kubernetes/kubernetes/blob/master/cmd/cloud-controller-manager/controller-manager.go) from Kubernestes core as a template for your main.go. As mentioned above, the only difference should be the cloud package that will be imported.
+3. Import your cloud package in `main.go`, ensure your package has an `init` block to run [cloudprovider.RegisterCloudProvider](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/plugins.go#L42-L52).
+
+Using existing out-of-tree cloud providers as an example may be helpful. You can find the list [here](/docs/tasks/administer-cluster/running-cloud-controller.md#examples).
+
+### In Tree
+
+For in-tree cloud providers, you can run the in-tree cloud controller manager as a [Daemonset](/docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml) in your cluster. See the [running cloud controller manager docs](/docs/tasks/administer-cluster/running-cloud-controller.md) for more details.
@@ -0,0 +1,281 @@
+---
+approvers:
+- pipejakob
+- luxas
+- roberthbailey
+- jbeda
+title: Upgrading kubeadm clusters from 1.7 to 1.8
+---
+
+{% capture overview %}
+
+This guide is for upgrading `kubeadm` clusters from version 1.7.x to 1.8.x, as well as 1.7.x to 1.7.y and 1.8.x to 1.8.y where `y > x`.
+See also [upgrading kubeadm clusters from 1.6 to 1.7](/docs/tasks/administer-cluster/kubeadm-upgrade-1-7/) if you're on a 1.6 cluster currently.
+
+{% endcapture %}
+
+{% capture prerequisites %}
+
+Before proceeding:
+
+- You need to have a functional `kubeadm` Kubernetes cluster running version 1.7.0 or higher in order to use the process described here.
+- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#v180-beta1) carefully.
+- As `kubeadm upgrade` does not upgrade etcd make sure to back it up. You can, for example, use `etcdctl backup` to take care of this.
+- Note that `kubeadm upgrade` will not touch any of your workloads, only Kubernetes-internal components. As a best-practice you should back up what's important to you. For example, any app-level state, such as a database an app might depend on (like MySQL or MongoDB) must be backed up beforehand.
+
+Also, note that only one minor version upgrade is supported. That is, you can only upgrade from, say 1.7 to 1.8, not from 1.7 to 1.9.
+
+{% endcapture %}
+
+{% capture steps %}
+
+## Upgrading your control plane
+
+You have to carry out the following steps by executing these commands on your master node:
+
+1. Install the most recent version of `kubeadm` using `curl` like so:
+
+```shell
+$ export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt) # or manually specify a released Kubernetes version
+$ export ARCH=amd64 # or: arm, arm64, ppc64le, s390x
+$ curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
+$ chmod a+rx /usr/bin/kubeadm
+```
+
+Verify that this download of kubeadm works, and has the expected version:
+
+```shell
+$ kubeadm version
+```
+
+2. If this the first time you use `kubeadm upgrade`, in order to preserve the configuration for future upgrades, do:
+
+Note that for below you will need to recall what CLI args you passed to `kubeadm init` the first time.
+
+If you used flags, do:
+
+```shell
+$ kubeadm config upload from-flags [flags]
+```
+
+Where `flags` can be empty.
+
+If you used a config file, do:
+
+```shell
+$ kubeadm config upload from-file --config [config]
+```
+
+Where the `config` is mandatory.
+
+3. On the master node, run the following:
+
+```shell
+$ kubeadm upgrade plan
+[preflight] Running pre-flight checks
+[upgrade] Making sure the cluster is healthy:
+[upgrade/health] Checking API Server health: Healthy
+[upgrade/health] Checking Node health: All Nodes are healthy
+[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
+[upgrade/config] Making sure the configuration is correct:
+[upgrade/config] Reading configuration from the cluster...
+[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
+[upgrade] Fetching available versions to upgrade to:
+[upgrade/versions] Cluster version: v1.7.1
+[upgrade/versions] kubeadm version: v1.8.0
+[upgrade/versions] Latest stable version: v1.8.0
+[upgrade/versions] Latest version in the v1.7 series: v1.7.6
+
+Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
+COMPONENT   CURRENT      AVAILABLE
+Kubelet     1 x v1.7.1   v1.7.6
+
+Upgrade to the latest version in the v1.7 series:
+
+COMPONENT            CURRENT   AVAILABLE
+API Server           v1.7.1    v1.7.6
+Controller Manager   v1.7.1    v1.7.6
+Scheduler            v1.7.1    v1.7.6
+Kube Proxy           v1.7.1    v1.7.6
+Kube DNS             1.14.4    1.14.4
+
+You can now apply the upgrade by executing the following command:
+
+	kubeadm upgrade apply v1.7.6
+
+_____________________________________________________________________
+
+Components that must be upgraded manually after you've upgraded the control plane with 'kubeadm upgrade apply':
+COMPONENT   CURRENT      AVAILABLE
+Kubelet     1 x v1.7.1   v1.8.0
+
+Upgrade to the latest experimental version:
+
+COMPONENT            CURRENT   AVAILABLE
+API Server           v1.7.1    v1.8.0
+Controller Manager   v1.7.1    v1.8.0
+Scheduler            v1.7.1    v1.8.0
+Kube Proxy           v1.7.1    v1.8.0
+Kube DNS             1.14.4    1.14.4
+
+You can now apply the upgrade by executing the following command:
+
+	kubeadm upgrade apply v1.8.0
+
+Note: Before you do can perform this upgrade, you have to update kubeadm to v1.8.0
+
+_____________________________________________________________________
+```
+
+The `kubeadm upgrade plan` checks that your cluster is in an upgradeable state and fetches the versions available to upgrade to in an user-friendly way.
+
+4. Pick a version to upgrade to and run, for example, `kubeadm upgrade apply` as follows:
+
+```shell
+$ kubeadm upgrade apply v1.8.0
+[preflight] Running pre-flight checks
+[upgrade] Making sure the cluster is healthy:
+[upgrade/health] Checking API Server health: Healthy
+[upgrade/health] Checking Node health: All Nodes are healthy
+[upgrade/health] Checking Static Pod manifests exists on disk: All manifests exist on disk
+[upgrade/config] Making sure the configuration is correct:
+[upgrade/config] Reading configuration from the cluster...
+[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
+[upgrade/version] You have chosen to upgrade to version "v1.8.0"
+[upgrade/versions] Cluster version: v1.7.1
+[upgrade/versions] kubeadm version: v1.8.0
+[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler]
+[upgrade/prepull] Prepulling image for component kube-scheduler.
+[upgrade/prepull] Prepulling image for component kube-apiserver.
+[upgrade/prepull] Prepulling image for component kube-controller-manager.
+[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
+[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
+[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
+[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
+[upgrade/prepull] Prepulled image for component kube-apiserver.
+[upgrade/prepull] Prepulled image for component kube-controller-manager.
+[upgrade/prepull] Prepulled image for component kube-scheduler.
+[upgrade/prepull] Successfully prepulled the images for all the control plane components
+[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.8.0"...
+[upgrade/staticpods] Writing upgraded Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769"
+[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-apiserver.yaml"
+[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-controller-manager.yaml"
+[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests432902769/kube-scheduler.yaml"
+[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-apiserver.yaml"
+[upgrade/staticpods] Waiting for the kubelet to restart the component
+[apiclient] Found 1 Pods for label selector component=kube-apiserver
+[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
+[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-controller-manager.yaml"
+[upgrade/staticpods] Waiting for the kubelet to restart the component
+[apiclient] Found 1 Pods for label selector component=kube-controller-manager
+[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
+[upgrade/staticpods] Moved upgraded manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests155856668/kube-scheduler.yaml"
+[upgrade/staticpods] Waiting for the kubelet to restart the component
+[apiclient] Found 1 Pods for label selector component=kube-scheduler
+[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
+[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
+[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
+[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
+[addons] Applied essential addon: kube-dns
+[addons] Applied essential addon: kube-proxy
+
+[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.8.0". Enjoy!
+
+[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.
+```
+
+`kubeadm upgrade apply` does the following:
+
+- It checks that your cluster is in an upgradeable state, that is:
+  - The API Server is reachable,
+  - All nodes are in the `Ready` state, and
+  - The control plane is healthy
+- It enforces the version skew policies.
+- It makes sure the control plane images are available or available to pull to the machine.
+- It upgrades the control plane components or rollbacks if any of them fails to come up.
+- It applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created.
+
+5. Manually upgrade your Software Defined Network (SDN).
+
+   Your Container Network Interface (CNI) provider might have its own upgrade instructions to follow now.
+   Check the [addons](/docs/concepts/cluster-administration/addons/) page to
+   find your CNI provider and see if there are additional upgrade steps
+   necessary.
+
+## Upgrading your master and node packages
+
+For each host (referred to as `$HOST` below) in your cluster, upgrade `kubelet` by executing the following commands:
+
+1. Prepare the host for maintenance, marking it unschedulable and evicting the workload:
+
+```shell
+$ kubectl drain $HOST --ignore-daemonsets
+```
+
+When running this command against the master host, this error is expected and can be safely ignored (since there are static pods running on the master):
+
+```shell
+node "master" already cordoned
+error: pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): etcd-kubeadm, kube-apiserver-kubeadm, kube-controller-manager-kubeadm, kube-scheduler-kubeadm
+```
+
+2. Upgrade the Kubernetes package versions on the `$HOST` node by using a Linux distribution-specific package manager:
+
+If the host is running a Debian-based distro such as Ubuntu, run:
+
+```shell
+$ apt-get update
+$ apt-get upgrade
+```
+
+If the host is running CentOS or the like, run:
+
+```shell
+$ yum update
+```
+
+Now the new version of the `kubelet` should be running on the host. Verify this using the following command on `$HOST`:
+
+```shell
+$ systemctl status kubelet
+```
+
+3. Since certificate rotation is enabled by default, you may need to manually approve the new kubelet's CertificateSigningRequest before it can rejoin the cluster:
+
+```shell
+$ kubectl get csr | grep -v Approved
+NAME                                                   AGE       REQUESTOR                 CONDITION
+node-csr-czl32tarZb_XYKnvXf0Q0o4spGUXzJhN2p4_ld7k1iM   2h        system:bootstrap:033abb   Pending
+```
+
+If you see any CSRs listed that aren't already approved, you can manually approve them using kubectl:
+
+```shell
+$ kubectl certificate approve node-csr-czl32tarZb_XYKnvXf0Q0o4spGUXzJhN2p4_ld7k1iM
+certificatesigningrequest "node-csr-czl32tarZb_XYKnvXf0Q0o4spGUXzJhN2p4_ld7k1iM" approved
+```
+
+4. Bring the host back online by marking it schedulable:
+
+```shell
+$ kubectl uncordon $HOST
+```
+
+5. After upgrading `kubelet` on each host in your cluster, verify that all nodes are available again by executing the following (from anywhere, for example, from outside the cluster):
+
+```shell
+$ kubectl get nodes
+```
+
+If the `STATUS` column of the above command shows `Ready` for all of your hosts, you are done.
+
+## Recovering from a bad state
+
+If `kubeadm upgrade` somehow fails and fails to roll back, due to an unexpected shutdown during execution for instance,
+you may run `kubeadm upgrade` again as it is idempotent and should eventually make sure the actual state is the desired state you are declaring.
+
+You can use `kubeadm upgrade` to change a running cluster with `x.x.x --> x.x.x` with `--force`, which can be used to recover from a bad state.
+
+{% endcapture %}
+
+{% include templates/task.md %}
@@ -9,7 +9,7 @@ This page shows how to specify opaque integer resources for a Node.
 Opaque integer resources allow cluster administrators to advertise node-level
 resources that would otherwise be unknown to Kubernetes.

-{% include feature-state-alpha.md %}
+{% include feature-state-deprecated.md %}

 {% endcapture %}

@@ -0,0 +1,13 @@
+kind: InitializerConfiguration
+apiVersion: admissionregistration.k8s.io/v1alpha1
+metadata:
+  name: pvlabel.kubernetes.io
+initializers:
+  - name: pvlabel.kubernetes.io
+    rules:
+    - apiGroups:
+      - ""
+      apiVersions:
+      - "*"
+      resources:
+      - persistentvolumes
@@ -0,0 +1,470 @@
+---
+approvers:
+- mtaufen
+- dawnchen
+title: Reconfigure a Node's Kubelet in a Live Cluster
+---
+
+{% capture overview %}
+{% include feature-state-alpha.md %}
+As of Kubernetes 1.8, the new
+[Dynamic Kubelet Configuration](https://github.com/kubernetes/features/issues/281)
+feature is available in alpha. This allows you to change the configuration of
+Kubelets in a live Kubernetes cluster via first-class Kubernetes concepts.
+Specifically, this feature allows you to configure individual Nodes' Kubelets
+via ConfigMaps.
+
+**Warning:** All Kubelet configuration parameters may be changed dynamically,
+but not all parameters are safe to change dynamically. This feature is intended
+for system experts who have a strong understanding of how configuration changes
+will affect behavior. No documentation currently exists which plainly lists
+"safe to change" fields, but we plan to add it before this feature graduates
+from alpha.
+{% endcapture %}
+
+{% capture prerequisites %}
+- A live Kubernetes cluster with both Master and Node at v1.8 or higher must be
+running, with the `DynamicKubeletConfig` feature gate enabled and the Kubelet's
+`--dynamic-config-dir` flag set to a writeable directory on the Node.
+This flag must be set to enable Dynamic Kubelet Configuration.
+- The kubectl command-line tool must be also be v1.8 or higher, and must be
+configured to communicate with the cluster.
+{% endcapture %}
+
+{% capture steps %}
+
+## Reconfiguring the Kubelet on a Live Node in your Cluster
+
+### Basic Workflow Overview
+
+The basic workflow for configuring a Kubelet in a live cluster is as follows:
+
+1. Write a YAML or JSON configuration file containing the
+Kubelet's configuration.
+2. Wrap this file in a ConfigMap and save it to the Kubernetes control plane.
+3. Update the Kubelet's correspoinding Node object to use this ConfigMap.
+
+Each Kubelet watches a configuration reference on its respective Node object.
+When this reference changes, the Kubelet downloads the new configuration and
+exits. For the feature to work correctly, you must be running a process manager
+(like systemd) which will restart the Kubelet when it exits. When the Kubelet is
+restarted, it will begin using the new configuration.
+
+The new configuration completely overrides the old configuration; unspecified
+fields in the new configuration will receive their canonical default values.
+Some CLI flags do not have an associated configuration field, and will not be
+affected by the new configuration. These fields are defined by the KubeletFlags
+structure, [here](https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go).
+
+The status of the Node's Kubelet configuration is reported via the `ConfigOK`
+condition in the Node status. Once you have updated a Node to use the new
+ConfigMap, you can observe this condition to confirm that the Node is using the
+intended configuration. A table describing the possible conditions can be found
+at the end of this article.
+
+This document describes editing Nodes using `kubectl edit`.
+There are other ways to modify a Node's spec, including `kubectl patch`, for
+example, which facilitate scripted workflows.
+
+This document only describes a single Node consuming each ConfigMap. Keep in
+mind that it is also valid for multiple Nodes to consume the same ConfigMap.
+
+### Node Authorizer Workarounds
+
+The Node Authorizer does not yet pay attention to which ConfigMaps are assigned
+to which Nodes. If you currently use the Node authorizer, your Kubelets will not
+be automatically granted permission to download their respective ConfigMaps.
+
+The temporary workaround used in this document is to manually create the RBAC
+Roles and RoleBindings for each ConfigMap. The Node Authorizer will be extended
+before the Dynamic Kubelet Configuration feature graduates from alpha, so doing
+this in production should never be necessary.
+
+### Generating a file that contains the current configuration
+
+The Dynamic Kubelet Configuration feature allows you to provide an override for
+the entire configuration object, rather than a per-field overlay. This is a
+simpler model that makes it easier to trace the source of configuration values
+and debug issues. The compromise, however, is that you must start with knowledge
+of the existing configuration to ensure that you only change the fields you
+intend to change.
+
+In the future, the Kubelet will be bootstrapped from a file on disk, and you
+will simply edit a copy of this file (which, as a best practice, should live in
+version control) while creating the first Kubelet ConfigMap. Today, however, the
+Kubelet is still bootstrapped with command-line flags. Fortunately, there is a
+dirty trick you can use to generate a config file containing a Node's current
+configuration. The trick involves hitting the Kubelet server's `configz`
+endpoint via the kubectl proxy. This endpoint, in its current implementation, is
+intended to be used only as a debugging aid, which is part of why this is a
+dirty trick. There is ongoing work to improve the endpoint, and in the future
+this will be a less "dirty" operation. This trick also requires the `jq` command
+to be installed on your machine, for unpacking and editing the JSON response
+from the endpoint.
+
+Do the following to generate the file:
+
+1. Pick a Node to reconfigure. We will refer to this Node's name as NODE_NAME.
+2. Start the kubectl proxy in the background with `kubectl proxy --port=8001 &`
+3. Run the following command to download and unpack the configuration from the
+configz endpoint:
+
+```
+$ export NODE_NAME=the-name-of-the-node-you-are-reconfiguring
+$ curl -sSL http://localhost:8001/api/v1/proxy/nodes/${NODE_NAME}/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubeletconfig/v1alpha1"' > kubelet_configz_${NODE_NAME}
+```
+
+Note that we have to manually add the `kind` and `apiVersion` to the downloaded
+object, as these are not reported by the configz endpoint. This is one of the
+limitations of the endpoint that is planned to be fixed in the future.
+
+### Edit the configuration file
+
+Using your editor of choice, change one of the parameters in the
+`kubelet_configz_${NODE_NAME}` file from the previous step. A QPS parameter,
+`eventRecordQPS` for example, is a good candidate.
+
+### Push the configuration file to the control plane
+
+Push the edited configuration file to the control plane with the
+following command:
+
+```
+$ kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
+```
+
+You should see a response similar to:
+
+```
+apiVersion: v1
+data:
+  kubelet: |
+    {...}
+kind: ConfigMap
+metadata:
+  creationTimestamp: 2017-09-14T20:23:33Z
+  name: my-node-config-gkt4c2m4b2
+  namespace: kube-system
+  resourceVersion: "119980"
+  selfLink: /api/v1/namespaces/kube-system/configmaps/my-node-config-gkt4c2m4b2
+  uid: 946d785e-998a-11e7-a8dd-42010a800006
+```
+
+Note that the configuration data must appear under the ConfigMap's
+`kubelet` key.
+
+We create the ConfigMap in the `kube-system` namespace, which is appropriate
+because this ConfigMap configures a Kubernetes system component - the Kubelet.
+
+The `--append-hash` option appends a short checksum of the ConfigMap contents
+to the name. This is convenient for an edit->push workflow, as it will
+automatically, yet deterministically, generate new names for new ConfigMaps. 
+
+We use the `-o yaml` output format so that the name, namespace, and uid are all
+reported following creation. We will need these in the next step. We will refer
+to the name as CONFIG_MAP_NAME and the uid as CONFIG_MAP_UID.
+
+### Authorize your Node to read the new ConfigMap
+
+Now that you've created a new ConfigMap, you need to authorize your node to
+read it. First, create a Role for your new ConfigMap with the 
+following commands:
+
+```
+$ export CONFIG_MAP_NAME=name-from-previous-output
+$ kubectl -n kube-system create role ${CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${CONFIG_MAP_NAME}
+```
+
+Next, create a RoleBinding to associate your Node with the new Role:
+
+```
+$ kubectl -n kube-system create rolebinding ${CONFIG_MAP_NAME}-reader --role=${CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
+```
+
+Once the Node Authorizer is updated to do this automatically, you will
+be able to skip this step. 
+
+### Set the Node to use the new configuration
+
+Edit the Node's reference to point to the new ConfigMap with the
+following command:
+
+```
+kubectl edit node ${NODE_NAME}
+```
+
+Once in your editor, add the following YAML under `spec`:
+
+```
+configSource:
+    configMapRef:
+        name: CONFIG_MAP_NAME
+        namespace: kube-system
+        uid: CONFIG_MAP_UID
+```
+
+Be sure to specify all three of `name`, `namespace`, and `uid`.
+
+### Observe that the Node begins using the new configuration
+
+Retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and look for the
+`ConfigOK` condition in `status.conditions`. You should see the message
+`Using current (UID: CONFIG_MAP_UID)` when the Kubelet starts using the new
+configuration.
+
+For convenience, you can use the following command (using `jq`) to filter down
+to the `ConfigOK` condition:
+
+```
+$ kubectl get no ${NODE_NAME} -o json | jq '.status.conditions|map(select(.type=="ConfigOK"))'
+[
+  {
+    "lastHeartbeatTime": "2017-09-20T18:08:29Z",
+    "lastTransitionTime": "2017-09-20T18:08:17Z",
+    "message": "using current (UID: \"2ebc8d1a-9e2a-11e7-a8dd-42010a800006\")",
+    "reason": "passing all checks",
+    "status": "True",
+    "type": "ConfigOK"
+  }
+]
+```
+
+If something goes wrong, you may see one of several different error conditions,
+detailed in the Table of ConfigOK Conditions, below. When this happens, you
+should check the Kubelet's log for more details.
+
+### Edit the configuration file again
+
+To change the configuration again, we simply repeat the above workflow.
+Try editing the `kubelet` file, changing the previously changed parameter to a
+new value.
+
+### Push the newly edited configuration to the control plane
+
+Push the new configuration to the control plane in a new ConfigMap with the
+following command:
+
+```
+$ kubectl create configmap my-node-config --namespace=kube-system --from-file=kubelet=kubelet_configz_${NODE_NAME} --append-hash -o yaml
+```
+
+This new ConfigMap will get a new name, as we have changed the contents.
+We will refer to the new name as NEW_CONFIG_MAP_NAME and the new uid
+as NEW_CONFIG_MAP_UID.
+
+### Authorize your Node to read the new ConfigMap
+
+Now that you've created a new ConfigMap, you need to authorize your node to
+read it. First, create a Role for your new ConfigMap with the 
+following commands:
+
+```
+$ export NEW_CONFIG_MAP_NAME=name-from-previous-output
+$ kubectl -n kube-system create role ${NEW_CONFIG_MAP_NAME}-reader --verb=get --resource=configmap --resource-name=${NEW_CONFIG_MAP_NAME}
+```
+
+Next, create a RoleBinding to associate your Node with the new Role:
+
+```
+$ kubectl -n kube-system create rolebinding ${NEW_CONFIG_MAP_NAME}-reader --role=${NEW_CONFIG_MAP_NAME}-reader --user=system:node:${NODE_NAME}
+```
+
+Once the Node Authorizer is updated to do this automatically, you will
+be able to skip this step. 
+
+### Configure the Node to use the new configuration
+
+Once more, edit the Node's `spec.configSource` with
+`kubectl edit node ${NODE_NAME}`. Your new `spec.configSource` should look like
+the following, with `name` and `uid` substituted as necessary:
+
+```
+configSource:
+    configMapRef:
+        name: NEW_CONFIG_MAP_NAME
+        namespace: kube-system
+        uid: NEW_CONFIG_MAP_UID
+```
+
+### Observe that the Kubelet is using the new configuration
+
+Once more, retrieve the Node with `kubectl get node ${NODE_NAME} -o yaml`, and
+look for the `ConfigOK` condition in `status.conditions`. You should the message
+`Using current (UID: NEW_CONFIG_MAP_UID)` when the Kubelet starts using the
+new configuration.
+
+### Deauthorize your Node fom reading the old ConfigMap
+
+Once you know your Node is using the new configuration and are confident that
+the new configuration has not caused any problems, it is a good idea to
+deauthorize the node from reading the old ConfigMap. Run the following
+commands to remove the RoleBinding and Role:
+
+```
+$ kubectl -n kube-system delete rolebinding ${CONFIG_MAP_NAME}-reader
+$ kubectl -n kube-system delete role ${CONFIG_MAP_NAME}-reader
+```
+
+Note that this does not necessarily prevent the Node from reverting to the old
+configuration, as it may locally cache the old ConfigMap for an indefinite
+period of time.
+
+You may optionally also choose to remove the old ConfigMap:
+
+```
+$ kubectl -n kube-system delete configmap ${CONFIG_MAP_NAME}
+```
+
+Once the Node Authorizer is updated to do this automatically, you will
+be able to skip this step.
+
+### Reset the Node to use its local default configuration
+
+Finally, if you wish to reset the Node to use the configuration it was
+provisioned with, simply edit the Node with `kubectl edit node ${NODE_NAME}` and
+remove the `spec.configSource` subfield.
+
+### Observe that the Node is using its local default configuration
+
+After removing this subfield, you should eventually observe that the ConfigOK
+condition's message reverts to either `using current (default)` or
+`using current (init)`, depending on how the Node was provisioned.
+
+### Deauthorize your Node fom reading the old ConfigMap
+
+Once you know your Node is using the default configuraiton again, it is a good
+idea to deauthorize the node from reading the old ConfigMap. Run the following
+commands to remove the RoleBinding and Role:
+
+```
+$ kubectl -n kube-system delete rolebinding ${NEW_CONFIG_MAP_NAME}-reader
+$ kubectl -n kube-system delete role ${NEW_CONFIG_MAP_NAME}-reader
+```
+
+Note that this does not necessarily prevent the Node from reverting to the old
+ConfigMap, as it may locally cache the old ConfigMap for an indefinite
+period of time.
+
+You may optionally also choose to remove the old ConfigMap:
+
+```
+$ kubectl -n kube-system delete configmap ${NEW_CONFIG_MAP_NAME}
+```
+
+Once the Node Authorizer is updated to do this automatically, you will
+be able to skip this step.
+
+{% endcapture %}
+
+{% capture discussion %}
+## Kubectl Patch Example
+As mentioned above, there are many ways to change a Node's configSource.
+Here is an example command that uses `kubectl patch`:
+
+```
+kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMapRef\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"uid\":\"${CONFIG_MAP_UID}\"}}}}"
+```
+
+## Understanding ConfigOK Conditions
+
+The following table describes several of the `ConfigOK` Node conditions you
+might encounter in a cluster that has Dynamic Kubelet Config enabled. If you
+observe a condition with `status=False`, you should check the Kubelet log for
+more error details by searching for the message or reason text.
+
+<table>
+
+
+<table align="left">
+<tr>
+    <th>Possible Messages</th>
+    <th>Possible Reasons</th>
+    <th>Status</th>
+</tr>
+<tr>
+    <td><p>using current (default)</p></td>
+    <td><p>current is set to the local default, and no init config was provided</p></td>
+    <td><p>True</p></td>
+</tr>
+<tr>
+    <td><p>using current (init)</p></td>
+    <td><p>current is set to the local default, and an init config was provided</p></td>
+    <td><p>True</p></td>
+</tr>
+<tr>
+    <td><p>using current (UID: CURRENT_CONFIG_MAP_UID)</p></td>
+    <td><p>passing all checks</p></td>
+    <td><p>True</p></td>
+</tr>
+<tr>
+    <td><p>using last-known-good (default)</p></td>
+    <td>
+        <ul>
+            <li>failed to load current (UID: CURRENT_CONFIG_MAP_UID)</li>
+            <li>failed to parse current (UID: CURRENT_CONFIG_MAP_UID)</li>
+            <li>failed to validate current (UID: CURRENT_CONFIG_MAP_UID)</li>
+        </ul>
+    </td>
+    <td><p>False</p></td>
+</tr>
+<tr>
+    <td><p>using last-known-good (init)</p></td>
+    <td>
+        <ul>
+            <li>failed to load current (UID: CURRENT_CONFIG_MAP_UID)</li>
+            <li>failed to parse current (UID: CURRENT_CONFIG_MAP_UID)</li>
+            <li>failed to validate current (UID: CURRENT_CONFIG_MAP_UID)</li>
+        </ul>
+    </td>
+    <td><p>False</p></td>
+</tr>
+<tr>
+    <td><p>using last-known-good (UID: LAST_KNOWN_GOOD_CONFIG_MAP_UID)</p></td>
+    <td>
+        <ul>
+            <li>failed to load current (UID: CURRENT_CONFIG_MAP_UID)</li>
+            <li>failed to parse current (UID: CURRENT_CONFIG_MAP_UID)</li>
+            <li>failed to validate current (UID: CURRENT_CONFIG_MAP_UID)</li>
+        </ul>
+    </td>
+    <td><p>False</p></td>
+</tr>
+<tr>
+    <td>
+        <p>
+            The reasons in the next column could potentially appear for any of
+            the above messages.
+        </p>
+        <p>
+            This condition indicates that the Kubelet is having trouble
+            reconciling `spec.configSource`, and thus no change to the in-use
+            configuration has occurred.
+        </p>
+        <p>
+            The "failed to sync" reasons are specific to the failure that
+            occurred, and the next column does not necessarily contain all
+            possible failure reasons.
+        </p>
+    </td>
+    <td>
+    <p>failed to sync, reason:</p>
+    <ul>
+        <li>failed to read Node from informer object cache</li>
+        <li>failed to reset to local (default or init) config</li>
+        <li>invalid NodeConfigSource, exactly one subfield must be non-nil, but all were nil</li>
+        <li>invalid ObjectReference, all of UID, Name, and Namespace must be specified</li>
+        <li>invalid ObjectReference, UID SOME_UID does not match UID of downloaded ConfigMap SOME_OTHER_UID</li>
+        <li>failed to determine whether object with UID SOME_UID was already checkpointed</li>
+        <li>failed to download ConfigMap with name SOME_NAME from namespace SOME_NAMESPACE</li>
+        <li>failed to save config checkpoint for object with UID SOME_UID</li>
+        <li>failed to set current config checkpoint to default</li>
+        <li>failed to set current config checkpoint to object with UID SOME_UID</li>
+    </ul>
+    </td>
+    <td><p>False</p></td>
+</tr>
+</table>
+{% endcapture %}
+
+
+{% include templates/task.md %}
@@ -42,7 +42,7 @@ on each node.

 `Allocatable` on a Kubernetes node is defined as the amount of compute resources
 that are available for pods. The scheduler does not over-subscribe
-`Allocatable`. `CPU`, `memory` and `storage` are supported as of now.
+`Allocatable`. `CPU`, `memory` and `ephemeral-storage` are supported as of now.

 Node Allocatable is exposed as part of `v1.Node` object in the API and as part
 of `kubectl describe node` in the CLI.
@@ -77,7 +77,7 @@ be configured to use the `systemd` cgroup driver.

 ### Kube Reserved

- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi][,][storage=1Gi]`
+- **Kubelet Flag**: `--kube-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi]`
 - **Kubelet Flag**: `--kube-reserved-cgroup=`

 `kube-reserved` is meant to capture resource reservation for kubernetes system
@@ -107,7 +107,7 @@ exist. Kubelet will fail if an invalid cgroup is specified.

 ### System Reserved

- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi][,][storage=1Gi]`
+- **Kubelet Flag**: `--system-reserved=[cpu=100mi][,][memory=100Mi][,][ephemeral-storage=1Gi]`
 - **Kubelet Flag**: `--system-reserved-cgroup=`


@@ -135,7 +135,7 @@ Memory pressure at the node level leads to System OOMs which affects the entire
 node and all pods running on it. Nodes can go offline temporarily until memory
 has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
 provides [`Out of Resource`](./out-of-resource.md) management. Evictions are
-supported for `memory` and `storage` only. By reserving some memory via
+supported for `memory` and `ephemeral-storage` only. By reserving some memory via
 `--eviction-hard` flag, the `kubelet` attempts to `evict` pods whenever memory
 availability on the node drops below the reserved value. Hypothetically, if
 system daemons did not exist on a node, pods cannot use more than `capacity -
@@ -191,8 +191,8 @@ So expect a drop in `Allocatable` capacity in future releases.
 Here is an example to illustrate Node Allocatable computation:

 * Node has `32Gi` of `memory`, `16 CPUs` and `100Gi` of `Storage`
-* `--kube-reserved` is set to `cpu=1,memory=2Gi,storage=1Gi`
-* `--system-reserved` is set to `cpu=500m,memory=1Gi,storage=1Gi`
+* `--kube-reserved` is set to `cpu=1,memory=2Gi,ephemeral-storage=1Gi`
+* `--system-reserved` is set to `cpu=500m,memory=1Gi,ephemeral-storage=1Gi`
 * `--eviction-hard` is set to `memory.available<500Mi,nodefs.available<10%`

 Under this scenario, `Allocatable` will be `14.5 CPUs`, `28.5Gi` of memory and
@@ -1,34 +1,93 @@
 ---
 approvers:
+- luxas
 - thockin
-title: Build and Run cloud-controller-manager
+- wlan0
+title: Kubernetes Cloud Controller Manager
 ---

-Kubernetes version 1.6 contains a new binary called as `cloud-controller-manager`. `cloud-controller-manager` is a daemon that embeds cloud-specific control loops in Kubernetes. These cloud-specific control loops were originally in the kube-controller-manager. However, cloud providers move at a different pace and schedule compared to the Kubernetes project, and abstracting the provider-specific code to the `cloud-controller-manager` binary allows cloud provider vendors to evolve independently from the core Kubernetes code.
-
-The `cloud-controller-manager` can be linked to any cloud provider that satisifies the [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go).
-In future Kubernetes releases, cloud vendors should link code that satisfies the above interface to the `cloud-controller-manager` project and compile `cloud-controller-manager` for their own clouds. Cloud providers would also be responsible for maintaining and evolving their code.
+**Cloud Controller Manager is an alpha feature in 1.8. In upcoming releases it will be the preferred way to integrate Kubernetes with any cloud. This will ensure cloud providers can develop their features independantly from the core Kubernetes release cycles.**

 * TOC
 {:toc}

-### Building cloud-controller-manager for your cloud
+## Cloud Controller Manager

-To build cloud-controller-manager for your cloud, follow these steps:
+Kubernetes v1.6 contains a new binary called `cloud-controller-manager`. `cloud-controller-manager` is a daemon that embeds cloud-specific control loops. These cloud-specific control loops were originally in the `kube-controller-manager`. Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the `cloud-controller-manager` binary allows cloud vendors to evolve independently from the core Kubernetes code.

-1. Write a cloudprovider that satisfies the [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go).
-2. Link the cloudprovider to cloud-controller-manager.
+The `cloud-controller-manager` can be linked to any cloud provider that satisifies [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go). For backwards compatibility, the [cloud-controller-manager](https://github.com/kubernetes/kubernetes/tree/master/cmd/cloud-controller-manager) provided in the core Kubernetes project uses the same cloud libraries as `kube-controller-manager`. Cloud providers already supported in Kubernetes core are expected to use the in-tree cloud-controller-manager to transition out of Kubernetes core. In future Kubernetes releases, all cloud controller managers will be developed outside of the core Kubernetes project managed by sig leads or cloud vendors.

-The methods in [cloudprovider.Interface](https://git.k8s.io/kubernetes/pkg/cloudprovider/cloud.go) are self-explanatory. All of the
-[existing providers](https://git.k8s.io/kubernetes/pkg/cloudprovider/providers) satisfy this interface. If your cloud is already a part
-of the existing providers, you do not need to write a new provider; you can proceed directly with linking your cloud provider to the `cloud-controller-manager`.
+## Administration

-Once your code is ready, you must import that code into `cloud-controller-manager`. See the [rancher cloud sample](https://github.com/rancher/rancher-cloud-controller-manager) for a reference example. The import step in the sample is the only step required to link your cloud provider to the `cloud-controller-manager`.
+### Requirements
+
+Every cloud has their own set of requirements for running their own cloud provider integration, it should not be too different from the requirements when running `kube-controller-manager`. As a general rule of thumb you'll need:
+
+* cloud authentication/authorization: your cloud may require a token or IAM rules to allow access to their APIs
+* kubernetes authentication/authorization: cloud-controller-manager may need RBAC rules set to speak to the kubernetes apiserver
+* high availabilty: like kube-controller-manager, you may want a high available setup for cloud controller manager using leader election (on by default).

 ### Running cloud-controller-manager

-To run `cloud-controller-manager`, add it to your existing Kubernetes cluster as a Master component. All other master components except `kube-controller-manager` can be run without any changes.
+Successfully running cloud-controller-manager requires some changes to your cluster configuration.

-The `kube-controller-manager` should not run any cloud-specific controllers, since the `cloud-controller-manager` takes over this responsibility. To prevent the `kube-controller-manager` from running cloud-specific controllers, you must set the `--cloud-provider` flag in `kube-controller-manager` to `external`.
+* `kube-apiserver` and `kube-controller-manager` MUST NOT specify the `--cloud-provider` flag. This ensures that it does not run any cloud specific loops that would be run by cloud controller manager. In the future, this flag will be deprecated and removed.
+* `kubelet` must run with `--cloud-provider=external`. This is to ensure that the kubelet is aware that it must be initialized by the cloud controller manager before it is scheduled any work.
+* `kube-apiserver` SHOULD NOT run the `PersistentVolumeLabel` admission controller since the cloud controller manager takes over labeling persistent volumes. To prevent the PersistentVolumeLabel admission plugin from running, make sure the `kube-apiserver` has a `--admission-control` flag with a value that does not include `PersistentVolumeLabel`.
+* For the `cloud-controller-manager` to label persistent volumes, initializers will need to be enabled and an InitializerConifguration needs to be added to the system.  Follow [these instructions](/docs/admin/extensible-admission-controllers.md#enable-initializers-alpha-feature) to enable initializers.  Use the following YAML to create the InitializerConfiguration:

-The `kube-apiserver` should not run the Persistent Volume Label admission controller either since the `cloud-controller-manager` takes over labeling persistent volumes.  To prevent the Persistent Volume Label admission plugin from running, make sure the `kube-apiserver` has a `--admission-control` flag with a value that does not include `PersistentVolumeLabel`.
+{% include code.html language="yaml" file="persistent-volume-label-initializer-config.yaml" ghlink="/docs/tasks/administer-cluster/persistent-volume-label-initializer-config.yaml" %}
+
+Keep in mind that setting up your cluster to use cloud controller manager will change your cluster behaviour in a few ways:
+
+* kubelets specifying `--cloud-provider=external` will add a taint `node.cloudprovider.kubernetes.io/uninitialized` with an effect `NoSchedule` during initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work. Note that in the event that cloud controller manager is not available, new nodes in the cluster will be left unscheduable. The taint is important since the scheduler may require cloud specific information about nodes such as it's region or type (high cpu, gpu, high memory, spot instance, etc).
+* cloud information about nodes in the cluster will no longer be retrieved using local metadata, but instead all API calls to retreive node information will go through cloud controller manager. This may mean you can restrict access to your cloud API on the kubelets for better security. For larger clusters you may want to consider if cloud controller manager will hit rate limits since it is now responsible for almost all API calls to your cloud from within the cluster.
+
+
+As of v1.8, cloud controller manager can implement:
+
+* node controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
+* service controller - responsible for loadbalancers on your cloud against services of type LoadBalancer.
+* route controller - responsible for setting up network routes on your cloud
+* [PersistentVolumeLabel Admission Controller](/docs/admin/admission-controllers#persistentvolumelabel) - responsible for labeling persistent volumes on your cloud - ensure that the persistent volume label admission plugin is not enabled on your kube-apiserver.
+* any other features you would like to implement if you are running an out-of-tree provider.
+
+
+## Examples
+
+If you are using a cloud that is currently supported in Kubernetes core and would like to adopt cloud controller manager, see the [cloud controller manager in kubernetes core](https://github.com/kubernetes/kubernetes/tree/master/cmd/cloud-controller-manager).
+
+For cloud controller managers not in Kubernetes core, you can find the respective projects in repos maintained by cloud vendors or sig leads.
+
+* [DigitalOcean](https://github.com/digitalocean/digitalocean-cloud-controller-manager)
+* [keepalived](https://github.com/munnerz/keepalived-cloud-provider)
+* [Rancher](https://github.com/rancher/rancher-cloud-controller-manager)
+
+For providers already in Kubernetes core, you can run the in-tree cloud controller manager as a Daemonset in your cluster, use the following as a guideline:
+
+{% include code.html language="yaml" file="cloud-controller-manager-daemonset-example.yaml" ghlink="/docs/tasks/administer-cluster/cloud-controller-manager-daemonset-example.yaml" %}
+
+
+## Limitations
+
+Running cloud controller manager comes with a few possible limitations. Although these limitations are being addressed in upcoming releases, it's important that you are aware of these limitations for production workloads.
+
+### Support for Volumes
+
+Cloud controller manager does not implement any of the volume controllers found in `kube-controller-manager` as the volume integrations also require coordination with kubelets. As we evolve CSI (container storage interface) and add stronger support for flex volume plugins, necessary support will be added to cloud controller manager so that clouds can fully integrate with volumes. Learn more about out-of-tree CSI volume plugins [here](https://github.com/kubernetes/features/issues/178).
+
+### Scalability
+
+In the previous architecture for cloud providers, we relied on kubelets using a local metadata service to retreive node information about itself. With this new architecture, we now fully rely on the cloud controller managers to retrieve information for all nodes. For very larger clusters, you should consider possible bottle necks such as resource requirements and API rate limiting.
+
+### Chicken and Egg
+
+The goal of the cloud controller manager project is to decouple development of cloud features from the core Kubernetes project. Unforunately, many aspects of the Kubernetes project has assumptions that cloud provider features are tightly integrated into the project. As a result, adopting this new architecture can create several situations where a request is being made for information from a cloud provider, but the cloud controller manager may not be able to return that information without the original request being complete.
+
+A good example of this is the TLS bootstrapping feature in the Kubelet. Currently, TLS bootstraping assumes that the Kubelet has the ability to ask the cloud provider (or a local metadata service) for all its address types (private, public, etc) but cloud controller manager cannot set a node's address types without being initialzed in the first place which requires that the kubelet has TLS certificates to communicate with the apiserver.
+
+As this initiative evolves, changes will be made to address these issues in upcoming releases.
+
+## Developing your own Cloud Controller Manager
+
+To build and develop your own cloud controller manager, read the [Developing Cloud Controller Manager](/docs/tasks/administer-cluster/developing-cloud-controller-manager.md) doc.