Merge branch 'master' into release-1.13
This commit is contained in:
Jim Angel
@@ -1,3 +1,5 @@
|
||||
# See the OWNERS docs at https://go.k8s.io/owners
|
||||
|
||||
# This is the directory for English source content.
|
||||
# Teams and members are visible at https://github.com/orgs/kubernetes/teams.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ cid: home
|
||||
|
||||
{{< blocks/section id="oceanNodes" >}}
|
||||
{{% blocks/feature image="flower" %}}
|
||||
### [Kubernetes]({{< relref "/docs/concepts/overview/what-is-kubernetes" >}}) is an open-source system for automating deployment, scaling, and management of containerized applications.
|
||||
### [Kubernetes (K8s)]({{< relref "/docs/concepts/overview/what-is-kubernetes" >}}) is an open-source system for automating deployment, scaling, and management of containerized applications.
|
||||
|
||||
It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon [15 years of experience of running production workloads at Google](http://queue.acm.org/detail.cfm?id=2898444), combined with best-of-breed ideas and practices from the community.
|
||||
{{% /blocks/feature %}}
|
||||
@@ -44,12 +44,12 @@ Kubernetes is open source giving you the freedom to take advantage of on-premise
|
||||
<br>
|
||||
<br>
|
||||
<br>
|
||||
<a href="https://www.lfasiallc.com/events/kubecon-cloudnativecon-china-2018/" button id="desktopKCButton">Attend KubeCon in Shanghai on Nov. 13-15, 2018</a>
|
||||
<a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019" button id="desktopKCButton">Attend KubeCon in Barcelona on May 20-23, 2019</a>
|
||||
<br>
|
||||
<br>
|
||||
<br>
|
||||
<br>
|
||||
<a href="https://events.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2018/" button id="desktopKCButton">Attend KubeCon in Seattle on Dec. 11-13, 2018</a>
|
||||
<a href="https://www.lfasiallc.com/events/kubecon-cloudnativecon-china-2019" button id="desktopKCButton">Attend KubeCon in Shanghai on June 24-26, 2019</a>
|
||||
</div>
|
||||
<div id="videoPlayer">
|
||||
<iframe data-url="https://www.youtube.com/embed/H06qrNmGqyE?autoplay=1" frameborder="0" allowfullscreen></iframe>
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# See the OWNERS docs at https://go.k8s.io/owners
|
||||
|
||||
# Owned by Kubernetes Blog reviewers.
|
||||
options:
|
||||
no_parent_owners: false
|
||||
|
||||
@@ -37,11 +37,11 @@ Every week the Kubernetes contributing community meet virtually over Google Hang
|
||||
* additional status - additive, backward compatible
|
||||
* elimination of phase - won't make it for v1
|
||||
* Service discussion - Public IPs
|
||||
* with public ips as it exists we can't go to v1
|
||||
* with public IPs as it exists we can't go to v1
|
||||
* Tim has been developing a mitigation if we can't get Justin's overhaul in (but hopefully we will)
|
||||
* Justin's fix will describe public IPs in a much better way
|
||||
* The general problem is it's too flexible and you can do things that are scary, the mitigation is to restrict public ip usage to specific use cases -- validated public ips would be copied to status, which is what kube-proxy would use
|
||||
* public ips used for -
|
||||
* The general problem is it's too flexible and you can do things that are scary, the mitigation is to restrict public ip usage to specific use cases -- validated public IPs would be copied to status, which is what kube-proxy would use
|
||||
* public IPs used for -
|
||||
* binding to nodes / node
|
||||
* request a specific load balancer IP (GCE only)
|
||||
* emulate multi-port services -- now we support multi-port services, so no longer necessary
|
||||
|
||||
@@ -27,7 +27,7 @@ E2E issues and LGTM process
|
||||
|
||||
* Question/concern to work out is securing Jenkins. Short term conclusion: Will look at pushing Jenkins logs into GCS bucket. Lavalamp will follow up with Jeff Grafton.
|
||||
|
||||
* Longer term solution may be a merge queue, where e2e runs for each merge (as opposed to multiple merges). This exists in Openshift today.
|
||||
* Longer term solution may be a merge queue, where e2e runs for each merge (as opposed to multiple merges). This exists in OpenShift today.
|
||||
|
||||
Cluster Upgrades for Kubernetes as final v1 feature
|
||||
|
||||
|
||||
@@ -131,7 +131,7 @@ Cluster Federation is a control plane of cluster federation in Kubernetes. It of
|
||||
|
||||
|
||||
|
||||
Each Kubernetes cluster exposes an API endpoint and gets registered to Cluster Federation as a part of Federation object. Then using Cluster Federation API, you can create federated services. Those objects are comprised of multiple equivalent underlying Kubernetes resources. Assuming that the 3 clusters on the picture above belong to the same Federation object, each Service created via Cluster Federation, will get equivalent service created in each of the clusters. Besides that, a Cluster Federation service will get publicly resolvable DNS name resolvable to Kuberentes service’s public ip addresses (DNS record gets programmed to a one of the public DNS providers below):
|
||||
Each Kubernetes cluster exposes an API endpoint and gets registered to Cluster Federation as a part of Federation object. Then using Cluster Federation API, you can create federated services. Those objects are comprised of multiple equivalent underlying Kubernetes resources. Assuming that the 3 clusters on the picture above belong to the same Federation object, each Service created via Cluster Federation, will get equivalent service created in each of the clusters. Besides that, a Cluster Federation service will get publicly resolvable DNS name resolvable to Kubernetes service’s public ip addresses (DNS record gets programmed to a one of the public DNS providers below):
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -4,6 +4,8 @@ date: 2016-08-31
|
||||
slug: security-best-practices-kubernetes-deployment
|
||||
url: /blog/2016/08/Security-Best-Practices-Kubernetes-Deployment
|
||||
---
|
||||
_Note: some of the recommendations in this post are no longer current. Current cluster hardening options are described in this [documentation](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/)._
|
||||
|
||||
_Editor’s note: today’s post is by Amir Jerbi and Michael Cherny of Aqua Security, describing security best practices for Kubernetes deployments, based on data they’ve collected from various use-cases seen in both on-premises and cloud deployments._
|
||||
|
||||
Kubernetes provides many controls that can greatly improve your application security. Configuring them requires intimate knowledge with Kubernetes and the deployment’s security requirements. The best practices we highlight here are aligned to the container lifecycle: build, ship and run, and are specifically tailored to Kubernetes deployments. We adopted these best practices in [our own SaaS deployment](http://blog.aquasec.com/running-a-security-service-in-google-cloud-real-world-example) that runs Kubernetes on Google Cloud Platform.
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ We see kompose as a terrific way to expose Kubernetes principles to Docker users
|
||||
|
||||
Over the summer, Kompose has found a new gear with help from Tomas Kral and Suraj Deshmukh from Red Hat, and Janet Kuo from Google. Together with our own lead kompose developer Nguyen An-Tu they are making kompose even more exciting. We proposed Kompose to the Kubernetes Incubator within the SIG-apps and we received approval from the general Kubernetes community; you can now find kompose in the [Kubernetes Incubator](https://github.com/kubernetes-incubator/kompose).
|
||||
|
||||
Kompose now supports Docker-compose v2 format, persistent volume claims have been added recently, as well as multiple container per pods. It can also be used to target Openshift deployments, by specifying a different provider than the default Kubernetes. Kompose is also now available in Fedora packages and we look forward to see it in CentOS distributions in the coming weeks.
|
||||
Kompose now supports Docker-compose v2 format, persistent volume claims have been added recently, as well as multiple container per pods. It can also be used to target OpenShift deployments, by specifying a different provider than the default Kubernetes. Kompose is also now available in Fedora packages and we look forward to see it in CentOS distributions in the coming weeks.
|
||||
|
||||
kompose is a single Golang binary that you build or install from the [release on GitHub](https://github.com/kubernetes-incubator/kompose). Let’s skip the build instructions and dive straight into an example.
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ I was able to run more processes on a single physical server than I could using
|
||||
|
||||
|
||||
|
||||
To orchestrate container deployment, we are using[Armada infrastructure](https://console.bluemix.net/containers-kubernetes/launch), a Kubernetes implementation by IBM for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure.
|
||||
To orchestrate container deployment, we are using [IBM Cloud Kubernetes Service infrastructure](https://cloud.ibm.com/containers-kubernetes/landing), a Kubernetes implementation by IBM for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure.
|
||||
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ Here is a snapshot of Watson Care Manager, running inside a Kubernetes cluster:
|
||||
|
||||
|
||||
|
||||
Before deploying an app, a user must create a worker node cluster. I can create a cluster using the kubectl cli commands or create it from[a Bluemix](http://bluemix.net/) dashboard.
|
||||
Before deploying an app, a user must create a worker node cluster. I can create a cluster using the kubectl cli commands or create it from the [IBM Cloud](https://cloud.ibm.com/) dashboard.
|
||||
|
||||
|
||||
|
||||
@@ -107,16 +107,16 @@ If needed, run a rolling update to update the existing pod.
|
||||
|
||||
|
||||
|
||||
Deploying the application in Armada:
|
||||
Deploying the application in IBM Cloud Kubernetes Service:
|
||||
|
||||
|
||||
|
||||
Provision a cluster in Armada with \<x\> worker nodes. Create Kubernetes controllers for deploying the containers in worker nodes, the Armada infrastructure pulls the Docker images from IBM Bluemix Docker registry to create containers. We tried deploying an application container and running a logmet agent (see Reading and displaying logs using logmet container, below) inside the containers that forwards the application logs to an IBM cloud logging service. As part of the process, YAML files are used to create a controller resource for the UrbanCode Deploy (UCD). UCD agent is deployed as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) controller, which is used to connect to the UCD server. The whole process of deployment of application happens in UCD. To support the application for public access, we created a service resource to interact between pods and access container services. For storage support, we created persistent volume claims and mounted the volume for the containers.
|
||||
Provision a cluster in IBM Cloud Kubernetes Service with \<x\> worker nodes. Create Kubernetes controllers for deploying the containers in worker nodes, the IBM Cloud Kubernetes Service infrastructure pulls the Docker images from IBM Cloud Container Registry to create containers. We tried deploying an application container and running a logmet agent (see Reading and displaying logs using logmet container, below) inside the containers that forwards the application logs to an IBM Cloud logging service. As part of the process, YAML files are used to create a controller resource for the UrbanCode Deploy (UCD). UCD agent is deployed as a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) controller, which is used to connect to the UCD server. The whole process of deployment of application happens in UCD. To support the application for public access, we created a service resource to interact between pods and access container services. For storage support, we created persistent volume claims and mounted the volume for the containers.
|
||||
|
||||
|
||||
|
||||
|  |
|
||||
| UCD: IBM UrbanCode Deploy is a tool for automating application deployments through your environments. Armada: Kubernetes implementation of IBM. WH Docker Registry: Docker Private image registry. Common agent containers: We expect to configure our services to use the WHC mandatory agents. We deployed all ion containers. |
|
||||
| UCD: IBM UrbanCode Deploy is a tool for automating application deployments through your environments. IBM Cloud Kubernetes Service: Kubernetes implementation of IBM. WH Docker Registry: Docker Private image registry. Common agent containers: We expect to configure our services to use the WHC mandatory agents. We deployed all ion containers. |
|
||||
|
||||
|
||||
|
||||
@@ -142,7 +142,7 @@ Exposing services with Ingress:
|
||||
|
||||
|
||||
|
||||
To expose our services to outside the cluster, we used Ingress. In Armada, if we create a paid cluster, an Ingress controller is automatically installed for us to use. We were able to access services through Ingress by creating a YAML resource file that specifies the service path.
|
||||
To expose our services to outside the cluster, we used Ingress. In IBM Cloud Kubernetes Service, if we create a paid cluster, an Ingress controller is automatically installed for us to use. We were able to access services through Ingress by creating a YAML resource file that specifies the service path.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -94,7 +94,7 @@ If you’d like to try out Kubeflow, we have a number of options for you:
|
||||
|
||||
1. You can use sample walkthroughs hosted on [Katacoda](https://www.katacoda.com/kubeflow)
|
||||
2. You can follow a guided tutorial with existing models from the [examples repository](https://github.com/kubeflow/examples). These include the [Github Issue Summarization](https://github.com/kubeflow/examples/tree/master/github_issue_summarization), [MNIST](https://github.com/kubeflow/examples/tree/master/mnist) and [Reinforcement Learning with Agents](https://github.com/kubeflow/examples/tree/master/agents).
|
||||
3. You can start a cluster on your own and try your own model. Any Kubernetes conformant cluster will support Kubeflow including those from contributors [Caicloud](https://www.prnewswire.com/news-releases/caicloud-releases-its-kubernetes-based-cluster-as-a-service-product-claas-20-and-the-first-tensorflow-as-a-service-taas-11-while-closing-6m-series-a-funding-300418071.html), [Canonical](https://jujucharms.com/canonical-kubernetes/), [Google](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-container-cluster), [Heptio](https://heptio.com/products/kubernetes-subscription/), [Mesosphere](https://github.com/mesosphere/dcos-kubernetes-quickstart), [Microsoft](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough), [IBM](https://console.bluemix.net/docs/containers/cs_tutorials.html#cs_cluster_tutorial), [Red Hat/Openshift ](https://docs.openshift.com/container-platform/3.3/install_config/install/quick_install.html#install-config-install-quick-install)and [Weaveworks](https://www.weave.works/product/cloud/).
|
||||
3. You can start a cluster on your own and try your own model. Any Kubernetes conformant cluster will support Kubeflow including those from contributors [Caicloud](https://www.prnewswire.com/news-releases/caicloud-releases-its-kubernetes-based-cluster-as-a-service-product-claas-20-and-the-first-tensorflow-as-a-service-taas-11-while-closing-6m-series-a-funding-300418071.html), [Canonical](https://jujucharms.com/canonical-kubernetes/), [Google](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-container-cluster), [Heptio](https://heptio.com/products/kubernetes-subscription/), [Mesosphere](https://github.com/mesosphere/dcos-kubernetes-quickstart), [Microsoft](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough), [IBM](https://cloud.ibm.com/docs/containers?topic=containers-cs_cluster_tutorial#cs_cluster_tutorial), [Red Hat/Openshift ](https://docs.openshift.com/container-platform/3.3/install_config/install/quick_install.html#install-config-install-quick-install)and [Weaveworks](https://www.weave.works/product/cloud/).
|
||||
|
||||
There were also a number of sessions at KubeCon + CloudNativeCon EU 2018 covering Kubeflow. The links to the talks are here; the associated videos will be posted in the coming days.
|
||||
|
||||
|
||||
@@ -109,7 +109,7 @@ For a detailed list of changes in the containerd 1.1 release, please see the rel
|
||||
To setup a Kubernetes cluster using containerd as the container runtime:
|
||||
|
||||
* For a production quality cluster on GCE brought up with kube-up.sh, see [here](https://github.com/containerd/cri/blob/v1.0.0/docs/kube-up.md).
|
||||
* For a multi-node cluster installer and bring up steps using ansible and kubeadm, see [here](https://github.com/containerd/cri/blob/v1.0.0/contrib/ansible/README.md).
|
||||
* For a multi-node cluster installer and bring up steps using Ansible and kubeadm, see [here](https://github.com/containerd/cri/blob/v1.0.0/contrib/ansible/README.md).
|
||||
* For creating a cluster from scratch on Google Cloud, see [Kubernetes the Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way).
|
||||
* For a custom installation from release tarball, see [here](https://github.com/containerd/cri/blob/v1.0.0/docs/installation.md).
|
||||
* To install using LinuxKit on a local VM, see [here](https://github.com/linuxkit/linuxkit/tree/master/projects/kubernetes).
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
layout: blog
|
||||
title: 'gRPC Load Balancing on Kubernetes without Tears'
|
||||
date: 2018-11-07
|
||||
---
|
||||
|
||||
**Author**: William Morgan (Buoyant)
|
||||
|
||||
Many new gRPC users are surprised to find that Kubernetes's default load
|
||||
balancing often doesn't work out of the box with gRPC. For example, here's what
|
||||
happens when you take a [simple gRPC Node.js microservices
|
||||
app](https://github.com/sourishkrout/nodevoto) and deploy it on Kubernetes:
|
||||
|
||||
|
||||
|
||||
While the `voting` service displayed here has several pods, it's clear from
|
||||
Kubernetes's CPU graphs that only one of the pods is actually doing any
|
||||
work—because only one of the pods is receiving any traffic. Why?
|
||||
|
||||
In this blog post, we describe why this happens, and how you can easily fix it
|
||||
by adding gRPC load balancing to any Kubernetes app with
|
||||
[Linkerd](https://linkerd.io), a [CNCF](https://cncf.io) service mesh and service sidecar.
|
||||
|
||||
# Why does gRPC need special load balancing?
|
||||
|
||||
First, let's understand why we need to do something special for gRPC.
|
||||
|
||||
gRPC is an increasingly common choice for application developers. Compared to
|
||||
alternative protocols such as JSON-over-HTTP, gRPC can provide some significant
|
||||
benefits, including dramatically lower (de)serialization costs, automatic type
|
||||
checking, formalized APIs, and less TCP management overhead.
|
||||
|
||||
However, gRPC also breaks the standard connection-level load balancing,
|
||||
including what's provided by Kubernetes. This is because gRPC is built on
|
||||
HTTP/2, and HTTP/2 is designed to have a single long-lived TCP connection,
|
||||
across which all requests are *multiplexed*—meaning multiple requests can be
|
||||
active on the same connection at any point in time. Normally, this is great, as
|
||||
it reduces the overhead of connection management. However, it also means that
|
||||
(as you might imagine) connection-level balancing isn't very useful. Once the
|
||||
connection is established, there's no more balancing to be done. All requests
|
||||
will get pinned to a single destination pod, as shown below:
|
||||
|
||||
|
||||
|
||||
# Why doesn't this affect HTTP/1.1?
|
||||
|
||||
The reason why this problem doesn't occur in HTTP/1.1, which also has the
|
||||
concept of long-lived connections, is because HTTP/1.1 has several features
|
||||
that naturally result in cycling of TCP connections. Because of this,
|
||||
connection-level balancing is "good enough", and for most HTTP/1.1 apps we
|
||||
don't need to do anything more.
|
||||
|
||||
To understand why, let's take a deeper look at HTTP/1.1. In contrast to HTTP/2,
|
||||
HTTP/1.1 cannot multiplex requests. Only one HTTP request can be active at a
|
||||
time per TCP connection. The client makes a request, e.g. `GET /foo`, and then
|
||||
waits until the server responds. While that request-response cycle is
|
||||
happening, no other requests can be issued on that connection.
|
||||
|
||||
Usually, we want lots of requests happening in parallel. Therefore, to have
|
||||
concurrent HTTP/1.1 requests, we need to make multiple HTTP/1.1 connections,
|
||||
and issue our requests across all of them. Additionally, long-lived HTTP/1.1
|
||||
connections typically expire after some time, and are torn down by the client
|
||||
(or server). These two factors combined mean that HTTP/1.1 requests typically
|
||||
cycle across multiple TCP connections, and so connection-level balancing works.
|
||||
|
||||
# So how do we load balance gRPC?
|
||||
|
||||
Now back to gRPC. Since we can't balance at the connection level, in order to
|
||||
do gRPC load balancing, we need to shift from connection balancing to *request*
|
||||
balancing. In other words, we need to open an HTTP/2 connection to each
|
||||
destination, and balance *requests* across these connections, as shown below:
|
||||
|
||||
|
||||
|
||||
In network terms, this means we need to make decisions at L5/L7 rather than
|
||||
L3/L4, i.e. we need to understand the protocol sent over the TCP connections.
|
||||
|
||||
How do we accomplish this? There are a couple options. First, our application
|
||||
code could manually maintain its own load balancing pool of destinations, and
|
||||
we could configure our gRPC client to [use this load balancing
|
||||
pool](https://godoc.org/google.golang.org/grpc/balancer). This approach gives
|
||||
us the most control, but it can be very complex in environments like Kubernetes
|
||||
where the pool changes over time as Kubernetes reschedules pods. Our
|
||||
application would have to watch the Kubernetes API and keep itself up to date
|
||||
with the pods.
|
||||
|
||||
Alternatively, in Kubernetes, we could deploy our app as [headless
|
||||
services](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services).
|
||||
In this case, Kubernetes [will create multiple A
|
||||
records](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services)
|
||||
in the DNS entry for the service. If our gRPC client is sufficiently advanced,
|
||||
it can automatically maintain the load balancing pool from those DNS entries.
|
||||
But this approach restricts us to certain gRPC clients, and it's rarely
|
||||
possible to only use headless services.
|
||||
|
||||
Finally, we can take a third approach: use a lightweight proxy.
|
||||
|
||||
# gRPC load balancing on Kubernetes with Linkerd
|
||||
|
||||
[Linkerd](https://linkerd.io) is a [CNCF](https://cncf.io)-hosted *service
|
||||
mesh* for Kubernetes. Most relevant to our purposes, Linkerd also functions as
|
||||
a *service sidecar*, where it can be applied to a single service—even without
|
||||
cluster-wide permissions. What this means is that when we add Linkerd to our
|
||||
service, it adds a tiny, ultra-fast proxy to each pod, and these proxies watch
|
||||
the Kubernetes API and do gRPC load balancing automatically. Our deployment
|
||||
then looks like this:
|
||||
|
||||
|
||||
|
||||
Using Linkerd has a couple advantages. First, it works with services written in
|
||||
any language, with any gRPC client, and any deployment model (headless or not).
|
||||
Because Linkerd's proxies are completely transparent, they auto-detect HTTP/2
|
||||
and HTTP/1.x and do L7 load balancing, and they pass through all other traffic
|
||||
as pure TCP. This means that everything will *just work.*
|
||||
|
||||
Second, Linkerd's load balancing is very sophisticated. Not only does Linkerd
|
||||
maintain a watch on the Kubernetes API and automatically update the load
|
||||
balancing pool as pods get rescheduled, Linkerd uses an *exponentially-weighted
|
||||
moving average* of response latencies to automatically send requests to the
|
||||
fastest pods. If one pod is slowing down, even momentarily, Linkerd will shift
|
||||
traffic away from it. This can reduce end-to-end tail latencies.
|
||||
|
||||
Finally, Linkerd's Rust-based proxies are incredibly fast and small. They
|
||||
introduce <1ms of p99 latency and require <10mb of RSS per pod, meaning that
|
||||
the impact on system performance will be negligible.
|
||||
|
||||
# gRPC Load Balancing in 60 seconds
|
||||
|
||||
Linkerd is very easy to try. Just follow the steps in the [Linkerd Getting
|
||||
Started Instructions](https://linkerd.io/2/getting-started/)—install the
|
||||
CLI on your laptop, install the control plane on your cluster, and "mesh" your
|
||||
service (inject the proxies into each pod). You'll have Linkerd running on your
|
||||
service in no time, and should see proper gRPC balancing immediately.
|
||||
|
||||
Let's take a look at our sample `voting` service again, this time after
|
||||
installing Linkerd:
|
||||
|
||||
|
||||
|
||||
As we can see, the CPU graphs for all pods are active, indicating that all pods
|
||||
are now taking traffic—without having to change a line of code. Voila,
|
||||
gRPC load balancing as if by magic!
|
||||
|
||||
Linkerd also gives us built-in traffic-level dashboards, so we don't even need
|
||||
to guess what's happening from CPU charts any more. Here's a Linkerd graph
|
||||
that's showing the success rate, request volume, and latency percentiles of
|
||||
each pod:
|
||||
|
||||
|
||||
|
||||
We can see that each pod is getting around 5 RPS. We can also see that, while
|
||||
we've solved our load balancing problem, we still have some work to do on our
|
||||
success rate for this service. (The demo app is built with an intentional
|
||||
failure—as an exercise to the reader, see if you can figure it out by
|
||||
using the Linkerd dashboard!)
|
||||
|
||||
# Wrapping it up
|
||||
|
||||
If you're interested in a dead simple way to add gRPC load balancing to your
|
||||
Kubernetes services, regardless of what language it's written in, what gRPC
|
||||
client you're using, or how it's deployed, you can use Linkerd to add gRPC load
|
||||
balancing in a few commands.
|
||||
|
||||
There's a lot more to Linkerd, including security, reliability, and debugging
|
||||
and diagnostics features, but those are topics for future blog posts.
|
||||
|
||||
Want to learn more? We’d love to have you join our rapidly-growing community!
|
||||
Linkerd is a [CNCF](https://cncf.io) project, [hosted on
|
||||
GitHub](https://github.com/linkerd/linkerd2), and has a thriving community
|
||||
on [Slack](https://slack.linkerd.io), [Twitter](https://twitter.com/linkerd),
|
||||
and the [mailing lists](https://lists.cncf.io/g/cncf-linkerd-users). Come and
|
||||
join the fun!
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
layout: blog
|
||||
title: 'New Contributor Workshop Shanghai'
|
||||
date: 2018-12-05
|
||||
---
|
||||
|
||||
**Authors**: Josh Berkus (Red Hat), Yang Li (The Plant), Puja Abbassi (Giant Swarm), XiangPeng Zhao (ZTE)
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/attendees.png" caption="Kubecon Shanghai New Contributor Summit attendees. Photo by Jerry Zhang" >}}
|
||||
|
||||
We recently completed our first New Contributor Summit in China, at the first KubeCon in China. It was very exciting to see all of the Chinese and Asian developers (plus a few folks from around the world) interested in becoming contributors. Over the course of a long day, they learned how, why, and where to contribute to Kubernetes, created pull requests, attended a panel of current contributors, and got their CLAs signed.
|
||||
|
||||
This was our second New Contributor Workshop (NCW), building on the one created and led by SIG Contributor Experience members in Copenhagen. Because of the audience, it was held in both Chinese and English, taking advantage of the superb simultaneous interpretation services the CNCF sponsored. Likewise, the NCW team included both English and Chinese-speaking members of the community: Yang Li, XiangPeng Zhao, Puja Abbassi, Noah Abrahams, Tim Pepper, Zach Corleissen, Sen Lu, and Josh Berkus. In addition to presenting and helping students, the bilingual members of the team translated all of the slides into Chinese. Fifty-one students attended.
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/noahabrahams.png" caption="Noah Abrahams explains Kubernetes communications channels. Photo by Jerry Zhang" >}}
|
||||
|
||||
The NCW takes participants through the stages of contributing to Kubernetes, starting from deciding where to contribute, followed by an introduction to the SIG system and our repository structure. We also have "guest speakers" from Docs and Test Infrastructure who cover contributing in those areas. We finally wind up with some hands-on exercises in filing issues and creating and approving PRs.
|
||||
|
||||
Those hands-on exercises use a repository known as [the contributor playground](https://github.com/kubernetes-sigs/contributor-playground), created by SIG Contributor Experience as a place for new contributors to try out performing various actions on a Kubernetes repo. It has modified Prow and Tide automation, uses Owners files like in the real repositories. This lets students learn how the mechanics of contributing to our repositories work without disrupting normal development.
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/yangli.png" caption="Yang Li talks about getting your PRs reviewed. Photo by Josh Berkus" >}}
|
||||
|
||||
Both the "Great Firewall" and the language barrier prevent contributing Kubernetes from China from being straightforward. What's more, because open source business models are not mature in China, the time for employees work on open source projects is limited.
|
||||
|
||||
Chinese engineers are eager to participate in the development of Kubernetes, but many of them don't know where to start since Kubernetes is such a large project. With this workshop, we hope to help those who want to contribute, whether they wish to fix some bugs they encountered, improve or localize documentation, or they need to work with Kubernetes at their work. We are glad to see more and more Chinese contributors joining the community in the past few years, and we hope to see more of them in the future.
|
||||
|
||||
"I have been participating in the Kubernetes community for about three years," said XiangPeng Zhao. "In the community, I notice that more and more Chinese developers are showing their interest in contributing to Kubernetes. However, it's not easy to start contributing to such a project. I tried my best to help those who I met in the community, but I think there might still be some new contributors leaving the community due to not knowing where to get help when in trouble. Fortunately, the community initiated NCW at KubeCon Copenhagen and held a second one at KubeCon Shanghai. I was so excited to be invited by Josh Berkus to help organize this workshop. During the workshop, I met community friends in person, mentored attendees in the exercises, and so on. All of this was a memorable experience for me. I also learned a lot as a contributor who already has years of contributing experience. I wish I had attended such a workshop when I started contributing to Kubernetes years ago."
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/panel.png" caption="Panel of contributors. Photo by Jerry Zhang" >}}
|
||||
|
||||
The workshop ended with a panel of current contributors, featuring Lucas Käldström, Janet Kuo, Da Ma, Pengfei Ni, Zefeng Wang, and Chao Xu. The panel aimed to give both new and current contributors a look behind the scenes on the day-to-day of some of the most active contributors and maintainers, both from China and around the world. Panelists talked about where to begin your contributor's journey, but also how to interact with reviewers and maintainers. They further touched upon the main issues of contributing from China and gave attendees an outlook into exciting features they can look forward to in upcoming releases of Kubernetes.
|
||||
|
||||
After the workshop, Xiang Peng Zhao chatted with some attendees on WeChat and Twitter about their experiences. They were very glad to have attended the NCW and had some suggestions on improving the workshop. One attendee, Mohammad, said, "I had a great time at the workshop and learned a lot about the entire process of k8s for a contributor." Another attendee, Jie Jia, said, "The workshop was wonderful. It systematically explained how to contribute to Kubernetes. The attendee could understand the process even if s/he knew nothing about that before. For those who were already contributors, they could also learn something new. Furthermore, I could make new friends from inside or outside of China in the workshop. It was awesome!"
|
||||
|
||||
SIG Contributor Experience will continue to run New Contributor Workshops at each upcoming Kubecon, including Seattle, Barcelona, and the return to Shanghai in June 2019. If you failed to get into one this year, register for one at a future Kubecon. And, when you meet an NCW attendee, make sure to welcome them to the community.
|
||||
|
||||
Links:
|
||||
|
||||
* English versions of the slides: [PDF](https://gist.github.com/jberkus/889be25c234b01761ce44eccff816380#file-kubernetes-shanghai-english-pdf) or [Google Docs with speaker notes](https://docs.google.com/presentation/d/1l5f_iAFsKg50LFq3N80KbZKUIEL_tyCaUoWPzSxColo/edit?usp=sharing)
|
||||
* Chinese version of the slides: [PDF](https://gist.github.com/jberkus/889be25c234b01761ce44eccff816380#file-kubernetes-shanghai-cihinese-pdf)
|
||||
* [Contributor playground](https://github.com/kubernetes-sigs/contributor-playground)
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
layout: blog
|
||||
title: Kubernetes Federation Evolution
|
||||
date: 2018-12-12
|
||||
---
|
||||
|
||||
**Authors**: Irfan Ur Rehman (Huawei), Paul Morie (RedHat) and Shashidhara T D (Huawei)
|
||||
|
||||
Kubernetes provides great primitives for deploying applications to a cluster: it can be as simple as `kubectl create -f app.yaml`. Deploy apps across multiple clusters has never been that simple. How should app workloads be distributed? Should the app resources be replicated into all clusters, replicated into selected clusters, or partitioned into clusters? How is access to the clusters managed? What happens if some of the resources that a user wants to distribute pre-exist, in some or all of the clusters, in some form?
|
||||
|
||||
In SIG Multicluster, our journey has revealed that there are multiple possible models to solve these problems and there probably is no single best-fit, all-scenario solution. [Federation](https://kubernetes.io/docs/concepts/cluster-administration/federation/), however, is the single biggest Kubernetes open source sub-project, and has seen the maximum interest and contribution from the community in this problem space. The project initially reused the Kubernetes API to do away with any added usage complexity for an existing Kubernetes user. This approach was not viable, because of the problems summarised below:
|
||||
|
||||
* Difficulties in re-implementing the Kubernetes API at the cluster level, as federation-specific extensions were stored in annotations.
|
||||
* Limited flexibility in federated types, placement and reconciliation, due to 1:1 emulation of the Kubernetes API.
|
||||
* No settled path to GA, and general confusion on API maturity; for example, Deployments are GA in Kubernetes but not even Beta in Federation v1.
|
||||
|
||||
|
||||
The ideas have evolved further with a federation-specific API architecture and a community effort which now continues as Federation v2.
|
||||
|
||||
# Conceptual Overview
|
||||
Because Federation attempts to address a complex set of problems, it pays to break the different parts of those problems down. Let’s take a look at the different high-level areas involved:
|
||||
{{< figure src="/images/blog/2018-12-11-Kubernetes-Federation-Evolution/concepts.png" caption="Kubernetes Federation v2 Concepts" >}}
|
||||
|
||||
## Federating arbitrary resources
|
||||
One of the main goals of Federation is to be able to define the APIs and API groups which encompass basic tenets needed to federate any given Kubernetes resource. This is crucial, due to the popularity of CustomResourceDefinitions as a way to extend Kubernetes with new APIs.
|
||||
|
||||
The workgroup arrived at a common definition of the federation API and API groups as _'a mechanism that distributes “normal” Kubernetes API resources into different clusters'_. The distribution in its most simple form could be imagined as ***simple propagation*** of this _'normal Kubernetes API resource'_ across the federated clusters. A thoughtful reader can certainly discern more complicated mechanisms, other than this simple propagation of the Kubernetes resources.
|
||||
|
||||
During the journey of defining building blocks of the federation APIs, one of the near term goals also evolved as _'to be able to create a simple federation a.k.a. simple propagation of any Kubernetes resource or a CRD, writing almost zero code'_. What ensued further was a core API group defining the building blocks as a `Template` resource, a `Placement` resource and an `Override` resource per given Kubernetes resource, a `TypeConfig` to specify sync or no sync for the given resource and associated controller(s) to carry out the sync. More details follow [in the next section](#federating-resources-the-details). Further sections will also talk about being able to follow a layered behaviour with higher-level federation APIs consuming the behaviour of these core building blocks, and users being able to consume whole or part of the API and associated controllers. Lastly, this architecture also allows the users to write additional controllers or replace the available reference controllers with their own, to carry out desired behaviour.
|
||||
|
||||
The ability to _'easily federate arbitrary Kubernetes resources'_, and a decoupled API, divided into building blocks APIs, higher level APIs and possible user intended types, presented such that different users can consume parts and write controllers composing solutions specific to them, makes a compelling case for Federation v2.
|
||||
|
||||
## Federating resources: the details
|
||||
Fundamentally, federation must be configured with two types of information:
|
||||
|
||||
* Which API types federation should handle
|
||||
* Which clusters federation should target for distributing those resources.
|
||||
|
||||
For each API type that federation handles, different parts of the declared state live in different API resources:
|
||||
|
||||
* A `Template` type holds the base specification of the resource - for example, a type called `FederatedReplicaSet` holds the base specification of a `ReplicaSet` that should be distributed to the targeted clusters
|
||||
* A `Placement` type holds the specification of the clusters the resource should be distributed to - for example, a type called `FederatedReplicaSetPlacement` holds information about which clusters `FederatedReplicaSets` should be distributed to
|
||||
* An optional `Overrides` type holds the specification of how the `Template` resource should be varied in some clusters - for example, a type called `FederatedReplicaSetOverrides` holds information about how a `FederatedReplicaSet` should be varied in certain clusters.
|
||||
|
||||
These types are all associated by name - meaning that for a particular Template resource with name `foo`, the Placement and Override information for that resource are contained by the Override and Placement resources with the name `foo` and in the same namespace as the Template.
|
||||
|
||||
## Higher-level behaviour
|
||||
The architecture of the v2 API allows higher-level APIs to be constructed using the mechanics provided by the core API types (`Template`, `Placement` and `Override`), and associated controllers, for a given resource. In the community we uncovered a few use cases and implemented the higher-level APIs and associated controllers useful for those cases. Some of these types described in further sections also provide an useful reference to anybody interested in solving more complex use cases, building on top of the mechanics already available with the v2 API.
|
||||
|
||||
### ReplicaSchedulingPreference
|
||||
`ReplicaSchedulingPreference` provides an automated mechanism of distributing and maintaining total number of replicas for Deployment or ReplicaSet-based federated workloads into federated clusters. This is based on high-level user preferences given by the user. These preferences include the semantics of _weighted distribution_ and _limits_ (min and max) for distributing the replicas. These also include semantics to allow redistribution of replicas dynamically in case some replica Pods remain unscheduled in some clusters, for example due to insufficient resources in that cluster.
|
||||
More details can be found at the [user guide for ReplicaSchedulingPreferences](https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md#replicaschedulingpreference).
|
||||
|
||||
### Federated services & cross-cluster service discovery
|
||||
Kubernetes Services are very useful in constructing a microservices architecture. There is a clear desire to deploy services across cluster, zone, region and cloud boundaries. Services that span clusters provide geographic distribution, enable hybrid and multi-cloud scenarios and improve the level of high availability beyond single cluster deployments. Customers who want their services to span one or more (possibly remote) clusters, need them to be reachable in a consistent manner from both within and outside their clusters.
|
||||
|
||||
Federated `Service`, at its core, contains a `Template` (a definition of a Kubernetes Service), a `Placement` (which clusters to be deployed into), an `Override` (optional variation in particular clusters) and a `ServiceDNSRecord` (specifying details on how to discover it).
|
||||
|
||||
Note: The federated service has to be of type `LoadBalancer` in order for it to be discoverable across clusters.
|
||||
|
||||
#### Discovering a federated service from Pods inside your federated clusters
|
||||
By default, Kubernetes clusters come preconfigured with a cluster-local DNS server, as well as an intelligently constructed DNS search path, which together ensure that DNS queries like `myservice`, `myservice.mynamespace`, or `some-other-service.other-namespace`, issued by software running inside Pods, are automatically expanded and resolved correctly to the appropriate IP of Services running in the local cluster.
|
||||
|
||||
With the introduction of federated services and cross-cluster service discovery, this concept is extended to cover Kubernetes Services running in any other cluster across your cluster federation, globally. To take advantage of this extended range, you use a slightly different DNS name (e.g. `myservice.mynamespace.myfederation`) to resolve federated services. Using a different DNS name also avoids having your existing applications accidentally traversing cross-zone or cross-region networks and you incurring perhaps unwanted network charges or latency, without you explicitly opting in to this behavior.
|
||||
|
||||
Lets consider an example, using a service named `nginx`.
|
||||
|
||||
A Pod in a cluster in the `us-central1-a` availability zone needs to contact our `nginx` service. Rather than use the service’s traditional cluster-local DNS name (`nginx.mynamespace`, which is automatically expanded to `nginx.mynamespace.svc.cluster.local`) it can now use the service’s federated DNS name, which is `nginx.mynamespace.myfederation`. This will be automatically expanded and resolved to the closest healthy shard of my `nginx` service, wherever in the world that may be. If a healthy shard exists in the local cluster, that service’s cluster-local IP address will be returned (by the cluster-local DNS). This is exactly equivalent to non-federated service resolution.
|
||||
|
||||
If the Service does not exist in the local cluster (or it exists but has no healthy backend pods), the DNS query is automatically expanded to `nginx.mynamespace.myfederation.svc.us-central1-a.us-central1.example.com`. Behind the scenes, this finds the external IP of one of the shards closest to my availability zone. This expansion is performed automatically by the cluster-local DNS server, which returns the associated CNAME record. This results in a traversal of the hierarchy of DNS records, and ends up at one of the external IP’s of the federated service nearby.
|
||||
|
||||
It is also possible to target service shards in availability zones and regions other than the ones local to a Pod by specifying the appropriate DNS names explicitly, and not relying on automatic DNS expansion. For example, `nginx.mynamespace.myfederation.svc.europe-west1.example.com`will resolve to all of the currently healthy service shards in Europe, even if the Pod issuing the lookup is located in the U.S., and irrespective of whether or not there are healthy shards of the service in the U.S. This is useful for remote monitoring and other similar applications.
|
||||
|
||||
#### Discovering a federated service from other clients outside your federated clusters
|
||||
For external clients, automatic DNS expansion described is not currently possible. External clients need to specify one of the fully qualified DNS names of the federated service, be that a zonal, regional or global name. For convenience reasons, it is often a good idea to manually configure additional static CNAME records in your service, for example:
|
||||
|
||||
| SHORT NAME | CNAME |
|
||||
|-------------------|-------------------------------------------------------------|
|
||||
| eu.nginx.acme.com | nginx.mynamespace.myfederation.svc.europe-west1.example.com |
|
||||
| us.nginx.acme.com | nginx.mynamespace.myfederation.svc.us-central1.example.com |
|
||||
| nginx.acme.com | nginx.mynamespace.myfederation.svc.example.com |
|
||||
|
||||
That way, your clients can always use the short form on the left, and always be automatically routed to the closest healthy shard on their home continent. All of the required failover is handled for you automatically by Kubernetes cluster federation.
|
||||
|
||||
As further reading, a more elaborate example for users is available in the [Multi-Cluster Service DNS with ExternalDNS guide](https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/servicedns-with-externaldns.md).
|
||||
|
||||
# Try it yourself
|
||||
To get started with Federation v2, please refer to the [user guide](https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md). Deployment can be accomplished with a [Helm chart](https://github.com/kubernetes-sigs/federation-v2/blob/master/charts/federation-v2/README.md), and once the control plane is available, the [user guide’s example](https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md#example) can be used to get some hands-on experience with using Federation V2.
|
||||
|
||||
Federation v2 can be deployed in both _cluster-scoped_ and _namespace-scoped_ configurations. A cluster-scoped deployment will require cluster-admin privileges to both host and member clusters, and may be a good fit for evaluating federation on clusters that are not running critical workloads. Namespace-scoped deployment requires access to only a single namespace on host and member clusters, and is a better fit for evaluating federation on clusters running workloads. Most of the user guide refers to cluster-scoped deployment, with the [namespaced federation](https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/userguide.md#namespaced-federation) section documenting how use of a namespaced deployment differs. The same cluster can host multiple federations, and clusters can be part of multiple federations when using namespaced federation.
|
||||
|
||||
# Next Steps
|
||||
As we noted in the beginning of this post, the multicluster problem space is extremely broad. It can be difficult to know exactly how to handle such broad problem spaces without concrete pieces of software to frame those conversations around. Our hope in the Federation working group is that Federation v2 can be a concrete artifact to frame discussions around. We would love to know experiences that folks have had in this problem space, how they feel about Federation v2, and what use-cases they’re interested in exploring in the future.
|
||||
|
||||
Please feel welcome to join us at the [sig-multicluster slack channel](https://kubernetes.slack.com/messages/C09R1PJR3) or at [Federation working group meetings](https://docs.google.com/document/d/1FQx0BPlkkl1Bn0c9ocVBxYIKojpmrS1CFP5h0DI68AE/edit) on Wednesdays at 07:30 PST.
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
layout: blog
|
||||
title: 'etcd: Current status and future roadmap'
|
||||
date: 2018-12-11
|
||||
---
|
||||
|
||||
**Author**: Gyuho Lee (Amazon Container OSS Team, @gyuho), Joe Betz (Google Cloud, @jpbetz)
|
||||
|
||||
etcd is a distributed key value store that provides a reliable way to manage the coordination state of distributed systems. etcd was first announced in June 2013 by CoreOS (part of Red Hat as of 2018). Since its adoption in Kubernetes in 2014, etcd has become a fundamental part of the Kubernetes cluster management software design, and the etcd community has grown exponentially. etcd is now being used in production environments of multiple companies, including large cloud provider environments such as AWS, Google Cloud Platform, Azure, and other on-premises Kubernetes implementations. CNCF currently has [32 conformant Kubernetes platforms and distributions](https://www.cncf.io/announcement/2017/11/13/cloud-native-computing-foundation-launches-certified-kubernetes-program-32-conformant-distributions-platforms/), all of which use etcd as the datastore.
|
||||
|
||||
In this blog post, we’ll review some of the milestones achieved in latest etcd releases, and go over the future roadmap for etcd. Share your thoughts and feedback on features you consider important on the mailing list: etcd-dev@googlegroups.com.
|
||||
|
||||
## etcd, 2013
|
||||
|
||||
In June 2014, Kubernetes was released with etcd as a backing storage for all master states. Kubernetes v0.4 used etcd v0.2 API, which was in an alpha stage at the time. As Kubernetes reached the v1.0 milestone in 2015, etcd stabilized its v2.0 API. The widespread adoption of Kubernetes led to a dramatic increase in the scalability requirements for etcd. To handle large number of workloads and the growing requirements on scale, etcd released v3.0 API in June 2016. Kubernetes v1.13 finally [dropped support for etcd v2.0 API](https://github.com/kubernetes/enhancements/issues/622) and adopted the etcd v3.0 API. The table below gives a visual snapshot of the release cycles of etcd and Kubernetes.
|
||||
|
||||
| | etcd | Kubernetes |
|
||||
|---|---|---|
|
||||
| Initial Commit | June 2, 2013 | June 1, 2014 |
|
||||
| First Stable Release | January 28, 2015 (v2.0.0) | July 13, 2015 (v1.0.0) |
|
||||
| Latest Release | October 10, 2018 (v3.3.10) | December 3, 2018 (v1.13.0) |
|
||||
|
||||
## etcd v3.1, early 2017
|
||||
|
||||
etcd v3.1 features provide better read performance and better availability during version upgrades. Given the high use of etcd in production even to this day, these features were very useful for users. It implements Raft read index, which bypasses [Raft WAL](https://godoc.org/github.com/etcd-io/etcd/wal) disk writes for linearizable reads. The follower requests read index from the leader. Responses from the leader indicate whether a follower has advanced as much as the leader. When the follower's logs are up-to-date, quorum read is served locally without going through the full Raft protocol. Thus, no disk write is required for read requests. etcd v3.1 introduces automatic leadership transfer. When etcd leader receives an interrupt signal, it automatically transfers its leadership to a follower. This provides higher availability when the cluster adds or loses a member.
|
||||
|
||||
## etcd v3.2 (summer 2017)
|
||||
|
||||
etcd v3.2 focuses on stability. Its client was shipped in Kubernetes v1.10, v1.11, and v1.12. The etcd team still actively maintains the branch by backporting all the bug fixes. This release introduces gRPC proxy to support, watch, and coalesce all watch event broadcasts into one gRPC stream. These event broadcasts can go up to one million events per second.
|
||||
|
||||
etcd v3.2 also introduces changes such as `“snapshot-count”` default value from 10,000 to 100,000. With higher snapshot count, etcd server holds Raft entries in-memory for longer periods before compacting the old ones. etcd v3.2 default configuration shows higher memory usage, while giving more time for slow followers to catch up. It is a trade-off between less frequent snapshot sends and higher memory usage. Users can employ lower `etcd --snapshot-count` value to reduce the memory usage or higher `“snapshot-count”` value to increase the availability of slow followers.
|
||||
|
||||
Another new feature backported to etcd v3.2.19 was `etcd --initial-election-tick-advance` flag. By default, a rejoining follower fast-forwards election ticks to speed up its initial cluster bootstrap. For example, the starting follower node only waits 200ms instead of full election timeout 1-second before starting an election. Ideally, within the 200ms, it receives a leader heartbeat and immediately joins the cluster as a follower. However, if network partition happens, heartbeat may drop and thus leadership election will be triggered. A vote request from a partitioned node is quite disruptive. If it contains a higher Raft term, current leader is forced to step down. With “initial-election-tick-advance” set to false, a rejoining node has [more chance to receive leader heartbeats](https://github.com/etcd-io/etcd/pull/9591) before disrupting the cluster.
|
||||
|
||||
## etcd v3.3 (early 2018)
|
||||
|
||||
etcd v3.3 continues the theme of stability. Its client is included in [Kubernetes v1.13](https://github.com/kubernetes/kubernetes/pull/69322). Previously, etcd client carelessly retried on network disconnects without any backoff or failover logic. The client was often stuck with a partitioned node, [affecting several production users](https://github.com/etcd-io/etcd/issues/7321). v3.3 client balancer now maintains a list of unhealthy endpoints using gRPC health checking protocol, making more efficient retries and failover in the face of transient disconnects and [network partitions](https://github.com/etcd-io/etcd/issues/8711). This was backported to etcd v3.2 and also [included in Kubernetes v1.10 API server](https://github.com/kubernetes/kubernetes/pull/57480). etcd v3.3 also provides more predictable database size. etcd used to maintain a separate freelist DB to track pages that were no longer in use and freed after transactions, so that following transactions can reuse them. However, it turns out persisting freelist demands high disk space and introduces high latency for Kubernetes workloads. Especially when there were frequent snapshots with lots of read transactions, etcd database size quickly grew from 16 MB to 4 GB. etcd v3.3 disables freelist sync and rebuilds the freelist on restart. The overhead is so small that it is unnoticeable to most users. See ["database space exceeded" issue](https://github.com/etcd-io/etcd/issues/8009) for more information on this.
|
||||
|
||||
## etcd v3.4 and beyond
|
||||
|
||||
etcd v3.4 focuses on improving the operational experience. It adds [Raft pre-vote feature](https://github.com/etcd-io/etcd/pull/9352) to improve the robustness of leadership election. When a node becomes isolated (e.g. network partition), this member will start an election requesting votes with increased Raft terms. When a leader receives a vote request with a higher term, it steps down to a follower. With pre-vote, Raft runs an additional election phase to check if the candidate can get enough votes to win an election. The isolated follower's vote request is rejected because it does not contain the latest log entries.
|
||||
|
||||
etcd v3.4 adds a [Raft learner](https://etcd.readthedocs.io/en/latest/server-learner.html#server-learner) that joins the cluster as a non-voting member that still receives all the updates from leader. Adding a learner node does not increase the size of quorum and hence improves the cluster availability during membership reconfiguration. It only serves as a standby node until it gets promoted to a voting member. Moreover, to handle unexpected upgrade failures, v3.4 introduces [etcd downgrade](https://groups.google.com/forum/?hl=en#!topic/etcd-dev/Hq6zru44L74) feature.
|
||||
|
||||
etcd v3 storage uses multi-version concurrency control model to preserve key updates as event history. Kubernetes runs compaction to discard the event history that is no longer needed, and reclaims the storage space. etcd v3.4 will improve this storage compact operation, boost backend [concurrency for large read transactions](https://github.com/etcd-io/etcd/pull/9384), and [optimize storage commit interval](https://github.com/etcd-io/etcd/pull/10283) for Kubernetes use-case.
|
||||
|
||||
To further improve etcd client load balancer, the v3.4 balancer was rewritten to leverage the newly introduced gRPC load balancing API. By leveraging gPRC, the etcd client load balancer codebase was substantially simplified while retaining feature parity with the v3.3 implementation and improving overall load balancing by round-robining requests across healthy endpoints. See [Client Architecture](https://etcd.readthedocs.io/en/latest/client-architecture.html#client-architecture) for more details.
|
||||
|
||||
Additionally, etcd maintainers will continue to make improvements to Kubernetes test frameworks: kubemark integration for scalability tests, Kubernetes API server conformance tests with etcd to provide release recommends and version skew policy, specifying conformance testing requirements for each cloud provider, etc.
|
||||
|
||||
## etcd Joins CNCF
|
||||
|
||||
etcd now has a new home at [etcd-io](https://github.com/etcd-io) and [joined CNCF as an incubating project](https://www.cncf.io/blog/2018/12/11/cncf-to-host-etcd/).
|
||||
|
||||
The synergistic efforts with Kubernetes have driven the evolution of etcd. Without community feedback and contribution, etcd could not have achieved its maturity and reliability. We’re looking forward to continuing the growth of etcd as an open source project and are excited to work with the Kubernetes and the wider CNCF community.
|
||||
|
||||
Finally, we’d like to thank all contributors with special thanks to [Xiang Li](https://github.com/xiang90) for his leadership in etcd and Kubernetes.
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
layout: blog
|
||||
title: 'APIServer dry-run and kubectl diff'
|
||||
date: 2019-01-14
|
||||
---
|
||||
|
||||
**Author**: Antoine Pelisse (Google Cloud, @apelisse)
|
||||
|
||||
Declarative configuration management, also known as configuration-as-code, is
|
||||
one of the key strengths of Kubernetes. It allows users to commit the desired state of
|
||||
the cluster, and to keep track of the different versions, improve auditing and
|
||||
automation through CI/CD pipelines. The [Apply working-group](https://groups.google.com/forum/#!forum/kubernetes-wg-apply)
|
||||
is working on fixing some of the gaps, and is happy to announce that Kubernetes
|
||||
1.13 promoted server-side dry-run and `kubectl diff` to beta. These
|
||||
two features are big improvements for the Kubernetes declarative model.
|
||||
|
||||
## Challenges
|
||||
|
||||
A few pieces are still missing in order to have a seamless declarative
|
||||
experience with Kubernetes, and we tried to address some of these:
|
||||
|
||||
- While compilers and linters do a good job to detect errors in pull-requests
|
||||
for code, a good validation is missing for Kubernetes configuration files.
|
||||
The existing solution is to run `kubectl apply --dry-run`, but this runs a
|
||||
*local* dry-run that doesn't talk to the server: it doesn't have server
|
||||
validation and doesn't go through validating admission controllers. As an
|
||||
example, Custom resource names are only validated on the server so a local
|
||||
dry-run won't help.
|
||||
- It can be difficult to know how your object is going to be applied by the
|
||||
server for multiple reasons:
|
||||
- Defaulting will set some fields to potentially unexpected values,
|
||||
- Mutating webhooks might set fields or clobber/change some values.
|
||||
- Patch and merges can have surprising effects and result in unexpected
|
||||
objects. For example, it can be hard to know how lists are going to be
|
||||
ordered once merged.
|
||||
|
||||
The working group has tried to address these problems.
|
||||
|
||||
## APIServer dry-run
|
||||
|
||||
[APIServer dry-run](https://kubernetes.io/docs/reference/using-api/api-concepts/#dry-run) was implemented to address these two problems:
|
||||
|
||||
- it allows individual requests to the apiserver to be marked as "dry-run",
|
||||
- the apiserver guarantees that dry-run requests won't be persisted to storage,
|
||||
- the request is still processed as typical request: the fields are
|
||||
defaulted, the object is validated, it goes through the validation admission
|
||||
chain, and through the mutating admission chain, and then the final object is
|
||||
returned to the user as it normally would, without being persisted.
|
||||
|
||||
While dynamic admission controllers are not supposed to have side-effects on
|
||||
each request, dry-run requests are only processed if all admission controllers
|
||||
explicitly announce that they don't have any dry-run side-effects.
|
||||
|
||||
### How to enable it
|
||||
|
||||
Server-side dry-run is enabled through a feature-gate. Now that the feature is
|
||||
Beta in 1.13, it should be enabled by default, but still can be enabled/disabled
|
||||
using `kube-apiserver --feature-gates DryRun=true`.
|
||||
|
||||
If you have dynamic admission controllers, you might have to fix them to:
|
||||
|
||||
- Remove any side-effects when the dry-run parameter is specified on the webhook request,
|
||||
- Specify in the [`sideEffects`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#webhook-v1beta1-admissionregistration)
|
||||
field of the `admissionregistration.k8s.io/v1beta1.Webhook` object to indicate that the object doesn't
|
||||
have side-effects on dry-run (or at all).
|
||||
|
||||
### How to use it
|
||||
|
||||
You can trigger the feature from kubectl by using `kubectl apply
|
||||
--server-dry-run`, which will decorate the request with the dryRun flag
|
||||
and return the object as it would have been applied, or an error if it would
|
||||
have failed.
|
||||
|
||||
## Kubectl diff
|
||||
|
||||
APIServer dry-run is convenient because it lets you see how the object would be
|
||||
processed, but it can be hard to identify exactly what changed if the object is
|
||||
big. `kubectl diff` does exactly what you want by showing the differences between
|
||||
the current "live" object and the new "dry-run" object. It makes it very
|
||||
convenient to focus on only the changes that are made to the object, how the
|
||||
server has merged these and how the mutating webhooks affects the output.
|
||||
|
||||
### How to use it
|
||||
|
||||
`kubectl diff` is meant to be as similar as possible to `kubectl apply`:
|
||||
`kubectl diff -f some-resources.yaml` will show a diff for the resources in the yaml file. One can even use the diff program of their choice by using the KUBECTL_EXTERNAL_DIFF environment variable, for example:
|
||||
```
|
||||
KUBECTL_EXTERNAL_DIFF=meld kubectl diff -f some-resources.yaml
|
||||
```
|
||||
|
||||
## What's next
|
||||
|
||||
The working group is still busy trying to improve some of these things:
|
||||
|
||||
- Server-side apply is trying to improve the apply scenario, by adding owner
|
||||
semantics to fields! It's also going to improve support for CRDs and unions!
|
||||
- Some kubectl apply features are missing from diff and could be useful, like the ability
|
||||
to filter by label, or to display pruned resources.
|
||||
- Eventually, kubectl diff will use server-side apply!
|
||||
@@ -0,0 +1,208 @@
|
||||
---
|
||||
title: Container Storage Interface (CSI) for Kubernetes GA
|
||||
date: 2019-01-15
|
||||
slug: container-storage-interface-ga
|
||||
---
|
||||
|
||||
|
||||
|
||||
|
||||
**Author:** Saad Ali, Senior Software Engineer, Google
|
||||
|
||||
The Kubernetes implementation of the [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI) has been promoted to GA in the Kubernetes v1.13 release. Support for CSI was [introduced as alpha](http://blog.kubernetes.io/2018/01/introducing-container-storage-interface.html) in Kubernetes v1.9 release, and [promoted to beta](https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/) in the Kubernetes v1.10 release.
|
||||
|
||||
The GA milestone indicates that Kubernetes users may depend on the feature and its API without fear of backwards incompatible changes in future causing regressions. GA features are protected by the [Kubernetes deprecation policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/).
|
||||
|
||||
## Why CSI?
|
||||
|
||||
Although prior to CSI Kubernetes provided a powerful volume plugin system, it was challenging to add support for new volume plugins to Kubernetes: volume plugins were “in-tree” meaning their code was part of the core Kubernetes code and shipped with the core Kubernetes binaries—vendors wanting to add support for their storage system to Kubernetes (or even fix a bug in an existing volume plugin) were forced to align with the Kubernetes release process. In addition, third-party storage code caused reliability and security issues in core Kubernetes binaries and the code was often difficult (and in some cases impossible) for Kubernetes maintainers to test and maintain.
|
||||
|
||||
CSI was developed as a standard for exposing arbitrary block and file storage storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. With the adoption of the Container Storage Interface, the Kubernetes volume layer becomes truly extensible. Using CSI, third-party storage providers can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code. This gives Kubernetes users more options for storage and makes the system more secure and reliable.
|
||||
|
||||
## What’s new?
|
||||
|
||||
With the promotion to GA, the Kubernetes implementation of CSI introduces the following changes:
|
||||
|
||||
- Kubernetes is now compatible with CSI spec [v1.0](https://github.com/container-storage-interface/spec/releases/tag/v1.0.0) and [v0.3](https://github.com/container-storage-interface/spec/releases/tag/v0.3.0) (instead of CSI spec [v0.2](https://github.com/container-storage-interface/spec/releases/tag/v0.2.0)).
|
||||
- There were breaking changes between CSI spec v0.3.0 and v1.0.0, but Kubernetes v1.13 supports both versions so either version will work with Kubernetes v1.13.
|
||||
- Please note that with the release of the CSI 1.0 API, support for CSI drivers using 0.3 and older releases of the CSI API is deprecated, and is planned to be removed in Kubernetes v1.15.
|
||||
- There were no breaking changes between CSI spec v0.2 and v0.3, so v0.2 drivers should also work with Kubernetes v1.10.0+.
|
||||
- There were breaking changes between the CSI spec v0.1 and v0.2, so very old drivers implementing CSI 0.1 must be updated to be at least 0.2 compatible before use with Kubernetes v1.10.0+.
|
||||
- The Kubernetes `VolumeAttachment` object (introduced in v1.9 in the storage v1alpha1 group, and added to the v1beta1 group in v1.10) has been added to the storage v1 group in v1.13.
|
||||
- The Kubernetes `CSIPersistentVolumeSource` volume type has been promoted to GA.
|
||||
- The [Kubelet device plugin registration mechanism](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-registration), which is the means by which kubelet discovers new CSI drivers, has been promoted to GA in Kubernetes v1.13.
|
||||
|
||||
## How to deploy a CSI driver?
|
||||
|
||||
Kubernetes users interested in how to deploy or manage an existing CSI driver on Kubernetes should look at the documentation provided by the author of the CSI driver.
|
||||
|
||||
## How to use a CSI volume?
|
||||
|
||||
Assuming a CSI storage plugin is already deployed on a Kubernetes cluster, users can use CSI volumes through the familiar Kubernetes storage API objects: `PersistentVolumeClaims`, `PersistentVolumes`, and `StorageClasses`. Documented [here](https://kubernetes.io/docs/concepts/storage/volumes/#csi).
|
||||
|
||||
Although the Kubernetes implementation of CSI is a GA feature in Kubernetes v1.13, it may require the following flag:
|
||||
|
||||
- API server binary and kubelet binaries:
|
||||
- `--allow-privileged=true`
|
||||
- Most CSI plugins will require bidirectional mount propagation, which can only be enabled for privileged pods. Privileged pods are only permitted on clusters where this flag has been set to true (this is the default in some environments like GCE, GKE, and kubeadm).
|
||||
|
||||
### Dynamic Provisioning
|
||||
|
||||
You can enable automatic creation/deletion of volumes for CSI Storage plugins that support dynamic provisioning by creating a `StorageClass` pointing to the CSI plugin.
|
||||
|
||||
The following StorageClass, for example, enables dynamic creation of “`fast-storage`” volumes by a CSI volume plugin called “`csi-driver.example.com`”.
|
||||
|
||||
```
|
||||
kind: StorageClass
|
||||
apiVersion: storage.k8s.io/v1
|
||||
metadata:
|
||||
name: fast-storage
|
||||
provisioner: csi-driver.example.com
|
||||
parameters:
|
||||
type: pd-ssd
|
||||
csi.storage.k8s.io/provisioner-secret-name: mysecret
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: mynamespace
|
||||
```
|
||||
|
||||
New for GA, the [CSI external-provisioner](https://github.com/kubernetes-csi/external-provisioner) (v1.0.1+) reserves the parameter keys prefixed with `csi.storage.k8s.io/`. If the keys do not correspond to a set of known keys the values are simply ignored (and not passed to the CSI driver). The older secret parameter keys (`csiProvisionerSecretName`, `csiProvisionerSecretNamespace`, etc.) are also supported by CSI external-provisioner v1.0.1 but are deprecated and may be removed in future releases of the CSI external-provisioner.
|
||||
|
||||
Dynamic provisioning is triggered by the creation of a `PersistentVolumeClaim` object. The following `PersistentVolumeClaim`, for example, triggers dynamic provisioning using the `StorageClass` above.
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: my-request-for-storage
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
storageClassName: fast-storage
|
||||
```
|
||||
|
||||
When volume provisioning is invoked, the parameter type: `pd-ssd` and the secret any referenced secret(s) are passed to the CSI plugin `csi-driver.example.com` via a `CreateVolume` call. In response, the external volume plugin provisions a new volume and then automatically create a `PersistentVolume` object to represent the new volume. Kubernetes then binds the new `PersistentVolume` object to the `PersistentVolumeClaim`, making it ready to use.
|
||||
|
||||
If the `fast-storage StorageClass` is marked as “default”, there is no need to include the `storageClassName` in the `PersistentVolumeClaim`, it will be used by default.
|
||||
|
||||
### Pre-Provisioned Volumes
|
||||
|
||||
You can always expose a pre-existing volume in Kubernetes by manually creating a PersistentVolume object to represent the existing volume. The following `PersistentVolume`, for example, exposes a volume with the name “`existingVolumeName`” belonging to a CSI storage plugin called “`csi-driver.example.com`”.
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: my-manually-created-pv
|
||||
spec:
|
||||
capacity:
|
||||
storage: 5Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
csi:
|
||||
driver: csi-driver.example.com
|
||||
volumeHandle: existingVolumeName
|
||||
readOnly: false
|
||||
fsType: ext4
|
||||
volumeAttributes:
|
||||
foo: bar
|
||||
controllerPublishSecretRef:
|
||||
name: mysecret1
|
||||
namespace: mynamespace
|
||||
nodeStageSecretRef:
|
||||
name: mysecret2
|
||||
namespace: mynamespace
|
||||
nodePublishSecretRef
|
||||
name: mysecret3
|
||||
namespace: mynamespace
|
||||
```
|
||||
|
||||
### Attaching and Mounting
|
||||
|
||||
You can reference a `PersistentVolumeClaim` that is bound to a CSI volume in any pod or pod template.
|
||||
|
||||
```
|
||||
kind: Pod
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: my-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: my-frontend
|
||||
image: nginx
|
||||
volumeMounts:
|
||||
- mountPath: "/var/www/html"
|
||||
name: my-csi-volume
|
||||
volumes:
|
||||
- name: my-csi-volume
|
||||
persistentVolumeClaim:
|
||||
claimName: my-request-for-storage
|
||||
```
|
||||
|
||||
When the pod referencing a CSI volume is scheduled, Kubernetes will trigger the appropriate operations against the external CSI plugin (`ControllerPublishVolume`, `NodeStageVolume`, `NodePublishVolume`, etc.) to ensure the specified volume is attached, mounted, and ready to use by the containers in the pod.
|
||||
|
||||
For more details please see the CSI implementation [design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) and [documentation](https://kubernetes.io/docs/concepts/storage/volumes/#csi).
|
||||
|
||||
## How to write a CSI Driver?
|
||||
|
||||
The [kubernetes-csi](https://kubernetes-csi.github.io/) site details how to develop, deploy, and test a CSI driver on Kubernetes. In general, CSI Drivers should be deployed on Kubernetes along with the following sidecar (helper) containers:
|
||||
|
||||
- [external-attacher](https://github.com/kubernetes-csi/external-attacher)
|
||||
- Watches Kubernetes `VolumeAttachment` objects and triggers `ControllerPublish` and `ControllerUnpublish` operations against a CSI endpoint.
|
||||
- [external-provisioner](https://github.com/kubernetes-csi/external-provisioner)
|
||||
- Watches Kubernetes `PersistentVolumeClaim` objects and triggers `CreateVolume` and `DeleteVolume` operations against a CSI endpoint.
|
||||
- [node-driver-registrar](https://github.com/kubernetes-csi/node-driver-registrar)
|
||||
- Registers the CSI driver with kubelet using the [Kubelet device plugin mechanism](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-registration).
|
||||
- [cluster-driver-registrar](https://github.com/kubernetes-csi/cluster-driver-registrar) (Alpha)
|
||||
- Registers a CSI Driver with the Kubernetes cluster by creating a `CSIDriver` object which enables the driver to customize how Kubernetes interacts with it.
|
||||
- [external-snapshotter](https://github.com/kubernetes-csi/external-snapshotter) (Alpha)
|
||||
- Watches Kubernetes `VolumeSnapshot` CRD objects and triggers `CreateSnapshot` and `DeleteSnapshot` operations against a CSI endpoint.
|
||||
- [livenessprobe](https://github.com/kubernetes-csi/livenessprobe)
|
||||
- May be included in a CSI plugin pod to enable the [Kubernetes Liveness Probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) mechanism.
|
||||
|
||||
Storage vendors can build Kubernetes deployments for their plugins using these components, while leaving their CSI driver completely unaware of Kubernetes.
|
||||
|
||||
## List of CSI Drivers
|
||||
|
||||
CSI drivers are developed and maintained by third parties. You can find a non-definitive list of CSI drivers [here](https://kubernetes-csi.github.io/docs/Drivers.html).
|
||||
|
||||
## What about in-tree volume plugins?
|
||||
|
||||
There is a plan to migrate most of the persistent, remote in-tree volume plugins to CSI. For more details see [design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-migration.md).
|
||||
|
||||
## Limitations of GA
|
||||
|
||||
The GA implementation of CSI has the following limitations:
|
||||
|
||||
- Ephemeral local volumes must create a PVC (pod inline referencing of CSI volumes is not supported).
|
||||
|
||||
## What’s next?
|
||||
|
||||
- Work on moving Kubernetes CSI features that are still alpha to beta:
|
||||
- Raw block volumes
|
||||
- Topology awareness (the ability for Kubernetes to understand and influence where a CSI volume is provisioned (zone, regions, etc.).
|
||||
- Features depending on CSI CRDs (e.g. “Skip attach” and “Pod info on mount”).
|
||||
- Volume Snapshots
|
||||
- Work on completing support for local ephemeral volumes.
|
||||
- Work on migrating remote persistent in-tree volume plugins to CSI.
|
||||
|
||||
## How to get involved?
|
||||
The Kubernetes Slack channel [wg-csi](https://kubernetes.slack.com/messages/C8EJ01Z46/details/) and the Google group [kubernetes-sig-storage-wg-csi](https://groups.google.com/forum/#!forum/kubernetes-sig-storage-wg-csi) along with any of the standard [SIG storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are all great mediums to reach out to the SIG Storage team.
|
||||
|
||||
This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together. We offer a huge thank you to the new contributors who stepped up this quarter to help the project reach GA:
|
||||
|
||||
- Saad Ali ([saad-ali](https://github.com/saad-ali))
|
||||
- Michelle Au ([msau42](https://github.com/msau42))
|
||||
- Serguei Bezverkhi ([sbezverk](https://github.com/sbezverk))
|
||||
- Masaki Kimura ([mkimuram](https://github.com/mkimuram))
|
||||
- Patrick Ohly ([pohly](https://github.com/pohly))
|
||||
- Luis Pabón ([lpabon](https://github.com/lpabon))
|
||||
- Jan Šafránek ([jsafrane](https://github.com/jsafrane))
|
||||
- Vladimir Vivien ([vladimirvivien](https://github.com/vladimirvivien))
|
||||
- Cheng Xing ([verult](https://github.com/verult))
|
||||
- Xing Yang ([xing-yang](https://github.com/xing-yang))
|
||||
- David Zhu ([davidz627](https://github.com/davidz627))
|
||||
|
||||
If you’re interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG). We’re rapidly growing and always welcome new contributors.
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: Update on Volume Snapshot Alpha for Kubernetes
|
||||
date: 2019-01-17
|
||||
---
|
||||
|
||||
**Authors:** Jing Xu (Google), Xing Yang (Huawei), Saad Ali (Google)
|
||||
|
||||
Volume snapshotting support was introduced in Kubernetes v1.12 as an alpha feature. In Kubernetes v1.13, it remains an alpha feature, but a few enhancements were added and some breaking changes were made. This post summarizes the changes.
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
[CSI spec v1.0](https://github.com/container-storage-interface/spec/releases/tag/v1.0.0) introduced a few breaking changes to the volume snapshot feature. CSI driver maintainers should be aware of these changes as they upgrade their drivers to support v1.0.
|
||||
|
||||
## SnapshotStatus replaced with Boolean ReadyToUse
|
||||
|
||||
CSI v0.3.0, defined a `SnapshotStatus` enum in `CreateSnapshotResponse` which indicates whether the snapshot is `READY`, `UPLOADING`, or `ERROR_UPLOADING`. In CSI v1.0, `SnapshotStatus` has been removed from `CreateSnapshotResponse` and replaced with a `boolean ReadyToUse`. A `ReadyToUse` value of `true` indicates that post snapshot processing (such as uploading) is complete and the snapshot is ready to be used as a source to create a volume.
|
||||
|
||||
Storage systems that need to do post snapshot processing (such as uploading after the snapshot is cut) should return a successful `CreateSnapshotResponse` with the `ReadyToUse` field set to `false` as soon as the snapshot has been taken. This indicates that the Container Orchestration System (CO) can resume any workload that was quiesced for the snapshot to be taken. The CO can then repeatedly call `CreateSnapshot` until the `ReadyToUse` field is set to `true` or the call returns an error indicating a problem in processing. The CSI `ListSnapshot` call could be used along with `snapshot_id` filtering to determine if the snapshot is ready to use, but is not recommended because it provides no way to detect errors during processing (the `ReadyToUse` field simply remains `false` indefinitely).
|
||||
|
||||
The [v1.x.x releases](https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v1.0.1) of the CSI external-snapshotter sidecar container already handle this change by calling `CreateSnapshot` instead of `ListSnapshots` to check if a snapshot is ready to use. When upgrading their drivers to CSI 1.0, driver maintainers should use the appropriate 1.0 compatible sidecar container.
|
||||
|
||||
To be consistent with the change in the CSI spec, the `Ready` field in the `VolumeSnapshot` API object has been renamed to `ReadyToUse`. This change is visible to the user when running `kubectl describe volumesnapshot` to view the details of a snapshot.
|
||||
|
||||
## Timestamp Data Type
|
||||
|
||||
The creation time of a snapshot is available to Kubernetes admins as part of the `VolumeSnapshotContent` API object. This field is populated using the `creation_time` field in the CSI `CreateSnapshotResponse`. In CSI v1.0, this `creation_time` field type was changed to [`.google.protobuf.Timestamp`](https://godoc.org/github.com/golang/protobuf/ptypes/timestamp) instead of `int64`. When upgrading drivers to CSI 1.0, driver maintainers must make changes accordingly. The [v1.x.x releases](https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v1.0.1) of the CSI external-snapshotter sidecar container has been updated to handle this change.
|
||||
|
||||
## Deprecations
|
||||
|
||||
The following `VolumeSnapshotClass` parameters are deprecated and will be removed in a future release. They will be replaced with parameters listed in the `Replacement` section below.
|
||||
|
||||
Deprecated
|
||||
Replacement
|
||||
csiSnapshotterSecretName
|
||||
csi.storage.k8s.io/snapshotter-secret-name
|
||||
csiSnapshotterSecretNameSpace
|
||||
csi.storage.k8s.io/snapshotter-secret-namespace
|
||||
|
||||
## New Features
|
||||
|
||||
### SnapshotContent Deletion/Retain Policy
|
||||
|
||||
As described in the [initial blog post announcing the snapshot alpha](https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/), the Kubernetes snapshot APIs are similar to the PV/PVC APIs: just like a volume is represented by a bound PVC and PV pair, a snapshot is represented by a bound `VolumeSnapshot` and `VolumeSnapshotContent` pair.
|
||||
|
||||
With PV/PVC pairs, when a user is done with a volume, they can delete the PVC. And the reclaim policy on the PV determines what happens to the PV (whether it is also deleted or retained).
|
||||
|
||||
In the initial alpha release, snapshots did not support the ability to specify a reclaim policy. Instead when a snapshot object was deleted it always resulted in the snapshot being deleted. In Kubernetes v1.13, a snapshot content `DeletionPolicy` was added. It enables an admin to configure what what happens to a `VolumeSnapshotContent` after the `VolumeSnapshot` object it is bound to is deleted. The `DeletionPolicy` of a volume snapshot can either be `Retain` or `Delete`. If the value is not specified, the default depends on whether the `SnapshotContent` object was created via static binding or dynamic provisioning.
|
||||
|
||||
### Retain
|
||||
|
||||
The `Retain` policy allows for manual reclamation of the resource. If a `VolumeSnapshotContent` is statically created and bound, the default `DeletionPolicy` is `Retain`. When the `VolumeSnapshot` is deleted, the `VolumeSnapshotContent` continues to exist and the `VolumeSnapshotContent` is considered “released”. But it is not available for binding to other `VolumeSnapshot` objects because it contains data. It is up to an administrator to decide how to handle the remaining API object and resource cleanup.
|
||||
|
||||
### Delete
|
||||
|
||||
A `Delete` policy enables automatic deletion of the bound `VolumeSnapshotContent` object from Kubernetes and the associated storage asset in the external infrastructure (such as an AWS EBS snapshot or GCE PD snapshot, etc.). Snapshots that are dynamically provisioned inherit the deletion policy of their [`VolumeSnapshotClass`](https://kubernetes.io/docs/concepts/storage/volume-snapshot-classes/), which defaults to `Delete`. The administrator should configure the `VolumeSnapshotClass` with the desired retention policy. The policy may be changed for individual `VolumeSnapshotContent` after it is created by patching the object.
|
||||
|
||||
The following example demonstrates how to check the deletion policy of a dynamically provisioned `VolumeSnapshotContent`.
|
||||
|
||||
```
|
||||
$ kubectl create -f ./examples/kubernetes/demo-defaultsnapshotclass.yaml
|
||||
$ kubectl create -f ./examples/kubernetes/demo-snapshot.yaml
|
||||
$ kubectl get volumesnapshots demo-snapshot-podpvc -o yaml
|
||||
apiVersion: snapshot.storage.k8s.io/v1alpha1
|
||||
kind: VolumeSnapshot
|
||||
metadata:
|
||||
creationTimestamp: "2018-11-27T23:57:09Z"
|
||||
...
|
||||
spec:
|
||||
snapshotClassName: default-snapshot-class
|
||||
snapshotContentName: snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002
|
||||
source:
|
||||
apiGroup: null
|
||||
kind: PersistentVolumeClaim
|
||||
name: podpvc
|
||||
status:
|
||||
…
|
||||
$ kubectl get volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -o yaml
|
||||
apiVersion: snapshot.storage.k8s.io/v1alpha1
|
||||
kind: VolumeSnapshotContent
|
||||
…
|
||||
spec:
|
||||
csiVolumeSnapshotSource:
|
||||
creationTime: 1546469777852000000
|
||||
driver: pd.csi.storage.gke.io
|
||||
restoreSize: 6442450944
|
||||
snapshotHandle: projects/jing-k8s-dev/global/snapshots/snapshot-26cd0db3-f2a0-11e8-8be6-42010a800002
|
||||
deletionPolicy: Delete
|
||||
persistentVolumeRef:
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
name: pvc-853622a4-f28b-11e8-8be6-42010a800002
|
||||
resourceVersion: "21117"
|
||||
uid: ae400e9f-f28b-11e8-8be6-42010a800002
|
||||
snapshotClassName: default-snapshot-class
|
||||
volumeSnapshotRef:
|
||||
apiVersion: snapshot.storage.k8s.io/v1alpha1
|
||||
kind: VolumeSnapshot
|
||||
name: demo-snapshot-podpvc
|
||||
namespace: default
|
||||
resourceVersion: "6948065"
|
||||
uid: 26cd0db3-f2a0-11e8-8be6-42010a800002
|
||||
```
|
||||
|
||||
User can change the deletion policy by using patch:
|
||||
|
||||
```
|
||||
$ kubectl patch volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -p '{"spec":{"deletionPolicy":"Retain"}}' --type=merge
|
||||
|
||||
$ kubectl get volumesnapshotcontent snapcontent-26cd0db3-f2a0-11e8-8be6-42010a800002 -o yaml
|
||||
apiVersion: snapshot.storage.k8s.io/v1alpha1
|
||||
kind: VolumeSnapshotContent
|
||||
...
|
||||
spec:
|
||||
csiVolumeSnapshotSource:
|
||||
...
|
||||
deletionPolicy: Retain
|
||||
persistentVolumeRef:
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
name: pvc-853622a4-f28b-11e8-8be6-42010a800002
|
||||
...
|
||||
```
|
||||
|
||||
## Snapshot Object in Use Protection
|
||||
|
||||
The purpose of the Snapshot Object in Use Protection feature is to ensure that in-use snapshot API objects are not removed from the system (as this may result in data loss). There are two cases that require “in-use” protection:
|
||||
|
||||
1. If a volume snapshot is in active use by a persistent volume claim as a source to create a volume.
|
||||
2. If a `VolumeSnapshotContent` API object is bound to a VolumeSnapshot API object, the content object is considered in use.
|
||||
|
||||
If a user deletes a `VolumeSnapshot` API object in active use by a PVC, the `VolumeSnapshot` object is not removed immediately. Instead, removal of the `VolumeSnapshot` object is postponed until the `VolumeSnapshot` is no longer actively used by any PVCs. Similarly, if an admin deletes a `VolumeSnapshotContent` that is bound to a `VolumeSnapshot`, the `VolumeSnapshotContent` is not removed immediately. Instead, the `VolumeSnapshotContent` removal is postponed until the `VolumeSnapshotContent` is not bound to the `VolumeSnapshot` object.
|
||||
|
||||
## Which volume plugins support Kubernetes Snapshots?
|
||||
|
||||
Snapshots are only supported for CSI drivers (not for in-tree or Flexvolume). To use the Kubernetes snapshots feature, ensure that a CSI Driver that implements snapshots is deployed on your cluster.
|
||||
|
||||
As of the publishing of this blog post, the following CSI drivers support snapshots:
|
||||
|
||||
- [GCE Persistent Disk CSI Driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver)
|
||||
- [OpenSDS CSI Driver](https://github.com/opensds/nbp/tree/master/csi/server)
|
||||
- [Ceph RBD CSI Driver](https://github.com/ceph/ceph-csi/tree/master/pkg/rbd)
|
||||
- [Portworx CSI Driver](https://github.com/libopenstorage/openstorage/tree/master/csi)
|
||||
- [GlusterFS CSI Driver](https://github.com/gluster/gluster-csi-driver)
|
||||
- [Digital Ocean CSI Driver](https://github.com/digitalocean/csi-digitalocean)
|
||||
- [Ember CSI Driver](https://github.com/embercsi/ember-csi)
|
||||
- [Cinder CSI Driver](https://github.com/kubernetes/cloud-provider-openstack/tree/master/pkg/csi/cinder)
|
||||
- [Datera CSI Driver](https://github.com/Datera/datera-csi)
|
||||
- [NexentaStor CSI Driver](https://github.com/Nexenta/nexentastor-csi-driver)
|
||||
|
||||
Snapshot support for other [drivers](https://kubernetes-csi.github.io/docs/Drivers.html) is pending, and should be available soon. Read the “Container Storage Interface (CSI) for Kubernetes GA” blog post to learn more about CSI and how to deploy CSI drivers.
|
||||
|
||||
## What’s next?
|
||||
|
||||
Depending on feedback and adoption, the Kubernetes team plans to push the CSI Snapshot implementation to beta in either 1.15 or 1.16. Some of the features we are interested in supporting include consistency groups, application consistent snapshots, workload quiescing, in-place restores, and more.
|
||||
|
||||
## How can I learn more?
|
||||
|
||||
The code repository for snapshot APIs and controller is here: https://github.com/kubernetes-csi/external-snapshotter
|
||||
|
||||
Check out additional documentation on the snapshot feature here: http://k8s.io/docs/concepts/storage/volume-snapshots and https://kubernetes-csi.github.io/docs/
|
||||
|
||||
## How do I get involved?
|
||||
|
||||
This project, like all of Kubernetes, is the result of hard work by many contributors from diverse backgrounds working together.
|
||||
|
||||
Special thanks to all the contributors that helped add CSI v1.0 support and improve the snapshot feature in this release, including Saad Ali ([saadali](https://github.com/saadali)), Michelle Au ([msau42](https://github.com/msau42)), Deep Debroy ([ddebroy](https://github.com/ddebroy)), James DeFelice ([jdef](https://github.com/jdef)), John Griffith ([j-griffith](https://github.com/j-griffith)), Julian Hjortshoj ([julian-hj](https://github.com/julian-hj)), Tim Hockin ([thockin](https://github.com/thockin)), Patrick Ohly ([pohly](https://github.com/pohly)), Luis Pabon ([lpabon](https://github.com/lpabon)), Cheng Xing ([verult](https://github.com/verult)), Jing Xu ([jingxu97](https://github.com/jingxu97)), Shiwei Xu ([wackxu](https://github.com/wackxu)), Xing Yang ([xing-yang](https://github.com/xing-yang)), Jie Yu ([jieyu](https://github.com/jieyu)), David Zhu ([davidz627](https://github.com/davidz627)).
|
||||
|
||||
Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) (SIG). We’re rapidly growing and always welcome new contributors.
|
||||
|
||||
We also hold regular [SIG-Storage Snapshot Working Group meetings](https://docs.google.com/document/d/1qdfvAj5O-tTAZzqJyz3B-yczLLxOiQd-XKpJmTEMazs/edit?usp=sharing). New attendees are welcome to join for design and development discussions.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
title: Poseidon-Firmament Scheduler – Flow Network Graph Based Scheduler
|
||||
date: 2019-02-06
|
||||
---
|
||||
|
||||
**Authors:** Deepak Vij (Huawei), Shivram Shrivastava (Huawei)
|
||||
|
||||
## Introduction
|
||||
|
||||
Cluster Management systems such as Mesos, Google Borg, Kubernetes etc. in a cloud scale datacenter environment (also termed as ***Datacenter-as-a-Computer*** or ***Warehouse-Scale Computing - WSC***) typically manage application workloads by performing tasks such as tracking machine live-ness, starting, monitoring, terminating workloads and more importantly using a **Cluster Scheduler** to decide on workload placements.
|
||||
|
||||
A **Cluster Scheduler** essentially performs the scheduling of workloads to compute resources – combining the global placement of work across the WSC environment makes the “warehouse-scale computer” more efficient, increases utilization, and saves energy. **Cluster Scheduler** examples are Google Borg, Kubernetes, Firmament, Mesos, Tarcil, Quasar, Quincy, Swarm, YARN, Nomad, Sparrow, Apollo etc.
|
||||
|
||||
In this blog post, we briefly describe the novel Firmament flow network graph based scheduling approach ([OSDI paper](https://www.usenix.org/conference/osdi16/technical-sessions/presentation/gog)) in Kubernetes. We specifically describe the Firmament Scheduler and how it integrates with the Kubernetes cluster manager using Poseidon as the integration glue. We have seen extremely impressive scheduling throughput performance benchmarking numbers with this novel scheduling approach. Originally, Firmament Scheduler was conceptualized, designed and implemented by University of Cambridge researchers, [Malte Schwarzkopf](http://www.malteschwarzkopf.de/) & [Ionel Gog](http://ionelgog.org/).
|
||||
|
||||
## Poseidon-Firmament Scheduler – How It Works
|
||||
|
||||
At a very high level, [Poseidon-Firmament scheduler](https://kubernetes.io/docs/concepts/extend-kubernetes/poseidon-firmament-alternate-scheduler/) augments the current Kubernetes scheduling capabilities by incorporating novel flow network graph based scheduling capabilities alongside the default Kubernetes Scheduler. It models the scheduling problem as a constraint-based optimization over a flow network graph – by reducing scheduling to a min-cost max-flow optimization problem. Due to the inherent rescheduling capabilities, the new scheduler enables a globally optimal scheduling environment that constantly keeps refining the workloads placements dynamically.
|
||||
|
||||
## Key Advantages
|
||||
|
||||
Flow graph scheduling based [Poseidon-Firmament scheduler](https://kubernetes.io/docs/concepts/extend-kubernetes/poseidon-firmament-alternate-scheduler/) provides the following key advantages:
|
||||
|
||||
* Workloads (pods) are bulk scheduled to enable scheduling decisions at massive scale.
|
||||
|
||||
* Based on the extensive performance test results, Poseidon-Firmament scales much better than Kubernetes default scheduler as the number of nodes increase in a cluster. This is due to the fact that Poseidon-Firmament is able to amortize more and more work across workloads.
|
||||
|
||||
* Poseidon-Firmament Scheduler outperforms the Kubernetes default scheduler by a wide margin when it comes to throughput performance numbers for scenarios where compute resource requirements are somewhat uniform across jobs (Replicasets/Deployments/Jobs). Poseidon-Firmament scheduler end-to-end throughput performance numbers, including bind time, consistently get better as the number of nodes in a cluster increase. For example, for a 2,700 node cluster (shown in the graphs [here](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/benchmark/README.md)), Poseidon-Firmament scheduler achieves a 7X or greater end-to-end throughput than the Kubernetes default scheduler, which includes bind time.
|
||||
|
||||
* Availability of complex rule constraints.
|
||||
|
||||
* Scheduling in Poseidon-Firmament is very dynamic; it keeps cluster resources in a global optimal state during every scheduling run.
|
||||
|
||||
* Highly efficient resource utilizations.
|
||||
|
||||
## Firmament Flow Network Graph – An Overview
|
||||
|
||||
Firmament scheduler runs a min-cost flow algorithm over the flow network to find an optimal flow, from which it extracts the implied workload (pod placements). A flow network is a directed graph whose arcs carry flow from source nodes (i.e. pod nodes) to a sink node. A cost and capacity associated with each arc constrain the flow, and specify preferential routes for it.
|
||||
|
||||
Figure 1 below shows an example of a flow network for a cluster with two tasks (workloads or pods) and four machines (nodes) – each workload on the left hand side, is a source of one unit of flow. All such flow must be drained into the sink node (S) for a feasible solution to the optimization problem.
|
||||
|
||||
{{<figure width="600" src="/images/blog/2019-02-03-poseidon-firmament-scheduler/example-of-a-flow-network.png" caption="Figure 1. Example of a Flow Network">}}
|
||||
|
||||
|
||||
|
||||
## Poseidon Mediation Layer – An Overview
|
||||
|
||||
Poseidon is a service that acts as the integration glue for the Firmament scheduler with Kubernetes. It augments the current Kubernetes scheduling capabilities by incorporating new flow network graph based Firmament scheduling capabilities alongside the default Kubernetes Scheduler; multiple schedulers running simultaneously. Figure 2 below describes the high level overall design as far as how Poseidon integration glue works in conjunction with the underlying Firmament flow network graph based scheduler.
|
||||
|
||||
{{<figure width="600" src="/images/blog/2019-02-03-poseidon-firmament-scheduler/firmament-kubernetes-integration-overview.png" caption="Figure 2. Firmament Kubernetes Integration Overview">}}
|
||||
|
||||
As part of the Kubernetes multiple schedulers support, each new pod is typically scheduled by the default scheduler, but Kubernetes can be instructed to use another scheduler by specifying the name of another custom scheduler (in our case, [Poseidon-Firmament](https://kubernetes.io/docs/concepts/extend-kubernetes/poseidon-firmament-alternate-scheduler/)) at the time of pod deployment. In this case, the default scheduler will ignore that Pod and allow Poseidon scheduler to schedule the Pod to a relevant node.
|
||||
|
||||
{{< note >}}
|
||||
For details about the design of this project see the [design document](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/design/README.md).
|
||||
{{< /note >}}
|
||||
|
||||
## Possible Use Case Scenarios – When To Use It
|
||||
|
||||
[Poseidon-Firmament scheduler](https://kubernetes.io/docs/concepts/extend-kubernetes/poseidon-firmament-alternate-scheduler/) enables extremely high throughput scheduling environment at scale due to its bulk scheduling approach superiority versus K8s pod-at-a-time approach. In our extensive tests, we have observed substantial throughput benefits as long as resource requirements (CPU/Memory) for incoming Pods is uniform across jobs (Replicasets/Deployments/Jobs), mainly due to efficient amortization of work across jobs.
|
||||
|
||||
Although, [Poseidon-Firmament scheduler](https://kubernetes.io/docs/concepts/extend-kubernetes/poseidon-firmament-alternate-scheduler/) is capable of scheduling various types of workloads (service, batch, etc.), following are the few use cases where it excels the most:
|
||||
|
||||
1. For “Big Data/AI” jobs consisting of a large number of tasks, throughput benefits are tremendous.
|
||||
|
||||
2. Substantial throughput benefits also for service or batch job scenarios where workload resource requirements are uniform across jobs (Replicasets/Deplyments/Jobs).
|
||||
|
||||
## Current Project Stage
|
||||
|
||||
Currently Poseidon-Firmament project is an incubation project. Alpha Release is available at https://github.com/kubernetes-sigs/poseidon.
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: Runc and CVE-2019-5736
|
||||
date: 2019-02-11
|
||||
---
|
||||
|
||||
This morning [a container escape vulnerability in runc was announced](https://www.openwall.com/lists/oss-security/2019/02/11/2). We wanted to provide some guidance to Kubernetes users to ensure everyone is safe and secure.
|
||||
|
||||
## What Is Runc?
|
||||
|
||||
Very briefly, runc is the low-level tool which does the heavy lifting of spawning a Linux container. Other tools like Docker, Containerd, and CRI-O sit on top of runc to deal with things like data formatting and serialization, but runc is at the heart of all of these systems.
|
||||
|
||||
Kubernetes in turn sits on top of those tools, and so while no part of Kubernetes itself is vulnerable, most Kubernetes installations are using runc under the hood.
|
||||
|
||||
### What Is The Vulnerability?
|
||||
|
||||
While full details are still embargoed to give people time to patch, the rough version is that when running a process as root (UID 0) inside a container, that process can exploit a bug in runc to gain root privileges on the host running the container. This then allows them unlimited access to the server as well as any other containers on that server.
|
||||
|
||||
If the process inside the container is either trusted (something you know is not hostile) or is not running as UID 0, then the vulnerability does not apply. It can also be prevented by SELinux, if an appropriate policy has been applied. RedHat Enterprise Linux and CentOS both include appropriate SELinux permissions with their packages and so are believed to be unaffected if SELinux is enabled.
|
||||
|
||||
The most common source of risk is attacker-controller container images, such as unvetted images from public repositories.
|
||||
|
||||
### What Should I Do?
|
||||
|
||||
As with all security issues, the two main options are to mitigate the vulnerability or upgrade your version of runc to one that includes the fix.
|
||||
|
||||
As the exploit requires UID 0 within the container, a direct mitigation is to ensure all your containers are running as a non-0 user. This can be set within the container image, or via your pod specification:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: run-as-uid-1000
|
||||
spec:
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
# ...
|
||||
```
|
||||
|
||||
This can also be enforced globally using a PodSecurityPolicy:
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: non-root
|
||||
spec:
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
runAsUser:
|
||||
# Require the container to run without root privileges.
|
||||
rule: 'MustRunAsNonRoot'
|
||||
```
|
||||
|
||||
Setting a policy like this is highly encouraged given the overall risks of running as UID 0 inside a container.
|
||||
|
||||
Another potential mitigation is to ensure all your container images are vetted and trusted. This can be accomplished by building all your images yourself, or by vetting the contents of an image and then pinning to the image version hash (`image: external/someimage@sha256:7832659873hacdef`).
|
||||
|
||||
Upgrading runc can generally be accomplished by upgrading the package `runc` for your distribution or by upgrading your OS image if using immutable images. This is a list of known safe versions for various distributions and platforms:
|
||||
|
||||
* Ubuntu - [`runc 1.0.0~rc4+dfsg1-6ubuntu0.18.10.1`](https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-5736.html)
|
||||
* Debian - [`runc 1.0.0~rc6+dfsg1-2`](https://security-tracker.debian.org/tracker/CVE-2019-5736)
|
||||
* RedHat Enterprise Linux - [`docker 1.13.1-91.git07f3374.el7`](https://access.redhat.com/security/vulnerabilities/runcescape) (if SELinux is disabled)
|
||||
* Amazon Linux - [`docker 18.06.1ce-7.25.amzn1.x86_64`](https://alas.aws.amazon.com/ALAS-2019-1156.html)
|
||||
* CoreOS - Stable: [`1967.5.0`](https://coreos.com/releases/#1967.5.0) / Beta: [`2023.2.0`](https://coreos.com/releases/#2023.2.0) / Alpha: [`2051.0.0`](https://coreos.com/releases/#2051.0.0)
|
||||
* Kops Debian - [in progress](https://github.com/kubernetes/kops/pull/6460) (see [advisory](https://github.com/kubernetes/kops/blob/master/docs/advisories/cve_2019_5736.md) for how to address until Kops Debian is patched)
|
||||
* Docker - [`18.09.2`](https://github.com/docker/docker-ce/releases/tag/v18.09.2)
|
||||
|
||||
Some platforms have also posted more specific instructions:
|
||||
|
||||
#### Google Container Engine (GKE)
|
||||
|
||||
Google has issued a [security bulletin](https://cloud.google.com/kubernetes-engine/docs/security-bulletins#february-11-2019-runc) with more detailed information but in short, if you are using the default GKE node image then you are safe. If you are using an Ubuntu node image then you will need to mitigate or upgrade to an image with a fixed version of runc.
|
||||
|
||||
#### Amazon Elastic Container Service for Kubernetes (EKS)
|
||||
|
||||
Amazon has also issued a [security bulletin](https://aws.amazon.com/security/security-bulletins/AWS-2019-002/) with more detailed information. All EKS users should mitigate the issue or upgrade to a new node image.
|
||||
|
||||
#### Azure Kubernetes Service (AKS)
|
||||
|
||||
Microsoft has issued a [security bulletin](https://azure.microsoft.com/en-us/updates/cve-2019-5736-and-runc-vulnerability/) with detailed information on mitigating the issue. Microsoft recommends all AKS users to upgrade their cluster to mitigate the issue.
|
||||
|
||||
#### Kops
|
||||
|
||||
Kops has issued an [advisory](https://github.com/kubernetes/kops/blob/master/docs/advisories/cve_2019_5736.md) with detailed information on mitigating this issue.
|
||||
|
||||
### Docker
|
||||
|
||||
We don't have specific confirmation that Docker for Mac and Docker for Windows are vulnerable, however it seems likely. Docker has released a fix in [version 18.09.2](https://github.com/docker/docker-ce/releases/tag/v18.09.2) and it is recommended you upgrade to it. This also applies to other deploy systems using Docker under the hood.
|
||||
|
||||
If you are unable to upgrade Docker, the Rancher team has provided backports of the fix for many older versions at [github.com/rancher/runc-cve](https://github.com/rancher/runc-cve).
|
||||
|
||||
## Getting More Information
|
||||
|
||||
If you have any further questions about how this vulnerability impacts Kubernetes, please join us at [discuss.kubernetes.io](https://discuss.kubernetes.io/).
|
||||
|
||||
If you would like to get in contact with the [runc team](https://github.com/opencontainers/org/blob/master/README.md#communications), you can reach them on [Google Groups](https://groups.google.com/a/opencontainers.org/forum/#!forum/dev) or `#opencontainers` on Freenode IRC.
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
---
|
||||
title: Building a Kubernetes Edge (Ingress) Control Plane for Envoy v2
|
||||
date: 2019-02-12
|
||||
slug: building-a-kubernetes-edge-control-plane-for-envoy-v2
|
||||
---
|
||||
|
||||
|
||||
**Author:**
|
||||
Daniel Bryant, Product Architect, Datawire;
|
||||
Flynn, Ambassador Lead Developer, Datawire;
|
||||
Richard Li, CEO and Co-founder, Datawire
|
||||
|
||||
|
||||
Kubernetes has become the de facto runtime for container-based microservice applications, but this orchestration framework alone does not provide all of the infrastructure necessary for running a distributed system. Microservices typically communicate through Layer 7 protocols such as HTTP, gRPC, or WebSockets, and therefore having the ability to make routing decisions, manipulate protocol metadata, and observe at this layer is vital. However, traditional load balancers and edge proxies have predominantly focused on L3/4 traffic. This is where the [Envoy Proxy](https://www.envoyproxy.io/) comes into play.
|
||||
|
||||
Envoy proxy was designed as a [universal data plane](https://blog.envoyproxy.io/the-universal-data-plane-api-d15cec7a) from the ground-up by the Lyft Engineering team for today's distributed, L7-centric world, with broad support for L7 protocols, a real-time API for managing its configuration, first-class observability, and high performance within a small memory footprint. However, Envoy's vast feature set and flexibility of operation also makes its configuration highly complicated -- this is evident from looking at its rich but verbose [control plane](https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc) syntax.
|
||||
|
||||
With the open source [Ambassador API Gateway](https://www.getambassador.io), we wanted to tackle the challenge of creating a new control plane that focuses on the use case of deploying Envoy as an forward-facing edge proxy within a Kubernetes cluster, in a way that is idiomatic to Kubernetes operators. In this article, we'll walk through two major iterations of the Ambassador design, and how we integrated Ambassador with Kubernetes.
|
||||
|
||||
|
||||
## Ambassador pre-2019: Envoy v1 APIs, Jinja Template Files, and Hot Restarts
|
||||
|
||||
Ambassador itself is deployed within a container as a Kubernetes service, and uses annotations added to Kubernetes Services as its [core configuration model](https://www.getambassador.io/reference/configuration). This approach [enables application developers to manage routing](https://www.getambassador.io/concepts/developers) as part of the Kubernetes service definition. We explicitly decided to go down this route because of [limitations](https://blog.getambassador.io/kubernetes-ingress-nodeport-load-balancers-and-ingress-controllers-6e29f1c44f2d) in the current [Ingress API spec](https://kubernetes.io/docs/concepts/services-networking/ingress/), and we liked the simplicity of extending Kubernetes services, rather than introducing another custom resource type. An example of an Ambassador annotation can be seen here:
|
||||
|
||||
|
||||
```
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: my-service
|
||||
annotations:
|
||||
getambassador.io/config: |
|
||||
---
|
||||
apiVersion: ambassador/v0
|
||||
kind: Mapping
|
||||
name: my_service_mapping
|
||||
prefix: /my-service/
|
||||
service: my-service
|
||||
spec:
|
||||
selector:
|
||||
app: MyApp
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 80
|
||||
targetPort: 9376
|
||||
```
|
||||
|
||||
|
||||
Translating this simple Ambassador annotation config into valid [Envoy v1](https://www.envoyproxy.io/docs/envoy/v1.6.0/configuration/overview/v1_overview) config was not a trivial task. By design, Ambassador's configuration isn't based on the same conceptual model as Envoy's configuration -- we deliberately wanted to aggregate and simplify operations and config. Therefore, translating between one set of concepts to the other involves a fair amount of logic within Ambassador.
|
||||
|
||||
In this first iteration of Ambassador we created a Python-based service that watched the Kubernetes API for changes to Service objects. When new or updated Ambassador annotations were detected, these were translated from the Ambassador syntax into an intermediate representation (IR) which embodied our core configuration model and concepts. Next, Ambassador translated this IR into a representative Envoy configuration which was saved as a file within pods associated with the running Ambassador k8s Service. Ambassador then "hot-restarted" the Envoy process running within the Ambassador pods, which triggered the loading of the new configuration.
|
||||
|
||||
There were many benefits with this initial implementation. The mechanics involved were fundamentally simple, the transformation of Ambassador config into Envoy config was reliable, and the file-based hot restart integration with Envoy was dependable.
|
||||
|
||||
However, there were also notable challenges with this version of Ambassador. First, although the hot restart was effective for the majority of our customers' use cases, it was not very fast, and some customers (particularly those with huge application deployments) found it was limiting the frequency with which they could change their configuration. Hot restart can also drop connections, especially long-lived connections like WebSockets or gRPC streams.
|
||||
|
||||
More crucially, though, the first implementation of the IR allowed rapid prototyping but was primitive enough that it proved very difficult to make substantial changes. While this was a pain point from the beginning, it became a critical issue as Envoy shifted to the [Envoy v2 API](https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview). It was clear that the v2 API would offer Ambassador many benefits -- as Matt Klein outlined in his blog post, "[The universal data plane API](https://blog.envoyproxy.io/the-universal-data-plane-api-d15cec7a)" -- including access to new features and a solution to the connection-drop problem noted above, but it was also clear that the existing IR implementation was not capable of making the leap.
|
||||
|
||||
|
||||
## Ambassador >= v0.50: Envoy v2 APIs (ADS), Testing with KAT, and Golang
|
||||
|
||||
In consultation with the [Ambassador community](http://d6e.co/slack), the [Datawire](www.datawire.io) team undertook a redesign of the internals of Ambassador in 2018. This was driven by two key goals. First, we wanted to integrate Envoy's v2 configuration format, which would enable the support of features such as [SNI](https://www.getambassador.io/user-guide/sni/), [rate limiting](https://www.getambassador.io/user-guide/rate-limiting) and [gRPC authentication APIs](https://www.getambassador.io/user-guide/auth-tutorial). Second, we also wanted to do much more robust semantic validation of Envoy configuration due to its increasing complexity (particularly when operating with large-scale application deployments).
|
||||
|
||||
|
||||
### Initial stages
|
||||
|
||||
We started by restructuring the Ambassador internals more along the lines of a multipass compiler. The class hierarchy was made to more closely mirror the separation of concerns between the Ambassador configuration resources, the IR, and the Envoy configuration resources. Core parts of Ambassador were also redesigned to facilitate contributions from the community outside Datawire. We decided to take this approach for several reasons. First, Envoy Proxy is a very fast moving project, and we realized that we needed an approach where a seemingly minor Envoy configuration change didn't result in days of reengineering within Ambassador. In addition, we wanted to be able to provide semantic verification of configuration.
|
||||
|
||||
As we started working more closely with Envoy v2, a testing challenge was quickly identified. As more and more features were being supported in Ambassador, more and more bugs appeared in Ambassador's handling of less common but completely valid combinations of features. This drove to creation of a new testing requirement that meant Ambassador's test suite needed to be reworked to automatically manage many combinations of features, rather than relying on humans to write each test individually. Moreover, we wanted the test suite to be fast in order to maximize engineering productivity.
|
||||
|
||||
Thus, as part of the Ambassador rearchitecture, we introduced the [Kubernetes Acceptance Test (KAT)](https://github.com/datawire/ambassador/tree/master/kat) framework. KAT is an extensible test framework that:
|
||||
|
||||
|
||||
|
||||
1. Deploys a bunch of services (along with Ambassador) to a Kubernetes cluster
|
||||
1. Run a series of verification queries against the spun up APIs
|
||||
1. Perform a bunch of assertions on those query results
|
||||
|
||||
KAT is designed for performance -- it batches test setup upfront, and then runs all the queries in step 3 asynchronously with a high performance client. The traffic driver in KAT runs locally using [Telepresence](https://www.telepresence.io), which makes it easier to debug issues.
|
||||
|
||||
### Introducing Golang to the Ambassador Stack
|
||||
|
||||
With the KAT test framework in place, we quickly ran into some issues with Envoy v2 configuration and hot restart, which presented the opportunity to switch to use Envoy’s Aggregated Discovery Service (ADS) APIs instead of hot restart. This completely eliminated the requirement for restart on configuration changes, which we found could lead to dropped connection under high loads or long-lived connections.
|
||||
|
||||
However, we faced an interesting question as we considered the move to the ADS. The ADS is not as simple as one might expect: there are explicit ordering dependencies when sending updates to Envoy. The Envoy project has reference implementations of the ordering logic, but only in Go and Java, where Ambassador was primarily in Python. We agonized a bit, and decided that the simplest way forward was to accept the polyglot nature of our world, and do our ADS implementation in Go.
|
||||
|
||||
We also found, with KAT, that our testing had reached the point where Python’s performance with many network connections was a limitation, so we took advantage of Go here, as well, writing KAT’s querying and backend services primarily in Go. After all, what’s another Golang dependency when you’ve already taken the plunge?
|
||||
|
||||
With a new test framework, new IR generating valid Envoy v2 configuration, and the ADS, we thought we were done with the major architectural changes in Ambassador 0.50. Alas, we hit one more issue. On the Azure Kubernetes Service, Ambassador annotation changes were no longer being detected.
|
||||
|
||||
Working with the highly-responsive AKS engineering team, we were able to identify the issue -- namely, the Kubernetes API server in AKS is exposed through a chain of proxies, requiring clients to be updating to understand how to connect using the FQDN of the API server, which is provided through a mutating webhook in AKS. Unfortunately, support for this feature was not available in the official Kubernetes Python client, so this was the third spot where we chose to switch to Go instead of Python.
|
||||
|
||||
This raises the interesting question of, “why not ditch all the Python code, and just rewrite Ambassador entirely in Go?” It’s a valid question. The main concern with a rewrite is that Ambassador and Envoy operate at different conceptual levels rather than simply expressing the same concepts with different syntax. Being certain that we’ve expressed the conceptual bridges in a new language is not a trivial challenge, and not something to undertake without already having really excellent test coverage in place
|
||||
|
||||
At this point, we use Go to coverage very specific, well-contained functions that can be verified for correctness much more easily that we could verify a complete Golang rewrite. In the future, who knows? But for 0.50.0, this functional split let us both take advantage of Golang’s strengths, while letting us retain more confidence about all the changes already in 0.50.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
We've learned a lot in the process of building [Ambassador 0.50](https://blog.getambassador.io/ambassador-0-50-ga-release-notes-sni-new-authservice-and-envoy-v2-support-3b30a4d04c81). Some of our key takeaways:
|
||||
|
||||
* Kubernetes and Envoy are very powerful frameworks, but they are also extremely fast moving targets -- there is sometimes no substitute for reading the source code and talking to the maintainers (who are fortunately all quite accessible!)
|
||||
* The best supported libraries in the Kubernetes / Envoy ecosystem are written in Go. While we love Python, we have had to adopt Go so that we're not forced to maintain too many components ourselves.
|
||||
* Redesigning a test harness is sometimes necessary to move your software forward.
|
||||
* The real cost in redesigning a test harness is often in porting your old tests to the new harness implementation.
|
||||
* Designing (and implementing) an effective control plane for the edge proxy use case has been challenging, and the feedback from the open source community around Kubernetes, Envoy and Ambassador has been extremely useful.
|
||||
|
||||
Migrating Ambassador to the Envoy v2 configuration and ADS APIs was a long and difficult journey that required lots of architecture and design discussions and plenty of coding, but early feedback from results have been positive. [Ambassador 0.50 is available now](https://blog.getambassador.io/announcing-ambassador-0-50-8dffab5b05e0), so you can take it for a test run and share your feedback with the community on our [Slack channel](http://d6e.co/slack) or on [Twitter](https://www.twitter.com/getambassadorio).
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: Automate Operations on your Cluster with OperatorHub.io
|
||||
date: 2019-02-28
|
||||
---
|
||||
|
||||
**Author:**
|
||||
Diane Mueller, Director of Community Development, Cloud Platforms, Red Hat
|
||||
|
||||
One of the important challenges facing developers and Kubernetes administrators has been a lack of ability to quickly find common services that are operationally ready for Kubernetes. Typically, the presence of an Operator for a specific service - a pattern that was introduced in 2016 and has gained momentum - is a good signal for the operational readiness of the service on Kubernetes. However, there has to date not existed a registry of Operators to simplify the discovery of such services.
|
||||
|
||||
To help address this challenge, today Red Hat is launching OperatorHub.io in collaboration with AWS, Google Cloud and Microsoft. OperatorHub.io enables developers and Kubernetes administrators to find and install curated Operator-backed services with a base level of documentation, active maintainership by communities or vendors, basic testing, and packaging for optimized life-cycle management on Kubernetes.
|
||||
|
||||
The Operators currently in OperatorHub.io are just the start. We invite the Kubernetes community to join us in building a vibrant community for Operators by developing, packaging, and publishing Operators on OperatorHub.io.
|
||||
|
||||
## What does OperatorHub.io provide?
|
||||
|
||||
OperatorHub.io is designed to address the needs of both Kubernetes developers and users. For the former it provides a common registry where they can publish their Operators alongside with descriptions, relevant details like version, image, code repository and have them be readily packaged for installation. They can also update already published Operators to new versions when they are released.
|
||||
|
||||
|
||||
Users get the ability to discover and download Operators at a central location, that has content which has been screened for the previously mentioned criteria and scanned for known vulnerabilities. In addition, developers can guide users of their Operators with prescriptive examples of the `CustomResources` that they introduce to interact with the application.
|
||||
|
||||
## What is an Operator?
|
||||
|
||||
Operators were first introduced in 2016 by CoreOS and have been used by Red Hat and the Kubernetes community as a way to package, deploy and manage a Kubernetes-native application. A Kubernetes-native application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and well-known tooling, like kubectl.
|
||||
|
||||
An Operator is implemented as a custom controller that watches for certain Kubernetes resources to appear, be modified or deleted. These are typically `CustomResourceDefinitions` that the Operator “owns.” In the spec properties of these objects the user declares the desired state of the application or the operation. The Operator’s reconciliation loop will pick these up and perform the required actions to achieve the desired state. For example, the intent to create a highly available etcd cluster could be expressed by creating an new resource of type `EtcdCluster`:
|
||||
|
||||
```
|
||||
apiVersion: "etcd.database.coreos.com/v1beta2"
|
||||
kind: "EtcdCluster"
|
||||
metadata:
|
||||
name: "my-etcd-cluster"
|
||||
spec:
|
||||
size: 3
|
||||
version: "3.3.12"
|
||||
```
|
||||
|
||||
The `EtcdOperator` would be responsible for creating a 3-node etcd cluster running version v3.3.12 as a result. Similarly, an object of type `EtcdBackup` could be defined to express the intent to create a consistent backup of the etcd database to an S3 bucket.
|
||||
|
||||
## How do I create and run an Operator?
|
||||
|
||||
One way to get started is with the [Operator Framework](https://github.com/operator-framework), an open source toolkit that provides an SDK, lifecycle management, metering and monitoring capabilities. It enables developers to build, test, and package Operators. Operators can be implemented in several programming and automation languages, including Go, Helm, and Ansible, all three of which are supported directly by the SDK.
|
||||
|
||||
If you are interested in creating your own Operator, we recommend checking out the Operator Framework to [get started](https://github.com/operator-framework/getting-started).
|
||||
|
||||
Operators vary in where they fall along [the capability spectrum](https://github.com/operator-framework/operator-sdk/blob/master/doc/images/operator-maturity-model.png) ranging from basic functionality to having specific operational logic for an application to automate advanced scenarios like backup, restore or tuning. Beyond basic installation, advanced Operators are designed to handle upgrades more seamlessly and react to failures automatically. Currently, Operators on OperatorHub.io span the maturity spectrum, but we anticipate their continuing maturation over time.
|
||||
|
||||
While Operators on OperatorHub.io don’t need to be implemented using the SDK, they are packaged for deployment through the [Operator Lifecycle Manager](https://github.com/operator-framework/operator-lifecycle-manager) (OLM). The format mainly consists of a YAML manifest referred to as `[ClusterServiceVersion]`(https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/building-your-csv.md) which provides information about the `CustomResourceDefinitions` the Operator owns or requires, which RBAC definition it needs, where the image is stored, etc. This file is usually accompanied by additional YAML files which define the Operators’ own CRDs. This information is processed by OLM at the time a user requests to install an Operator to provide dependency resolution and automation.
|
||||
|
||||
## What does listing of an Operator on OperatorHub.io mean?
|
||||
|
||||
To be listed, Operators must successfully show cluster lifecycle features, be packaged as a CSV to be maintained through OLM, and have acceptable documentation for its intended users.
|
||||
|
||||
Some examples of Operators that are currently listed on OperatorHub.io include: Amazon Web Services Operator, Couchbase Autonomous Operator, CrunchyData’s PostgreSQL, etcd Operator, Jaeger Operator for Kubernetes, Kubernetes Federation Operator, MongoDB Enterprise Operator, Percona MySQL Operator, PlanetScale’s Vitess Operator, Prometheus Operator, and Redis Operator.
|
||||
|
||||
## Want to add your Operator to OperatorHub.io? Follow these steps
|
||||
|
||||
If you have an existing Operator, follow the [contribution guide](https://www.operatorhub.io/contribute) using a fork of the [community-operators](https://github.com/operator-framework/community-operators/) repository. Each contribution contains the CSV, all of the `CustomResourceDefinitions`, access control rules and references to the container image needed to install and run your Operator, plus other info like a description of its features and supported Kubernetes versions. A complete example, including multiple versions of the Operator, can be found with the [EtcdOperator](https://github.com/operator-framework/community-operators/tree/master/community-operators/etcd).
|
||||
|
||||
After testing out your Operator on your own cluster, submit a PR to the [community repository](https://github.com/operator-framework/community-operators) with all of YAML files following [this directory structure](https://github.com/operator-framework/community-operators#adding-your-operator). Subsequent versions of the Operator can be published in the same way. At first this will be reviewed manually, but automation is on the way. After it’s merged by the maintainers, it will show up on OperatorHub.io along with its documentation and a convenient installation method.
|
||||
|
||||
## Want to learn more?
|
||||
|
||||
- Attend one of the upcoming Kubernetes Operator Framework hands-on workshops at [ScaleX](https://www.socallinuxexpo.org/scale/17x/presentations/workshop-kubernetes-operator-framework) in Pasadena on March 7 and at the [OpenShift Commons Gathering on Operating at Scale in Santa Clara on March 11](https://commons.openshift.org/gatherings/Santa_Clara_2019.html)
|
||||
- Listen to this [OpenShift Commons Briefing on “The State of Operators” with Daniel Messer and Diane Mueller](https://www.youtube.com/watch?v=GgEKEYH9MMM&feature=youtu.be)
|
||||
- Join in on the online conversations in the community [Kubernetes-Operator Slack Channel](https://kubernetes.slack.com/messages/CAW0GV7A5) and the [Operator Framework Google Group](https://groups.google.com/forum/#!forum/operator-framework)
|
||||
- Finally, read up on how to add your Operator to OperatorHub.io: https://operatorhub.io/contribute
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
title: Raw Block Volume support to Beta
|
||||
date: 2019-03-07
|
||||
---
|
||||
|
||||
**Authors:**
|
||||
Ben Swartzlander (NetApp), Saad Ali (Google)
|
||||
|
||||
Kubernetes v1.13 moves raw block volume support to beta. This feature allows persistent volumes to be exposed inside containers as a block device instead of as a mounted file system.
|
||||
|
||||
## What are block devices?
|
||||
|
||||
Block devices enable random access to data in fixed-size blocks. Hard drives, SSDs, and CD-ROMs drives are all examples of block devices.
|
||||
|
||||
Typically persistent storage is implemented in a layered maner with a file system (like ext4) on top of a block device (like a spinning disk or SSD). Applications then read and write files instead of operating on blocks. The operating systems take care of reading and writing files, using the specified filesystem, to the underlying device as blocks.
|
||||
|
||||
It's worth noting that while whole disks are block devices, so are disk partitions, and so are LUNs from a storage area network (SAN) device.
|
||||
|
||||
## Why add raw block volumes to kubernetes?
|
||||
|
||||
There are some specialized applications that require direct access to a block device because, for example, the file system layer introduces unneeded overhead. The most common case is databases, which prefer to organize their data directly on the underlying storage. Raw block devices are also commonly used by any software which itself implements some kind of storage service (software defined storage systems).
|
||||
|
||||
From a programmer's perspective, a block device is a very large array of bytes, usually with some minimum granularity for reads and writes, often 512 bytes, but frequently 4K or larger.
|
||||
|
||||
As it becomes more common to run database software and storage infrastructure software inside of Kubernetes, the need for raw block device support in Kubernetes becomes more important.
|
||||
|
||||
## Which volume plugins support raw blocks?
|
||||
|
||||
As of the publishing of this blog, the following in-tree volumes types support raw blocks:
|
||||
|
||||
- AWS EBS
|
||||
- Azure Disk
|
||||
- Cinder
|
||||
- Fibre Channel
|
||||
- GCE PD
|
||||
- iSCSI
|
||||
- Local volumes
|
||||
- RBD (Ceph)
|
||||
- Vsphere
|
||||
|
||||
Out-of-tree [CSI volume drivers](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) may also support raw block volumes. Kubernetes CSI support for raw block volumes is currently alpha. See documentation [here](https://kubernetes-csi.github.io/docs/raw-block.html).
|
||||
|
||||
## Kubernetes raw block volume API
|
||||
|
||||
Raw block volumes share a lot in common with ordinary volumes. Both are requested by creating `PersistentVolumeClaim` objects which bind to `PersistentVolume` objects, and are attached to Pods in Kubernetes by including them in the volumes array of the `PodSpec`.
|
||||
|
||||
There are 2 important differences however. First, to request a raw block `PersistentVolumeClaim`, you must set `volumeMode = "Block"` in the `PersistentVolumeClaimSpec`. Leaving `volumeMode` blank is the same as specifying `volumeMode = "Filesystem"` which results in the traditional behavior. `PersistentVolumes` also have a `volumeMode` field in their `PersistentVolumeSpec`, and `"Block"` type PVCs can only bind to `"Block"` type PVs and `"Filesystem"` PVCs can only bind to `"Filesystem"` PVs.
|
||||
|
||||
Secondly, when using a raw block volume in your Pods, you must specify a `VolumeDevice` in the Container portion of the `PodSpec` rather than a `VolumeMount`. `VolumeDevices` have `devicePaths` instead of `mountPaths`, and inside the container, applications will see a device at that path instead of a mounted file system.
|
||||
|
||||
Applications open, read, and write to the device node inside the container just like they would interact with any block device on a system in a non-containerized or virtualized context.
|
||||
|
||||
## Creating a new raw block PVC
|
||||
|
||||
First, ensure that the provisioner associated with the storage class you choose is one that support raw blocks. Then create the PVC.
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: my-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
volumeMode: Block
|
||||
storageClassName: my-sc
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
```
|
||||
|
||||
## Using a raw block PVC
|
||||
|
||||
When you use the PVC in a pod definition, you get to choose the device path for the block device rather than the mount path for the file system.
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: my-pod
|
||||
spec:
|
||||
containers:
|
||||
- name: my-container
|
||||
image: busybox
|
||||
command:
|
||||
- sleep
|
||||
- “3600”
|
||||
volumeDevices:
|
||||
- devicePath: /dev/block
|
||||
name: my-volume
|
||||
imagePullPolicy: IfNotPresent
|
||||
volumes:
|
||||
- name: my-volume
|
||||
persistentVolumeClaim:
|
||||
claimName: my-pvc
|
||||
```
|
||||
|
||||
## As a storage vendor, how do I add support for raw block devices to my CSI plugin?
|
||||
|
||||
Raw block support for CSI plugins is still alpha, but support can be added today. The [CSI specification](https://github.com/container-storage-interface/spec/blob/master/spec.md) details how to handle requests for volume that have the `BlockVolume` capability instead of the `MountVolume` capability. CSI plugins can support both kinds of volumes, or one or the other. For more details see [documentation here](https://kubernetes-csi.github.io/docs/raw-block.html).
|
||||
|
||||
|
||||
## Issues/gotchas
|
||||
|
||||
Because block devices are actually devices, it’s possible to do low-level actions on them from inside containers that wouldn’t be possible with file system volumes. For example, block devices that are actually SCSI disks support sending SCSI commands to the device using Linux ioctls.
|
||||
|
||||
By default, Linux won’t allow containers to send SCSI commands to disks from inside containers though. In order to do so, you must grant the `SYS_RAWIO` capability to the container security context to allow this. See documentation [here](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-capabilities-for-a-container).
|
||||
|
||||
Also, while Kubernetes is guaranteed to deliver a block device to the container, there’s no guarantee that it’s actually a SCSI disk or any other kind of disk for that matter. The user must either ensure that the desired disk type is used with his pods, or only deploy applications that can handle a variety of block device types.
|
||||
|
||||
## How can I learn more?
|
||||
|
||||
Check out additional documentation on the snapshot feature here: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#raw-block-volume-support
|
||||
|
||||
How do I get involved?
|
||||
|
||||
Join the Kubernetes storage SIG and the CSI community and help us add more great features and improve existing ones like raw block storage!
|
||||
|
||||
https://github.com/kubernetes/community/tree/master/sig-storage
|
||||
https://github.com/container-storage-interface/community/blob/master/README.md
|
||||
|
||||
Special thanks to all the contributors who helped add block volume support to Kubernetes including:
|
||||
|
||||
- Ben Swartzlander (https://github.com/bswartz)
|
||||
- Brad Childs (https://github.com/childsb)
|
||||
- Erin Boyd (https://github.com/erinboyd)
|
||||
- Masaki Kimura (https://github.com/mkimuram)
|
||||
- Matthew Wong (https://github.com/wongma7)
|
||||
- Michelle Au (https://github.com/msau42)
|
||||
- Mitsuhiro Tanino (https://github.com/mtanino)
|
||||
- Saad Ali (https://github.com/saad-ali)
|
||||
@@ -0,0 +1,253 @@
|
||||
---
|
||||
layout: blog
|
||||
title: Kubernetes Setup Using Ansible and Vagrant
|
||||
date: 2019-03-15
|
||||
---
|
||||
|
||||
**Author:** Naresh L J (Infosys)
|
||||
|
||||
## Objective
|
||||
This blog post describes the steps required to setup a multi node Kubernetes cluster for development purposes. This setup provides a production-like cluster that can be setup on your local machine.
|
||||
|
||||
## Why do we require multi node cluster setup?
|
||||
Multi node Kubernetes clusters offer a production-like environment which has various advantages. Even though Minikube provides an excellent platform for getting started, it doesn't provide the opportunity to work with multi node clusters which can help solve problems or bugs that are related to application design and architecture. For instance, Ops can reproduce an issue in a multi node cluster environment, Testers can deploy multiple versions of an application for executing test cases and verifying changes. These benefits enable teams to resolve issues faster which make the more agile.
|
||||
|
||||
## Why use Vagrant and Ansible?
|
||||
Vagrant is a tool that will allow us to create a virtual environment easily and it eliminates pitfalls that cause the works-on-my-machine phenomenon. It can be used with multiple providers such as Oracle VirtualBox, VMware, Docker, and so on. It allows us to create a disposable environment by making use of configuration files.
|
||||
|
||||
Ansible is an infrastructure automation engine that automates software configuration management. It is agentless and allows us to use SSH keys for connecting to remote machines. Ansible playbooks are written in yaml and offer inventory management in simple text files.
|
||||
|
||||
|
||||
### Prerequisites
|
||||
- Vagrant should be installed on your machine. Installation binaries can be found [here](https://www.vagrantup.com/downloads.html).
|
||||
- Oracle VirtualBox can be used as a Vagrant provider or make use of similar providers as described in Vagrant's official [documentation](https://www.vagrantup.com/docs/providers/).
|
||||
- Ansible should be installed in your machine. Refer to the [Ansible installation guide](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) for platform specific installation.
|
||||
|
||||
## Setup overview
|
||||
We will be setting up a Kubernetes cluster that will consist of one master and two worker nodes. All the nodes will run Ubuntu Xenial 64-bit OS and Ansible playbooks will be used for provisioning.
|
||||
|
||||
#### Step 1: Creating a Vagrantfile
|
||||
Use the text editor of your choice and create a file with named `Vagrantfile`, inserting the code below. The value of N denotes the number of nodes present in the cluster, it can be modified accordingly. In the below example, we are setting the value of N as 2.
|
||||
|
||||
```ruby
|
||||
IMAGE_NAME = "bento/ubuntu-16.04"
|
||||
N = 2
|
||||
|
||||
Vagrant.configure("2") do |config|
|
||||
config.ssh.insert_key = false
|
||||
|
||||
config.vm.provider "virtualbox" do |v|
|
||||
v.memory = 1024
|
||||
v.cpus = 2
|
||||
end
|
||||
|
||||
config.vm.define "k8s-master" do |master|
|
||||
master.vm.box = IMAGE_NAME
|
||||
master.vm.network "private_network", ip: "192.168.50.10"
|
||||
master.vm.hostname = "k8s-master"
|
||||
master.vm.provision "ansible" do |ansible|
|
||||
ansible.playbook = "kubernetes-setup/master-playbook.yml"
|
||||
end
|
||||
end
|
||||
|
||||
(1..N).each do |i|
|
||||
config.vm.define "node-#{i}" do |node|
|
||||
node.vm.box = IMAGE_NAME
|
||||
node.vm.network "private_network", ip: "192.168.50.#{i + 10}"
|
||||
node.vm.hostname = "node-#{i}"
|
||||
node.vm.provision "ansible" do |ansible|
|
||||
ansible.playbook = "kubernetes-setup/node-playbook.yml"
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Step 2: Create an Ansible playbook for Kubernetes master.
|
||||
Create a directory named `kubernetes-setup` in the same directory as the `Vagrantfile`. Create two files named `master-playbook.yml` and `node-playbook.yml` in the directory `kubernetes-setup`.
|
||||
|
||||
In the file `master-playbook.yml`, add the code below.
|
||||
|
||||
#### Step 2.1: Install Docker and its dependent components.
|
||||
|
||||
We will be installing the following packages, and then adding a user named “vagrant” to the “docker” group.
|
||||
- docker-ce
|
||||
- docker-ce-cli
|
||||
- containerd.io
|
||||
|
||||
```yaml
|
||||
---
|
||||
- hosts: all
|
||||
become: true
|
||||
tasks:
|
||||
- name: Install packages that allow apt to be used over HTTPS
|
||||
apt:
|
||||
name: "{{ packages }}"
|
||||
state: present
|
||||
update_cache: yes
|
||||
vars:
|
||||
packages:
|
||||
- apt-transport-https
|
||||
- ca-certificates
|
||||
- curl
|
||||
- gnupg-agent
|
||||
- software-properties-common
|
||||
|
||||
- name: Add an apt signing key for Docker
|
||||
apt_key:
|
||||
url: https://download.docker.com/linux/ubuntu/gpg
|
||||
state: present
|
||||
|
||||
- name: Add apt repository for stable version
|
||||
apt_repository:
|
||||
repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
|
||||
state: present
|
||||
|
||||
- name: Install docker and its dependecies
|
||||
apt:
|
||||
name: "{{ packages }}"
|
||||
state: present
|
||||
update_cache: yes
|
||||
vars:
|
||||
packages:
|
||||
- docker-ce
|
||||
- docker-ce-cli
|
||||
- containerd.io
|
||||
notify:
|
||||
- docker status
|
||||
|
||||
- name: Add vagrant user to docker group
|
||||
user:
|
||||
name: vagrant
|
||||
group: docker
|
||||
```
|
||||
|
||||
#### Step 2.2: Kubelet will not start if the system has swap enabled, so we are disabling swap using the below code.
|
||||
|
||||
```yaml
|
||||
- name: Remove swapfile from /etc/fstab
|
||||
mount:
|
||||
name: "{{ item }}"
|
||||
fstype: swap
|
||||
state: absent
|
||||
with_items:
|
||||
- swap
|
||||
- none
|
||||
|
||||
- name: Disable swap
|
||||
command: swapoff -a
|
||||
when: ansible_swaptotal_mb > 0
|
||||
```
|
||||
|
||||
#### Step 2.3: Installing kubelet, kubeadm and kubectl using the below code.
|
||||
|
||||
```yaml
|
||||
- name: Add an apt signing key for Kubernetes
|
||||
apt_key:
|
||||
url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
|
||||
state: present
|
||||
|
||||
- name: Adding apt repository for Kubernetes
|
||||
apt_repository:
|
||||
repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
|
||||
state: present
|
||||
filename: kubernetes.list
|
||||
|
||||
- name: Install Kubernetes binaries
|
||||
apt:
|
||||
name: "{{ packages }}"
|
||||
state: present
|
||||
update_cache: yes
|
||||
vars:
|
||||
packages:
|
||||
- kubelet
|
||||
- kubeadm
|
||||
- kubectl
|
||||
```
|
||||
|
||||
#### Step 2.3: Initialize the Kubernetes cluster with kubeadm using the below code (applicable only on master node).
|
||||
|
||||
```yaml
|
||||
- name: Initialize the Kubernetes cluster using kubeadm
|
||||
command: kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10" --node-name k8s-master --pod-network-cidr=192.168.0.0/16
|
||||
```
|
||||
|
||||
#### Step 2.4: Setup the kube config file for the vagrant user to access the Kubernetes cluster using the below code.
|
||||
|
||||
```yaml
|
||||
- name: Setup kubeconfig for vagrant user
|
||||
command: "{{ item }}"
|
||||
with_items:
|
||||
- mkdir -p /home/vagrant/.kube
|
||||
- cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
|
||||
- chown vagrant:vagrant /home/vagrant/.kube/config
|
||||
```
|
||||
|
||||
#### Step 2.5: Setup the container networking provider and the network policy engine using the below code.
|
||||
|
||||
```yaml
|
||||
- name: Install calico pod network
|
||||
become: false
|
||||
command: kubectl create -f https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml
|
||||
```
|
||||
|
||||
#### Step 2.6: Generate kube join command for joining the node to the Kubernetes cluster and store the command in the file named `join-command`.
|
||||
|
||||
```yaml
|
||||
- name: Generate join command
|
||||
command: kubeadm token create --print-join-command
|
||||
register: join_command
|
||||
|
||||
- name: Copy join command to local file
|
||||
local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="./join-command"
|
||||
```
|
||||
|
||||
#### Step 2.7: Setup a handler for checking Docker daemon using the below code.
|
||||
|
||||
```yaml
|
||||
handlers:
|
||||
- name: docker status
|
||||
service: name=docker state=started
|
||||
```
|
||||
|
||||
#### Step 3: Create the Ansible playbook for Kubernetes node.
|
||||
Create a file named `node-playbook.yml` in the directory `kubernetes-setup`.
|
||||
|
||||
Add the code below into `node-playbook.yml`
|
||||
|
||||
#### Step 3.1: Start adding the code from Steps 2.1 till 2.3.
|
||||
|
||||
#### Step 3.2: Join the nodes to the Kubernetes cluster using below code.
|
||||
|
||||
```yaml
|
||||
- name: Copy the join command to server location
|
||||
copy: src=join-command dest=/tmp/join-command.sh mode=0777
|
||||
|
||||
- name: Join the node to cluster
|
||||
command: sh /tmp/join-command.sh
|
||||
```
|
||||
|
||||
#### Step 3.3: Add the code from step 2.7 to finish this playbook.
|
||||
|
||||
#### Step 4: Upon completing the Vagrantfile and playbooks follow the below steps.
|
||||
|
||||
```shell
|
||||
$ cd /path/to/Vagrantfile
|
||||
$ vagrant up
|
||||
```
|
||||
|
||||
Upon completion of all the above steps, the Kubernetes cluster should be up and running.
|
||||
We can login to the master or worker nodes using Vagrant as follows:
|
||||
|
||||
```shell
|
||||
$ ## Accessing master
|
||||
$ vagrant ssh k8s-master
|
||||
vagrant@k8s-master:~$ kubectl get nodes
|
||||
NAME STATUS ROLES AGE VERSION
|
||||
k8s-master Ready master 18m v1.13.3
|
||||
node-1 Ready <none> 12m v1.13.3
|
||||
node-2 Ready <none> 6m22s v1.13.3
|
||||
|
||||
$ ## Accessing nodes
|
||||
$ vagrant ssh node-1
|
||||
$ vagrant ssh node-2
|
||||
```
|
||||
@@ -1,3 +1,5 @@
|
||||
# See the OWNERS docs at https://go.k8s.io/owners
|
||||
|
||||
# Owned by Kubernetes Blog reviewers.
|
||||
options:
|
||||
no_parent_owners: false
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 7.6 KiB After Width: | Height: | Size: 6.9 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 8.3 KiB After Width: | Height: | Size: 11 KiB |
@@ -0,0 +1,86 @@
|
||||
---
|
||||
title: NetEase Case Study
|
||||
case_study_styles: true
|
||||
cid: caseStudies
|
||||
css: /css/style_case_studies.css
|
||||
---
|
||||
|
||||
|
||||
<div class="banner1" style="background-image: url('/images/CaseStudy_netease_banner1.jpg')">
|
||||
<h1> CASE STUDY:<img src="/images/netease_logo.png" class="header_logo" style="width:22%;margin-bottom:-1%"><br> <div class="subhead" style="margin-top:1%"> How NetEase Leverages Kubernetes to Support Internet Business Worldwide</div></h1>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="details" style="font-size:1em">
|
||||
Company <b>NetEase</b> Location <b>Hangzhou, China</b> Industry <b>Internet technology</b>
|
||||
</div>
|
||||
|
||||
<hr>
|
||||
<section class="section1">
|
||||
<div class="cols">
|
||||
<div class="col1" style="margin-left:-1.5% !important">
|
||||
<h2>Challenge</h2>
|
||||
Its gaming business is one of the largest in the world, but that’s not all that <a href="https://netease-na.com/">NetEase</a> provides to Chinese consumers. The company also operates e-commerce, advertising, music streaming, online education, and email platforms; the last of which serves almost a billion users with free email services through sites like <a href="https://www.163.com/">163.com</a>. In 2015, the NetEase Cloud team providing the infrastructure for all of these systems realized that their R&D process was slowing down developers. “Our users needed to prepare all of the infrastructure by themselves,” says Feng Changjian, Architect for NetEase Cloud and Container Service. “We were eager to provide the infrastructure and tools for our users automatically via serverless container service.”
|
||||
<br><br>
|
||||
<h2>Solution</h2>
|
||||
After considering building its own orchestration solution, NetEase decided to base its private cloud platform on <a href="https://kubernetes.io/">Kubernetes</a>. The fact that the technology came out of Google gave the team confidence that it could keep up with NetEase’s scale. “After our 2-to-3-month evaluation, we believed it could satisfy our needs,” says Feng. The team started working with Kubernetes in 2015, before it was even 1.0. Today, the NetEase internal cloud platform—which also leverages the CNCF projects <a href="https://prometheus.io/">Prometheus</a>, <a href="https://www.envoyproxy.io/">Envoy</a>, <a href="https://goharbor.io/">Harbor</a>, <a href="https://grpc.io/">gRPC</a>, and <a href="https://helm.sh/">Helm</a>—runs 10,000 nodes in a production cluster and can support up to 30,000 nodes in a cluster. Based on its learnings from its internal platform, the company introduced a Kubernetes-based cloud and microservices-oriented PaaS product, <a href="https://landscape.cncf.io/selected=netease-qingzhou-microservice">NetEase Qingzhou Microservice</a>, to outside customers.
|
||||
|
||||
|
||||
<br><br>
|
||||
<h2>Impact</h2>
|
||||
The NetEase team reports that Kubernetes has increased R&D efficiency by more than 100%. Deployment efficiency has improved by 280%. “In the past, if we wanted to do upgrades, we needed to work with other teams, even in other departments,” says Feng. “We needed special staff to prepare everything, so it took about half an hour. Now we can do it in only 5 minutes.” The new platform also allows for mixed deployments using GPU and CPU resources. “Before, if we put all the resources toward the GPU, we won’t have spare resources for the CPU. But now we have improvements thanks to the mixed deployments,” he says. Those improvements have also brought an increase in resource utilization.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</section>
|
||||
<div class="banner2">
|
||||
<div class="banner2text">
|
||||
"The system can support 30,000 nodes in a single cluster. In production, we have gotten the data of 10,000 nodes in a single cluster. The whole internal system is using this system for development, test, and production."<br style="height:25px"><span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br>— Zeng Yuxing, Architect, NetEase</span>
|
||||
</div>
|
||||
</div>
|
||||
<section class="section2">
|
||||
<div class="fullcol">
|
||||
<h2>Its gaming business is the <a href="https://newzoo.com/insights/rankings/top-25-companies-game-revenues/">fifth-largest</a> in the world, but that’s not all that <a href="https://netease-na.com/">NetEase</a> provides consumers.</h2>The company also operates e-commerce, advertising, music streaming, online education, and email platforms in China; the last of which serves almost a billion users with free email services through popular sites like <a href="https://www.163.com/">163.com</a> and <a href="https://www.126.com/">126.com</a>. With that kind of scale, the NetEase Cloud team providing the infrastructure for all of these systems realized in 2015 that their R&D process was making it hard for developers to keep up with demand. “Our users needed to prepare all of the infrastructure by themselves,” says Feng Changjian, Architect for NetEase Cloud and Container Service. “We were eager to provide the infrastructure and tools for our users automatically via serverless container service.”<br><br>
|
||||
After considering building its own orchestration solution, NetEase decided to base its private cloud platform on <a href="https://kubernetes.io/">Kubernetes</a>. The fact that the technology came out of Google gave the team confidence that it could keep up with NetEase’s scale. “After our 2-to-3-month evaluation, we believed it could satisfy our needs,” says Feng.
|
||||
</div>
|
||||
</section>
|
||||
<div class="banner3" style="background-image: url('/images/CaseStudy_netease_banner3.jpg')">
|
||||
<div class="banner3text">
|
||||
"We leveraged the programmability of Kubernetes so that we can build a platform to satisfy the needs of our internal customers for upgrades and deployment."
|
||||
<br style="height:25px"><span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br>- Feng Changjian, Architect for NetEase Cloud and Container Service, NetEase</span>
|
||||
</div>
|
||||
</div>
|
||||
<section class="section3">
|
||||
<div class="fullcol">
|
||||
The team started adopting Kubernetes in 2015, before it was even 1.0, because it was relatively easy to use and enabled DevOps at the company. “We abandoned some of the concepts of Kubernetes; we only wanted to use the standardized framework,” says Feng. “We leveraged the programmability of Kubernetes so that we can build a platform to satisfy the needs of our internal customers for upgrades and deployment.”<br><br>
|
||||
The team first focused on building the container platform to manage resources better, and then turned their attention to improving its support of microservices by adding internal systems such as monitoring. That has meant integrating the CNCF projects <a href="https://prometheus.io/">Prometheus</a>, <a href="https://www.envoyproxy.io/">Envoy</a>, <a href="https://goharbor.io/">Harbor</a>, <a href="https://grpc.io/">gRPC</a>, and <a href="https://helm.sh/">Helm</a>. “We are trying to provide a simplified and standardized process, so our users and customers can leverage our best practices,” says Feng.<br><br>
|
||||
And the team is continuing to make improvements. For example, the e-commerce part of the business needs to leverage mixed deployments, which in the past required using two separate platforms: the infrastructure-as-a-service platform and the Kubernetes platform. More recently, NetEase has created a cross-platform application that enables using both with one-command deployment.
|
||||
</div>
|
||||
</section>
|
||||
<div class="banner4" style="background-image: url('/images/CaseStudy_netease_banner4.jpg')">
|
||||
<div class="banner4text">
|
||||
"As long as a company has a mature team and enough developers, I think Kubernetes is a very good technology that can help them."
|
||||
<span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br><br>- Li Lanqing, Kubernetes Developer, NetEase</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<section class="section5" style="padding:0px !important">
|
||||
<div class="fullcol">
|
||||
Today, the NetEase internal cloud platform “can support 30,000 nodes in a single cluster,” says Architect Zeng Yuxing. “In production, we have gotten the data of 10,000 nodes in a single cluster. The whole internal system is using this system for development, test, and production.” <br><br>
|
||||
The NetEase team reports that Kubernetes has increased R&D efficiency by more than 100%. Deployment efficiency has improved by 280%. “In the past, if we wanted to do upgrades, we needed to work with other teams, even in other departments,” says Feng. “We needed special staff to prepare everything, so it took about half an hour. Now we can do it in only 5 minutes.” The new platform also allows for mixed deployments using GPU and CPU resources. “Before, if we put all the resources toward the GPU, we won’t have spare resources for the CPU. But now we have improvements thanks to the mixed deployments.” Those improvements have also brought an increase in resource utilization.
|
||||
|
||||
</div>
|
||||
|
||||
<div class="banner5">
|
||||
<div class="banner5text">
|
||||
"By engaging with this community, we can gain some experience from it and we can also benefit from it. We can see what are the concerns and the challenges faced by the community, so we can get involved."<span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br><br>- Li Lanqing, Kubernetes Developer, NetEase</span>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
<div class="fullcol">
|
||||
Based on the results and learnings from using its internal platform, the company introduced a Kubernetes-based cloud and microservices-oriented PaaS product, <a href="https://landscape.cncf.io/selected=netease-qingzhou-microservice">NetEase Qingzhou Microservice</a>, to outside customers. “The idea is that we can find the problems encountered by our game and e-commerce and cloud music providers, so we can integrate their experiences and provide a platform to satisfy the needs of our users,” says Zeng. <br><br>
|
||||
With or without the use of the NetEase product, the team encourages other companies to try Kubernetes. “As long as a company has a mature team and enough developers, I think Kubernetes is a very good technology that can help them,” says Kubernetes developer Li Lanqing.<br><br>
|
||||
As an end user as well as a vendor, NetEase has become more involved in the community, learning from other companies and sharing what they’ve done. The team has been contributing to the Harbor and Envoy projects, providing feedback as the technologies are being tested at NetEase scale. “We are a team focusing on addressing the challenges of microservices architecture,” says Feng. “By engaging with this community, we can gain some experience from it and we can also benefit from it. We can see what are the concerns and the challenges faced by the community, so we can get involved.”
|
||||
</div>
|
||||
</section>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 5.5 KiB |
@@ -17,12 +17,11 @@ If you notice that this is out of date, please
|
||||
|
||||
If you notice a violation of the Code of Conduct at an event or meeting, in
|
||||
Slack, or in another communication mechanism, reach out to
|
||||
the [Kubernetes Code of Conduct Committee](https://github.com/kubernetes/community/tree/master/committee-code-of-conduct) <conduct@kubernetes.io>.
|
||||
the <a href="https://git.k8s.io/community/committee-code-of-conduct">Kubernetes Code of Conduct Committee</a>.
|
||||
You can reach us by email at <a href="mailto:conduct@kubernetes.io">conduct@kubernetes.io</a>.
|
||||
Your anonymity will be protected.
|
||||
|
||||
<div class="cncf_coc_container">
|
||||
{{< include "/static/cncf-code-of-conduct.md" >}}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
@@ -255,6 +255,8 @@ The following cloud providers have implemented CCMs:
|
||||
* [Azure](https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider/providers/azure)
|
||||
* [GCE](https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider/providers/gce)
|
||||
* [AWS](https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider/providers/aws)
|
||||
* [BaiduCloud](https://github.com/baidu/cloud-provider-baiducloud)
|
||||
* [Linode](https://github.com/linode/linode-cloud-controller-manager)
|
||||
|
||||
## Cluster Administration
|
||||
|
||||
|
||||
@@ -77,7 +77,7 @@ To verify this connection, use the `--kubelet-certificate-authority` flag to
|
||||
provide the apiserver with a root certificate bundle to use to verify the
|
||||
kubelet's serving certificate.
|
||||
|
||||
If that is not possible, use [SSH tunneling](/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
|
||||
If that is not possible, use [SSH tunneling](/docs/concepts/architecture/master-node-communication/#ssh-tunnels)
|
||||
between the apiserver and kubelet if required to avoid connecting over an
|
||||
untrusted or public network.
|
||||
|
||||
@@ -95,4 +95,15 @@ connection will be encrypted, it will not provide any guarantees of integrity.
|
||||
These connections **are not currently safe** to run over untrusted and/or
|
||||
public networks.
|
||||
|
||||
### SSH Tunnels
|
||||
|
||||
Kubernetes supports SSH tunnels to protect the Master -> Cluster communication
|
||||
paths. In this configuration, the apiserver initiates an SSH tunnel to each node
|
||||
in the cluster (connecting to the ssh server listening on port 22) and passes
|
||||
all traffic destined for a kubelet, node, pod, or service through the tunnel.
|
||||
This tunnel ensures that the traffic is not exposed outside of the network in
|
||||
which the nodes are running.
|
||||
|
||||
SSH tunnels are currently deprecated so you shouldn't opt to use them unless you know what you are doing. A replacement for this communication channel is being designed.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -74,7 +74,7 @@ the `Terminating` or `Unknown` state. In cases where Kubernetes cannot deduce fr
|
||||
permanently left a cluster, the cluster administrator may need to delete the node object by hand. Deleting the node object from
|
||||
Kubernetes causes all the Pod objects running on the node to be deleted from the apiserver, and frees up their names.
|
||||
|
||||
In version 1.12, `TaintNodesByCondition` feature is promoted to beta，so node lifecycle controller automatically creates
|
||||
In version 1.12, `TaintNodesByCondition` feature is promoted to beta, so node lifecycle controller automatically creates
|
||||
[taints](/docs/concepts/configuration/taint-and-toleration/) that represent conditions.
|
||||
Similarly the scheduler ignores conditions when considering a Node; instead
|
||||
it looks at the Node's taints and a Pod's tolerations.
|
||||
@@ -272,27 +272,8 @@ The Kubernetes scheduler ensures that there are enough resources for all the pod
|
||||
checks that the sum of the requests of containers on the node is no greater than the node capacity. It
|
||||
includes all containers started by the kubelet, but not containers started directly by the [container runtime](/docs/concepts/overview/components/#node-components) nor any process running outside of the containers.
|
||||
|
||||
If you want to explicitly reserve resources for non-pod processes, you can create a placeholder
|
||||
pod. Use the following template:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: resource-reserver
|
||||
spec:
|
||||
containers:
|
||||
- name: sleep-forever
|
||||
image: k8s.gcr.io/pause:0.8.0
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 100Mi
|
||||
```
|
||||
|
||||
Set the `cpu` and `memory` values to the amount of resources you want to reserve.
|
||||
Place the file in the manifest directory (`--config=DIR` flag of kubelet). Do this
|
||||
on each kubelet where you want to reserve resources.
|
||||
If you want to explicitly reserve resources for non-Pod processes, follow this tutorial to
|
||||
[reserve resources for system daemons](/docs/tasks/administer-cluster/reserve-compute-resources/#system-reserved).
|
||||
|
||||
|
||||
## API Object
|
||||
|
||||
@@ -26,10 +26,11 @@ Add-ons in each section are sorted alphabetically - the ordering does not imply
|
||||
* [Cilium](https://github.com/cilium/cilium) is a L3 network and network policy plugin that can enforce HTTP/API/L7 policies transparently. Both routing and overlay/encapsulation mode are supported.
|
||||
* [CNI-Genie](https://github.com/Huawei-PaaS/CNI-Genie) enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, Romana, or Weave.
|
||||
* [Contiv](http://contiv.github.io) provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully [open sourced](http://github.com/contiv). The [installer](http://github.com/contiv/install) provides both kubeadm and non-kubeadm based installation options.
|
||||
* [Contrail](http://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is a open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads.
|
||||
* [Flannel](https://github.com/coreos/flannel/blob/master/Documentation/kubernetes.md) is an overlay network provider that can be used with Kubernetes.
|
||||
* [Knitter](https://github.com/ZTE/Knitter/) is a network solution supporting multiple networking in Kubernetes.
|
||||
* [Multus](https://github.com/Intel-Corp/multus-cni) is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
|
||||
* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and Openshift.
|
||||
* [NSX-T](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.
|
||||
* [Nuage](https://github.com/nuagenetworks/nuage-kubernetes/blob/v5.1.1-1/docs/kubernetes-1-installation.rst) is an SDN platform that provides policy-based networking between Kubernetes Pods and non-Kubernetes environments with visibility and security monitoring.
|
||||
* [Romana](http://romana.io) is a Layer 3 networking solution for pod networks that also supports the [NetworkPolicy API](/docs/concepts/services-networking/network-policies/). Kubeadm add-on installation details available [here](https://github.com/romana/romana/tree/master/containerize).
|
||||
* [Weave Net](https://www.weave.works/docs/net/latest/kube-addon/) provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database.
|
||||
|
||||
@@ -183,7 +183,7 @@ Finally, add the same parameters into the API server start parameters.
|
||||
|
||||
../cfssl gencert -initca ca-csr.json | ../cfssljson -bare ca
|
||||
1. Create a JSON config file for generating keys and certificates for the API
|
||||
server as shown below. Be sure to replace the values in angle brackets with
|
||||
server, for example, `server-csr.json`. Be sure to replace the values in angle brackets with
|
||||
real values you want to use. The `MASTER_CLUSTER_IP` is the service cluster
|
||||
IP for the API server as described in previous subsection.
|
||||
The sample below also assumes that you are using `cluster.local` as the default
|
||||
@@ -231,8 +231,11 @@ refresh the local list for valid certificates.
|
||||
On each client, perform the following operations:
|
||||
|
||||
```bash
|
||||
$ sudo cp ca.crt /usr/local/share/ca-certificates/kubernetes.crt
|
||||
$ sudo update-ca-certificates
|
||||
sudo cp ca.crt /usr/local/share/ca-certificates/kubernetes.crt
|
||||
sudo update-ca-certificates
|
||||
```
|
||||
|
||||
```
|
||||
Updating certificates in /etc/ssl/certs...
|
||||
1 added, 0 removed; done.
|
||||
Running hooks in /etc/ca-certificates/update.d....
|
||||
|
||||
@@ -251,11 +251,11 @@ file:
|
||||
monitor for the Neutron load balancer. Valid values are `true` and `false`.
|
||||
The default is `false`. When `true` is specified then `monitor-delay`,
|
||||
`monitor-timeout`, and `monitor-max-retries` must also be set.
|
||||
* `monitor-delay` (Optional): The time, in seconds, between sending probes to
|
||||
members of the load balancer.
|
||||
* `monitor-timeout` (Optional): Maximum number of seconds for a monitor to wait
|
||||
* `monitor-delay` (Optional): The time between sending probes to
|
||||
members of the load balancer. Ensure that you specify a valid time unit. The valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h"
|
||||
* `monitor-timeout` (Optional): Maximum time for a monitor to wait
|
||||
for a ping reply before it times out. The value must be less than the delay
|
||||
value.
|
||||
value. Ensure that you specify a valid time unit. The valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h"
|
||||
* `monitor-max-retries` (Optional): Number of permissible ping failures before
|
||||
changing the load balancer member's status to INACTIVE. Must be a number
|
||||
between 1 and 10.
|
||||
@@ -367,14 +367,21 @@ The `--hostname-override` parameter is ignored by the VSphere cloud provider.
|
||||
## IBM Cloud Kubernetes Service
|
||||
|
||||
### Compute nodes
|
||||
By using the IBM Cloud Kubernetes Service provider, you can create clusters with a mixture of virtual and physical (bare metal) nodes in a single zone or across multiple zones in a region. For more information, see [Planning your cluster and worker node setup](https://console.bluemix.net/docs/containers/cs_clusters_planning.html#plan_clusters).
|
||||
By using the IBM Cloud Kubernetes Service provider, you can create clusters with a mixture of virtual and physical (bare metal) nodes in a single zone or across multiple zones in a region. For more information, see [Planning your cluster and worker node setup](https://cloud.ibm.com/docs/containers?topic=containers-plan_clusters#plan_clusters).
|
||||
|
||||
The name of the Kubernetes Node object is the private IP address of the IBM Cloud Kubernetes Service worker node instance.
|
||||
|
||||
### Networking
|
||||
The IBM Cloud Kubernetes Service provider provides VLANs for quality network performance and network isolation for nodes. You can set up custom firewalls and Calico network policies to add an extra layer of security for your cluster, or connect your cluster to your on-prem data center via VPN. For more information, see [Planning in-cluster and private networking](https://console.bluemix.net/docs/containers/cs_network_cluster.html#planning).
|
||||
The IBM Cloud Kubernetes Service provider provides VLANs for quality network performance and network isolation for nodes. You can set up custom firewalls and Calico network policies to add an extra layer of security for your cluster, or connect your cluster to your on-prem data center via VPN. For more information, see [Planning in-cluster and private networking](https://cloud.ibm.com/docs/containers?topic=containers-cs_network_cluster#cs_network_cluster).
|
||||
|
||||
To expose apps to the public or within the cluster, you can leverage NodePort, LoadBalancer, or Ingress services. You can also customize the Ingress application load balancer with annotations. For more information, see [Planning to expose your apps with external networking](https://console.bluemix.net/docs/containers/cs_network_planning.html#planning).
|
||||
To expose apps to the public or within the cluster, you can leverage NodePort, LoadBalancer, or Ingress services. You can also customize the Ingress application load balancer with annotations. For more information, see [Planning to expose your apps with external networking](https://cloud.ibm.com/docs/containers?topic=containers-cs_network_planning#cs_network_planning).
|
||||
|
||||
### Storage
|
||||
The IBM Cloud Kubernetes Service provider leverages Kubernetes-native persistent volumes to enable users to mount file, block, and cloud object storage to their apps. You can also use database-as-a-service and third-party add-ons for persistent storage of your data. For more information, see [Planning highly available persistent storage](https://console.bluemix.net/docs/containers/cs_storage_planning.html#storage_planning).
|
||||
The IBM Cloud Kubernetes Service provider leverages Kubernetes-native persistent volumes to enable users to mount file, block, and cloud object storage to their apps. You can also use database-as-a-service and third-party add-ons for persistent storage of your data. For more information, see [Planning highly available persistent storage](https://cloud.ibm.com/docs/containers?topic=containers-storage_planning#storage_planning).
|
||||
|
||||
## Baidu Cloud Container Engine
|
||||
|
||||
### Node Name
|
||||
|
||||
The Baidu cloud provider uses the private IP address of the node (as determined by the kubelet or overridden with `--hostname-override`) as the name of the Kubernetes Node object.
|
||||
Note that the Kubernetes Node name must match the Baidu VM private IP.
|
||||
|
||||
@@ -6,7 +6,9 @@ weight: 80
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
{{< include "federation-current-state.md" >}}
|
||||
{{< deprecationfilewarning >}}
|
||||
{{< include "federation-deprecation-warning-note.md" >}}
|
||||
{{< /deprecationfilewarning >}}
|
||||
|
||||
This page explains why and how to manage multiple Kubernetes clusters using
|
||||
federation.
|
||||
|
||||
@@ -35,14 +35,14 @@ a container that writes some text to standard output once per second.
|
||||
To run this pod, use the following command:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/examples/debug/counter-pod.yaml
|
||||
kubectl create -f https://k8s.io/examples/debug/counter-pod.yaml
|
||||
pod/counter created
|
||||
```
|
||||
|
||||
To fetch the logs, use the `kubectl logs` command, as follows:
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter
|
||||
kubectl logs counter
|
||||
0: Mon Jan 1 00:00:00 UTC 2001
|
||||
1: Mon Jan 1 00:00:01 UTC 2001
|
||||
2: Mon Jan 1 00:00:02 UTC 2001
|
||||
@@ -76,8 +76,7 @@ and the former approach is used in any other environment. In both cases, by
|
||||
default rotation is configured to take place when log file exceeds 10MB.
|
||||
|
||||
As an example, you can find detailed information about how `kube-up.sh` sets
|
||||
up logging for COS image on GCP in the corresponding [script]
|
||||
[cosConfigureHelper].
|
||||
up logging for COS image on GCP in the corresponding [script][cosConfigureHelper].
|
||||
|
||||
When you run [`kubectl logs`](/docs/reference/generated/kubectl/kubectl-commands#logs) as in
|
||||
the basic logging example, the kubelet on the node handles the request and
|
||||
@@ -89,9 +88,9 @@ only the contents of the latest log file will be available through
|
||||
`kubectl logs`. E.g. if there's a 10MB file, `logrotate` performs
|
||||
the rotation and there are two files, one 10MB in size and one empty,
|
||||
`kubectl logs` will return an empty response.
|
||||
{{< /note >}}
|
||||
|
||||
[cosConfigureHelper]: https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/cluster/gce/gci/configure-helper.sh
|
||||
{{< /note >}}
|
||||
|
||||
### System component logs
|
||||
|
||||
@@ -104,7 +103,7 @@ that do not run in a container. For example:
|
||||
On machines with systemd, the kubelet and container runtime write to journald. If
|
||||
systemd is not present, they write to `.log` files in the `/var/log` directory.
|
||||
System components inside containers always write to the `/var/log` directory,
|
||||
bypassing the default logging mechanism. They use the [glog][glog]
|
||||
bypassing the default logging mechanism. They use the [klog][klog]
|
||||
logging library. You can find the conventions for logging severity for those
|
||||
components in the [development docs on logging](https://git.k8s.io/community/contributors/devel/logging.md).
|
||||
|
||||
@@ -113,7 +112,7 @@ directory should be rotated. In Kubernetes clusters brought up by
|
||||
the `kube-up.sh` script, those logs are configured to be rotated by
|
||||
the `logrotate` tool daily or once the size exceeds 100MB.
|
||||
|
||||
[glog]: https://godoc.org/github.com/golang/glog
|
||||
[klog]: https://github.com/kubernetes/klog
|
||||
|
||||
## Cluster-level logging architectures
|
||||
|
||||
@@ -179,7 +178,9 @@ Now when you run this pod, you can access each log stream separately by
|
||||
running the following commands:
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter count-log-1
|
||||
kubectl logs counter count-log-1
|
||||
```
|
||||
```
|
||||
0: Mon Jan 1 00:00:00 UTC 2001
|
||||
1: Mon Jan 1 00:00:01 UTC 2001
|
||||
2: Mon Jan 1 00:00:02 UTC 2001
|
||||
@@ -187,7 +188,9 @@ $ kubectl logs counter count-log-1
|
||||
```
|
||||
|
||||
```shell
|
||||
$ kubectl logs counter count-log-2
|
||||
kubectl logs counter count-log-2
|
||||
```
|
||||
```
|
||||
Mon Jan 1 00:00:00 UTC 2001 INFO 0
|
||||
Mon Jan 1 00:00:01 UTC 2001 INFO 1
|
||||
Mon Jan 1 00:00:02 UTC 2001 INFO 2
|
||||
|
||||
@@ -26,7 +26,10 @@ Many applications require multiple resources to be created, such as a Deployment
|
||||
Multiple resources can be created the same way as a single resource:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/examples/application/nginx-app.yaml
|
||||
kubectl create -f https://k8s.io/examples/application/nginx-app.yaml
|
||||
```
|
||||
|
||||
```shell
|
||||
service/my-nginx-svc created
|
||||
deployment.apps/my-nginx created
|
||||
```
|
||||
@@ -36,13 +39,13 @@ The resources will be created in the order they appear in the file. Therefore, i
|
||||
`kubectl create` also accepts multiple `-f` arguments:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/examples/application/nginx/nginx-svc.yaml -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml
|
||||
kubectl create -f https://k8s.io/examples/application/nginx/nginx-svc.yaml -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml
|
||||
```
|
||||
|
||||
And a directory can be specified rather than or in addition to individual files:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/examples/application/nginx/
|
||||
kubectl create -f https://k8s.io/examples/application/nginx/
|
||||
```
|
||||
|
||||
`kubectl` will read any files with suffixes `.yaml`, `.yml`, or `.json`.
|
||||
@@ -52,7 +55,10 @@ It is a recommended practice to put resources related to the same microservice o
|
||||
A URL can also be specified as a configuration source, which is handy for deploying directly from configuration files checked into github:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/application/nginx/nginx-deployment.yaml
|
||||
kubectl create -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/application/nginx/nginx-deployment.yaml
|
||||
```
|
||||
|
||||
```shell
|
||||
deployment.apps/my-nginx created
|
||||
```
|
||||
|
||||
@@ -61,7 +67,10 @@ deployment.apps/my-nginx created
|
||||
Resource creation isn't the only operation that `kubectl` can perform in bulk. It can also extract resource names from configuration files in order to perform other operations, in particular to delete the same resources you created:
|
||||
|
||||
```shell
|
||||
$ kubectl delete -f https://k8s.io/examples/application/nginx-app.yaml
|
||||
kubectl delete -f https://k8s.io/examples/application/nginx-app.yaml
|
||||
```
|
||||
|
||||
```shell
|
||||
deployment.apps "my-nginx" deleted
|
||||
service "my-nginx-svc" deleted
|
||||
```
|
||||
@@ -69,13 +78,16 @@ service "my-nginx-svc" deleted
|
||||
In the case of just two resources, it's also easy to specify both on the command line using the resource/name syntax:
|
||||
|
||||
```shell
|
||||
$ kubectl delete deployments/my-nginx services/my-nginx-svc
|
||||
kubectl delete deployments/my-nginx services/my-nginx-svc
|
||||
```
|
||||
|
||||
For larger numbers of resources, you'll find it easier to specify the selector (label query) specified using `-l` or `--selector`, to filter resources by their labels:
|
||||
|
||||
```shell
|
||||
$ kubectl delete deployment,services -l app=nginx
|
||||
kubectl delete deployment,services -l app=nginx
|
||||
```
|
||||
|
||||
```shell
|
||||
deployment.apps "my-nginx" deleted
|
||||
service "my-nginx-svc" deleted
|
||||
```
|
||||
@@ -83,7 +95,10 @@ service "my-nginx-svc" deleted
|
||||
Because `kubectl` outputs resource names in the same syntax it accepts, it's easy to chain operations using `$()` or `xargs`:
|
||||
|
||||
```shell
|
||||
$ kubectl get $(kubectl create -f docs/concepts/cluster-administration/nginx/ -o name | grep service)
|
||||
kubectl get $(kubectl create -f docs/concepts/cluster-administration/nginx/ -o name | grep service)
|
||||
```
|
||||
|
||||
```shell
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
my-nginx-svc LoadBalancer 10.0.0.208 <pending> 80/TCP 0s
|
||||
```
|
||||
@@ -108,14 +123,20 @@ project/k8s/development
|
||||
By default, performing a bulk operation on `project/k8s/development` will stop at the first level of the directory, not processing any subdirectories. If we had tried to create the resources in this directory using the following command, we would have encountered an error:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f project/k8s/development
|
||||
kubectl create -f project/k8s/development
|
||||
```
|
||||
|
||||
```shell
|
||||
error: you must provide one or more resources by argument or filename (.json|.yaml|.yml|stdin)
|
||||
```
|
||||
|
||||
Instead, specify the `--recursive` or `-R` flag with the `--filename,-f` flag as such:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f project/k8s/development --recursive
|
||||
kubectl create -f project/k8s/development --recursive
|
||||
```
|
||||
|
||||
```shell
|
||||
configmap/my-config created
|
||||
deployment.apps/my-deployment created
|
||||
persistentvolumeclaim/my-pvc created
|
||||
@@ -126,7 +147,10 @@ The `--recursive` flag works with any operation that accepts the `--filename,-f`
|
||||
The `--recursive` flag also works when multiple `-f` arguments are provided:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f project/k8s/namespaces -f project/k8s/development --recursive
|
||||
kubectl create -f project/k8s/namespaces -f project/k8s/development --recursive
|
||||
```
|
||||
|
||||
```shell
|
||||
namespace/development created
|
||||
namespace/staging created
|
||||
configmap/my-config created
|
||||
@@ -169,8 +193,11 @@ and
|
||||
The labels allow us to slice and dice our resources along any dimension specified by a label:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml
|
||||
$ kubectl get pods -Lapp -Ltier -Lrole
|
||||
kubectl create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml
|
||||
kubectl get pods -Lapp -Ltier -Lrole
|
||||
```
|
||||
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE APP TIER ROLE
|
||||
guestbook-fe-4nlpb 1/1 Running 0 1m guestbook frontend <none>
|
||||
guestbook-fe-ght6d 1/1 Running 0 1m guestbook frontend <none>
|
||||
@@ -180,7 +207,12 @@ guestbook-redis-slave-2q2yf 1/1 Running 0 1m guestboo
|
||||
guestbook-redis-slave-qgazl 1/1 Running 0 1m guestbook backend slave
|
||||
my-nginx-divi2 1/1 Running 0 29m nginx <none> <none>
|
||||
my-nginx-o0ef1 1/1 Running 0 29m nginx <none> <none>
|
||||
$ kubectl get pods -lapp=guestbook,role=slave
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl get pods -lapp=guestbook,role=slave
|
||||
```
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
guestbook-redis-slave-2q2yf 1/1 Running 0 3m
|
||||
guestbook-redis-slave-qgazl 1/1 Running 0 3m
|
||||
@@ -240,7 +272,10 @@ Sometimes existing pods and other resources need to be relabeled before creating
|
||||
For example, if you want to label all your nginx pods as frontend tier, simply run:
|
||||
|
||||
```shell
|
||||
$ kubectl label pods -l app=nginx tier=fe
|
||||
kubectl label pods -l app=nginx tier=fe
|
||||
```
|
||||
|
||||
```shell
|
||||
pod/my-nginx-2035384211-j5fhi labeled
|
||||
pod/my-nginx-2035384211-u2c7e labeled
|
||||
pod/my-nginx-2035384211-u3t6x labeled
|
||||
@@ -250,7 +285,9 @@ This first filters all pods with the label "app=nginx", and then labels them wit
|
||||
To see the pods you just labeled, run:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l app=nginx -L tier
|
||||
kubectl get pods -l app=nginx -L tier
|
||||
```
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE TIER
|
||||
my-nginx-2035384211-j5fhi 1/1 Running 0 23m fe
|
||||
my-nginx-2035384211-u2c7e 1/1 Running 0 23m fe
|
||||
@@ -266,8 +303,10 @@ For more information, please see [labels](/docs/concepts/overview/working-with-o
|
||||
Sometimes you would want to attach annotations to resources. Annotations are arbitrary non-identifying metadata for retrieval by API clients such as tools, libraries, etc. This can be done with `kubectl annotate`. For example:
|
||||
|
||||
```shell
|
||||
$ kubectl annotate pods my-nginx-v4-9gw19 description='my frontend running nginx'
|
||||
$ kubectl get pods my-nginx-v4-9gw19 -o yaml
|
||||
kubectl annotate pods my-nginx-v4-9gw19 description='my frontend running nginx'
|
||||
kubectl get pods my-nginx-v4-9gw19 -o yaml
|
||||
```
|
||||
```shell
|
||||
apiversion: v1
|
||||
kind: pod
|
||||
metadata:
|
||||
@@ -283,14 +322,18 @@ For more information, please see [annotations](/docs/concepts/overview/working-w
|
||||
When load on your application grows or shrinks, it's easy to scale with `kubectl`. For instance, to decrease the number of nginx replicas from 3 to 1, do:
|
||||
|
||||
```shell
|
||||
$ kubectl scale deployment/my-nginx --replicas=1
|
||||
kubectl scale deployment/my-nginx --replicas=1
|
||||
```
|
||||
```shell
|
||||
deployment.extensions/my-nginx scaled
|
||||
```
|
||||
|
||||
Now you only have one pod managed by the deployment.
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l app=nginx
|
||||
kubectl get pods -l app=nginx
|
||||
```
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
my-nginx-2035384211-j5fhi 1/1 Running 0 30m
|
||||
```
|
||||
@@ -298,7 +341,9 @@ my-nginx-2035384211-j5fhi 1/1 Running 0 30m
|
||||
To have the system automatically choose the number of nginx replicas as needed, ranging from 1 to 3, do:
|
||||
|
||||
```shell
|
||||
$ kubectl autoscale deployment/my-nginx --min=1 --max=3
|
||||
kubectl autoscale deployment/my-nginx --min=1 --max=3
|
||||
```
|
||||
```shell
|
||||
horizontalpodautoscaler.autoscaling/my-nginx autoscaled
|
||||
```
|
||||
|
||||
@@ -320,7 +365,9 @@ Then, you can use [`kubectl apply`](/docs/reference/generated/kubectl/kubectl-co
|
||||
This command will compare the version of the configuration that you're pushing with the previous version and apply the changes you've made, without overwriting any automated changes to properties you haven't specified.
|
||||
|
||||
```shell
|
||||
$ kubectl apply -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml
|
||||
kubectl apply -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml
|
||||
```
|
||||
```shell
|
||||
deployment.apps/my-nginx configured
|
||||
```
|
||||
|
||||
@@ -339,18 +386,20 @@ To use apply, always create resource initially with either `kubectl apply` or `k
|
||||
Alternatively, you may also update resources with `kubectl edit`:
|
||||
|
||||
```shell
|
||||
$ kubectl edit deployment/my-nginx
|
||||
kubectl edit deployment/my-nginx
|
||||
```
|
||||
|
||||
This is equivalent to first `get` the resource, edit it in text editor, and then `apply` the resource with the updated version:
|
||||
|
||||
```shell
|
||||
$ kubectl get deployment my-nginx -o yaml > /tmp/nginx.yaml
|
||||
$ vi /tmp/nginx.yaml
|
||||
kubectl get deployment my-nginx -o yaml > /tmp/nginx.yaml
|
||||
vi /tmp/nginx.yaml
|
||||
# do some edit, and then save the file
|
||||
$ kubectl apply -f /tmp/nginx.yaml
|
||||
|
||||
kubectl apply -f /tmp/nginx.yaml
|
||||
deployment.apps/my-nginx configured
|
||||
$ rm /tmp/nginx.yaml
|
||||
|
||||
rm /tmp/nginx.yaml
|
||||
```
|
||||
|
||||
This allows you to do more significant changes more easily. Note that you can specify the editor with your `EDITOR` or `KUBE_EDITOR` environment variables.
|
||||
@@ -370,7 +419,9 @@ and
|
||||
In some cases, you may need to update resource fields that cannot be updated once initialized, or you may just want to make a recursive change immediately, such as to fix broken pods created by a Deployment. To change such fields, use `replace --force`, which deletes and re-creates the resource. In this case, you can simply modify your original configuration file:
|
||||
|
||||
```shell
|
||||
$ kubectl replace -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml --force
|
||||
kubectl replace -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml --force
|
||||
```
|
||||
```shell
|
||||
deployment.apps/my-nginx deleted
|
||||
deployment.apps/my-nginx replaced
|
||||
```
|
||||
@@ -385,14 +436,16 @@ you should read [how to use `kubectl rolling-update`](/docs/tasks/run-applicatio
|
||||
Let's say you were running version 1.7.9 of nginx:
|
||||
|
||||
```shell
|
||||
$ kubectl run my-nginx --image=nginx:1.7.9 --replicas=3
|
||||
kubectl run my-nginx --image=nginx:1.7.9 --replicas=3
|
||||
```
|
||||
```shell
|
||||
deployment.apps/my-nginx created
|
||||
```
|
||||
|
||||
To update to version 1.9.1, simply change `.spec.template.spec.containers[0].image` from `nginx:1.7.9` to `nginx:1.9.1`, with the kubectl commands we learned above.
|
||||
|
||||
```shell
|
||||
$ kubectl edit deployment/my-nginx
|
||||
kubectl edit deployment/my-nginx
|
||||
```
|
||||
|
||||
That's it! The Deployment will declaratively update the deployed nginx application progressively behind the scene. It ensures that only a certain number of old replicas may be down while they are being updated, and only a certain number of new replicas may be created above the desired number of pods. To learn more details about it, visit [Deployment page](/docs/concepts/workloads/controllers/deployment/).
|
||||
|
||||
@@ -7,8 +7,9 @@ weight: 50
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
Kubernetes approaches networking somewhat differently than Docker does by
|
||||
default. There are 4 distinct networking problems to solve:
|
||||
Networking is a central part of Kubernetes, but it can be challenging to
|
||||
understand exactly how it is expected to work. There are 4 distinct networking
|
||||
problems to address:
|
||||
|
||||
1. Highly-coupled container-to-container communications: this is solved by
|
||||
[pods](/docs/concepts/workloads/pods/pod/) and `localhost` communications.
|
||||
@@ -21,80 +22,56 @@ default. There are 4 distinct networking problems to solve:
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
Kubernetes assumes that pods can communicate with other pods, regardless of
|
||||
which host they land on. Every pod gets its own IP address so you do not
|
||||
need to explicitly create links between pods and you almost never need to deal
|
||||
with mapping container ports to host ports. This creates a clean,
|
||||
backwards-compatible model where pods can be treated much like VMs or physical
|
||||
hosts from the perspectives of port allocation, naming, service discovery, load
|
||||
balancing, application configuration, and migration.
|
||||
Kubernetes is all about sharing machines between applications. Typically,
|
||||
sharing machines requires ensuring that two applications do not try to use the
|
||||
same ports. Coordinating ports across multiple developers is very difficult to
|
||||
do at scale and exposes users to cluster-level issues outside of their control.
|
||||
|
||||
There are requirements imposed on how you set up your cluster networking to
|
||||
achieve this.
|
||||
|
||||
## Docker model
|
||||
|
||||
Before discussing the Kubernetes approach to networking, it is worthwhile to
|
||||
review the "normal" way that networking works with Docker. By default, Docker
|
||||
uses host-private networking. It creates a virtual bridge, called `docker0` by
|
||||
default, and allocates a subnet from one of the private address blocks defined
|
||||
in [RFC1918](https://tools.ietf.org/html/rfc1918) for that bridge. For each
|
||||
container that Docker creates, it allocates a virtual Ethernet device (called
|
||||
`veth`) which is attached to the bridge. The veth is mapped to appear as `eth0`
|
||||
in the container, using Linux namespaces. The in-container `eth0` interface is
|
||||
given an IP address from the bridge's address range.
|
||||
|
||||
The result is that Docker containers can talk to other containers only if they
|
||||
are on the same machine (and thus the same virtual bridge). Containers on
|
||||
different machines can not reach each other - in fact they may end up with the
|
||||
exact same network ranges and IP addresses.
|
||||
|
||||
In order for Docker containers to communicate across nodes, there must
|
||||
be allocated ports on the machine’s own IP address, which are then
|
||||
forwarded or proxied to the containers. This obviously means that
|
||||
containers must either coordinate which ports they use very carefully
|
||||
or ports must be allocated dynamically.
|
||||
|
||||
## Kubernetes model
|
||||
|
||||
Coordinating ports across multiple developers is very difficult to do at
|
||||
scale and exposes users to cluster-level issues outside of their control.
|
||||
Dynamic port allocation brings a lot of complications to the system - every
|
||||
application has to take ports as flags, the API servers have to know how to
|
||||
insert dynamic port numbers into configuration blocks, services have to know
|
||||
how to find each other, etc. Rather than deal with this, Kubernetes takes a
|
||||
different approach.
|
||||
|
||||
## The Kubernetes network model
|
||||
|
||||
Every `Pod` gets its own IP address. This means you do not need to explicitly
|
||||
create links between `Pods` and you almost never need to deal with mapping
|
||||
container ports to host ports. This creates a clean, backwards-compatible
|
||||
model where `Pods` can be treated much like VMs or physical hosts from the
|
||||
perspectives of port allocation, naming, service discovery, load balancing,
|
||||
application configuration, and migration.
|
||||
|
||||
Kubernetes imposes the following fundamental requirements on any networking
|
||||
implementation (barring any intentional network segmentation policies):
|
||||
|
||||
* all containers can communicate with all other containers without NAT
|
||||
* all nodes can communicate with all containers (and vice-versa) without NAT
|
||||
* the IP that a container sees itself as is the same IP that others see it as
|
||||
* pods on a node can communicate with all pods on all nodes without NAT
|
||||
* agents on a node (e.g. system daemons, kubelet) can communicate with all
|
||||
pods on that node
|
||||
|
||||
What this means in practice is that you can not just take two computers
|
||||
running Docker and expect Kubernetes to work. You must ensure that the
|
||||
fundamental requirements are met.
|
||||
Note: For those platforms that support `Pods` running in the host network (e.g.
|
||||
Linux):
|
||||
|
||||
* pods in the host network of a node can communicate with all pods on all
|
||||
nodes without NAT
|
||||
|
||||
This model is not only less complex overall, but it is principally compatible
|
||||
with the desire for Kubernetes to enable low-friction porting of apps from VMs
|
||||
to containers. If your job previously ran in a VM, your VM had an IP and could
|
||||
talk to other VMs in your project. This is the same basic model.
|
||||
|
||||
Until now this document has talked about containers. In reality, Kubernetes
|
||||
applies IP addresses at the `Pod` scope - containers within a `Pod` share their
|
||||
network namespaces - including their IP address. This means that containers
|
||||
within a `Pod` can all reach each other's ports on `localhost`. This does imply
|
||||
that containers within a `Pod` must coordinate port usage, but this is no
|
||||
different than processes in a VM. This is called the "IP-per-pod" model. This
|
||||
is implemented, using Docker, as a "pod container" which holds the network namespace
|
||||
open while "app containers" (the things the user specified) join that namespace
|
||||
with Docker's `--net=container:<id>` function.
|
||||
Kubernetes IP addresses exist at the `Pod` scope - containers within a `Pod`
|
||||
share their network namespaces - including their IP address. This means that
|
||||
containers within a `Pod` can all reach each other's ports on `localhost`. This
|
||||
also means that containers within a `Pod` must coordinate port usage, but this
|
||||
is no different than processes in a VM. This is called the "IP-per-pod" model.
|
||||
|
||||
As with Docker, it is possible to request host ports, but this is reduced to a
|
||||
very niche operation. In this case a port will be allocated on the host `Node`
|
||||
and traffic will be forwarded to the `Pod`. The `Pod` itself is blind to the
|
||||
existence or non-existence of host ports.
|
||||
How this is implemented is a detail of the particular container runtime in use.
|
||||
|
||||
It is possible to request ports on the `Node` itself which forward to your `Pod`
|
||||
(called host ports), but this is a very niche operation. How that forwarding is
|
||||
implemented is also a detail of the container runtime. The `Pod` itself is
|
||||
blind to the existence or non-existence of host ports.
|
||||
|
||||
## How to implement the Kubernetes networking model
|
||||
|
||||
@@ -125,7 +102,7 @@ Details on how the AOS system works can be accessed here: http://www.apstra.com/
|
||||
|
||||
[Big Cloud Fabric](https://www.bigswitch.com/container-network-automation) is a cloud native networking architecture, designed to run Kubernetes in private cloud/on-premises environments. Using unified physical & virtual SDN, Big Cloud Fabric tackles inherent container networking problems such as load balancing, visibility, troubleshooting, security policies & container traffic monitoring.
|
||||
|
||||
With the help of the Big Cloud Fabric's virtual pod multi-tenant architecture, container orchestration systems such as Kubernetes, RedHat Openshift, Mesosphere DC/OS & Docker Swarm will be natively integrated along side with VM orchestration systems such as VMware, OpenStack & Nutanix. Customers will be able to securely inter-connect any number of these clusters and enable inter-tenant communication between them if needed.
|
||||
With the help of the Big Cloud Fabric's virtual pod multi-tenant architecture, container orchestration systems such as Kubernetes, RedHat OpenShift, Mesosphere DC/OS & Docker Swarm will be natively integrated along side with VM orchestration systems such as VMware, OpenStack & Nutanix. Customers will be able to securely inter-connect any number of these clusters and enable inter-tenant communication between them if needed.
|
||||
|
||||
BCF was recognized by Gartner as a visionary in the latest [Magic Quadrant](http://go.bigswitch.com/17GatedDocuments-MagicQuadrantforDataCenterNetworking_Reg.html). One of the BCF Kubernetes on-premises deployments (which includes Kubernetes, DC/OS & VMware running on multiple DCs across different geographic regions) is also referenced [here](https://portworx.com/architects-corner-kubernetes-satya-komala-nio/).
|
||||
|
||||
@@ -143,13 +120,29 @@ addressing.
|
||||
|
||||
CNI-Genie also supports [assigning multiple IP addresses to a pod](https://github.com/Huawei-PaaS/CNI-Genie/blob/master/docs/multiple-ips/README.md#feature-2-extension-cni-genie-multiple-ip-addresses-per-pod), each from a different CNI plugin.
|
||||
|
||||
### cni-ipvlan-vpc-k8s
|
||||
[cni-ipvlan-vpc-k8s](https://github.com/lyft/cni-ipvlan-vpc-k8s) contains a set
|
||||
of CNI and IPAM plugins to provide a simple, host-local, low latency, high
|
||||
throughput, and compliant networking stack for Kubernetes within Amazon Virtual
|
||||
Private Cloud (VPC) environments by making use of Amazon Elastic Network
|
||||
Interfaces (ENI) and binding AWS-managed IPs into Pods using the Linux kernel's
|
||||
IPvlan driver in L2 mode.
|
||||
|
||||
The plugins are designed to be straightforward to configure and deploy within a
|
||||
VPC. Kubelets boot and then self-configure and scale their IP usage as needed
|
||||
without requiring the often recommended complexities of administering overlay
|
||||
networks, BGP, disabling source/destination checks, or adjusting VPC route
|
||||
tables to provide per-instance subnets to each host (which is limited to 50-100
|
||||
entries per VPC). In short, cni-ipvlan-vpc-k8s significantly reduces the
|
||||
network complexity required to deploy Kubernetes at scale within AWS.
|
||||
|
||||
### Contiv
|
||||
|
||||
[Contiv](https://github.com/contiv/netplugin) provides configurable networking (native l3 using BGP, overlay using vxlan, classic l2, or Cisco-SDN/ACI) for various use cases. [Contiv](http://contiv.io) is all open sourced.
|
||||
|
||||
### Contrail
|
||||
### Contrail / Tungsten Fabric
|
||||
|
||||
[Contrail](http://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [OpenContrail](http://www.opencontrail.org), is a truly open, multi-cloud network virtualization and policy management platform. Contrail / OpenContrail is integrated with various orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provides different isolation modes for virtual machines, containers/pods and bare metal workloads.
|
||||
[Contrail](http://www.juniper.net/us/en/products-services/sdn/contrail/contrail-networking/), based on [Tungsten Fabric](https://tungsten.io), is a truly open, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with various orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide different isolation modes for virtual machines, containers/pods and bare metal workloads.
|
||||
|
||||
### DANM
|
||||
|
||||
@@ -246,7 +239,7 @@ Multus supports all [reference plugins](https://github.com/containernetworking/p
|
||||
|
||||
[VMware NSX-T](https://docs.vmware.com/en/VMware-NSX-T/index.html) is a network virtualization and security platform. NSX-T can provide network virtualization for a multi-cloud and multi-hypervisor environment and is focused on emerging application frameworks and architectures that have heterogeneous endpoints and technology stacks. In addition to vSphere hypervisors, these environments include other hypervisors such as KVM, containers, and bare metal.
|
||||
|
||||
[NSX-T Container Plug-in (NCP)](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) provides integration between NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and Openshift.
|
||||
[NSX-T Container Plug-in (NCP)](https://docs.vmware.com/en/VMware-NSX-T/2.0/nsxt_20_ncp_kubernetes.pdf) provides integration between NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.
|
||||
|
||||
### Nuage Networks VCS (Virtualized Cloud Services)
|
||||
|
||||
@@ -272,9 +265,9 @@ at [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes).
|
||||
|
||||
[Project Calico](http://docs.projectcalico.org/) is an open source container networking provider and network policy engine.
|
||||
|
||||
Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet. Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking. Calico also provides fine-grained, intent based network security policy for Kubernetes pods via its distributed firewall.
|
||||
Calico provides a highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet, for both Linux (open source) and Windows (proprietary - available from [Tigera](https://www.tigera.io/essentials/)). Calico can be deployed without encapsulation or overlays to provide high-performance, high-scale data center networking. Calico also provides fine-grained, intent based network security policy for Kubernetes pods via its distributed firewall.
|
||||
|
||||
Calico can also be run in policy enforcement mode in conjunction with other networking solutions such as Flannel, aka [canal](https://github.com/tigera/canal), or native GCE networking.
|
||||
Calico can also be run in policy enforcement mode in conjunction with other networking solutions such as Flannel, aka [canal](https://github.com/tigera/canal), or native GCE, AWS or Azure networking.
|
||||
|
||||
### Romana
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ weight: 30
|
||||
{{% capture overview %}}
|
||||
|
||||
You can constrain a [pod](/docs/concepts/workloads/pods/pod/) to only be able to run on particular [nodes](/docs/concepts/architecture/nodes/) or to prefer to
|
||||
run on particular nodes. There are several ways to do this, and they all use
|
||||
run on particular nodes. There are several ways to do this, and the recommended approaches all use
|
||||
[label selectors](/docs/concepts/overview/working-with-objects/labels/) to make the selection.
|
||||
Generally such constraints are unnecessary, as the scheduler will automatically do a reasonable placement
|
||||
(e.g. spread your pods across nodes, not place the pod on a node with insufficient free resources, etc.)
|
||||
@@ -29,7 +29,7 @@ repo here](https://github.com/kubernetes/website/tree/{{< param "docsbranch" >}}
|
||||
|
||||
## nodeSelector
|
||||
|
||||
`nodeSelector` is the simplest form of constraint.
|
||||
`nodeSelector` is the simplest recommended form of node selection constraint.
|
||||
`nodeSelector` is a field of PodSpec. It specifies a map of key-value pairs. For the pod to be eligible
|
||||
to run on a node, the node must have each of the indicated key-value pairs as labels (it can have
|
||||
additional labels as well). The most common usage is one key-value pair.
|
||||
@@ -46,7 +46,7 @@ Run `kubectl get nodes` to get the names of your cluster's nodes. Pick out the o
|
||||
|
||||
If this fails with an "invalid command" error, you're likely using an older version of kubectl that doesn't have the `label` command. In that case, see the [previous version](https://github.com/kubernetes/kubernetes/blob/a053dbc313572ed60d89dae9821ecab8bfd676dc/examples/node-selection/README.md) of this guide for instructions on how to manually set labels on a node.
|
||||
|
||||
You can verify that it worked by re-running `kubectl get nodes --show-labels` and checking that the node now has a label.
|
||||
You can verify that it worked by re-running `kubectl get nodes --show-labels` and checking that the node now has a label. You can also use `kubectl describe node "nodename"` to see the full list of labels of the given node.
|
||||
|
||||
### Step Two: Add a nodeSelector field to your pod configuration
|
||||
|
||||
@@ -362,6 +362,41 @@ For more information on inter-pod affinity/anti-affinity, see the
|
||||
You may want to check [Taints](/docs/concepts/configuration/taint-and-toleration/)
|
||||
as well, which allow a *node* to *repel* a set of pods.
|
||||
|
||||
## nodeName
|
||||
|
||||
`nodeName` is the simplest form of node selection constraint, but due
|
||||
to its limitations it is typically not used. `nodeName` is a field of
|
||||
PodSpec. If it is non-empty, the scheduler ignores the pod and the
|
||||
kubelet running on the named node tries to run the pod. Thus, if
|
||||
`nodeName` is provided in the PodSpec, it takes precedence over the
|
||||
above methods for node selection.
|
||||
|
||||
Some of the limitations of using `nodeName` to select nodes are:
|
||||
|
||||
- If the named node does not exist, the pod will not be run, and in
|
||||
some cases may be automatically deleted.
|
||||
- If the named node does not have the resources to accommodate the
|
||||
pod, the pod will fail and its reason will indicate why,
|
||||
e.g. OutOfmemory or OutOfcpu.
|
||||
- Node names in cloud environments are not always predictable or
|
||||
stable.
|
||||
|
||||
Here is an example of a pod config file using the `nodeName` field:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: nginx
|
||||
spec:
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx
|
||||
nodeName: kube-01
|
||||
```
|
||||
|
||||
The above pod will run on the node kube-01.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
@@ -189,7 +189,9 @@ unscheduled until a place can be found. An event is produced each time the
|
||||
scheduler fails to find a place for the Pod, like this:
|
||||
|
||||
```shell
|
||||
$ kubectl describe pod frontend | grep -A 3 Events
|
||||
kubectl describe pod frontend | grep -A 3 Events
|
||||
```
|
||||
```
|
||||
Events:
|
||||
FirstSeen LastSeen Count From Subobject PathReason Message
|
||||
36s 5s 6 {scheduler } FailedScheduling Failed for reason PodExceedsFreeCPU and possibly others
|
||||
@@ -210,7 +212,9 @@ You can check node capacities and amounts allocated with the
|
||||
`kubectl describe nodes` command. For example:
|
||||
|
||||
```shell
|
||||
$ kubectl describe nodes e2e-test-minion-group-4lw4
|
||||
kubectl describe nodes e2e-test-minion-group-4lw4
|
||||
```
|
||||
```
|
||||
Name: e2e-test-minion-group-4lw4
|
||||
[ ... lines removed for clarity ...]
|
||||
Capacity:
|
||||
@@ -260,7 +264,9 @@ whether a Container is being killed because it is hitting a resource limit, call
|
||||
`kubectl describe pod` on the Pod of interest:
|
||||
|
||||
```shell
|
||||
[12:54:41] $ kubectl describe pod simmemleak-hra99
|
||||
kubectl describe pod simmemleak-hra99
|
||||
```
|
||||
```
|
||||
Name: simmemleak-hra99
|
||||
Namespace: default
|
||||
Image(s): saadali/simmemleak
|
||||
@@ -304,7 +310,9 @@ You can call `kubectl get pod` with the `-o go-template=...` option to fetch the
|
||||
of previously terminated Containers:
|
||||
|
||||
```shell
|
||||
[13:59:01] $ kubectl get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-hra99
|
||||
kubectl get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}' simmemleak-hra99
|
||||
```
|
||||
```
|
||||
Container Name: simmemleak
|
||||
LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]]
|
||||
```
|
||||
|
||||
@@ -103,7 +103,7 @@ Here are the rules that `kubectl` uses when it merges kubeconfig files:
|
||||
|
||||
1. Determine the context to use based on the first hit in this chain:
|
||||
|
||||
1. Use the `--context` command-line flag if it exits.
|
||||
1. Use the `--context` command-line flag if it exists.
|
||||
1. Use the `current-context` from the merged kubeconfig files.
|
||||
|
||||
An empty context is allowed at this point.
|
||||
|
||||
@@ -202,7 +202,7 @@ Node is found that satisfies all the specified requirements of the Pod,
|
||||
preemption logic is triggered for the pending Pod. Let's call the pending Pod P.
|
||||
Preemption logic tries to find a Node where removal of one or more Pods with
|
||||
lower priority than P would enable P to be scheduled on that Node. If such a
|
||||
Node is found, one or more lower priority Pods get deleted from the Node. After
|
||||
Node is found, one or more lower priority Pods get evicted from the Node. After
|
||||
the Pods are gone, P can be scheduled on the Node.
|
||||
|
||||
### User exposed information
|
||||
@@ -322,7 +322,7 @@ When a Pod is preempted, there will be events recorded for the preempted Pod.
|
||||
Preemption should happen only when a cluster does not have enough resources for
|
||||
a Pod. In such cases, preemption happens only when the priority of the pending
|
||||
Pod (preemptor) is higher than the victim Pods. Preemption must not happen when
|
||||
there is no pending Pod, or when the pending Pods have equal or higher priority
|
||||
there is no pending Pod, or when the pending Pods have equal or lower priority
|
||||
than the victims. If preemption happens in such scenarios, please file an issue.
|
||||
|
||||
#### Pods are preempted, but the preemptor is not scheduled
|
||||
@@ -364,11 +364,11 @@ Pod priority and
|
||||
[QoS](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md)
|
||||
are two orthogonal features with few interactions and no default restrictions on
|
||||
setting the priority of a Pod based on its QoS classes. The scheduler's
|
||||
preemption logic does consider QoS when choosing preemption targets. Preemption
|
||||
considers Pod priority and attempts to choose a set of targets with the lowest
|
||||
priority. Higher-priority Pods are considered for preemption only if the removal
|
||||
of the lowest priority Pods is not sufficient to allow the scheduler to schedule
|
||||
the preemptor Pod, or if the lowest priority Pods are protected by
|
||||
preemption logic does not consider QoS when choosing preemption targets.
|
||||
Preemption considers Pod priority and attempts to choose a set of targets with
|
||||
the lowest priority. Higher-priority Pods are considered for preemption only if
|
||||
the removal of the lowest priority Pods is not sufficient to allow the scheduler
|
||||
to schedule the preemptor Pod, or if the lowest priority Pods are protected by
|
||||
`PodDisruptionBudget`.
|
||||
|
||||
The only component that considers both QoS and Pod priority is
|
||||
|
||||
@@ -13,10 +13,10 @@ weight: 50
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
Objects of type `secret` are intended to hold sensitive information, such as
|
||||
passwords, OAuth tokens, and ssh keys. Putting this information in a `secret`
|
||||
is safer and more flexible than putting it verbatim in a `pod` definition or in
|
||||
a docker image. See [Secrets design document](https://git.k8s.io/community/contributors/design-proposals/auth/secrets.md) for more information.
|
||||
Kubernetes `secret` objects let you store and manage sensitive information, such
|
||||
as passwords, OAuth tokens, and ssh keys. Putting this information in a `secret`
|
||||
is safer and more flexible than putting it verbatim in a
|
||||
{{< glossary_tooltip term_id="pod" >}} definition or in a {{< glossary_tooltip text="container image" term_id="image" >}}. See [Secrets design document](https://git.k8s.io/community/contributors/design-proposals/auth/secrets.md) for more information.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -32,7 +32,8 @@ more control over how it is used, and reduces the risk of accidental exposure.
|
||||
Users can create secrets, and the system also creates some secrets.
|
||||
|
||||
To use a secret, a pod needs to reference the secret.
|
||||
A secret can be used with a pod in two ways: as files in a [volume](/docs/concepts/storage/volumes/) mounted on one or more of
|
||||
A secret can be used with a pod in two ways: as files in a
|
||||
{{< glossary_tooltip text="volume" term_id="volume" >}} mounted on one or more of
|
||||
its containers, or used by kubelet when pulling images for the pod.
|
||||
|
||||
### Built-in Secrets
|
||||
@@ -60,8 +61,8 @@ username and password that the pods should use is in the files
|
||||
|
||||
```shell
|
||||
# Create files needed for rest of example.
|
||||
$ echo -n 'admin' > ./username.txt
|
||||
$ echo -n '1f2d1e2e67df' > ./password.txt
|
||||
echo -n 'admin' > ./username.txt
|
||||
echo -n '1f2d1e2e67df' > ./password.txt
|
||||
```
|
||||
|
||||
The `kubectl create secret` command
|
||||
@@ -69,18 +70,25 @@ packages these files into a Secret and creates
|
||||
the object on the Apiserver.
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt
|
||||
kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt
|
||||
```
|
||||
```
|
||||
secret "db-user-pass" created
|
||||
```
|
||||
|
||||
You can check that the secret was created like this:
|
||||
|
||||
```shell
|
||||
$ kubectl get secrets
|
||||
kubectl get secrets
|
||||
```
|
||||
```
|
||||
NAME TYPE DATA AGE
|
||||
db-user-pass Opaque 2 51s
|
||||
|
||||
$ kubectl describe secrets/db-user-pass
|
||||
```
|
||||
```shell
|
||||
kubectl describe secrets/db-user-pass
|
||||
```
|
||||
```
|
||||
Name: db-user-pass
|
||||
Namespace: default
|
||||
Labels: <none>
|
||||
@@ -94,11 +102,14 @@ password.txt: 12 bytes
|
||||
username.txt: 5 bytes
|
||||
```
|
||||
|
||||
Note that neither `get` nor `describe` shows the contents of the file by default.
|
||||
This is to protect the secret from being exposed accidentally to someone looking
|
||||
{{< note >}}
|
||||
`kubectl get` and `kubectl describe` avoid showing the contents of a secret by
|
||||
default.
|
||||
This is to protect the secret from being exposed accidentally to an onlooker,
|
||||
or from being stored in a terminal log.
|
||||
{{< /note >}}
|
||||
|
||||
See [decoding a secret](#decoding-a-secret) for how to see the contents.
|
||||
See [decoding a secret](#decoding-a-secret) for how to see the contents of a secret.
|
||||
|
||||
#### Creating a Secret Manually
|
||||
|
||||
@@ -135,7 +146,9 @@ data:
|
||||
Now create the Secret using [`kubectl create`](/docs/reference/generated/kubectl/kubectl-commands#create):
|
||||
|
||||
```shell
|
||||
$ kubectl create -f ./secret.yaml
|
||||
kubectl create -f ./secret.yaml
|
||||
```
|
||||
```
|
||||
secret "mysecret" created
|
||||
```
|
||||
|
||||
@@ -246,7 +259,9 @@ the option `-w 0` to `base64` commands or the pipeline `base64 | tr -d '\n'` if
|
||||
Secrets can be retrieved via the `kubectl get secret` command. For example, to retrieve the secret created in the previous section:
|
||||
|
||||
```shell
|
||||
$ kubectl get secret mysecret -o yaml
|
||||
kubectl get secret mysecret -o yaml
|
||||
```
|
||||
```
|
||||
apiVersion: v1
|
||||
data:
|
||||
username: YWRtaW4=
|
||||
@@ -265,14 +280,17 @@ type: Opaque
|
||||
Decode the password field:
|
||||
|
||||
```shell
|
||||
$ echo 'MWYyZDFlMmU2N2Rm' | base64 --decode
|
||||
echo 'MWYyZDFlMmU2N2Rm' | base64 --decode
|
||||
```
|
||||
```
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
### Using Secrets
|
||||
|
||||
Secrets can be mounted as data volumes or be exposed as environment variables to
|
||||
be used by a container in a pod. They can also be used by other parts of the
|
||||
Secrets can be mounted as data volumes or be exposed as
|
||||
{{< glossary_tooltip text="environment variables" term_id="container-env-variables" >}}
|
||||
to be used by a container in a pod. They can also be used by other parts of the
|
||||
system, without being directly exposed to the pod. For example, they can hold
|
||||
credentials that other parts of the system should use to interact with external
|
||||
systems on your behalf.
|
||||
@@ -424,12 +442,22 @@ This is the result of commands
|
||||
executed inside the container from the example above:
|
||||
|
||||
```shell
|
||||
$ ls /etc/foo/
|
||||
ls /etc/foo/
|
||||
```
|
||||
```
|
||||
username
|
||||
password
|
||||
$ cat /etc/foo/username
|
||||
```
|
||||
```shell
|
||||
cat /etc/foo/username
|
||||
```
|
||||
```
|
||||
admin
|
||||
$ cat /etc/foo/password
|
||||
```
|
||||
```shell
|
||||
cat /etc/foo/password
|
||||
```
|
||||
```
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
@@ -458,7 +486,8 @@ Secret updates.
|
||||
|
||||
#### Using Secrets as Environment Variables
|
||||
|
||||
To use a secret in an environment variable in a pod:
|
||||
To use a secret in an {{< glossary_tooltip text="environment variable" term_id="container-env-variables" >}}
|
||||
in a pod:
|
||||
|
||||
1. Create a secret or use an existing one. Multiple pods can reference the same secret.
|
||||
1. Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret's name and key in `env[].valueFrom.secretKeyRef`.
|
||||
@@ -496,9 +525,15 @@ normal environment variables containing the base-64 decoded values of the secret
|
||||
This is the result of commands executed inside the container from the example above:
|
||||
|
||||
```shell
|
||||
$ echo $SECRET_USERNAME
|
||||
echo $SECRET_USERNAME
|
||||
```
|
||||
```
|
||||
admin
|
||||
$ echo $SECRET_PASSWORD
|
||||
```
|
||||
```shell
|
||||
echo $SECRET_PASSWORD
|
||||
```
|
||||
```
|
||||
1f2d1e2e67df
|
||||
```
|
||||
|
||||
@@ -534,10 +569,10 @@ Secret volume sources are validated to ensure that the specified object
|
||||
reference actually points to an object of type `Secret`. Therefore, a secret
|
||||
needs to be created before any pods that depend on it.
|
||||
|
||||
Secret API objects reside in a namespace. They can only be referenced by pods
|
||||
in that same namespace.
|
||||
Secret API objects reside in a {{< glossary_tooltip text="namespace" term_id="namespace" >}}.
|
||||
They can only be referenced by pods in that same namespace.
|
||||
|
||||
Individual secrets are limited to 1MB in size. This is to discourage creation
|
||||
Individual secrets are limited to 1MiB in size. This is to discourage creation
|
||||
of very large secrets which would exhaust apiserver and kubelet memory.
|
||||
However, creation of many smaller secrets could also exhaust memory. More
|
||||
comprehensive limits on memory usage due to secrets is a planned feature.
|
||||
@@ -549,8 +584,8 @@ controller. It does not include pods created via the kubelets
|
||||
not common ways to create pods.)
|
||||
|
||||
Secrets must be created before they are consumed in pods as environment
|
||||
variables unless they are marked as optional. References to Secrets that do not exist will prevent
|
||||
the pod from starting.
|
||||
variables unless they are marked as optional. References to Secrets that do
|
||||
not exist will prevent the pod from starting.
|
||||
|
||||
References via `secretKeyRef` to keys that do not exist in a named Secret
|
||||
will prevent the pod from starting.
|
||||
@@ -563,7 +598,9 @@ invalid keys that were skipped. The example shows a pod which refers to the
|
||||
default/mysecret that contains 2 invalid keys, 1badkey and 2alsobad.
|
||||
|
||||
```shell
|
||||
$ kubectl get events
|
||||
kubectl get events
|
||||
```
|
||||
```
|
||||
LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON
|
||||
0s 0s 1 dapi-test-pod Pod Warning InvalidEnvironmentVariableNames kubelet, 127.0.0.1 Keys [1badkey, 2alsobad] from the EnvFrom secret default/mysecret were skipped since they are considered invalid environment variable names.
|
||||
```
|
||||
@@ -586,7 +623,10 @@ start until all the pod's volumes are mounted.
|
||||
Create a secret containing some ssh keys:
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub
|
||||
kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa
|
||||
```
|
||||
```
|
||||
--from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub
|
||||
```
|
||||
|
||||
{{< caution >}}
|
||||
@@ -636,9 +676,18 @@ credentials.
|
||||
Make the secrets:
|
||||
|
||||
```shell
|
||||
$ kubectl create secret generic prod-db-secret --from-literal=username=produser --from-literal=password=Y4nys7f11
|
||||
kubectl create secret generic prod-db-secret --from-literal=username=produser
|
||||
--from-literal=password=Y4nys7f11
|
||||
```
|
||||
|
||||
```
|
||||
secret "prod-db-secret" created
|
||||
$ kubectl create secret generic test-db-secret --from-literal=username=testuser --from-literal=password=iluvtests
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl create secret generic test-db-secret --from-literal=username=testuser --from-literal=password=iluvtests
|
||||
```
|
||||
```
|
||||
secret "test-db-secret" created
|
||||
```
|
||||
{{< note >}}
|
||||
@@ -821,6 +870,7 @@ be available in future releases of Kubernetes.
|
||||
|
||||
## Security Properties
|
||||
|
||||
|
||||
### Protections
|
||||
|
||||
Because `secret` objects can be created independently of the `pods` that use
|
||||
@@ -829,51 +879,52 @@ creating, viewing, and editing pods. The system can also take additional
|
||||
precautions with `secret` objects, such as avoiding writing them to disk where
|
||||
possible.
|
||||
|
||||
A secret is only sent to a node if a pod on that node requires it. It is not
|
||||
written to disk. It is stored in a tmpfs. It is deleted once the pod that
|
||||
depends on it is deleted.
|
||||
|
||||
On most Kubernetes-project-maintained distributions, communication between user
|
||||
to the apiserver, and from apiserver to the kubelets, is protected by SSL/TLS.
|
||||
Secrets are protected when transmitted over these channels.
|
||||
|
||||
Secret data on nodes is stored in tmpfs volumes and thus does not come to rest
|
||||
on the node.
|
||||
A secret is only sent to a node if a pod on that node requires it.
|
||||
Kubelet stores the secret into a `tmpfs` so that the secret is not written
|
||||
to disk storage. Once the Pod that depends on the secret is deleted, kubelet
|
||||
will delete its local copy of the secret data as well.
|
||||
|
||||
There may be secrets for several pods on the same node. However, only the
|
||||
secrets that a pod requests are potentially visible within its containers.
|
||||
Therefore, one Pod does not have access to the secrets of another pod.
|
||||
Therefore, one Pod does not have access to the secrets of another Pod.
|
||||
|
||||
There may be several containers in a pod. However, each container in a pod has
|
||||
to request the secret volume in its `volumeMounts` for it to be visible within
|
||||
the container. This can be used to construct useful [security partitions at the
|
||||
Pod level](#use-case-secret-visible-to-one-container-in-a-pod).
|
||||
|
||||
On most Kubernetes-project-maintained distributions, communication between user
|
||||
to the apiserver, and from apiserver to the kubelets, is protected by SSL/TLS.
|
||||
Secrets are protected when transmitted over these channels.
|
||||
|
||||
{{< feature-state for_k8s_version="v1.13" state="beta" >}}
|
||||
|
||||
You can enable [encryption at rest](/docs/tasks/administer-cluster/encrypt-data/)
|
||||
for secret data, so that the secrets are not stored in the clear into {{< glossary_tooltip term_id="etcd" >}}.
|
||||
|
||||
### Risks
|
||||
|
||||
- In the API server secret data is stored as plaintext in etcd; therefore:
|
||||
- In the API server secret data is stored in {{< glossary_tooltip term_id="etcd" >}};
|
||||
therefore:
|
||||
- Administrators should enable encryption at rest for cluster data (requires v1.13 or later)
|
||||
- Administrators should limit access to etcd to admin users
|
||||
- Secret data in the API server is at rest on the disk that etcd uses; admins may want to wipe/shred disks
|
||||
used by etcd when no longer in use
|
||||
- Administrators may want to wipe/shred disks used by etcd when no longer in use
|
||||
- If running etcd in a cluster, administrators should make sure to use SSL/TLS
|
||||
for etcd peer-to-peer communication.
|
||||
- If you configure the secret through a manifest (JSON or YAML) file which has
|
||||
the secret data encoded as base64, sharing this file or checking it in to a
|
||||
source repository means the secret is compromised. Base64 encoding is not an
|
||||
source repository means the secret is compromised. Base64 encoding is _not_ an
|
||||
encryption method and is considered the same as plain text.
|
||||
- Applications still need to protect the value of secret after reading it from the volume,
|
||||
such as not accidentally logging it or transmitting it to an untrusted party.
|
||||
- A user who can create a pod that uses a secret can also see the value of that secret. Even
|
||||
if apiserver policy does not allow that user to read the secret object, the user could
|
||||
run a pod which exposes the secret.
|
||||
- If multiple replicas of etcd are run, then the secrets will be shared between them.
|
||||
By default, etcd does not secure peer-to-peer communication with SSL/TLS, though this can be configured.
|
||||
- Currently, anyone with root on any node can read any secret from the apiserver,
|
||||
- Currently, anyone with root on any node can read _any_ secret from the apiserver,
|
||||
by impersonating the kubelet. It is a planned feature to only send secrets to
|
||||
nodes that actually require them, to restrict the impact of a root exploit on a
|
||||
single node.
|
||||
|
||||
{{< note >}}
|
||||
As of 1.7 [encryption of secret data at rest is supported](/docs/tasks/administer-cluster/encrypt-data/).
|
||||
{{< /note >}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
|
||||
@@ -191,7 +191,7 @@ on the special hardware nodes. This will make sure that these special hardware
|
||||
nodes are dedicated for pods requesting such hardware and you don't have to
|
||||
manually add tolerations to your pods.
|
||||
|
||||
* **Taint based Evictions (alpha feature)**: A per-pod-configurable eviction behavior
|
||||
* **Taint based Evictions (beta feature)**: A per-pod-configurable eviction behavior
|
||||
when there are node problems, which is described in the next section.
|
||||
|
||||
## Taint based Evictions
|
||||
@@ -279,7 +279,7 @@ which matches the behavior when this feature is disabled.
|
||||
In version 1.12, `TaintNodesByCondition` feature is promoted to beta, so node lifecycle controller automatically creates taints corresponding to
|
||||
Node conditions.
|
||||
Similarly the scheduler does not check Node conditions; instead the scheduler checks taints. This assures that Node conditions don't affect what's scheduled onto the Node. The user can choose to ignore some of the Node's problems (represented as Node conditions) by adding appropriate Pod tolerations.
|
||||
Note that `TaintNodesByCondition` only taints nodes with `NoSchedule` effect. `NoExecute` effect is controlled by `TaintBasedEviction` which is an alpha feature and disabled by default.
|
||||
Note that `TaintNodesByCondition` only taints nodes with `NoSchedule` effect. `NoExecute` effect is controlled by `TaintBasedEviction` which is a beta feature and enabled by default since version 1.13.
|
||||
|
||||
Starting in Kubernetes 1.8, the DaemonSet controller automatically adds the
|
||||
following `NoSchedule` tolerations to all daemons, to prevent DaemonSets from
|
||||
|
||||
@@ -36,7 +36,7 @@ No parameters are passed to the handler.
|
||||
|
||||
`PreStop`
|
||||
|
||||
This hook is called immediately before a container is terminated.
|
||||
This hook is called immediately before a container is terminated due to an API request or management event such as liveness probe failure, preemption, resource contention and others. A call to the preStop hook fails if the container is already in terminated or completed state.
|
||||
It is blocking, meaning it is synchronous,
|
||||
so it must complete before the call to delete the container can be sent.
|
||||
No parameters are passed to the handler.
|
||||
@@ -99,17 +99,17 @@ Here is some example output of events from running this command:
|
||||
|
||||
```
|
||||
Events:
|
||||
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
|
||||
--------- -------- ----- ---- ------------- -------- ------ -------
|
||||
1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned test-1730497541-cq1d2 to gke-test-cluster-default-pool-a07e5d30-siqd
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Pulling pulling image "test:1.0"
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Created Created container with docker id 5c6a256a2567; Security:[seccomp=unconfined]
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Pulled Successfully pulled image "test:1.0"
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Started Started container with docker id 5c6a256a2567
|
||||
38s 38s 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Killing Killing container with docker id 5c6a256a2567: PostStart handler: Error executing in Docker Container: 1
|
||||
37s 37s 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Killing Killing container with docker id 8df9fdfd7054: PostStart handler: Error executing in Docker Container: 1
|
||||
38s 37s 2 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "main" with RunContainerError: "PostStart handler: Error executing in Docker Container: 1"
|
||||
1m 22s 2 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Warning FailedPostStartHook
|
||||
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
|
||||
--------- -------- ----- ---- ------------- -------- ------ -------
|
||||
1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned test-1730497541-cq1d2 to gke-test-cluster-default-pool-a07e5d30-siqd
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Pulling pulling image "test:1.0"
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Created Created container with docker id 5c6a256a2567; Security:[seccomp=unconfined]
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Pulled Successfully pulled image "test:1.0"
|
||||
1m 1m 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Started Started container with docker id 5c6a256a2567
|
||||
38s 38s 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Killing Killing container with docker id 5c6a256a2567: PostStart handler: Error executing in Docker Container: 1
|
||||
37s 37s 1 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Normal Killing Killing container with docker id 8df9fdfd7054: PostStart handler: Error executing in Docker Container: 1
|
||||
38s 37s 2 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "main" with RunContainerError: "PostStart handler: Error executing in Docker Container: 1"
|
||||
1m 22s 2 {kubelet gke-test-cluster-default-pool-a07e5d30-siqd} spec.containers{main} Warning FailedPostStartHook
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -149,9 +149,9 @@ Once you have those variables filled in you can
|
||||
### Using IBM Cloud Container Registry
|
||||
IBM Cloud Container Registry provides a multi-tenant private image registry that you can use to safely store and share your Docker images. By default, images in your private registry are scanned by the integrated Vulnerability Advisor to detect security issues and potential vulnerabilities. Users in your IBM Cloud account can access your images, or you can create a token to grant access to registry namespaces.
|
||||
|
||||
To install the IBM Cloud Container Registry CLI plug-in and create a namespace for your images, see [Getting started with IBM Cloud Container Registry](https://console.bluemix.net/docs/services/Registry/index.html#index).
|
||||
To install the IBM Cloud Container Registry CLI plug-in and create a namespace for your images, see [Getting started with IBM Cloud Container Registry](https://cloud.ibm.com/docs/services/Registry?topic=registry-index#index).
|
||||
|
||||
You can use the IBM Cloud Container Registry to deploy containers from [IBM Cloud public images](https://console.bluemix.net/docs/services/RegistryImages/index.html#ibm_images) and your private images into the `default` namespace of your IBM Cloud Kubernetes Service cluster. To deploy a container into other namespaces, or to use an image from a different IBM Cloud Container Registry region or IBM Cloud account, create a Kubernetes `imagePullSecret`. For more information, see [Building containers from images](https://console.bluemix.net/docs/containers/cs_images.html#images).
|
||||
You can use the IBM Cloud Container Registry to deploy containers from [IBM Cloud public images](https://cloud.ibm.com/docs/services/Registry?topic=registry-public_images#public_images) and your private images into the `default` namespace of your IBM Cloud Kubernetes Service cluster. To deploy a container into other namespaces, or to use an image from a different IBM Cloud Container Registry region or IBM Cloud account, create a Kubernetes `imagePullSecret`. For more information, see [Building containers from images](https://cloud.ibm.com/docs/containers?topic=containers-images#images).
|
||||
|
||||
### Configuring Nodes to Authenticate to a Private Registry
|
||||
|
||||
@@ -170,6 +170,11 @@ will not work reliably on GCE, and any other cloud provider that does automatic
|
||||
node replacement.
|
||||
{{< /note >}}
|
||||
|
||||
{{< note >}}
|
||||
Kubernetes as of now only supports the `auths` and `HttpHeaders` section of docker config. This means credential helpers (`credHelpers` or `credsStore`) are not supported.
|
||||
{{< /note >}}
|
||||
|
||||
|
||||
Docker stores keys for private registries in the `$HOME/.dockercfg` or `$HOME/.docker/config.json` file. If you put the same file
|
||||
in the search paths list below, kubelet uses it as the credential provider when pulling images.
|
||||
|
||||
@@ -278,42 +283,17 @@ kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGIS
|
||||
secret/myregistrykey created.
|
||||
```
|
||||
|
||||
If you need access to multiple registries, you can create one secret for each registry.
|
||||
Kubelet will merge any `imagePullSecrets` into a single virtual `.docker/config.json`
|
||||
when pulling images for your Pods.
|
||||
If you already have a Docker credentials file then, rather than using the above
|
||||
command, you can import the credentials file as a Kubernetes secret.
|
||||
[Create a Secret based on existing Docker credentials](/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) explains how to set this up.
|
||||
This is particularly useful if you are using multiple private container
|
||||
registries, as `kubectl create secret docker-registry` creates a Secret that will
|
||||
only work with a single private registry.
|
||||
|
||||
{{< note >}}
|
||||
Pods can only reference image pull secrets in their own namespace,
|
||||
so this process needs to be done one time per namespace.
|
||||
|
||||
##### Bypassing kubectl create secrets
|
||||
|
||||
If for some reason you need multiple items in a single `.docker/config.json` or need
|
||||
control not given by the above command, then you can [create a secret using
|
||||
json or yaml](/docs/user-guide/secrets/#creating-a-secret-manually).
|
||||
|
||||
Be sure to:
|
||||
|
||||
- set the name of the data item to `.dockerconfigjson`
|
||||
- base64 encode the docker file and paste that string, unbroken
|
||||
as the value for field `data[".dockerconfigjson"]`
|
||||
- set `type` to `kubernetes.io/dockerconfigjson`
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: myregistrykey
|
||||
namespace: awesomeapps
|
||||
data:
|
||||
.dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
|
||||
type: kubernetes.io/dockerconfigjson
|
||||
```
|
||||
|
||||
If you get the error message `error: no objects passed to create`, it may mean the base64 encoded string is invalid.
|
||||
If you get an error message like `Secret "myregistrykey" is invalid: data[.dockerconfigjson]: invalid value ...`, it means
|
||||
the data was successfully un-base64 encoded, but could not be parsed as a `.docker/config.json` file.
|
||||
{{< /note >}}
|
||||
|
||||
#### Referring to an imagePullSecrets on a Pod
|
||||
|
||||
@@ -355,7 +335,7 @@ common use cases and suggested solutions.
|
||||
1. Cluster running some proprietary images which should be hidden to those outside the company, but
|
||||
visible to all cluster users.
|
||||
- Use a hosted private [Docker registry](https://docs.docker.com/registry/).
|
||||
- It may be hosted on the [Docker Hub](https://hub.docker.com/account/signup/), or elsewhere.
|
||||
- It may be hosted on the [Docker Hub](https://hub.docker.com/signup), or elsewhere.
|
||||
- Manually configure .docker/config.json on each node as described above.
|
||||
- Or, run an internal private registry behind your firewall with open read access.
|
||||
- No Kubernetes configuration is required.
|
||||
@@ -372,3 +352,6 @@ common use cases and suggested solutions.
|
||||
- The tenant adds that secret to imagePullSecrets of each namespace.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
If you need access to multiple registries, you can create one secret for each registry.
|
||||
Kubelet will merge any `imagePullSecrets` into a single virtual `.docker/config.json`
|
||||
|
||||
@@ -9,40 +9,48 @@ weight: 20
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
This page explains *custom resources*, which are extensions of the Kubernetes
|
||||
API, including when to add a custom resource to your Kubernetes cluster and when
|
||||
to use a standalone service. It describes the two methods for adding custom
|
||||
resources and how to choose between them.
|
||||
*Custom resources* are extensions of the Kubernetes API. This page discusses when to add a custom
|
||||
resource to your Kubernetes cluster and when to use a standalone service. It describes the two
|
||||
methods for adding custom resources and how to choose between them.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
## Custom resources
|
||||
|
||||
A *resource* is an endpoint in the [Kubernetes API](/docs/reference/using-api/api-overview/) that stores a collection of [API objects](/docs/concepts/overview/working-with-objects/kubernetes-objects/) of a certain kind. For example, the built-in *pods* resource contains a collection of Pod objects.
|
||||
A *resource* is an endpoint in the [Kubernetes API](/docs/reference/using-api/api-overview/) that stores a collection of
|
||||
[API objects](/docs/concepts/overview/working-with-objects/kubernetes-objects/) of a certain kind. For example, the built-in *pods* resource contains a collection of Pod objects.
|
||||
|
||||
A *custom resource* is an extension of the Kubernetes API that is not necessarily available on every
|
||||
Kubernetes cluster.
|
||||
In other words, it represents a customization of a particular Kubernetes installation.
|
||||
A *custom resource* is an extension of the Kubernetes API that is not necessarily available in a default
|
||||
Kubernetes installation. It represents a customization of a particular Kubernetes installation. However,
|
||||
many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
|
||||
|
||||
Custom resources can appear and disappear in a running cluster through dynamic registration,
|
||||
and cluster admins can update custom resources independently of the cluster itself.
|
||||
Once a custom resource is installed, users can create and access its objects with
|
||||
[kubectl](/docs/user-guide/kubectl-overview/), just as they do for built-in resources like *pods*.
|
||||
Once a custom resource is installed, users can create and access its objects using
|
||||
[kubectl](/docs/user-guide/kubectl-overview/), just as they do for built-in resources like
|
||||
*Pods*.
|
||||
|
||||
### Custom controllers
|
||||
## Custom controllers
|
||||
|
||||
On their own, custom resources simply let you store and retrieve structured data.
|
||||
It is only when combined with a *controller* that they become a true declarative API.
|
||||
When you combine a custom resource with a *custom controller*, custom resources
|
||||
provide a true _declarative API_.
|
||||
|
||||
A [declarative API](/docs/concepts/overview/working-with-objects/kubernetes-objects/#understanding-kubernetes-objects)
|
||||
allows you to _declare_ or specify the desired state of your resource and tries
|
||||
to match the actual state to this desired state.
|
||||
Here, the controller interprets the structured data as a record of the user's
|
||||
desired state, and continually takes action to achieve and maintain this state.
|
||||
allows you to _declare_ or specify the desired state of your resource and tries to
|
||||
keep the current state of Kubernetes objects in sync with the desired state.
|
||||
The controller interprets the structured data as a record of the user's
|
||||
desired state, and continually maintains this state.
|
||||
|
||||
A *custom controller* is a controller that users can deploy and update on a running cluster, independently of the cluster's own lifecycle. Custom controllers can work with any kind of resource, but they are especially effective when combined with custom resources. The [Operator](https://coreos.com/blog/introducing-operators.html) pattern is one example of such a combination. It allows developers to encode domain knowledge for specific applications into an extension of the Kubernetes API.
|
||||
You can deploy and update a custom controller on a running cluster, independently
|
||||
of the cluster's own lifecycle. Custom controllers can work with any kind of resource,
|
||||
but they are especially effective when combined with custom resources. The
|
||||
[Operator pattern](https://coreos.com/blog/introducing-operators.html) combines custom
|
||||
resources and custom controllers. You can use custom controllers to encode domain knowledge
|
||||
for specific applications into an extension of the Kubernetes API.
|
||||
|
||||
### Should I add a custom resource to my Kubernetes Cluster?
|
||||
## Should I add a custom resource to my Kubernetes Cluster?
|
||||
|
||||
When creating a new API, consider whether to [aggregate your API with the Kubernetes cluster APIs](/docs/concepts/api-extension/apiserver-aggregation/) or let your API stand alone.
|
||||
|
||||
@@ -56,7 +64,7 @@ When creating a new API, consider whether to [aggregate your API with the Kubern
|
||||
| Your resources are naturally scoped to a cluster or to namespaces of a cluster. | Cluster or namespace scoped resources are a poor fit; you need control over the specifics of resource paths. |
|
||||
| You want to reuse [Kubernetes API support features](#common-features). | You don't need those features. |
|
||||
|
||||
#### Declarative APIs
|
||||
### Declarative APIs
|
||||
|
||||
In a Declarative API, typically:
|
||||
|
||||
@@ -80,7 +88,7 @@ Signs that your API might not be declarative include:
|
||||
- The API is not easily modeled as objects.
|
||||
- You chose to represent pending operations with an operation ID or an operation object.
|
||||
|
||||
### Should I use a configMap or a custom resource?
|
||||
## Should I use a configMap or a custom resource?
|
||||
|
||||
Use a ConfigMap if any of the following apply:
|
||||
|
||||
@@ -126,13 +134,9 @@ This frees you from writing your own API server to handle the custom resource,
|
||||
but the generic nature of the implementation means you have less flexibility than with
|
||||
[API server aggregation](#api-server-aggregation).
|
||||
|
||||
Refer to the [Custom Controller example, which uses Custom Resources](https://github.com/kubernetes/sample-controller)
|
||||
for a demonstration of how to register a new custom resource, work with instances of your new resource type,
|
||||
and setup a controller to handle events.
|
||||
|
||||
{{< note >}}
|
||||
CRD is the successor to the deprecated *ThirdPartyResource* (TPR) API, and is available as of Kubernetes 1.7.
|
||||
{{< /note >}}
|
||||
Refer to the [custom controller example](https://github.com/kubernetes/sample-controller)
|
||||
for an example of how to register a new custom resource, work with instances of your new resource type,
|
||||
and use a controller to handle events.
|
||||
|
||||
## API server aggregation
|
||||
|
||||
@@ -143,7 +147,7 @@ implementations for your custom resources by writing and deploying your own stan
|
||||
The main API server delegates requests to you for the custom resources that you handle,
|
||||
making them available to all of its clients.
|
||||
|
||||
### Choosing a method for adding custom resources
|
||||
## Choosing a method for adding custom resources
|
||||
|
||||
CRDs are easier to use. Aggregated APIs are more flexible. Choose the method that best meets your needs.
|
||||
|
||||
@@ -152,7 +156,7 @@ Typically, CRDs are a good fit if:
|
||||
* You have a handful of fields
|
||||
* You are using the resource within your company, or as part of a small open-source project (as opposed to a commercial product)
|
||||
|
||||
#### Comparing ease of use
|
||||
### Comparing ease of use
|
||||
|
||||
CRDs are easier to create than Aggregated APIs.
|
||||
|
||||
@@ -181,7 +185,7 @@ Aggregated APIs offer more advanced API features and customization of other feat
|
||||
| Protocol Buffers | The new resource supports clients that want to use Protocol Buffers | No | Yes |
|
||||
| OpenAPI Schema | Is there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an `int` in a `string` field?) | No, but planned | Yes |
|
||||
|
||||
#### Common Features
|
||||
### Common Features
|
||||
|
||||
When you create a custom resource, either via a CRDs or an AA, you get many features for your API, compared to implementing it outside the Kubernetes platform:
|
||||
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: Poseidon-Firmament - An alternate scheduler
|
||||
content_template: templates/concept
|
||||
weight: 80
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
**Current release of Poseidon-Firmament scheduler is an <code> alpha </code> release.**
|
||||
|
||||
Poseidon-Firmament scheduler is an alternate scheduler that can be deployed alongside the default Kubernetes scheduler.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
Poseidon is a service that acts as the integration glue for the [Firmament scheduler](https://github.com/Huawei-PaaS/firmament) with Kubernetes. Poseidon-Firmament scheduler augments the current Kubernetes scheduling capabilities. It incorporates novel flow network graph based scheduling capabilities alongside the default Kubernetes Scheduler. Firmament scheduler models workloads and clusters as flow networks and runs min-cost flow optimizations over these networks to make scheduling decisions.
|
||||
|
||||
It models the scheduling problem as a constraint-based optimization over a flow network graph. This is achieved by reducing scheduling to a min-cost max-flow optimization problem. The Poseidon-Firmament scheduler dynamically refines the workload placements.
|
||||
|
||||
Poseidon-Firmament scheduler runs alongside the default Kubernetes Scheduler as an alternate scheduler, so multiple schedulers run simultaneously.
|
||||
|
||||
## Key Advantages
|
||||
|
||||
### Flow graph scheduling based Poseidon-Firmament scheduler provides the following key advantages:
|
||||
- Workloads (pods) are bulk scheduled to enable scheduling at massive scale..
|
||||
- Based on the extensive performance test results, Poseidon-Firmament scales much better than the Kubernetes default scheduler as the number of nodes increase in a cluster. This is due to the fact that Poseidon-Firmament is able to amortize more and more work across workloads.
|
||||
- Poseidon-Firmament Scheduler outperforms the Kubernetes default scheduler by a wide margin when it comes to throughput performance numbers for scenarios where compute resource requirements are somewhat uniform across jobs (Replicasets/Deployments/Jobs). Poseidon-Firmament scheduler end-to-end throughput performance numbers, including bind time, consistently get better as the number of nodes in a cluster increase. For example, for a 2,700 node cluster (shown in the graphs [here](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/benchmark/README.md)), Poseidon-Firmament scheduler achieves a 7X or greater end-to-end throughput than the Kubernetes default scheduler, which includes bind time.
|
||||
|
||||
- Availability of complex rule constraints.
|
||||
- Scheduling in Poseidon-Firmament is dynamic; it keeps cluster resources in a global optimal state during every scheduling run.
|
||||
- Highly efficient resource utilizations.
|
||||
|
||||
## Poseidon-Firmament Scheduler - How it works
|
||||
|
||||
As part of the Kubernetes multiple schedulers support, each new pod is typically scheduled by the default scheduler. Kubernetes can be instructed to use another scheduler by specifying the name of another custom scheduler (“poseidon” in our case) in the **schedulerName** field of the PodSpec at the time of pod creation. In this case, the default scheduler will ignore that Pod and allow Poseidon scheduler to schedule the Pod on a relevant node.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
|
||||
...
|
||||
spec:
|
||||
schedulerName: poseidon
|
||||
```
|
||||
|
||||
|
||||
{{< note >}}
|
||||
For details about the design of this project see the [design document](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/design/README.md).
|
||||
{{< /note >}}
|
||||
|
||||
## Possible Use Case Scenarios - When to use it
|
||||
|
||||
As mentioned earlier, Poseidon-Firmament scheduler enables an extremely high throughput scheduling environment at scale due to its bulk scheduling approach versus Kubernetes pod-at-a-time approach. In our extensive tests, we have observed substantial throughput benefits as long as resource requirements (CPU/Memory) for incoming Pods are uniform across jobs (Replicasets/Deployments/Jobs), mainly due to efficient amortization of work across jobs.
|
||||
|
||||
Although, Poseidon-Firmament scheduler is capable of scheduling various types of workloads, such as service, batch, etc., the following are a few use cases where it excels the most:
|
||||
|
||||
1. For “Big Data/AI” jobs consisting of large number of tasks, throughput benefits are tremendous.
|
||||
2. Service or batch jobs where workload resource requirements are uniform across jobs (Replicasets/Deployments/Jobs).
|
||||
|
||||
## Current Project Stage
|
||||
|
||||
- **Alpha Release - Incubation repo.** at https://github.com/kubernetes-sigs/poseidon.
|
||||
- Currently, Poseidon-Firmament scheduler **does not provide support for high availability**, our implementation assumes that the scheduler cannot fail. The [design document](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/design/README.md) describes possible ways to enable high availability, but we leave this to future work.
|
||||
- We are **not aware of any production deployment** of Poseidon-Firmament scheduler at this time.
|
||||
- Poseidon-Firmament is supported from Kubernetes release 1.6 and works with all subsequent releases.
|
||||
- Release process for Poseidon and Firmament repos are in lock step. The current Poseidon release can be found [here](https://github.com/kubernetes-sigs/poseidon/releases) and the corresponding Firmament release can be found [here](https://github.com/Huawei-PaaS/firmament/releases).
|
||||
|
||||
## Features Comparison Matrix
|
||||
|
||||
|
||||
|Feature|Kubernetes Default Scheduler|Poseidon-Firmament Scheduler|Notes|
|
||||
|--- |--- |--- |--- |
|
||||
|Node Affinity/Anti-Affinity|Y|Y||
|
||||
|Pod Affinity/Anti-Affinity - including support for pod anti-affinity symmetry|Y|Y|Currently, the default scheduler outperforms the Poseidon-Firmament scheduler pod affinity/anti-affinity functionality. We are working towards resolving this.|
|
||||
|Taints & Tolerations|Y|Y||
|
||||
|Baseline Scheduling capability in accordance to available compute resources (CPU & Memory) on a node|Y|Y**|Not all Predicates & Priorities are supported at this time.|
|
||||
|Extreme Throughput at scale|Y**|Y|Bulk scheduling approach scales or increases workload placement. Substantial throughput benefits using Firmament scheduler as long as resource requirements (CPU/Memory) for incoming Pods is uniform across Replicasets/Deployments/Jobs. This is mainly due to efficient amortization of work across Replicasets/Deployments/Jobs . 1) For “Big Data/AI” jobs consisting of large no. of tasks, throughput benefits are tremendous. 2) Substantial throughput benefits also for service or batch job scenarios where workload resource requirements are uniform across Replicasets/Deployments/Jobs.|
|
||||
|Optimal Scheduling|Pod-by-Pod scheduler, processes one pod at a time (may result into sub-optimal scheduling)|Bulk Scheduling (Optimal scheduling)|Pod-by-Pod Kubernetes default scheduler may assign tasks to a sub-optimal machine. By contrast, Firmament considers all unscheduled tasks at the same time together with their soft and hard constraints.|
|
||||
|Colocation Interference Avoidance|N|N**|Planned in Poseidon-Firmament.|
|
||||
|Priority Pre-emption|Y|N**|Partially exists in Poseidon-Firmament versus extensive support in Kubernetes default scheduler.|
|
||||
|Inherent Re-Scheduling|N|Y**|Poseidon-Firmament scheduler supports workload re-scheduling. In each scheduling run it considers all the pods, including running pods, and as a result can migrate or evict pods – a globally optimal scheduling environment.|
|
||||
|Gang Scheduling|N|Y||
|
||||
|Support for Pre-bound Persistence Volume Scheduling|Y|Y||
|
||||
|Support for Local Volume & Dynamic Persistence Volume Binding Scheduling|Y|N**|Planned.|
|
||||
|High Availability|Y|N**|Planned.|
|
||||
|Real-time metrics based scheduling|N|Y**|Initially supported using Heapster (now deprecated) for placing pods using actual cluster utilization statistics rather than reservations. Plans to switch over to "metric server".|
|
||||
|Support for Max-Pod per node|Y|Y|Poseidon-Firmament scheduler seamlessly co-exists with Kubernetes default scheduler.|
|
||||
|Support for Ephemeral Storage, in addition to CPU/Memory|Y|Y||
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
For in-cluster installation of Poseidon, please start at the [Installation instructions](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/install/README.md).
|
||||
|
||||
|
||||
## Development
|
||||
|
||||
For developers, please refer to the [Developer Setup instructions](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/devel/README.md).
|
||||
|
||||
## Latest Throughput Performance Testing Results
|
||||
|
||||
Pod-by-pod schedulers, such as the Kubernetes default scheduler, typically process one pod at a time. These schedulers have the following crucial drawbacks:
|
||||
|
||||
1. The scheduler commits to a pod placement early and restricts the choices for other pods that wait to be placed.
|
||||
2. There is limited opportunities for amortizing work across pods because they are considered for placement individually.
|
||||
|
||||
These downsides of pod-by-pod schedulers are addressed by batching or bulk scheduling in Poseidon-Firmament scheduler. Processing several pods in a batch allows the scheduler to jointly consider their placement, and thus to find the best trade-off for the whole batch instead of one pod. At the same time it amortizes work across pods resulting in much higher throughput.
|
||||
|
||||
{{< note >}}
|
||||
Please refer to the [latest benchmark results](https://github.com/kubernetes-sigs/poseidon/blob/master/docs/benchmark/README.md) for detailed throughput performance comparison test results between Poseidon-Firmament scheduler and the Kubernetes default scheduler.
|
||||
{{< /note >}}
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -4,6 +4,9 @@ reviewers:
|
||||
title: Kubernetes Components
|
||||
content_template: templates/concept
|
||||
weight: 20
|
||||
card:
|
||||
name: concepts
|
||||
weight: 20
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -52,7 +55,7 @@ These controllers include:
|
||||
|
||||
cloud-controller-manager runs cloud-provider-specific controller loops only. You must disable these controller loops in the kube-controller-manager. You can disable the controller loops by setting the `--cloud-provider` flag to `external` when starting the kube-controller-manager.
|
||||
|
||||
cloud-controller-manager allows cloud vendors code and the Kubernetes core to evolve independent of each other. In prior releases, the core Kubernetes code was dependent upon cloud-provider-specific code for functionality. In future releases, code specific to cloud vendors should be maintained by the cloud vendor themselves, and linked to cloud-controller-manager while running Kubernetes.
|
||||
cloud-controller-manager allows cloud vendors code and the Kubernetes code to evolve independent of each other. In prior releases, the core Kubernetes code was dependent upon cloud-provider-specific code for functionality. In future releases, code specific to cloud vendors should be maintained by the cloud vendor themselves, and linked to cloud-controller-manager while running Kubernetes.
|
||||
|
||||
The following controllers have cloud provider dependencies:
|
||||
|
||||
@@ -76,7 +79,8 @@ network rules on the host and performing connection forwarding.
|
||||
|
||||
### Container Runtime
|
||||
|
||||
The container runtime is the software that is responsible for running containers. Kubernetes supports several runtimes: [Docker](http://www.docker.com), [rkt](https://coreos.com/rkt/), [runc](https://github.com/opencontainers/runc) and any OCI [runtime-spec](https://github.com/opencontainers/runtime-spec) implementation.
|
||||
The container runtime is the software that is responsible for running containers.
|
||||
Kubernetes supports several runtimes: [Docker](http://www.docker.com), [containerd](https://containerd.io), [cri-o](https://cri-o.io/), [rktlet](https://github.com/kubernetes-incubator/rktlet) and any implementation of the [Kubernetes CRI (Container Runtime Interface)](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md).
|
||||
|
||||
## Addons
|
||||
|
||||
|
||||
@@ -4,6 +4,9 @@ reviewers:
|
||||
title: The Kubernetes API
|
||||
content_template: templates/concept
|
||||
weight: 30
|
||||
card:
|
||||
name: concepts
|
||||
weight: 30
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -33,17 +36,19 @@ What constitutes a compatible change and how to change the API are detailed by t
|
||||
|
||||
## OpenAPI and Swagger definitions
|
||||
|
||||
Complete API details are documented using [Swagger v1.2](http://swagger.io/) and [OpenAPI](https://www.openapis.org/). The Kubernetes apiserver (aka "master") exposes an API that can be used to retrieve the Swagger v1.2 Kubernetes API spec located at `/swaggerapi`.
|
||||
Complete API details are documented using [OpenAPI](https://www.openapis.org/).
|
||||
|
||||
Starting with Kubernetes 1.10, OpenAPI spec is served in a single `/openapi/v2` endpoint. The format-separated endpoints (`/swagger.json`, `/swagger-2.0.0.json`, `/swagger-2.0.0.pb-v1`, `/swagger-2.0.0.pb-v1.gz`) are deprecated and will get removed in Kubernetes 1.14.
|
||||
|
||||
Requested format is specified by setting HTTP headers:
|
||||
Starting with Kubernetes 1.10, the Kubernetes API server serves an OpenAPI spec via the `/openapi/v2` endpoint.
|
||||
The requested format is specified by setting HTTP headers:
|
||||
|
||||
Header | Possible Values
|
||||
------ | ---------------
|
||||
Accept | `application/json`, `application/com.github.proto-openapi.spec.v2@v1.0+protobuf` (the default content-type is `application/json` for `*/*` or not passing this header)
|
||||
Accept-Encoding | `gzip` (not passing this header is acceptable)
|
||||
|
||||
Prior to 1.14, format-separated endpoints (`/swagger.json`, `/swagger-2.0.0.json`, `/swagger-2.0.0.pb-v1`, `/swagger-2.0.0.pb-v1.gz`)
|
||||
serve the OpenAPI spec in different formats. These endpoints are deprecated, and will be removed in Kubernetes 1.14.
|
||||
|
||||
**Examples of getting OpenAPI spec**:
|
||||
|
||||
Before 1.10 | Starting with Kubernetes 1.10
|
||||
@@ -52,9 +57,12 @@ GET /swagger.json | GET /openapi/v2 **Accept**: application/json
|
||||
GET /swagger-2.0.0.pb-v1 | GET /openapi/v2 **Accept**: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
|
||||
GET /swagger-2.0.0.pb-v1.gz | GET /openapi/v2 **Accept**: application/com.github.proto-openapi.spec.v2@v1.0+protobuf **Accept-Encoding**: gzip
|
||||
|
||||
|
||||
Kubernetes implements an alternative Protobuf based serialization format for the API that is primarily intended for intra-cluster communication, documented in the [design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/protobuf.md) and the IDL files for each schema are located in the Go packages that define the API objects.
|
||||
|
||||
Prior to 1.14, the Kubernetes apiserver also exposes an API that can be used to retrieve
|
||||
the [Swagger v1.2](http://swagger.io/) Kubernetes API spec at `/swaggerapi`.
|
||||
This endpoint is deprecated, and will be removed in Kubernetes 1.14.
|
||||
|
||||
## API versioning
|
||||
|
||||
To make it easier to eliminate fields or restructure resource representations, Kubernetes supports
|
||||
@@ -125,9 +133,9 @@ to pick up the `--runtime-config` changes.
|
||||
|
||||
## Enabling resources in the groups
|
||||
|
||||
DaemonSets, Deployments, HorizontalPodAutoscalers, Ingress, Jobs and ReplicaSets are enabled by default.
|
||||
DaemonSets, Deployments, HorizontalPodAutoscalers, Ingresses, Jobs and ReplicaSets are enabled by default.
|
||||
Other extensions resources can be enabled by setting `--runtime-config` on
|
||||
apiserver. `--runtime-config` accepts comma separated values. For example: to disable deployments and ingress, set
|
||||
`--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/ingress=false`
|
||||
`--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/ingresses=false`
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -73,7 +73,7 @@ Run `kubectl diff` to print the object that will be created:
|
||||
kubectl diff -f https://k8s.io/examples/application/simple_deployment.yaml
|
||||
```
|
||||
{{< note >}}
|
||||
**Note:** `diff` uses [server-side dry-run](/docs/reference/using-api/api-concepts/#dry-run), which needs to be enabled on `kube-apiserver`.
|
||||
`diff` uses [server-side dry-run](/docs/reference/using-api/api-concepts/#dry-run), which needs to be enabled on `kube-apiserver`.
|
||||
{{< /note >}}
|
||||
|
||||
Create the object using `kubectl apply`:
|
||||
@@ -625,7 +625,7 @@ Add, delete, or update individual elements. This does not preserve ordering.
|
||||
|
||||
This merge strategy uses a special tag on each field called a `patchMergeKey`. The
|
||||
`patchMergeKey` is defined for each field in the Kubernetes source code:
|
||||
[types.go](https://git.k8s.io/api/core/v1/types.go#L2565)
|
||||
[types.go](https://github.com/kubernetes/api/blob/d04500c8c3dda9c980b668c57abc2ca61efcf5c4/core/v1/types.go#L2747)
|
||||
When merging a list of maps, the field specified as the `patchMergeKey` for a given element
|
||||
is used like a map key for that element.
|
||||
|
||||
@@ -700,7 +700,7 @@ As of Kubernetes 1.5, merging lists of primitive elements is not supported.
|
||||
|
||||
{{< note >}}
|
||||
Which of the above strategies is chosen for a given field is controlled by
|
||||
the `patchStrategy` tag in [types.go](https://git.k8s.io/api/core/v1/types.go#L2565)
|
||||
the `patchStrategy` tag in [types.go](https://github.com/kubernetes/api/blob/d04500c8c3dda9c980b668c57abc2ca61efcf5c4/core/v1/types.go#L2748)
|
||||
If no `patchStrategy` is specified for a field of type list, then
|
||||
the list is replaced.
|
||||
{{< /note >}}
|
||||
|
||||
@@ -73,7 +73,7 @@ that must be set:
|
||||
The `kubectl` command also supports update commands driven by an aspect of the object.
|
||||
Setting this aspect may set different fields for different object types:
|
||||
|
||||
- `set` <field>: Set an aspect of an object.
|
||||
- `set` `<field>`: Set an aspect of an object.
|
||||
|
||||
{{< note >}}
|
||||
In Kubernetes version 1.5, not every verb-driven command has an associated aspect-driven command.
|
||||
@@ -160,5 +160,3 @@ kubectl create --edit -f /tmp/srv.yaml
|
||||
- [Kubectl Command Reference](/docs/reference/generated/kubectl/kubectl/)
|
||||
- [Kubernetes API Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/)
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -5,6 +5,9 @@ reviewers:
|
||||
title: What is Kubernetes?
|
||||
content_template: templates/concept
|
||||
weight: 10
|
||||
card:
|
||||
name: concepts
|
||||
weight: 10
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
@@ -53,11 +53,22 @@ Here are some examples of information that could be recorded in annotations:
|
||||
* Phone or pager numbers of persons responsible, or directory entries that
|
||||
specify where that information can be found, such as a team web site.
|
||||
|
||||
* Directives from the end-user to the implementations to modify behavior or
|
||||
engage non-standard features.
|
||||
|
||||
Instead of using annotations, you could store this type of information in an
|
||||
external database or directory, but that would make it much harder to produce
|
||||
shared client libraries and tools for deployment, management, introspection,
|
||||
and the like.
|
||||
|
||||
## Syntax and character set
|
||||
|
||||
_Annotations_ are key/value pairs. Valid annotation keys have two segments: an optional prefix and name, separated by a slash (`/`). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (`.`), not longer than 253 characters in total, followed by a slash (`/`).
|
||||
|
||||
If the prefix is omitted, the annotation Key is presumed to be private to the user. Automated system components (e.g. `kube-scheduler`, `kube-controller-manager`, `kube-apiserver`, `kubectl`, or other third-party automation) which add annotations to end-user objects must specify a prefix.
|
||||
|
||||
The `kubernetes.io/` and `k8s.io/` prefixes are reserved for Kubernetes core components.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
@@ -12,15 +12,15 @@ _Field selectors_ let you [select Kubernetes resources](/docs/concepts/overview/
|
||||
This `kubectl` command selects all Pods for which the value of the [`status.phase`](/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) field is `Running`:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods --field-selector status.phase=Running
|
||||
kubectl get pods --field-selector status.phase=Running
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
Field selectors are essentially resource *filters*. By default, no selectors/filters are applied, meaning that all resources of the specified type are selected. This makes the following `kubectl` queries equivalent:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
$ kubectl get pods --field-selector ""
|
||||
kubectl get pods
|
||||
kubectl get pods --field-selector ""
|
||||
```
|
||||
{{< /note >}}
|
||||
|
||||
@@ -29,7 +29,9 @@ $ kubectl get pods --field-selector ""
|
||||
Supported field selectors vary by Kubernetes resource type. All resource types support the `metadata.name` and `metadata.namespace` fields. Using unsupported field selectors produces an error. For example:
|
||||
|
||||
```shell
|
||||
$ kubectl get ingress --field-selector foo.bar=baz
|
||||
kubectl get ingress --field-selector foo.bar=baz
|
||||
```
|
||||
```
|
||||
Error from server (BadRequest): Unable to find "ingresses" that match label selector "", field selector "foo.bar=baz": "foo.bar" is not a known field selector: only "metadata.name", "metadata.namespace"
|
||||
```
|
||||
|
||||
@@ -38,7 +40,7 @@ Error from server (BadRequest): Unable to find "ingresses" that match label sele
|
||||
You can use the `=`, `==`, and `!=` operators with field selectors (`=` and `==` mean the same thing). This `kubectl` command, for example, selects all Kubernetes Services that aren't in the `default` namespace:
|
||||
|
||||
```shell
|
||||
$ kubectl get services --field-selector metadata.namespace!=default
|
||||
kubectl get services --field-selector metadata.namespace!=default
|
||||
```
|
||||
|
||||
## Chained selectors
|
||||
@@ -46,7 +48,7 @@ $ kubectl get services --field-selector metadata.namespace!=default
|
||||
As with [label](/docs/concepts/overview/working-with-objects/labels) and other selectors, field selectors can be chained together as a comma-separated list. This `kubectl` command selects all Pods for which the `status.phase` does not equal `Running` and the `spec.restartPolicy` field equals `Always`:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods --field-selector=status.phase!=Running,spec.restartPolicy=Always
|
||||
kubectl get pods --field-selector=status.phase!=Running,spec.restartPolicy=Always
|
||||
```
|
||||
|
||||
## Multiple resource types
|
||||
@@ -54,5 +56,5 @@ $ kubectl get pods --field-selector=status.phase!=Running,spec.restartPolicy=Alw
|
||||
You use field selectors across multiple resource types. This `kubectl` command selects all Statefulsets and Services that are not in the `default` namespace:
|
||||
|
||||
```shell
|
||||
$ kubectl get statefulsets,services --field-selector metadata.namespace!=default
|
||||
kubectl get statefulsets,services --field-selector metadata.namespace!=default
|
||||
```
|
||||
|
||||
@@ -2,6 +2,9 @@
|
||||
title: Understanding Kubernetes Objects
|
||||
content_template: templates/concept
|
||||
weight: 10
|
||||
card:
|
||||
name: concepts
|
||||
weight: 40
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -28,7 +31,7 @@ Every Kubernetes object includes two nested object fields that govern the object
|
||||
|
||||
For example, a Kubernetes Deployment is an object that can represent an application running on your cluster. When you create the Deployment, you might set the Deployment spec to specify that you want three replicas of the application to be running. The Kubernetes system reads the Deployment spec and starts three instances of your desired application--updating the status to match your spec. If any of those instances should fail (a status change), the Kubernetes system responds to the difference between spec and status by making a correction--in this case, starting a replacement instance.
|
||||
|
||||
For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://git.k8s.io/community/contributors/devel/api-conventions.md).
|
||||
For more information on the object spec, status, and metadata, see the [Kubernetes API Conventions](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md).
|
||||
|
||||
### Describing a Kubernetes Object
|
||||
|
||||
@@ -43,7 +46,7 @@ One way to create a Deployment using a `.yaml` file like the one above is to use
|
||||
in the `kubectl` command-line interface, passing the `.yaml` file as an argument. Here's an example:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/examples/application/deployment.yaml --record
|
||||
kubectl create -f https://k8s.io/examples/application/deployment.yaml --record
|
||||
```
|
||||
|
||||
The output is similar to this:
|
||||
|
||||
@@ -10,8 +10,8 @@ weight: 40
|
||||
|
||||
_Labels_ are key/value pairs that are attached to objects, such as pods.
|
||||
Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system.
|
||||
Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time.
|
||||
Each object can have a set of key/value labels defined. Each Key must be unique for a given object.
|
||||
Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time.
|
||||
Each object can have a set of key/value labels defined. Each Key must be unique for a given object.
|
||||
|
||||
```json
|
||||
"metadata": {
|
||||
@@ -22,7 +22,7 @@ Each object can have a set of key/value labels defined. Each Key must be unique
|
||||
}
|
||||
```
|
||||
|
||||
We'll eventually index and reverse-index labels for efficient queries and watches, use them to sort and group in UIs and CLIs, etc. We don't want to pollute labels with non-identifying, especially large and/or structured, data. Non-identifying information should be recorded using [annotations](/docs/concepts/overview/working-with-objects/annotations/).
|
||||
Labels allow for efficient queries and watches and are ideal for use in UIs and CLIs. Non-identifying information should be recorded using [annotations](/docs/concepts/overview/working-with-objects/annotations/).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -47,8 +47,11 @@ These are just examples of commonly used labels; you are free to develop your ow
|
||||
|
||||
## Syntax and character set
|
||||
|
||||
_Labels_ are key/value pairs. Valid label keys have two segments: an optional prefix and name, separated by a slash (`/`). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (`.`), not longer than 253 characters in total, followed by a slash (`/`).
|
||||
If the prefix is omitted, the label Key is presumed to be private to the user. Automated system components (e.g. `kube-scheduler`, `kube-controller-manager`, `kube-apiserver`, `kubectl`, or other third-party automation) which add labels to end-user objects must specify a prefix. The `kubernetes.io/` prefix is reserved for Kubernetes core components.
|
||||
_Labels_ are key/value pairs. Valid label keys have two segments: an optional prefix and name, separated by a slash (`/`). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (`.`), not longer than 253 characters in total, followed by a slash (`/`).
|
||||
|
||||
If the prefix is omitted, the label Key is presumed to be private to the user. Automated system components (e.g. `kube-scheduler`, `kube-controller-manager`, `kube-apiserver`, `kubectl`, or other third-party automation) which add labels to end-user objects must specify a prefix.
|
||||
|
||||
The `kubernetes.io/` and `k8s.io/` prefixes are reserved for Kubernetes core components.
|
||||
|
||||
Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between.
|
||||
|
||||
@@ -61,12 +64,12 @@ Via a _label selector_, the client/user can identify a set of objects. The label
|
||||
The API currently supports two types of selectors: _equality-based_ and _set-based_.
|
||||
A label selector can be made of multiple _requirements_ which are comma-separated. In the case of multiple requirements, all must be satisfied so the comma separator acts as a logical _AND_ (`&&`) operator.
|
||||
|
||||
An empty label selector (that is, one with zero requirements) selects every object in the collection.
|
||||
|
||||
A null label selector (which is only possible for optional selector fields) selects no objects.
|
||||
The semantics of empty or non-specified selectors are dependent on the context,
|
||||
and API types that use selectors should document the validity and meaning of
|
||||
them.
|
||||
|
||||
{{< note >}}
|
||||
The label selectors of two controllers must not overlap within a namespace, otherwise they will fight with each other.
|
||||
For some API types, such as ReplicaSets, the label selectors of two instances must not overlap within a namespace, or the controller can see that as conflicting instructions and fail to determine how many replicas should be present.
|
||||
{{< /note >}}
|
||||
|
||||
### _Equality-based_ requirement
|
||||
@@ -136,25 +139,25 @@ LIST and WATCH operations may specify label selectors to filter the sets of obje
|
||||
Both label selector styles can be used to list or watch resources via a REST client. For example, targeting `apiserver` with `kubectl` and using _equality-based_ one may write:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l environment=production,tier=frontend
|
||||
kubectl get pods -l environment=production,tier=frontend
|
||||
```
|
||||
|
||||
or using _set-based_ requirements:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l 'environment in (production),tier in (frontend)'
|
||||
kubectl get pods -l 'environment in (production),tier in (frontend)'
|
||||
```
|
||||
|
||||
As already mentioned _set-based_ requirements are more expressive. For instance, they can implement the _OR_ operator on values:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l 'environment in (production, qa)'
|
||||
kubectl get pods -l 'environment in (production, qa)'
|
||||
```
|
||||
|
||||
or restricting negative matching via _exists_ operator:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l 'environment,environment notin (frontend)'
|
||||
kubectl get pods -l 'environment,environment notin (frontend)'
|
||||
```
|
||||
|
||||
### Set references in API objects
|
||||
|
||||
@@ -46,7 +46,9 @@ for namespaces](/docs/admin/namespaces).
|
||||
You can list the current namespaces in a cluster using:
|
||||
|
||||
```shell
|
||||
$ kubectl get namespaces
|
||||
kubectl get namespaces
|
||||
```
|
||||
```
|
||||
NAME STATUS AGE
|
||||
default Active 1d
|
||||
kube-system Active 1d
|
||||
@@ -66,8 +68,8 @@ To temporarily set the namespace for a request, use the `--namespace` flag.
|
||||
For example:
|
||||
|
||||
```shell
|
||||
$ kubectl --namespace=<insert-namespace-name-here> run nginx --image=nginx
|
||||
$ kubectl --namespace=<insert-namespace-name-here> get pods
|
||||
kubectl --namespace=<insert-namespace-name-here> run nginx --image=nginx
|
||||
kubectl --namespace=<insert-namespace-name-here> get pods
|
||||
```
|
||||
|
||||
### Setting the namespace preference
|
||||
@@ -76,9 +78,9 @@ You can permanently save the namespace for all subsequent kubectl commands in th
|
||||
context.
|
||||
|
||||
```shell
|
||||
$ kubectl config set-context $(kubectl config current-context) --namespace=<insert-namespace-name-here>
|
||||
kubectl config set-context $(kubectl config current-context) --namespace=<insert-namespace-name-here>
|
||||
# Validate it
|
||||
$ kubectl config view | grep namespace:
|
||||
kubectl config view | grep namespace:
|
||||
```
|
||||
|
||||
## Namespaces and DNS
|
||||
@@ -101,10 +103,10 @@ To see which Kubernetes resources are and aren't in a namespace:
|
||||
|
||||
```shell
|
||||
# In a namespace
|
||||
$ kubectl api-resources --namespaced=true
|
||||
kubectl api-resources --namespaced=true
|
||||
|
||||
# Not in a namespace
|
||||
$ kubectl api-resources --namespaced=false
|
||||
kubectl api-resources --namespaced=false
|
||||
```
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -41,7 +41,7 @@ administrator to control the following:
|
||||
| Restricting escalation to root privileges | [`allowPrivilegeEscalation`, `defaultAllowPrivilegeEscalation`](#privilege-escalation) |
|
||||
| Linux capabilities | [`defaultAddCapabilities`, `requiredDropCapabilities`, `allowedCapabilities`](#capabilities) |
|
||||
| The SELinux context of the container | [`seLinux`](#selinux) |
|
||||
| The Allowed Proc Mount types for the container | [`allowedProcMountTypes`](#allowedProcMountTypes) |
|
||||
| The Allowed Proc Mount types for the container | [`allowedProcMountTypes`](#allowedprocmounttypes) |
|
||||
| The AppArmor profile used by containers | [annotations](#apparmor) |
|
||||
| The seccomp profile used by containers | [annotations](#seccomp) |
|
||||
| The sysctl profile used by containers | [annotations](#sysctl) |
|
||||
@@ -336,7 +336,6 @@ pause-7774d79b5-qrgcb 0/1 Pending 0 1s
|
||||
pause-7774d79b5-qrgcb 0/1 Pending 0 1s
|
||||
pause-7774d79b5-qrgcb 0/1 ContainerCreating 0 1s
|
||||
pause-7774d79b5-qrgcb 1/1 Running 0 2s
|
||||
^C
|
||||
```
|
||||
|
||||
### Clean up
|
||||
@@ -465,7 +464,7 @@ Please make sure [`volumes`](#volumes-and-file-systems) field contains the
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: allow-flex-volumes
|
||||
@@ -610,6 +609,6 @@ default cannot be changed.
|
||||
### Sysctl
|
||||
|
||||
Controlled via annotations on the PodSecurityPolicy. Refer to the [Sysctl documentation](
|
||||
/docs/concepts/cluster-administration/sysctl-cluster/#podsecuritypolicy-annotations).
|
||||
/docs/concepts/cluster-administration/sysctl-cluster/#podsecuritypolicy).
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -551,10 +551,10 @@ plugins:
|
||||
kind: Configuration
|
||||
limitedResources:
|
||||
- resource: pods
|
||||
matchScopes:
|
||||
- operator : In
|
||||
scopeName: PriorityClass
|
||||
values: ["cluster-services"]
|
||||
matchScopes:
|
||||
- scopeName: PriorityClass
|
||||
operator: In
|
||||
values: ["cluster-services"]
|
||||
```
|
||||
|
||||
Now, "cluster-services" pods will be allowed in only those namespaces where a quota object with a matching `scopeSelector` is present.
|
||||
@@ -562,8 +562,8 @@ For example:
|
||||
```yaml
|
||||
scopeSelector:
|
||||
matchExpressions:
|
||||
- operator : In
|
||||
scopeName: PriorityClass
|
||||
- scopeName: PriorityClass
|
||||
operator: In
|
||||
values: ["cluster-services"]
|
||||
```
|
||||
|
||||
|
||||
+3
-5
@@ -19,7 +19,7 @@ Modification not using HostAliases is not suggested because the file is managed
|
||||
|
||||
## Default Hosts File Content
|
||||
|
||||
Lets start an Nginx Pod which is assigned a Pod IP:
|
||||
Let's start an Nginx Pod which is assigned a Pod IP:
|
||||
|
||||
```shell
|
||||
kubectl run nginx --image nginx --generator=run-pod/v1
|
||||
@@ -107,10 +107,8 @@ fe00::2 ip6-allrouters
|
||||
10.200.0.5 hostaliases-pod
|
||||
|
||||
# Entries added by HostAliases.
|
||||
127.0.0.1 foo.local
|
||||
127.0.0.1 bar.local
|
||||
10.1.2.3 foo.remote
|
||||
10.1.2.3 bar.remote
|
||||
127.0.0.1 foo.local bar.local
|
||||
10.1.2.3 foo.remote bar.remote
|
||||
```
|
||||
|
||||
With the additional entries specified at the bottom.
|
||||
|
||||
@@ -17,7 +17,7 @@ Now that you have a continuously running, replicated application you can expose
|
||||
|
||||
By default, Docker uses host-private networking, so containers can talk to other containers only if they are on the same machine. In order for Docker containers to communicate across nodes, there must be allocated ports on the machine’s own IP address, which are then forwarded or proxied to the containers. This obviously means that containers must either coordinate which ports they use very carefully or ports must be allocated dynamically.
|
||||
|
||||
Coordinating ports across multiple developers is very difficult to do at scale and exposes users to cluster-level issues outside of their control. Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. We give every pod its own cluster-private-IP address so you do not need to explicitly create links between pods or mapping container ports to host ports. This means that containers within a Pod can all reach each other's ports on localhost, and all pods in a cluster can see each other without NAT. The rest of this document will elaborate on how you can run reliable services on such a networking model.
|
||||
Coordinating ports across multiple developers is very difficult to do at scale and exposes users to cluster-level issues outside of their control. Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. We give every pod its own cluster-private-IP address so you do not need to explicitly create links between pods or map container ports to host ports. This means that containers within a Pod can all reach each other's ports on localhost, and all pods in a cluster can see each other without NAT. The rest of this document will elaborate on how you can run reliable services on such a networking model.
|
||||
|
||||
This guide uses a simple nginx server to demonstrate proof of concept. The same principles are embodied in a more complete [Jenkins CI application](https://kubernetes.io/blog/2015/07/strong-simple-ssl-for-kubernetes).
|
||||
|
||||
@@ -35,8 +35,10 @@ Create an nginx Pod, and note that it has a container port specification:
|
||||
This makes it accessible from any node in your cluster. Check the nodes the Pod is running on:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f ./run-my-nginx.yaml
|
||||
$ kubectl get pods -l run=my-nginx -o wide
|
||||
kubectl create -f ./run-my-nginx.yaml
|
||||
kubectl get pods -l run=my-nginx -o wide
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
my-nginx-3800858182-jr4a2 1/1 Running 0 13s 10.244.3.4 kubernetes-minion-905m
|
||||
my-nginx-3800858182-kna2y 1/1 Running 0 13s 10.244.2.5 kubernetes-minion-ljyd
|
||||
@@ -45,7 +47,7 @@ my-nginx-3800858182-kna2y 1/1 Running 0 13s 10.244.2.5
|
||||
Check your pods' IPs:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -l run=my-nginx -o yaml | grep podIP
|
||||
kubectl get pods -l run=my-nginx -o yaml | grep podIP
|
||||
podIP: 10.244.3.4
|
||||
podIP: 10.244.2.5
|
||||
```
|
||||
@@ -63,7 +65,9 @@ A Kubernetes Service is an abstraction which defines a logical set of Pods runni
|
||||
You can create a Service for your 2 nginx replicas with `kubectl expose`:
|
||||
|
||||
```shell
|
||||
$ kubectl expose deployment/my-nginx
|
||||
kubectl expose deployment/my-nginx
|
||||
```
|
||||
```
|
||||
service/my-nginx exposed
|
||||
```
|
||||
|
||||
@@ -81,7 +85,9 @@ API object to see the list of supported fields in service definition.
|
||||
Check your Service:
|
||||
|
||||
```shell
|
||||
$ kubectl get svc my-nginx
|
||||
kubectl get svc my-nginx
|
||||
```
|
||||
```
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
my-nginx ClusterIP 10.0.162.149 <none> 80/TCP 21s
|
||||
```
|
||||
@@ -95,7 +101,9 @@ Check the endpoints, and note that the IPs are the same as the Pods created in
|
||||
the first step:
|
||||
|
||||
```shell
|
||||
$ kubectl describe svc my-nginx
|
||||
kubectl describe svc my-nginx
|
||||
```
|
||||
```
|
||||
Name: my-nginx
|
||||
Namespace: default
|
||||
Labels: run=my-nginx
|
||||
@@ -107,8 +115,11 @@ Port: <unset> 80/TCP
|
||||
Endpoints: 10.244.2.5:80,10.244.3.4:80
|
||||
Session Affinity: None
|
||||
Events: <none>
|
||||
|
||||
$ kubectl get ep my-nginx
|
||||
```
|
||||
```shell
|
||||
kubectl get ep my-nginx
|
||||
```
|
||||
```
|
||||
NAME ENDPOINTS AGE
|
||||
my-nginx 10.244.2.5:80,10.244.3.4:80 1m
|
||||
```
|
||||
@@ -131,7 +142,9 @@ each active Service. This introduces an ordering problem. To see why, inspect
|
||||
the environment of your running nginx Pods (your Pod name will be different):
|
||||
|
||||
```shell
|
||||
$ kubectl exec my-nginx-3800858182-jr4a2 -- printenv | grep SERVICE
|
||||
kubectl exec my-nginx-3800858182-jr4a2 -- printenv | grep SERVICE
|
||||
```
|
||||
```
|
||||
KUBERNETES_SERVICE_HOST=10.0.0.1
|
||||
KUBERNETES_SERVICE_PORT=443
|
||||
KUBERNETES_SERVICE_PORT_HTTPS=443
|
||||
@@ -147,9 +160,11 @@ replicas. This will give you scheduler-level Service spreading of your Pods
|
||||
variables:
|
||||
|
||||
```shell
|
||||
$ kubectl scale deployment my-nginx --replicas=0; kubectl scale deployment my-nginx --replicas=2;
|
||||
kubectl scale deployment my-nginx --replicas=0; kubectl scale deployment my-nginx --replicas=2;
|
||||
|
||||
$ kubectl get pods -l run=my-nginx -o wide
|
||||
kubectl get pods -l run=my-nginx -o wide
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
my-nginx-3800858182-e9ihh 1/1 Running 0 5s 10.244.2.7 kubernetes-minion-ljyd
|
||||
my-nginx-3800858182-j4rm4 1/1 Running 0 5s 10.244.3.8 kubernetes-minion-905m
|
||||
@@ -158,7 +173,9 @@ my-nginx-3800858182-j4rm4 1/1 Running 0 5s 10.244.3.8
|
||||
You may notice that the pods have different names, since they are killed and recreated.
|
||||
|
||||
```shell
|
||||
$ kubectl exec my-nginx-3800858182-e9ihh -- printenv | grep SERVICE
|
||||
kubectl exec my-nginx-3800858182-e9ihh -- printenv | grep SERVICE
|
||||
```
|
||||
```
|
||||
KUBERNETES_SERVICE_PORT=443
|
||||
MY_NGINX_SERVICE_HOST=10.0.162.149
|
||||
KUBERNETES_SERVICE_HOST=10.0.0.1
|
||||
@@ -171,19 +188,23 @@ KUBERNETES_SERVICE_PORT_HTTPS=443
|
||||
Kubernetes offers a DNS cluster addon Service that automatically assigns dns names to other Services. You can check if it's running on your cluster:
|
||||
|
||||
```shell
|
||||
$ kubectl get services kube-dns --namespace=kube-system
|
||||
kubectl get services kube-dns --namespace=kube-system
|
||||
```
|
||||
```
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 8m
|
||||
```
|
||||
|
||||
If it isn't running, you can [enable it](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/README.md#how-do-i-configure-it).
|
||||
If it isn't running, you can [enable it](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/addons/dns/kube-dns/README.md#how-do-i-configure-it).
|
||||
The rest of this section will assume you have a Service with a long lived IP
|
||||
(my-nginx), and a DNS server that has assigned a name to that IP (the CoreDNS
|
||||
cluster addon), so you can talk to the Service from any pod in your cluster using
|
||||
standard methods (e.g. gethostbyname). Let's run another curl application to test this:
|
||||
|
||||
```shell
|
||||
$ kubectl run curl --image=radial/busyboxplus:curl -i --tty
|
||||
kubectl run curl --image=radial/busyboxplus:curl -i --tty
|
||||
```
|
||||
```
|
||||
Waiting for pod default/curl-131556218-9fnch to be running, status is Pending, pod ready: false
|
||||
Hit enter for command prompt
|
||||
```
|
||||
@@ -210,10 +231,16 @@ Till now we have only accessed the nginx server from within the cluster. Before
|
||||
You can acquire all these from the [nginx https example](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/https-nginx/). This requires having go and make tools installed. If you don't want to install those, then follow the manual steps later. In short:
|
||||
|
||||
```shell
|
||||
$ make keys secret KEY=/tmp/nginx.key CERT=/tmp/nginx.crt SECRET=/tmp/secret.json
|
||||
$ kubectl create -f /tmp/secret.json
|
||||
make keys secret KEY=/tmp/nginx.key CERT=/tmp/nginx.crt SECRET=/tmp/secret.json
|
||||
kubectl create -f /tmp/secret.json
|
||||
```
|
||||
```
|
||||
secret/nginxsecret created
|
||||
$ kubectl get secrets
|
||||
```
|
||||
```shell
|
||||
kubectl get secrets
|
||||
```
|
||||
```
|
||||
NAME TYPE DATA AGE
|
||||
default-token-il9rc kubernetes.io/service-account-token 1 1d
|
||||
nginxsecret Opaque 2 1m
|
||||
@@ -242,8 +269,10 @@ data:
|
||||
Now create the secrets using the file:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f nginxsecrets.yaml
|
||||
$ kubectl get secrets
|
||||
kubectl create -f nginxsecrets.yaml
|
||||
kubectl get secrets
|
||||
```
|
||||
```
|
||||
NAME TYPE DATA AGE
|
||||
default-token-il9rc kubernetes.io/service-account-token 1 1d
|
||||
nginxsecret Opaque 2 1m
|
||||
@@ -263,13 +292,13 @@ Noteworthy points about the nginx-secure-app manifest:
|
||||
This is setup *before* the nginx server is started.
|
||||
|
||||
```shell
|
||||
$ kubectl delete deployments,svc my-nginx; kubectl create -f ./nginx-secure-app.yaml
|
||||
kubectl delete deployments,svc my-nginx; kubectl create -f ./nginx-secure-app.yaml
|
||||
```
|
||||
|
||||
At this point you can reach the nginx server from any node.
|
||||
|
||||
```shell
|
||||
$ kubectl get pods -o yaml | grep -i podip
|
||||
kubectl get pods -o yaml | grep -i podip
|
||||
podIP: 10.244.3.5
|
||||
node $ curl -k https://10.244.3.5
|
||||
...
|
||||
@@ -283,11 +312,15 @@ Let's test this from a pod (the same secret is being reused for simplicity, the
|
||||
{{< codenew file="service/networking/curlpod.yaml" >}}
|
||||
|
||||
```shell
|
||||
$ kubectl create -f ./curlpod.yaml
|
||||
$ kubectl get pods -l app=curlpod
|
||||
kubectl create -f ./curlpod.yaml
|
||||
kubectl get pods -l app=curlpod
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
curl-deployment-1515033274-1410r 1/1 Running 0 1m
|
||||
$ kubectl exec curl-deployment-1515033274-1410r -- curl https://my-nginx --cacert /etc/nginx/ssl/nginx.crt
|
||||
```
|
||||
```shell
|
||||
kubectl exec curl-deployment-1515033274-1410r -- curl https://my-nginx --cacert /etc/nginx/ssl/nginx.crt
|
||||
...
|
||||
<title>Welcome to nginx!</title>
|
||||
...
|
||||
@@ -302,7 +335,7 @@ so your nginx HTTPS replica is ready to serve traffic on the internet if your
|
||||
node has a public IP.
|
||||
|
||||
```shell
|
||||
$ kubectl get svc my-nginx -o yaml | grep nodePort -C 5
|
||||
kubectl get svc my-nginx -o yaml | grep nodePort -C 5
|
||||
uid: 07191fb3-f61a-11e5-8ae5-42010af00002
|
||||
spec:
|
||||
clusterIP: 10.0.162.149
|
||||
@@ -319,8 +352,9 @@ spec:
|
||||
targetPort: 443
|
||||
selector:
|
||||
run: my-nginx
|
||||
|
||||
$ kubectl get nodes -o yaml | grep ExternalIP -C 1
|
||||
```
|
||||
```shell
|
||||
kubectl get nodes -o yaml | grep ExternalIP -C 1
|
||||
- address: 104.197.41.11
|
||||
type: ExternalIP
|
||||
allocatable:
|
||||
@@ -338,12 +372,15 @@ $ curl https://<EXTERNAL-IP>:<NODE-PORT> -k
|
||||
Let's now recreate the Service to use a cloud load balancer, just change the `Type` of `my-nginx` Service from `NodePort` to `LoadBalancer`:
|
||||
|
||||
```shell
|
||||
$ kubectl edit svc my-nginx
|
||||
$ kubectl get svc my-nginx
|
||||
kubectl edit svc my-nginx
|
||||
kubectl get svc my-nginx
|
||||
```
|
||||
```
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
my-nginx ClusterIP 10.0.162.149 162.222.184.144 80/TCP,81/TCP,82/TCP 21s
|
||||
|
||||
$ curl https://<EXTERNAL-IP> -k
|
||||
```
|
||||
```
|
||||
curl https://<EXTERNAL-IP> -k
|
||||
...
|
||||
<title>Welcome to nginx!</title>
|
||||
```
|
||||
@@ -357,7 +394,7 @@ output, in fact, so you'll need to do `kubectl describe service my-nginx` to
|
||||
see it. You'll see something like this:
|
||||
|
||||
```shell
|
||||
$ kubectl describe service my-nginx
|
||||
kubectl describe service my-nginx
|
||||
...
|
||||
LoadBalancer Ingress: a320587ffd19711e5a37606cf4a74574-1142138393.us-east-1.elb.amazonaws.com
|
||||
...
|
||||
|
||||
@@ -251,7 +251,7 @@ options ndots:2 edns0
|
||||
For IPv6 setup, search path and name server should be setup like this:
|
||||
|
||||
```
|
||||
$ kubectl exec -it busybox -- cat /etc/resolv.conf
|
||||
$ kubectl exec -it dns-example -- cat /etc/resolv.conf
|
||||
nameserver fd00:79:30::a
|
||||
search default.svc.cluster.local svc.cluster.local cluster.local
|
||||
options ndots:5
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: Ingress Controllers
|
||||
reviewers:
|
||||
content_template: templates/concept
|
||||
weight: 40
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
In order for the Ingress resource to work, the cluster must have an ingress controller running.
|
||||
|
||||
Unlike other types of controllers which run as part of the `kube-controller-manager` binary, Ingress controllers
|
||||
are not started automatically with a cluster. Use this page to choose the ingress controller implementation
|
||||
that best fits your cluster.
|
||||
|
||||
Kubernetes as a project currently supports and maintains [GCE](https://git.k8s.io/ingress-gce/README.md) and
|
||||
[nginx](https://git.k8s.io/ingress-nginx/README.md) controllers.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Additional controllers
|
||||
|
||||
* [Ambassador](https://www.getambassador.io/) API Gateway is an [Envoy](https://www.envoyproxy.io) based ingress
|
||||
controller with [community](https://www.getambassador.io/docs) or
|
||||
[commercial](https://www.getambassador.io/pro/) support from [Datawire](https://www.datawire.io/).
|
||||
* [AppsCode Inc.](https://appscode.com) offers support and maintenance for the most widely used [HAProxy](http://www.haproxy.org/) based ingress controller [Voyager](https://appscode.com/products/voyager).
|
||||
* [Contour](https://github.com/heptio/contour) is an [Envoy](https://www.envoyproxy.io) based ingress controller
|
||||
provided and supported by Heptio.
|
||||
* Citrix provides an [Ingress Controller](https://github.com/citrix/citrix-k8s-ingress-controller) for its hardware (MPX), virtualized (VPX) and [free containerized (CPX) ADC](https://www.citrix.com/products/citrix-adc/cpx-express.html) for [baremetal](https://github.com/citrix/citrix-k8s-ingress-controller/tree/master/deployment/baremetal) and [cloud](https://github.com/citrix/citrix-k8s-ingress-controller/tree/master/deployment) deployments.
|
||||
* F5 Networks provides [support and maintenance](https://support.f5.com/csp/article/K86859508)
|
||||
for the [F5 BIG-IP Controller for Kubernetes](http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest).
|
||||
* [Gloo](https://gloo.solo.io) is an open-source ingress controller based on [Envoy](https://www.envoyproxy.io) which offers API Gateway functionality with enterprise support from [solo.io](https://www.solo.io).
|
||||
* [HAProxy](http://www.haproxy.org/) based ingress controller
|
||||
[jcmoraisjr/haproxy-ingress](https://github.com/jcmoraisjr/haproxy-ingress) which is mentioned on the blog post
|
||||
[HAProxy Ingress Controller for Kubernetes](https://www.haproxy.com/blog/haproxy_ingress_controller_for_kubernetes/).
|
||||
[HAProxy Technologies](https://www.haproxy.com/) offers support and maintenance for HAProxy Enterprise and
|
||||
the ingress controller [jcmoraisjr/haproxy-ingress](https://github.com/jcmoraisjr/haproxy-ingress).
|
||||
* [Istio](https://istio.io/) based ingress controller
|
||||
[Control Ingress Traffic](https://istio.io/docs/tasks/traffic-management/ingress/).
|
||||
* [Kong](https://konghq.com/) offers [community](https://discuss.konghq.com/c/kubernetes) or
|
||||
[commercial](https://konghq.com/kong-enterprise/) support and maintenance for the
|
||||
[Kong Ingress Controller for Kubernetes](https://github.com/Kong/kubernetes-ingress-controller).
|
||||
* [NGINX, Inc.](https://www.nginx.com/) offers support and maintenance for the
|
||||
[NGINX Ingress Controller for Kubernetes](https://www.nginx.com/products/nginx/kubernetes-ingress-controller).
|
||||
* [Traefik](https://github.com/containous/traefik) is a fully featured ingress controller
|
||||
([Let's Encrypt](https://letsencrypt.org), secrets, http2, websocket), and it also comes with commercial
|
||||
support by [Containous](https://containo.us/services).
|
||||
|
||||
## Using multiple Ingress controllers
|
||||
|
||||
You may deploy [any number of ingress controllers](https://git.k8s.io/ingress-nginx/docs/user-guide/multiple-ingress.md#multiple-ingress-controllers)
|
||||
within a cluster. When you create an ingress, you should annotate each ingress with the appropriate
|
||||
[`ingress.class`](https://git.k8s.io/ingress-gce/docs/faq/README.md#how-do-i-run-multiple-ingress-controllers-in-the-same-cluster)
|
||||
to indicate which ingress controller should be used if more than one exists within your cluster.
|
||||
|
||||
If you do not define a class, your cloud provider may use a default ingress provider.
|
||||
|
||||
Ideally, all ingress controllers should fulfill this specification, but the various ingress
|
||||
controllers operate slightly differently.
|
||||
|
||||
{{< note >}}
|
||||
Make sure you review your ingress controller's documentation to understand the caveats of choosing it.
|
||||
{{< /note >}}
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
* Learn more about [Ingress](/docs/concepts/services-networking/ingress/).
|
||||
* [Set up Ingress on Minikube with the NGINX Controller](/docs/tasks/access-application-cluster/ingress-minikube).
|
||||
|
||||
{{% /capture %}}
|
||||
@@ -25,7 +25,7 @@ For the sake of clarity, this guide defines the following terms:
|
||||
|
||||
Ingress, added in Kubernetes v1.1, exposes HTTP and HTTPS routes from outside the cluster to
|
||||
{{< link text="services" url="/docs/concepts/services-networking/service/" >}} within the cluster.
|
||||
Traffic routing is controlled by rules defined on the ingress resource.
|
||||
Traffic routing is controlled by rules defined on the Ingress resource.
|
||||
|
||||
```none
|
||||
internet
|
||||
@@ -35,9 +35,9 @@ Traffic routing is controlled by rules defined on the ingress resource.
|
||||
[ Services ]
|
||||
```
|
||||
|
||||
An ingress can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, and offer name based virtual hosting. An [ingress controller](#ingress-controllers) is responsible for fulfilling the ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic.
|
||||
An Ingress can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, and offer name based virtual hosting. An [Ingress controller](/docs/concepts/services-networking/ingress-controllers) is responsible for fulfilling the Ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic.
|
||||
|
||||
An ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically
|
||||
An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically
|
||||
uses a service of type [Service.Type=NodePort](/docs/concepts/services-networking/service/#nodeport) or
|
||||
[Service.Type=LoadBalancer](/docs/concepts/services-networking/service/#loadbalancer).
|
||||
|
||||
@@ -45,50 +45,19 @@ uses a service of type [Service.Type=NodePort](/docs/concepts/services-networkin
|
||||
|
||||
{{< feature-state for_k8s_version="v1.1" state="beta" >}}
|
||||
|
||||
Before you start using an ingress, there are a few things you should understand. The ingress is a beta resource. You will need an ingress controller to satisfy an ingress, simply creating the resource will have no effect.
|
||||
Before you start using an Ingress, there are a few things you should understand. The Ingress is a beta resource.
|
||||
|
||||
GCE/Google Kubernetes Engine deploys an [ingress controller](#ingress-controllers) on the master. Review the
|
||||
{{< note >}}
|
||||
You must have an [Ingress controller](/docs/concepts/services-networking/ingress-controllers) to satisfy an Ingress. Only creating an Ingress resource has no effect.
|
||||
{{< /note >}}
|
||||
|
||||
GCE/Google Kubernetes Engine deploys an Ingress controller on the master. Review the
|
||||
[beta limitations](https://github.com/kubernetes/ingress-gce/blob/master/BETA_LIMITATIONS.md#glbc-beta-limitations)
|
||||
of this controller if you are using GCE/GKE.
|
||||
|
||||
In environments other than GCE/Google Kubernetes Engine, you may need to
|
||||
[deploy an ingress controller](https://kubernetes.github.io/ingress-nginx/deploy/). There are a number of
|
||||
[ingress controller](#ingress-controllers) you may choose from.
|
||||
|
||||
## Ingress controllers
|
||||
|
||||
In order for the ingress resource to work, the cluster must have an ingress controller running. This is unlike other types of controllers, which run as part of the `kube-controller-manager` binary, and are typically started automatically with a cluster. Choose the ingress controller implementation that best fits your cluster.
|
||||
|
||||
* Kubernetes as a project currently supports and maintains [GCE](https://git.k8s.io/ingress-gce/README.md) and
|
||||
[nginx](https://git.k8s.io/ingress-nginx/README.md) controllers.
|
||||
|
||||
Additional controllers include:
|
||||
|
||||
* [Contour](https://github.com/heptio/contour) is an [Envoy](https://www.envoyproxy.io) based ingress controller
|
||||
provided and supported by Heptio.
|
||||
* F5 Networks provides [support and maintenance](https://support.f5.com/csp/article/K86859508)
|
||||
for the [F5 BIG-IP Controller for Kubernetes](http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest).
|
||||
* [HAProxy](http://www.haproxy.org/) based ingress controller
|
||||
[jcmoraisjr/haproxy-ingress](https://github.com/jcmoraisjr/haproxy-ingress) which is mentioned on the blog post
|
||||
[HAProxy Ingress Controller for Kubernetes](https://www.haproxy.com/blog/haproxy_ingress_controller_for_kubernetes/).
|
||||
[HAProxy Technologies](https://www.haproxy.com/) offers support and maintenance for HAProxy Enterprise and
|
||||
the ingress controller [jcmoraisjr/haproxy-ingress](https://github.com/jcmoraisjr/haproxy-ingress).
|
||||
* [Istio](https://istio.io/) based ingress controller
|
||||
[Control Ingress Traffic](https://istio.io/docs/tasks/traffic-management/ingress/).
|
||||
* [Kong](https://konghq.com/) offers [community](https://discuss.konghq.com/c/kubernetes) or
|
||||
[commercial](https://konghq.com/api-customer-success/) support and maintenance for the
|
||||
[Kong Ingress Controllerfor Kubernetes](https://konghq.com/blog/kubernetes-ingress-controller-for-kong/).
|
||||
* [NGINX, Inc.](https://www.nginx.com/) offers support and maintenance for the
|
||||
[NGINX Ingress Controller for Kubernetes](https://www.nginx.com/products/nginx/kubernetes-ingress-controller).
|
||||
* [Traefik](https://github.com/containous/traefik) is a fully featured ingress controller
|
||||
([Let's Encrypt](https://letsencrypt.org), secrets, http2, websocket), and it also comes with commercial
|
||||
support by [Containous](https://containo.us/services).
|
||||
|
||||
You may deploy [any number of ingress controllers](https://git.k8s.io/ingress-nginx/docs/user-guide/multiple-ingress.md#multiple-ingress-controllers) within a cluster.
|
||||
When you create an ingress, you should annotate each ingress with the appropriate
|
||||
[`ingress-class`](https://git.k8s.io/ingress-gce/examples/PREREQUISITES.md#ingress-class) to indicate which ingress
|
||||
controller should be used if more than one exists within your cluster.
|
||||
If you do not define a class, your cloud provider may use a default ingress provider.
|
||||
[ingress controllers](/docs/concepts/services-networking/ingress-controllers) you may choose from.
|
||||
|
||||
### Before you begin
|
||||
|
||||
@@ -120,14 +89,14 @@ spec:
|
||||
servicePort: 80
|
||||
```
|
||||
|
||||
As with all other Kubernetes resources, an ingress needs `apiVersion`, `kind`, and `metadata` fields.
|
||||
As with all other Kubernetes resources, an Ingress needs `apiVersion`, `kind`, and `metadata` fields.
|
||||
For general information about working with config files, see [deploying applications](/docs/tasks/run-application/run-stateless-application-deployment/), [configuring containers](/docs/tasks/configure-pod-container/configure-pod-configmap/), [managing resources](/docs/concepts/cluster-administration/manage-deployment/).
|
||||
Ingress frequently uses annotations to configure some options depending on the ingress controller, an example of which
|
||||
Ingress frequently uses annotations to configure some options depending on the Ingress controller, an example of which
|
||||
is the [rewrite-target annotation](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md).
|
||||
Different [ingress controller](#ingress-controllers) support different annotations. Review the documentation for
|
||||
your choice of ingress controller to learn which annotations are supported.
|
||||
Different [Ingress controller](/docs/concepts/services-networking/ingress-controllers) support different annotations. Review the documentation for
|
||||
your choice of Ingress controller to learn which annotations are supported.
|
||||
|
||||
The ingress [spec](https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status)
|
||||
The Ingress [spec](https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status)
|
||||
has all the information needed to configure a loadbalancer or proxy server. Most importantly, it
|
||||
contains a list of rules matched against all incoming requests. Ingress resource only supports rules
|
||||
for directing HTTP traffic.
|
||||
@@ -144,18 +113,17 @@ Each http rule contains the following information:
|
||||
loadbalancer will direct traffic to the referenced service.
|
||||
* A backend is a combination of service and port names as described in the
|
||||
[services doc](/docs/concepts/services-networking/service/). HTTP (and HTTPS) requests to the
|
||||
ingress matching the host and path of the rule will be sent to the listed backend.
|
||||
Ingress matching the host and path of the rule will be sent to the listed backend.
|
||||
|
||||
A default backend is often configured in an ingress controller that will service any requests that do not
|
||||
A default backend is often configured in an Ingress controller that will service any requests that do not
|
||||
match a path in the spec.
|
||||
|
||||
### Default Backend
|
||||
|
||||
An ingress with no rules sends all traffic to a single default backend. The default
|
||||
backend is typically a configuration option of the [ingress controller](#ingress-controllers)
|
||||
and is not specified in your ingress resources.
|
||||
An Ingress with no rules sends all traffic to a single default backend. The default
|
||||
backend is typically a configuration option of the [Ingress controller](/docs/concepts/services-networking/ingress-controllers) and is not specified in your Ingress resources.
|
||||
|
||||
If none of the hosts or paths match the HTTP request in the ingress objects, the traffic is
|
||||
If none of the hosts or paths match the HTTP request in the Ingress objects, the traffic is
|
||||
routed to your default backend.
|
||||
|
||||
## Types of Ingress
|
||||
@@ -163,7 +131,7 @@ routed to your default backend.
|
||||
### Single Service Ingress
|
||||
|
||||
There are existing Kubernetes concepts that allow you to expose a single Service
|
||||
(see [alternatives](#alternatives)). You can also do this with an ingress by specifying a
|
||||
(see [alternatives](#alternatives)). You can also do this with an Ingress by specifying a
|
||||
*default backend* with no rules.
|
||||
|
||||
{{< codenew file="service/networking/ingress.yaml" >}}
|
||||
@@ -179,8 +147,8 @@ NAME HOSTS ADDRESS PORTS AGE
|
||||
test-ingress * 107.178.254.228 80 59s
|
||||
```
|
||||
|
||||
Where `107.178.254.228` is the IP allocated by the ingress controller to satisfy
|
||||
this ingress.
|
||||
Where `107.178.254.228` is the IP allocated by the Ingress controller to satisfy
|
||||
this Ingress.
|
||||
|
||||
{{< note >}}
|
||||
Ingress controllers and load balancers may take a minute or two to allocate an IP address.
|
||||
@@ -190,7 +158,7 @@ Until that time you will often see the address listed as `<pending>`.
|
||||
### Simple fanout
|
||||
|
||||
A fanout configuration routes traffic from a single IP address to more than one service,
|
||||
based on the HTTP URI being requested. An ingress allows you to keep the number of loadbalancers
|
||||
based on the HTTP URI being requested. An Ingress allows you to keep the number of loadbalancers
|
||||
down to a minimum. For example, a setup like:
|
||||
|
||||
```shell
|
||||
@@ -198,7 +166,7 @@ foo.bar.com -> 178.91.123.132 -> / foo service1:4200
|
||||
/ bar service2:8080
|
||||
```
|
||||
|
||||
would require an ingress such as:
|
||||
would require an Ingress such as:
|
||||
|
||||
```yaml
|
||||
apiVersion: extensions/v1beta1
|
||||
@@ -222,7 +190,7 @@ spec:
|
||||
servicePort: 8080
|
||||
```
|
||||
|
||||
When you create the ingress with `kubectl create -f`:
|
||||
When you create the Ingress with `kubectl create -f`:
|
||||
|
||||
```shell
|
||||
kubectl describe ingress simple-fanout-example
|
||||
@@ -247,13 +215,13 @@ Events:
|
||||
Normal ADD 22s loadbalancer-controller default/test
|
||||
```
|
||||
|
||||
The ingress controller will provision an implementation specific loadbalancer
|
||||
that satisfies the ingress, as long as the services (`s1`, `s2`) exist.
|
||||
When it has done so, you will see the address of the loadbalancer at the
|
||||
The Ingress controller provisions an implementation specific loadbalancer
|
||||
that satisfies the Ingress, as long as the services (`s1`, `s2`) exist.
|
||||
When it has done so, you can see the address of the loadbalancer at the
|
||||
Address field.
|
||||
|
||||
{{< note >}}
|
||||
Depending on the [ingress controller](#ingress-controllers) you are using, you may need to
|
||||
Depending on the [Ingress controller](/docs/concepts/services-networking/ingress-controllers) you are using, you may need to
|
||||
create a default-http-backend [Service](/docs/concepts/services-networking/service/).
|
||||
{{< /note >}}
|
||||
|
||||
@@ -267,7 +235,7 @@ foo.bar.com --| |-> foo.bar.com s1:80
|
||||
bar.foo.com --| |-> bar.foo.com s2:80
|
||||
```
|
||||
|
||||
The following ingress tells the backing loadbalancer to route requests based on
|
||||
The following Ingress tells the backing loadbalancer to route requests based on
|
||||
the [Host header](https://tools.ietf.org/html/rfc7230#section-5.4).
|
||||
|
||||
```yaml
|
||||
@@ -291,10 +259,10 @@ spec:
|
||||
servicePort: 80
|
||||
```
|
||||
|
||||
If you create an ingress resource without any hosts defined in the rules, then any
|
||||
web traffic to the IP address of your ingress controller can be matched without a name based
|
||||
virtual host being required. For example, the following ingress resource will route traffic
|
||||
requested for `first.bar.com` to `service1`, `second.bar.com` to `service2`, and any traffic
|
||||
If you create an Ingress resource without any hosts defined in the rules, then any
|
||||
web traffic to the IP address of your Ingress controller can be matched without a name based
|
||||
virtual host being required. For example, the following Ingress resource will route traffic
|
||||
requested for `first.bar.com` to `service1`, `second.foo.com` to `service2`, and any traffic
|
||||
to the IP address without a hostname defined in request (that is, without a request header being
|
||||
presented) to `service3`.
|
||||
|
||||
@@ -326,12 +294,12 @@ spec:
|
||||
|
||||
### TLS
|
||||
|
||||
You can secure an ingress by specifying a [secret](/docs/concepts/configuration/secret)
|
||||
that contains a TLS private key and certificate. Currently the ingress only
|
||||
You can secure an Ingress by specifying a [secret](/docs/concepts/configuration/secret)
|
||||
that contains a TLS private key and certificate. Currently the Ingress only
|
||||
supports a single TLS port, 443, and assumes TLS termination. If the TLS
|
||||
configuration section in an ingress specifies different hosts, they will be
|
||||
configuration section in an Ingress specifies different hosts, they will be
|
||||
multiplexed on the same port according to the hostname specified through the
|
||||
SNI TLS extension (provided the ingress controller supports SNI). The TLS secret
|
||||
SNI TLS extension (provided the Ingress controller supports SNI). The TLS secret
|
||||
must contain keys named `tls.crt` and `tls.key` that contain the certificate
|
||||
and private key to use for TLS, e.g.:
|
||||
|
||||
@@ -344,10 +312,10 @@ kind: Secret
|
||||
metadata:
|
||||
name: testsecret-tls
|
||||
namespace: default
|
||||
type: Opaque
|
||||
type: kubernetes.io/tls
|
||||
```
|
||||
|
||||
Referencing this secret in an ingress will tell the ingress controller to
|
||||
Referencing this secret in an Ingress will tell the Ingress controller to
|
||||
secure the channel from the client to the loadbalancer using TLS. You need to make
|
||||
sure the TLS secret you created came from a certificate that contains a CN
|
||||
for `sslexample.foo.com`.
|
||||
@@ -373,24 +341,24 @@ spec:
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
There is a gap between TLS features supported by various ingress
|
||||
There is a gap between TLS features supported by various Ingress
|
||||
controllers. Please refer to documentation on
|
||||
[nginx](https://git.k8s.io/ingress-nginx/README.md#https),
|
||||
[GCE](https://git.k8s.io/ingress-gce/README.md#frontend-https), or any other
|
||||
platform specific ingress controller to understand how TLS works in your environment.
|
||||
platform specific Ingress controller to understand how TLS works in your environment.
|
||||
{{< /note >}}
|
||||
|
||||
### Loadbalancing
|
||||
|
||||
An ingress controller is bootstrapped with some load balancing policy settings
|
||||
that it applies to all ingress, such as the load balancing algorithm, backend
|
||||
An Ingress controller is bootstrapped with some load balancing policy settings
|
||||
that it applies to all Ingress, such as the load balancing algorithm, backend
|
||||
weight scheme, and others. More advanced load balancing concepts
|
||||
(e.g. persistent sessions, dynamic weights) are not yet exposed through the
|
||||
ingress. You can still get these features through the
|
||||
Ingress. You can still get these features through the
|
||||
[service loadbalancer](https://github.com/kubernetes/ingress-nginx).
|
||||
|
||||
It's also worth noting that even though health checks are not exposed directly
|
||||
through the ingress, there exist parallel concepts in Kubernetes such as
|
||||
through the Ingress, there exist parallel concepts in Kubernetes such as
|
||||
[readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
|
||||
which allow you to achieve the same end result. Please review the controller
|
||||
specific docs to see how they handle health checks (
|
||||
@@ -399,7 +367,7 @@ specific docs to see how they handle health checks (
|
||||
|
||||
## Updating an Ingress
|
||||
|
||||
To update an existing ingress to add a new Host, you can update it by editing the resource:
|
||||
To update an existing Ingress to add a new Host, you can update it by editing the resource:
|
||||
|
||||
```shell
|
||||
kubectl describe ingress test
|
||||
@@ -450,7 +418,7 @@ spec:
|
||||
```
|
||||
|
||||
Saving the yaml will update the resource in the API server, which should tell the
|
||||
ingress controller to reconfigure the loadbalancer.
|
||||
Ingress controller to reconfigure the loadbalancer.
|
||||
|
||||
```shell
|
||||
kubectl describe ingress test
|
||||
@@ -476,25 +444,24 @@ Events:
|
||||
Normal ADD 45s loadbalancer-controller default/test
|
||||
```
|
||||
|
||||
You can achieve the same by invoking `kubectl replace -f` on a modified ingress yaml file.
|
||||
You can achieve the same by invoking `kubectl replace -f` on a modified Ingress yaml file.
|
||||
|
||||
## Failing across availability zones
|
||||
|
||||
Techniques for spreading traffic across failure domains differs between cloud providers.
|
||||
Please check the documentation of the relevant [ingress controller](#ingress-controllers) for
|
||||
details. You can also refer to the [federation documentation](/docs/concepts/cluster-administration/federation/)
|
||||
for details on deploying ingress in a federated cluster.
|
||||
Please check the documentation of the relevant [Ingress controller](/docs/concepts/services-networking/ingress-controllers) for details. You can also refer to the [federation documentation](/docs/concepts/cluster-administration/federation/)
|
||||
for details on deploying Ingress in a federated cluster.
|
||||
|
||||
## Future Work
|
||||
|
||||
Track [SIG Network](https://github.com/kubernetes/community/tree/master/sig-network)
|
||||
for more details on the evolution of the ingress and related resources. You may also track the
|
||||
[ingress repository](https://github.com/kubernetes/ingress/tree/master) for more details on the
|
||||
evolution of various ingress controllers.
|
||||
[Ingress repository](https://github.com/kubernetes/ingress/tree/master) for more details on the
|
||||
evolution of various Ingress controllers.
|
||||
|
||||
## Alternatives
|
||||
|
||||
You can expose a Service in multiple ways that don't directly involve the ingress resource:
|
||||
You can expose a Service in multiple ways that don't directly involve the Ingress resource:
|
||||
|
||||
* Use [Service.Type=LoadBalancer](/docs/concepts/services-networking/service/#loadbalancer)
|
||||
* Use [Service.Type=NodePort](/docs/concepts/services-networking/service/#nodeport)
|
||||
@@ -503,6 +470,5 @@ You can expose a Service in multiple ways that don't directly involve the ingres
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
* [Set up Ingress on Minikube with the NGINX Controller](/docs/tasks/access-application-cluster/ingress-minikube)
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -92,11 +92,12 @@ __egress__: Each `NetworkPolicy` may include a list of whitelist `egress` rules.
|
||||
So, the example NetworkPolicy:
|
||||
|
||||
1. isolates "role=db" pods in the "default" namespace for both ingress and egress traffic (if they weren't already isolated)
|
||||
2. allows connections to TCP port 6379 of "role=db" pods in the "default" namespace from:
|
||||
2. (Ingress rules) allows connections to all pods in the “default” namespace with the label “role=db” on TCP port 6379 from:
|
||||
|
||||
* any pod in the "default" namespace with the label "role=frontend"
|
||||
* any pod in a namespace with the label "project=myproject"
|
||||
* IP addresses in the ranges 172.17.0.0–172.17.0.255 and 172.17.2.0–172.17.255.255 (ie, all of 172.17.0.0/16 except 172.17.1.0/24)
|
||||
3. allows connections from any pod in the "default" namespace with the label "role=db" to CIDR 10.0.0.0/24 on TCP port 5978
|
||||
3. (Egress rules) allows connections from any pod in the "default" namespace with the label "role=db" to CIDR 10.0.0.0/24 on TCP port 5978
|
||||
|
||||
See the [Declare Network Policy](/docs/tasks/administer-cluster/declare-network-policy/) walkthrough for further examples.
|
||||
|
||||
@@ -191,6 +192,8 @@ spec:
|
||||
podSelector: {}
|
||||
ingress:
|
||||
- {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
```
|
||||
|
||||
### Default deny all egress traffic
|
||||
@@ -264,4 +267,3 @@ The CNI plugin has to support SCTP as `protocol` value in `NetworkPolicy`.
|
||||
- See more [Recipes](https://github.com/ahmetb/kubernetes-network-policy-recipes) for common scenarios enabled by the NetworkPolicy resource.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
@@ -83,12 +83,9 @@ deploying and evolving your `Services`. For example, you can change the port
|
||||
number that pods expose in the next version of your backend software, without
|
||||
breaking clients.
|
||||
|
||||
Kubernetes `Services` support `TCP`, `UDP` and `SCTP` for protocols. The default
|
||||
is `TCP`.
|
||||
|
||||
{{< note >}}
|
||||
SCTP support is an alpha feature since Kubernetes 1.12
|
||||
{{< /note >}}
|
||||
`TCP` is the default protocol for services, and you can also use any other
|
||||
[supported protocol](#protocol-support). At the moment, you can only set a
|
||||
single `port` and `protocol` for a Service.
|
||||
|
||||
### Services without selectors
|
||||
|
||||
@@ -519,6 +516,16 @@ metadata:
|
||||
[...]
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{% tab name="Baidu Cloud" %}}
|
||||
```yaml
|
||||
[...]
|
||||
metadata:
|
||||
name: my-service
|
||||
annotations:
|
||||
service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
|
||||
[...]
|
||||
```
|
||||
{{% /tab %}}
|
||||
{{< /tabs >}}
|
||||
|
||||
|
||||
@@ -758,13 +765,10 @@ for supported instance types.
|
||||
|
||||
### Type ExternalName {#externalname}
|
||||
|
||||
{{< note >}}
|
||||
ExternalName Services are available only with `kube-dns` version 1.7 and later.
|
||||
{{< /note >}}
|
||||
Services of type ExternalName map a service to a DNS name, not to a typical selector such as
|
||||
`my-service` or `cassandra`. You specify these services with the `spec.externalName` parameter.
|
||||
|
||||
Services of type ExternalName map a service to a DNS name (specified using
|
||||
the `spec.externalName` parameter) rather than to a typical selector like
|
||||
`my-service` or `cassandra`. This Service definition, for example, would map
|
||||
This Service definition, for example, maps
|
||||
the `my-service` Service in the `prod` namespace to `my.database.example.com`:
|
||||
|
||||
```yaml
|
||||
@@ -777,6 +781,10 @@ spec:
|
||||
type: ExternalName
|
||||
externalName: my.database.example.com
|
||||
```
|
||||
{{< note >}}
|
||||
ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName
|
||||
is intended to specify a canonical DNS name. To hardcode an IP address, consider headless services.
|
||||
{{< /note >}}
|
||||
|
||||
When looking up the host `my-service.prod.svc.cluster.local`, the cluster DNS service
|
||||
will return a `CNAME` record with the value `my.database.example.com`. Accessing
|
||||
@@ -933,29 +941,74 @@ Service is a top-level resource in the Kubernetes REST API. More details about t
|
||||
API object can be found at:
|
||||
[Service API object](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#service-v1-core).
|
||||
|
||||
## SCTP support
|
||||
## Supported protocols {#protocol-support}
|
||||
|
||||
### TCP
|
||||
|
||||
{{< feature-state for_k8s_version="v1.0" state="stable" >}}
|
||||
|
||||
You can use TCP for any kind of service, and it's the default network protocol.
|
||||
|
||||
### UDP
|
||||
|
||||
{{< feature-state for_k8s_version="v1.0" state="stable" >}}
|
||||
|
||||
You can use UDP for most services. For type=LoadBalancer services, UDP support
|
||||
depends on the cloud provider offering this facility.
|
||||
|
||||
### HTTP
|
||||
|
||||
{{< feature-state for_k8s_version="v1.1" state="stable" >}}
|
||||
|
||||
If your cloud provider supports it, you can use a Service in LoadBalancer mode
|
||||
to set up external HTTP / HTTPS reverse proxying, forwarded to the Endpoints
|
||||
of the Service.
|
||||
|
||||
{{< note >}}
|
||||
You can also use {{< glossary_tooltip term_id="ingress" >}} in place of Service
|
||||
to expose HTTP / HTTPS services.
|
||||
{{< /note >}}
|
||||
|
||||
### PROXY protocol
|
||||
|
||||
{{< feature-state for_k8s_version="v1.1" state="stable" >}}
|
||||
|
||||
If your cloud provider supports it (eg, [AWS](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws)),
|
||||
you can use a Service in LoadBalancer mode to configure a load balancer outside
|
||||
of Kubernetes itself, that will forward connections prefixed with
|
||||
[PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt).
|
||||
|
||||
The load balancer will send an initial series of octets describing the
|
||||
incoming connection, similar to this example
|
||||
|
||||
```
|
||||
PROXY TCP4 192.0.2.202 10.0.42.7 12345 7\r\n
|
||||
```
|
||||
followed by the data from the client.
|
||||
|
||||
### SCTP
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="alpha" >}}
|
||||
|
||||
Kubernetes supports SCTP as a `protocol` value in `Service`, `Endpoint`, `NetworkPolicy` and `Pod` definitions as an alpha feature. To enable this feature, the cluster administrator needs to enable the `SCTPSupport` feature gate on the apiserver, for example, `“--feature-gates=SCTPSupport=true,...”`. When the feature gate is enabled, users can set the `protocol` field of a `Service`, `Endpoint`, `NetworkPolicy` and `Pod` to `SCTP`. Kubernetes sets up the network accordingly for the SCTP associations, just like it does for TCP connections.
|
||||
|
||||
### Warnings
|
||||
#### Warnings {#caveat-sctp-overview}
|
||||
|
||||
#### The support of multihomed SCTP associations
|
||||
##### Support for multihomed SCTP associations {#caveat-sctp-multihomed}
|
||||
|
||||
The support of multihomed SCTP associations requires that the CNI plugin can support the assignment of multiple interfaces and IP addresses to a `Pod`.
|
||||
|
||||
NAT for multihomed SCTP associations requires special logic in the corresponding kernel modules.
|
||||
|
||||
#### Service with type=LoadBalancer
|
||||
##### Service with type=LoadBalancer {#caveat-sctp-loadbalancer-service-type}
|
||||
|
||||
A `Service` with `type` LoadBalancer and `protocol` SCTP can be created only if the cloud provider's load balancer implementation supports SCTP as a protocol. Otherwise the `Service` creation request is rejected. The current set of cloud load balancer providers (`Azure`, `AWS`, `CloudStack`, `GCE`, `OpenStack`) do not support SCTP.
|
||||
|
||||
#### Windows
|
||||
##### Windows {#caveat-sctp-windows-os}
|
||||
|
||||
SCTP is not supported on Windows based nodes.
|
||||
|
||||
#### Userspace kube-proxy
|
||||
##### Userspace kube-proxy {#caveat-sctp-kube-proxy-userspace}
|
||||
|
||||
The kube-proxy does not support the management of SCTP associations when it is in userspace mode.
|
||||
|
||||
|
||||
@@ -309,9 +309,7 @@ Currently, storage size is the only resource that can be set or requested. Futu
|
||||
|
||||
### Volume Mode
|
||||
|
||||
{{< feature-state for_k8s_version="v1.9" state="alpha" >}}
|
||||
|
||||
To enable this feature, enable the `BlockVolume` feature gate on the apiserver, controller-manager and the kubelet.
|
||||
{{< feature-state for_k8s_version="v1.13" state="beta" >}}
|
||||
|
||||
Prior to Kubernetes 1.9, all volume plugins created a filesystem on the persistent volume.
|
||||
Now, you can set the value of `volumeMode` to `raw` to use a raw block device, or `filesystem`
|
||||
@@ -461,7 +459,7 @@ Claims use the same conventions as volumes when requesting storage with specific
|
||||
|
||||
### Volume Modes
|
||||
|
||||
Claims use the same convention as volumes to indicates the consumption of the volume as either a filesystem or block device.
|
||||
Claims use the same convention as volumes to indicate the consumption of the volume as either a filesystem or block device.
|
||||
|
||||
### Resources
|
||||
|
||||
@@ -548,10 +546,7 @@ spec:
|
||||
|
||||
## Raw Block Volume Support
|
||||
|
||||
{{< feature-state for_k8s_version="v1.9" state="alpha" >}}
|
||||
|
||||
To enable support for raw block volumes, enable the `BlockVolume` feature gate on the
|
||||
apiserver, controller-manager and the kubelet.
|
||||
{{< feature-state for_k8s_version="v1.13" state="beta" >}}
|
||||
|
||||
The following volume plugins support raw block volumes, including dynamic provisioning where
|
||||
applicable.
|
||||
|
||||
@@ -153,7 +153,7 @@ The following plugins support `WaitForFirstConsumer` with pre-created Persistent
|
||||
|
||||
### Allowed Topologies
|
||||
|
||||
When a cluster operactor specifies the `WaitForFirstConsumer` volume binding mode, it is no longer necessary
|
||||
When a cluster operator specifies the `WaitForFirstConsumer` volume binding mode, it is no longer necessary
|
||||
to restrict provisioning to specific topologies in most situations. However,
|
||||
if still required, `allowedTopologies` can be specified.
|
||||
|
||||
@@ -627,13 +627,13 @@ parameters:
|
||||
|
||||
```
|
||||
|
||||
* `fs`: filesystem to be laid out: [none/xfs/ext4] (default: `ext4`).
|
||||
* `fs`: filesystem to be laid out: `none/xfs/ext4` (default: `ext4`).
|
||||
* `block_size`: block size in Kbytes (default: `32`).
|
||||
* `repl`: number of synchronous replicas to be provided in the form of
|
||||
replication factor [1..3] (default: `1`) A string is expected here i.e.
|
||||
replication factor `1..3` (default: `1`) A string is expected here i.e.
|
||||
`"1"` and not `1`.
|
||||
* `io_priority`: determines whether the volume will be created from higher
|
||||
performance or a lower priority storage [high/medium/low] (default: `low`).
|
||||
performance or a lower priority storage `high/medium/low` (default: `low`).
|
||||
* `snap_interval`: clock/time interval in minutes for when to trigger snapshots.
|
||||
Snapshots are incremental based on difference with the prior snapshot, 0
|
||||
disables snaps (default: `0`). A string is expected here i.e.
|
||||
@@ -644,7 +644,7 @@ parameters:
|
||||
* `ephemeral`: specifies whether the volume should be cleaned-up after unmount
|
||||
or should be persistent. `emptyDir` use case can set this value to true and
|
||||
`persistent volumes` use case such as for databases like Cassandra should set
|
||||
to false, [true/false] (default `false`). A string is expected here i.e.
|
||||
to false, `true/false` (default `false`). A string is expected here i.e.
|
||||
`"true"` and not `true`.
|
||||
|
||||
### ScaleIO
|
||||
|
||||
@@ -75,6 +75,7 @@ Kubernetes supports several types of Volumes:
|
||||
* [downwardAPI](#downwardapi)
|
||||
* [emptyDir](#emptydir)
|
||||
* [fc (fibre channel)](#fc)
|
||||
* [flexVolume](#flexVolume)
|
||||
* [flocker](#flocker)
|
||||
* [gcePersistentDisk](#gcepersistentdisk)
|
||||
* [gitRepo (deprecated)](#gitrepo)
|
||||
@@ -789,9 +790,8 @@ receive updates for those volume sources.
|
||||
### portworxVolume {#portworxvolume}
|
||||
|
||||
A `portworxVolume` is an elastic block storage layer that runs hyperconverged with
|
||||
Kubernetes. Portworx fingerprints storage in a server, tiers based on capabilities,
|
||||
and aggregates capacity across multiple servers. Portworx runs in-guest in virtual
|
||||
machines or on bare metal Linux nodes.
|
||||
Kubernetes. [Portworx](https://portworx.com/use-case/kubernetes-storage/) fingerprints storage in a server, tiers based on capabilities,
|
||||
and aggregates capacity across multiple servers. Portworx runs in-guest in virtual machines or on bare metal Linux nodes.
|
||||
|
||||
A `portworxVolume` can be dynamically created through Kubernetes or it can also
|
||||
be pre-provisioned and referenced inside a Kubernetes Pod.
|
||||
@@ -834,7 +834,9 @@ You must have your own Quobyte setup running with the volumes
|
||||
created before you can use it.
|
||||
{{< /caution >}}
|
||||
|
||||
See the [Quobyte example](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/quobyte) for more details.
|
||||
Quobyte supports the {{< glossary_tooltip text="Container Storage Interface" term_id="csi" >}}.
|
||||
CSI is the recommended plugin to use Quobyte volumes inside Kubernetes. Quobyte's
|
||||
GitHub project has [instructions](https://github.com/quobyte/quobyte-csi#quobyte-csi) for deploying Quobyte using CSI, along with examples.
|
||||
|
||||
### rbd {#rbd}
|
||||
|
||||
@@ -1149,12 +1151,12 @@ CSI support was introduced as alpha in Kubernetes v1.9, moved to beta in
|
||||
Kubernetes v1.10, and is GA in Kubernetes v1.13.
|
||||
|
||||
{{< note >}}
|
||||
**Note:** Support for CSI spec versions 0.2 and 0.3 are deprecated in Kubernetes
|
||||
Support for CSI spec versions 0.2 and 0.3 are deprecated in Kubernetes
|
||||
v1.13 and will be removed in a future release.
|
||||
{{< /note >}}
|
||||
|
||||
{{< note >}}
|
||||
**Note:** CSI drivers may not be compatible across all Kubernetes releases.
|
||||
CSI drivers may not be compatible across all Kubernetes releases.
|
||||
Please check the specific CSI driver's documentation for supported
|
||||
deployments steps for each Kubernetes release and a compatibility matrix.
|
||||
{{< /note >}}
|
||||
@@ -1237,7 +1239,7 @@ Learn how to
|
||||
For more information on how to develop a CSI driver, refer to the [kubernetes-csi
|
||||
documentation](https://kubernetes-csi.github.io/docs/)
|
||||
|
||||
### Flexvolume
|
||||
### Flexvolume {#flexVolume}
|
||||
|
||||
Flexvolume is an out-of-tree plugin interface that has existed in Kubernetes
|
||||
since version 1.2 (before CSI). It uses an exec-based model to interface with
|
||||
@@ -1305,8 +1307,8 @@ MountFlags=shared
|
||||
```
|
||||
Or, remove `MountFlags=slave` if present. Then restart the Docker daemon:
|
||||
```shell
|
||||
$ sudo systemctl daemon-reload
|
||||
$ sudo systemctl restart docker
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ One CronJob object is like one line of a _crontab_ (cron table) file. It runs a
|
||||
on a given schedule, written in [Cron](https://en.wikipedia.org/wiki/Cron) format.
|
||||
|
||||
{{< note >}}
|
||||
All **CronJob** `schedule:` times are denoted in UTC.
|
||||
All **CronJob** `schedule:` times are based the timezone of the master where the job is initiated.
|
||||
{{< /note >}}
|
||||
|
||||
For instructions on creating and working with cron jobs, and for an example of a spec file for a cron job, see [Running automated tasks with cron jobs](/docs/tasks/job/automated-tasks-with-cron-jobs).
|
||||
@@ -46,11 +46,13 @@ It is important to note that if the `startingDeadlineSeconds` field is set (not
|
||||
|
||||
A CronJob is counted as missed if it has failed to be created at its scheduled time. For example, If `concurrencyPolicy` is set to `Forbid` and a CronJob was attempted to be scheduled when there was a previous schedule still running, then it would count as missed.
|
||||
|
||||
For example, suppose a cron job is set to start at exactly `08:30:00` and its
|
||||
`startingDeadlineSeconds` is set to 10, if the CronJob controller happens to
|
||||
be down from `08:29:00` to `08:42:00`, the job will not start.
|
||||
Set a longer `startingDeadlineSeconds` if starting later is better than not
|
||||
starting at all.
|
||||
For example, suppose a CronJob is set to schedule a new Job every one minute beginning at `08:30:00`, and its
|
||||
`startingDeadlineSeconds` field is not set. The default for this field is `100` seconds. If the CronJob controller happens to
|
||||
be down from `08:29:00` to `10:21:00`, the job will not start as the number of missed jobs which missed their schedule is greater than 100.
|
||||
|
||||
To illustrate this concept further, suppose a CronJob is set to schedule a new Job every one minute beginning at `08:30:00`, and its
|
||||
`startingDeadlineSeconds` is set to 200 seconds. If the CronJob controller happens to
|
||||
be down for the same period as the previous example (`08:29:00` to `10:21:00`,) the Job will still start at 10:22:00. This happens as the controller now checks how many missed schedules happened in the last 200 seconds (ie, 3 missed schedules), rather than from the last scheduled time until now.
|
||||
|
||||
The Cronjob is only responsible for creating Jobs that match its schedule, and
|
||||
the Job in turn is responsible for the management of the Pods it represents.
|
||||
|
||||
@@ -21,7 +21,7 @@ Some typical uses of a DaemonSet are:
|
||||
- running a cluster storage daemon, such as `glusterd`, `ceph`, on each node.
|
||||
- running a logs collection daemon on every node, such as `fluentd` or `logstash`.
|
||||
- running a node monitoring daemon on every node, such as [Prometheus Node Exporter](
|
||||
https://github.com/prometheus/node_exporter), `collectd`, [Dynatrace OneAgent](https://www.dynatrace.com/technologies/kubernetes-monitoring/), Datadog agent, New Relic agent, Ganglia `gmond` or Instana agent.
|
||||
https://github.com/prometheus/node_exporter), `collectd`, [Dynatrace OneAgent](https://www.dynatrace.com/technologies/kubernetes-monitoring/), [AppDynamics Agent](https://docs.appdynamics.com/display/CLOUD/Container+Visibility+with+Kubernetes), [Datadog agent](https://docs.datadoghq.com/agent/kubernetes/daemonset_setup/), [New Relic agent](https://docs.newrelic.com/docs/integrations/kubernetes-integration/installation/kubernetes-installation-configuration), Ganglia `gmond` or Instana agent.
|
||||
|
||||
In a simple case, one DaemonSet, covering all nodes, would be used for each type of daemon.
|
||||
A more complex setup might use multiple DaemonSets for a single type of daemon, but with
|
||||
|
||||
@@ -40,7 +40,6 @@ The following are typical use cases for Deployments:
|
||||
* [Use the status of the Deployment](#deployment-status) as an indicator that a rollout has stuck.
|
||||
* [Clean up older ReplicaSets](#clean-up-policy) that you don't need anymore.
|
||||
|
||||
|
||||
## Creating a Deployment
|
||||
|
||||
The following is an example of a Deployment. It creates a ReplicaSet to bring up three `nginx` Pods:
|
||||
@@ -55,9 +54,9 @@ In this example:
|
||||
In this case, you simply select a label that is defined in the Pod template (`app: nginx`).
|
||||
However, more sophisticated selection rules are possible,
|
||||
as long as the Pod template itself satisfies the rule.
|
||||
|
||||
|
||||
{{< note >}}
|
||||
`matchLabels` is a map of {key,value} pairs. A single {key,value} in the `matchLabels` map
|
||||
`matchLabels` is a map of {key,value} pairs. A single {key,value} in the `matchLabels` map
|
||||
is equivalent to an element of `matchExpressions`, whose key field is "key", the operator is "In",
|
||||
and the values array contains only "value". The requirements are ANDed.
|
||||
{{< /note >}}
|
||||
@@ -66,9 +65,9 @@ In this example:
|
||||
* The Pods are labeled `app: nginx`using the `labels` field.
|
||||
* The Pod template's specification, or `.template.spec` field, indicates that
|
||||
the Pods run one container, `nginx`, which runs the `nginx`
|
||||
[Docker Hub](https://hub.docker.com/) image at version 1.15.4.
|
||||
[Docker Hub](https://hub.docker.com/) image at version 1.7.9.
|
||||
* Create one container and name it `nginx` using the `name` field.
|
||||
* Run the `nginx` image at version `1.15.4`.
|
||||
* Run the `nginx` image at version `1.7.9`.
|
||||
* Open port `80` so that the container can send and accept traffic.
|
||||
|
||||
To create this Deployment, run the following command:
|
||||
@@ -128,18 +127,19 @@ To see the ReplicaSet (`rs`) created by the deployment, run `kubectl get rs`:
|
||||
|
||||
```shell
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-deployment-2035384211 3 3 3 18s
|
||||
nginx-deployment-75675f5897 3 3 3 18s
|
||||
```
|
||||
|
||||
Notice that the name of the ReplicaSet is always formatted as `[DEPLOYMENT-NAME]-[POD-TEMPLATE-HASH-VALUE]`. The hash value is automatically generated when the Deployment is created.
|
||||
Notice that the name of the ReplicaSet is always formatted as `[DEPLOYMENT-NAME]-[RANDOM-STRING]`. The random string is
|
||||
randomly generated and uses the pod-template-hash as a seed.
|
||||
|
||||
To see the labels automatically generated for each pod, run `kubectl get pods --show-labels`. The following output is returned:
|
||||
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE LABELS
|
||||
nginx-deployment-2035384211-7ci7o 1/1 Running 0 18s app=nginx,pod-template-hash=2035384211
|
||||
nginx-deployment-2035384211-kzszj 1/1 Running 0 18s app=nginx,pod-template-hash=2035384211
|
||||
nginx-deployment-2035384211-qqcnn 1/1 Running 0 18s app=nginx,pod-template-hash=2035384211
|
||||
nginx-deployment-75675f5897-7ci7o 1/1 Running 0 18s app=nginx,pod-template-hash=3123191453
|
||||
nginx-deployment-75675f5897-kzszj 1/1 Running 0 18s app=nginx,pod-template-hash=3123191453
|
||||
nginx-deployment-75675f5897-qqcnn 1/1 Running 0 18s app=nginx,pod-template-hash=3123191453
|
||||
```
|
||||
|
||||
The created ReplicaSet ensures that there are three `nginx` Pods running at all times.
|
||||
@@ -171,21 +171,27 @@ Suppose that you now want to update the nginx Pods to use the `nginx:1.9.1` imag
|
||||
instead of the `nginx:1.7.9` image.
|
||||
|
||||
```shell
|
||||
$ kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1 --record
|
||||
deployment.apps/nginx-deployment image updated
|
||||
kubectl --record deployment.apps/nginx-deployment set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1
|
||||
```
|
||||
```
|
||||
image updated
|
||||
```
|
||||
|
||||
Alternatively, you can `edit` the Deployment and change `.spec.template.spec.containers[0].image` from `nginx:1.7.9` to `nginx:1.9.1`:
|
||||
|
||||
```shell
|
||||
$ kubectl edit deployment.v1.apps/nginx-deployment
|
||||
kubectl edit deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment edited
|
||||
```
|
||||
|
||||
To see the rollout status, run:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
|
||||
deployment.apps/nginx-deployment successfully rolled out
|
||||
```
|
||||
@@ -193,7 +199,9 @@ deployment.apps/nginx-deployment successfully rolled out
|
||||
After the rollout succeeds, you may want to `get` the Deployment:
|
||||
|
||||
```shell
|
||||
$ kubectl get deployments
|
||||
kubectl get deployments
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
nginx-deployment 3 3 3 3 36s
|
||||
```
|
||||
@@ -206,7 +214,9 @@ You can run `kubectl get rs` to see that the Deployment updated the Pods by crea
|
||||
up to 3 replicas, as well as scaling down the old ReplicaSet to 0 replicas.
|
||||
|
||||
```shell
|
||||
$ kubectl get rs
|
||||
kubectl get rs
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-deployment-1564180365 3 3 3 6s
|
||||
nginx-deployment-2035384211 0 0 0 36s
|
||||
@@ -215,7 +225,9 @@ nginx-deployment-2035384211 0 0 0 36s
|
||||
Running `get pods` should now show only the new Pods:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
kubectl get pods
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
nginx-deployment-1564180365-khku8 1/1 Running 0 14s
|
||||
nginx-deployment-1564180365-nacti 1/1 Running 0 14s
|
||||
@@ -236,7 +248,9 @@ new Pods have come up, and does not create new Pods until a sufficient number of
|
||||
It makes sure that number of available Pods is at least 2 and the number of total Pods is at most 4.
|
||||
|
||||
```shell
|
||||
$ kubectl describe deployments
|
||||
kubectl describe deployments
|
||||
```
|
||||
```
|
||||
Name: nginx-deployment
|
||||
Namespace: default
|
||||
CreationTimestamp: Thu, 30 Nov 2017 10:56:25 +0000
|
||||
@@ -337,14 +351,18 @@ rolled back.
|
||||
Suppose that you made a typo while updating the Deployment, by putting the image name as `nginx:1.91` instead of `nginx:1.9.1`:
|
||||
|
||||
```shell
|
||||
$ kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.91 --record=true
|
||||
kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.91 --record=true
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment image updated
|
||||
```
|
||||
|
||||
The rollout will be stuck.
|
||||
|
||||
```shell
|
||||
$ kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
|
||||
```
|
||||
|
||||
@@ -354,7 +372,9 @@ Press Ctrl-C to stop the above rollout status watch. For more information on stu
|
||||
You will see that the number of old replicas (nginx-deployment-1564180365 and nginx-deployment-2035384211) is 2, and new replicas (nginx-deployment-3066724191) is 1.
|
||||
|
||||
```shell
|
||||
$ kubectl get rs
|
||||
kubectl get rs
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-deployment-1564180365 3 3 3 25s
|
||||
nginx-deployment-2035384211 0 0 0 36s
|
||||
@@ -364,7 +384,9 @@ nginx-deployment-3066724191 1 1 0 6s
|
||||
Looking at the Pods created, you will see that 1 Pod created by new ReplicaSet is stuck in an image pull loop.
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
kubectl get pods
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
nginx-deployment-1564180365-70iae 1/1 Running 0 25s
|
||||
nginx-deployment-1564180365-jbqqo 1/1 Running 0 25s
|
||||
@@ -379,7 +401,9 @@ Kubernetes by default sets the value to 25%.
|
||||
{{< /note >}}
|
||||
|
||||
```shell
|
||||
$ kubectl describe deployment
|
||||
kubectl describe deployment
|
||||
```
|
||||
```
|
||||
Name: nginx-deployment
|
||||
Namespace: default
|
||||
CreationTimestamp: Tue, 15 Mar 2016 14:48:04 -0700
|
||||
@@ -426,7 +450,9 @@ To fix this, you need to rollback to a previous revision of Deployment that is s
|
||||
First, check the revisions of this deployment:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout history deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout history deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
deployments "nginx-deployment"
|
||||
REVISION CHANGE-CAUSE
|
||||
1 kubectl create --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml --record=true
|
||||
@@ -442,7 +468,9 @@ REVISION CHANGE-CAUSE
|
||||
To further see the details of each revision, run:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout history deployment.v1.apps/nginx-deployment --revision=2
|
||||
kubectl rollout history deployment.v1.apps/nginx-deployment --revision=2
|
||||
```
|
||||
```
|
||||
deployments "nginx-deployment" revision 2
|
||||
Labels: app=nginx
|
||||
pod-template-hash=1159050644
|
||||
@@ -463,14 +491,18 @@ deployments "nginx-deployment" revision 2
|
||||
Now you've decided to undo the current rollout and rollback to the previous revision:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout undo deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout undo deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment
|
||||
```
|
||||
|
||||
Alternatively, you can rollback to a specific revision by specify that in `--to-revision`:
|
||||
Alternatively, you can rollback to a specific revision by specifying it with `--to-revision`:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout undo deployment.v1.apps/nginx-deployment --to-revision=2
|
||||
kubectl rollout undo deployment.v1.apps/nginx-deployment --to-revision=2
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment
|
||||
```
|
||||
|
||||
@@ -480,11 +512,17 @@ The Deployment is now rolled back to a previous stable revision. As you can see,
|
||||
for rolling back to revision 2 is generated from Deployment controller.
|
||||
|
||||
```shell
|
||||
$ kubectl get deployment nginx-deployment
|
||||
kubectl get deployment nginx-deployment
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
nginx-deployment 3 3 3 3 30m
|
||||
```
|
||||
|
||||
$ kubectl describe deployment nginx-deployment
|
||||
```shell
|
||||
kubectl describe deployment nginx-deployment
|
||||
```
|
||||
```
|
||||
Name: nginx-deployment
|
||||
Namespace: default
|
||||
CreationTimestamp: Sun, 02 Sep 2018 18:17:55 -0500
|
||||
@@ -533,7 +571,9 @@ Events:
|
||||
You can scale a Deployment by using the following command:
|
||||
|
||||
```shell
|
||||
$ kubectl scale deployment.v1.apps/nginx-deployment --replicas=10
|
||||
kubectl scale deployment.v1.apps/nginx-deployment --replicas=10
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment scaled
|
||||
```
|
||||
|
||||
@@ -542,7 +582,9 @@ in your cluster, you can setup an autoscaler for your Deployment and choose the
|
||||
Pods you want to run based on the CPU utilization of your existing Pods.
|
||||
|
||||
```shell
|
||||
$ kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80
|
||||
kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment scaled
|
||||
```
|
||||
|
||||
@@ -556,7 +598,9 @@ ReplicaSets (ReplicaSets with Pods) in order to mitigate risk. This is called *p
|
||||
For example, you are running a Deployment with 10 replicas, [maxSurge](#max-surge)=3, and [maxUnavailable](#max-unavailable)=2.
|
||||
|
||||
```shell
|
||||
$ kubectl get deploy
|
||||
kubectl get deploy
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
nginx-deployment 10 10 10 10 50s
|
||||
```
|
||||
@@ -564,7 +608,9 @@ nginx-deployment 10 10 10 10 50s
|
||||
You update to a new image which happens to be unresolvable from inside the cluster.
|
||||
|
||||
```shell
|
||||
$ kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:sometag
|
||||
kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:sometag
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment image updated
|
||||
```
|
||||
|
||||
@@ -572,7 +618,9 @@ The image update starts a new rollout with ReplicaSet nginx-deployment-198919819
|
||||
`maxUnavailable` requirement that you mentioned above.
|
||||
|
||||
```shell
|
||||
$ kubectl get rs
|
||||
kubectl get rs
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-deployment-1989198191 5 5 0 9s
|
||||
nginx-deployment-618515232 8 8 8 1m
|
||||
@@ -590,10 +638,17 @@ new ReplicaSet. The rollout process should eventually move all replicas to the n
|
||||
the new replicas become healthy.
|
||||
|
||||
```shell
|
||||
$ kubectl get deploy
|
||||
kubectl get deploy
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
nginx-deployment 15 18 7 8 7m
|
||||
$ kubectl get rs
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl get rs
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-deployment-1989198191 7 7 0 7m
|
||||
nginx-deployment-618515232 11 11 11 7m
|
||||
@@ -607,10 +662,16 @@ apply multiple fixes in between pausing and resuming without triggering unnecess
|
||||
For example, with a Deployment that was just created:
|
||||
|
||||
```shell
|
||||
$ kubectl get deploy
|
||||
kubectl get deploy
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
nginx 3 3 3 3 1m
|
||||
$ kubectl get rs
|
||||
```
|
||||
```shell
|
||||
kubectl get rs
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-2142116321 3 3 3 1m
|
||||
```
|
||||
@@ -618,26 +679,36 @@ nginx-2142116321 3 3 3 1m
|
||||
Pause by running the following command:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout pause deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout pause deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment paused
|
||||
```
|
||||
|
||||
Then update the image of the Deployment:
|
||||
|
||||
```shell
|
||||
$ kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1
|
||||
kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment image updated
|
||||
```
|
||||
|
||||
Notice that no new rollout started:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout history deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout history deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
deployments "nginx"
|
||||
REVISION CHANGE-CAUSE
|
||||
1 <none>
|
||||
```
|
||||
|
||||
$ kubectl get rs
|
||||
```shell
|
||||
kubectl get rs
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-2142116321 3 3 3 2m
|
||||
```
|
||||
@@ -645,7 +716,9 @@ nginx-2142116321 3 3 3 2m
|
||||
You can make as many updates as you wish, for example, update the resources that will be used:
|
||||
|
||||
```shell
|
||||
$ kubectl set resources deployment.v1.apps/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi
|
||||
kubectl set resources deployment.v1.apps/nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment resource requirements updated
|
||||
```
|
||||
|
||||
@@ -655,9 +728,18 @@ the Deployment will not have any effect as long as the Deployment is paused.
|
||||
Eventually, resume the Deployment and observe a new ReplicaSet coming up with all the new updates:
|
||||
|
||||
```shell
|
||||
$ kubectl rollout resume deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout resume deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
|
||||
```
|
||||
deployment.apps/nginx-deployment resumed
|
||||
$ kubectl get rs -w
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl get rs -w
|
||||
```
|
||||
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-2142116321 2 2 2 2m
|
||||
nginx-3926361531 2 2 0 6s
|
||||
@@ -673,8 +755,12 @@ nginx-2142116321 0 1 1 2m
|
||||
nginx-2142116321 0 1 1 2m
|
||||
nginx-2142116321 0 0 0 2m
|
||||
nginx-3926361531 3 3 3 20s
|
||||
^C
|
||||
$ kubectl get rs
|
||||
|
||||
```
|
||||
```shell
|
||||
kubectl get rs
|
||||
```
|
||||
```
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
nginx-2142116321 0 0 0 2m
|
||||
nginx-3926361531 3 3 3 28s
|
||||
@@ -713,7 +799,9 @@ You can check if a Deployment has completed by using `kubectl rollout status`. I
|
||||
successfully, `kubectl rollout status` returns a zero exit code.
|
||||
|
||||
```shell
|
||||
$ kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
Waiting for rollout to finish: 2 of 3 updated replicas are available...
|
||||
deployment.apps/nginx-deployment successfully rolled out
|
||||
$ echo $?
|
||||
@@ -741,7 +829,9 @@ The following `kubectl` command sets the spec with `progressDeadlineSeconds` to
|
||||
lack of progress for a Deployment after 10 minutes:
|
||||
|
||||
```shell
|
||||
$ kubectl patch deployment.v1.apps/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}'
|
||||
kubectl patch deployment.v1.apps/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}'
|
||||
```
|
||||
```
|
||||
deployment.apps/nginx-deployment patched
|
||||
```
|
||||
Once the deadline has been exceeded, the Deployment controller adds a DeploymentCondition with the following
|
||||
@@ -770,7 +860,9 @@ due to any other kind of error that can be treated as transient. For example, le
|
||||
insufficient quota. If you describe the Deployment you will notice the following section:
|
||||
|
||||
```shell
|
||||
$ kubectl describe deployment nginx-deployment
|
||||
kubectl describe deployment nginx-deployment
|
||||
```
|
||||
```
|
||||
<...>
|
||||
Conditions:
|
||||
Type Status Reason
|
||||
@@ -846,7 +938,9 @@ You can check if a Deployment has failed to progress by using `kubectl rollout s
|
||||
returns a non-zero exit code if the Deployment has exceeded the progression deadline.
|
||||
|
||||
```shell
|
||||
$ kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
kubectl rollout status deployment.v1.apps/nginx-deployment
|
||||
```
|
||||
```
|
||||
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
|
||||
error: deployment "nginx" exceeded its progress deadline
|
||||
$ echo $?
|
||||
@@ -988,15 +1082,12 @@ Field `.spec.rollbackTo` has been deprecated in API versions `extensions/v1beta1
|
||||
|
||||
### Revision History Limit
|
||||
|
||||
A Deployment's revision history is stored in the replica sets it controls.
|
||||
A Deployment's revision history is stored in the ReplicaSets it controls.
|
||||
|
||||
`.spec.revisionHistoryLimit` is an optional field that specifies the number of old ReplicaSets to retain
|
||||
to allow rollback. Its ideal value depends on the frequency and stability of new Deployments. All old
|
||||
ReplicaSets will be kept by default, consuming resources in `etcd` and crowding the output of `kubectl get rs`,
|
||||
if this field is not set. The configuration of each Deployment revision is stored in its ReplicaSets;
|
||||
therefore, once an old ReplicaSet is deleted, you lose the ability to rollback to that revision of Deployment.
|
||||
to allow rollback. These old ReplicaSets consume resources in `etcd` and crowd the output of `kubectl get rs`. The configuration of each Deployment revision is stored in its ReplicaSets; therefore, once an old ReplicaSet is deleted, you lose the ability to rollback to that revision of Deployment. By default, 10 old ReplicaSets will be kept, however its ideal value depends on the frequency and stability of new Deployments.
|
||||
|
||||
More specifically, setting this field to zero means that all old ReplicaSets with 0 replica will be cleaned up.
|
||||
More specifically, setting this field to zero means that all old ReplicaSets with 0 replicas will be cleaned up.
|
||||
In this case, a new Deployment rollout cannot be undone, since its revision history is cleaned up.
|
||||
|
||||
### Paused
|
||||
@@ -1015,5 +1106,3 @@ in a similar fashion. But Deployments are recommended, since they are declarativ
|
||||
additional features, such as rolling back to any previous revision even after the rolling update is done.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -60,6 +60,14 @@ metadata:
|
||||
...
|
||||
```
|
||||
|
||||
{{< note >}}
|
||||
Cross-namespace owner references is disallowed by design. This means:
|
||||
1) Namespace-scoped dependents can only specify owners in the same namespace,
|
||||
and owners that are cluster-scoped.
|
||||
2) Cluster-scoped dependents can only specify cluster-scoped owners, but not
|
||||
namespace-scoped owners.
|
||||
{{< /note >}}
|
||||
|
||||
## Controlling how the garbage collector deletes dependents
|
||||
|
||||
When you delete an object, you can specify whether the object's dependents are
|
||||
|
||||
@@ -13,16 +13,16 @@ weight: 70
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
A _job_ creates one or more pods and ensures that a specified number of them successfully terminate.
|
||||
As pods successfully complete, the _job_ tracks the successful completions. When a specified number
|
||||
of successful completions is reached, the job itself is complete. Deleting a Job will cleanup the
|
||||
pods it created.
|
||||
A Job creates one or more Pods and ensures that a specified number of them successfully terminate.
|
||||
As pods successfully complete, the Job tracks the successful completions. When a specified number
|
||||
of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up
|
||||
the Pods it created.
|
||||
|
||||
A simple case is to create one Job object in order to reliably run one Pod to completion.
|
||||
The Job object will start a new Pod if the first pod fails or is deleted (for example
|
||||
The Job object will start a new Pod if the first Pod fails or is deleted (for example
|
||||
due to a node hardware failure or a node reboot).
|
||||
|
||||
A Job can also be used to run multiple pods in parallel.
|
||||
You can also use a Job to run multiple Pods in parallel.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -36,17 +36,21 @@ It takes around 10s to complete.
|
||||
|
||||
{{< codenew file="controllers/job.yaml" >}}
|
||||
|
||||
Run the example job by downloading the example file and then running this command:
|
||||
You can run the example with this command:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/examples/controllers/job.yaml
|
||||
kubectl create -f https://k8s.io/examples/controllers/job.yaml
|
||||
```
|
||||
```
|
||||
job "pi" created
|
||||
```
|
||||
|
||||
Check on the status of the job using this command:
|
||||
Check on the status of the Job with `kubectl`:
|
||||
|
||||
```shell
|
||||
$ kubectl describe jobs/pi
|
||||
kubectl describe jobs/pi
|
||||
```
|
||||
```
|
||||
Name: pi
|
||||
Namespace: default
|
||||
Selector: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495
|
||||
@@ -78,18 +82,20 @@ Events:
|
||||
1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: pi-dtn4q
|
||||
```
|
||||
|
||||
To view completed pods of a job, use `kubectl get pods`.
|
||||
To view completed Pods of a Job, use `kubectl get pods`.
|
||||
|
||||
To list all the pods that belong to a job in a machine readable form, you can use a command like this:
|
||||
To list all the Pods that belong to a Job in a machine readable form, you can use a command like this:
|
||||
|
||||
```shell
|
||||
$ pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name})
|
||||
$ echo $pods
|
||||
pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath='{.items[*].metadata.name}')
|
||||
echo $pods
|
||||
```
|
||||
```
|
||||
pi-aiw0a
|
||||
```
|
||||
|
||||
Here, the selector is the same as the selector for the job. The `--output=jsonpath` option specifies an expression
|
||||
that just gets the name from each pod in the returned list.
|
||||
Here, the selector is the same as the selector for the Job. The `--output=jsonpath` option specifies an expression
|
||||
that just gets the name from each Pod in the returned list.
|
||||
|
||||
View the standard output of one of the pods:
|
||||
|
||||
@@ -102,7 +108,7 @@ $ kubectl logs $pods
|
||||
|
||||
As with all other Kubernetes config, a Job needs `apiVersion`, `kind`, and `metadata` fields.
|
||||
|
||||
A Job also needs a [`.spec` section](https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status).
|
||||
A Job also needs a [`.spec` section](https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status).
|
||||
|
||||
### Pod Template
|
||||
|
||||
@@ -110,7 +116,7 @@ The `.spec.template` is the only required field of the `.spec`.
|
||||
|
||||
The `.spec.template` is a [pod template](/docs/concepts/workloads/pods/pod-overview/#pod-templates). It has exactly the same schema as a [pod](/docs/user-guide/pods), except it is nested and does not have an `apiVersion` or `kind`.
|
||||
|
||||
In addition to required fields for a Pod, a pod template in a job must specify appropriate
|
||||
In addition to required fields for a Pod, a pod template in a Job must specify appropriate
|
||||
labels (see [pod selector](#pod-selector)) and an appropriate restart policy.
|
||||
|
||||
Only a [`RestartPolicy`](/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy) equal to `Never` or `OnFailure` is allowed.
|
||||
@@ -123,31 +129,30 @@ See section [specifying your own pod selector](#specifying-your-own-pod-selector
|
||||
|
||||
### Parallel Jobs
|
||||
|
||||
There are three main types of jobs:
|
||||
There are three main types of task suitable to run as a Job:
|
||||
|
||||
1. Non-parallel Jobs
|
||||
- normally only one pod is started, unless the pod fails.
|
||||
- job is complete as soon as Pod terminates successfully.
|
||||
- normally, only one Pod is started, unless the Pod fails.
|
||||
- the Job is complete as soon as its Pod terminates successfully.
|
||||
1. Parallel Jobs with a *fixed completion count*:
|
||||
- specify a non-zero positive value for `.spec.completions`.
|
||||
- the job is complete when there is one successful pod for each value in the range 1 to `.spec.completions`.
|
||||
- **not implemented yet:** Each pod passed a different index in the range 1 to `.spec.completions`.
|
||||
- the Job represents the overall task, and is complete when there is one successful Pod for each value in the range 1 to `.spec.completions`.
|
||||
- **not implemented yet:** Each Pod is passed a different index in the range 1 to `.spec.completions`.
|
||||
1. Parallel Jobs with a *work queue*:
|
||||
- do not specify `.spec.completions`, default to `.spec.parallelism`.
|
||||
- the pods must coordinate with themselves or an external service to determine what each should work on.
|
||||
- each pod is independently capable of determining whether or not all its peers are done, thus the entire Job is done.
|
||||
- when _any_ pod terminates with success, no new pods are created.
|
||||
- once at least one pod has terminated with success and all pods are terminated, then the job is completed with success.
|
||||
- once any pod has exited with success, no other pod should still be doing any work or writing any output. They should all be
|
||||
in the process of exiting.
|
||||
- do not specify `.spec.completions`, default to `.spec.parallelism`.
|
||||
- the Pods must coordinate amongst themselves or an external service to determine what each should work on. For example, a Pod might fetch a batch of up to N items from the work queue.
|
||||
- each Pod is independently capable of determining whether or not all its peers are done, and thus that the entire Job is done.
|
||||
- when _any_ Pod from the Job terminates with success, no new Pods are created.
|
||||
- once at least one Pod has terminated with success and all Pods are terminated, then the Job is completed with success.
|
||||
- once any Pod has exited with success, no other Pod should still be doing any work for this task or writing any output. They should all be in the process of exiting.
|
||||
|
||||
For a Non-parallel job, you can leave both `.spec.completions` and `.spec.parallelism` unset. When both are
|
||||
For a _non-parallel_ Job, you can leave both `.spec.completions` and `.spec.parallelism` unset. When both are
|
||||
unset, both are defaulted to 1.
|
||||
|
||||
For a Fixed Completion Count job, you should set `.spec.completions` to the number of completions needed.
|
||||
For a _fixed completion count_ Job, you should set `.spec.completions` to the number of completions needed.
|
||||
You can set `.spec.parallelism`, or leave it unset and it will default to 1.
|
||||
|
||||
For a Work Queue Job, you must leave `.spec.completions` unset, and set `.spec.parallelism` to
|
||||
For a _work queue_ Job, you must leave `.spec.completions` unset, and set `.spec.parallelism` to
|
||||
a non-negative integer.
|
||||
|
||||
For more information about how to make use of the different types of job, see the [job patterns](#job-patterns) section.
|
||||
@@ -162,28 +167,28 @@ If it is specified as 0, then the Job is effectively paused until it is increase
|
||||
Actual parallelism (number of pods running at any instant) may be more or less than requested
|
||||
parallelism, for a variety of reasons:
|
||||
|
||||
- For Fixed Completion Count jobs, the actual number of pods running in parallel will not exceed the number of
|
||||
- For _fixed completion count_ Jobs, the actual number of pods running in parallel will not exceed the number of
|
||||
remaining completions. Higher values of `.spec.parallelism` are effectively ignored.
|
||||
- For work queue jobs, no new pods are started after any pod has succeeded -- remaining pods are allowed to complete, however.
|
||||
- For _work queue_ Jobs, no new Pods are started after any Pod has succeeded -- remaining Pods are allowed to complete, however.
|
||||
- If the controller has not had time to react.
|
||||
- If the controller failed to create pods for any reason (lack of ResourceQuota, lack of permission, etc.),
|
||||
- If the controller failed to create Pods for any reason (lack of `ResourceQuota`, lack of permission, etc.),
|
||||
then there may be fewer pods than requested.
|
||||
- The controller may throttle new pod creation due to excessive previous pod failures in the same Job.
|
||||
- When a pod is gracefully shutdown, it takes time to stop.
|
||||
- The controller may throttle new Pod creation due to excessive previous pod failures in the same Job.
|
||||
- When a Pod is gracefully shut down, it takes time to stop.
|
||||
|
||||
## Handling Pod and Container Failures
|
||||
|
||||
A Container in a Pod may fail for a number of reasons, such as because the process in it exited with
|
||||
a non-zero exit code, or the Container was killed for exceeding a memory limit, etc. If this
|
||||
A container in a Pod may fail for a number of reasons, such as because the process in it exited with
|
||||
a non-zero exit code, or the container was killed for exceeding a memory limit, etc. If this
|
||||
happens, and the `.spec.template.spec.restartPolicy = "OnFailure"`, then the Pod stays
|
||||
on the node, but the Container is re-run. Therefore, your program needs to handle the case when it is
|
||||
on the node, but the container is re-run. Therefore, your program needs to handle the case when it is
|
||||
restarted locally, or else specify `.spec.template.spec.restartPolicy = "Never"`.
|
||||
See [pods-states](/docs/concepts/workloads/pods/pod-lifecycle/#example-states) for more information on `restartPolicy`.
|
||||
See [pod lifecycle](/docs/concepts/workloads/pods/pod-lifecycle/#example-states) for more information on `restartPolicy`.
|
||||
|
||||
An entire Pod can also fail, for a number of reasons, such as when the pod is kicked off the node
|
||||
(node is upgraded, rebooted, deleted, etc.), or if a container of the Pod fails and the
|
||||
`.spec.template.spec.restartPolicy = "Never"`. When a Pod fails, then the Job controller
|
||||
starts a new Pod. Therefore, your program needs to handle the case when it is restarted in a new
|
||||
starts a new Pod. This means that your application needs to handle the case when it is restarted in a new
|
||||
pod. In particular, it needs to handle temporary files, locks, incomplete output and the like
|
||||
caused by previous runs.
|
||||
|
||||
@@ -194,7 +199,7 @@ sometimes be started twice.
|
||||
If you do specify `.spec.parallelism` and `.spec.completions` both greater than 1, then there may be
|
||||
multiple pods running at once. Therefore, your pods must also be tolerant of concurrency.
|
||||
|
||||
### Pod Backoff failure policy
|
||||
### Pod backoff failure policy
|
||||
|
||||
There are situations where you want to fail a Job after some amount of retries
|
||||
due to a logical error in configuration etc.
|
||||
@@ -221,8 +226,7 @@ By default, a Job will run uninterrupted unless a Pod fails, at which point the
|
||||
Do this by setting the `.spec.activeDeadlineSeconds` field of the Job to a number of seconds.
|
||||
|
||||
The `activeDeadlineSeconds` applies to the duration of the job, no matter how many Pods are created.
|
||||
Once a Job reaches `activeDeadlineSeconds`, the Job and all of its Pods are terminated.
|
||||
The result is that the job has a status with `reason: DeadlineExceeded`.
|
||||
Once a Job reaches `activeDeadlineSeconds`, all of its Pods are terminated and the Job status will become `type: Failed` with `reason: DeadlineExceeded`.
|
||||
|
||||
Note that a Job's `.spec.activeDeadlineSeconds` takes precedence over its `.spec.backoffLimit`. Therefore, a Job that is retrying one or more failed Pods will not deploy additional Pods once it reaches the time limit specified by `activeDeadlineSeconds`, even if the `backoffLimit` is not yet reached.
|
||||
|
||||
@@ -245,7 +249,7 @@ spec:
|
||||
restartPolicy: Never
|
||||
```
|
||||
|
||||
Note that both the Job Spec and the [Pod Template Spec](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#detailed-behavior) within the Job have an `activeDeadlineSeconds` field. Ensure that you set this field at the proper level.
|
||||
Note that both the Job spec and the [Pod template spec](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#detailed-behavior) within the Job have an `activeDeadlineSeconds` field. Ensure that you set this field at the proper level.
|
||||
|
||||
## Clean Up Finished Jobs Automatically
|
||||
|
||||
@@ -317,7 +321,7 @@ The tradeoffs are:
|
||||
- One Job object for each work item, vs. a single Job object for all work items. The latter is
|
||||
better for large numbers of work items. The former creates some overhead for the user and for the
|
||||
system to manage large numbers of Job objects.
|
||||
- Number of pods created equals number of work items, vs. each pod can process multiple work items.
|
||||
- Number of pods created equals number of work items, vs. each Pod can process multiple work items.
|
||||
The former typically requires less modification to existing code and containers. The latter
|
||||
is better for large numbers of work items, for similar reasons to the previous bullet.
|
||||
- Several approaches use a work queue. This requires running a queue service,
|
||||
@@ -337,7 +341,7 @@ The pattern names are also links to examples and more detailed description.
|
||||
|
||||
When you specify completions with `.spec.completions`, each Pod created by the Job controller
|
||||
has an identical [`spec`](https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status). This means that
|
||||
all pods will have the same command line and the same
|
||||
all pods for a task will have the same command line and the same
|
||||
image, the same volumes, and (almost) the same environment variables. These patterns
|
||||
are different ways to arrange for pods to work on different things.
|
||||
|
||||
@@ -356,29 +360,29 @@ Here, `W` is the number of work items.
|
||||
|
||||
### Specifying your own pod selector
|
||||
|
||||
Normally, when you create a job object, you do not specify `.spec.selector`.
|
||||
The system defaulting logic adds this field when the job is created.
|
||||
Normally, when you create a Job object, you do not specify `.spec.selector`.
|
||||
The system defaulting logic adds this field when the Job is created.
|
||||
It picks a selector value that will not overlap with any other jobs.
|
||||
|
||||
However, in some cases, you might need to override this automatically set selector.
|
||||
To do this, you can specify the `.spec.selector` of the job.
|
||||
To do this, you can specify the `.spec.selector` of the Job.
|
||||
|
||||
Be very careful when doing this. If you specify a label selector which is not
|
||||
unique to the pods of that job, and which matches unrelated pods, then pods of the unrelated
|
||||
job may be deleted, or this job may count other pods as completing it, or one or both
|
||||
of the jobs may refuse to create pods or run to completion. If a non-unique selector is
|
||||
chosen, then other controllers (e.g. ReplicationController) and their pods may behave
|
||||
unique to the pods of that Job, and which matches unrelated Pods, then pods of the unrelated
|
||||
job may be deleted, or this Job may count other Pods as completing it, or one or both
|
||||
Jobs may refuse to create Pods or run to completion. If a non-unique selector is
|
||||
chosen, then other controllers (e.g. ReplicationController) and their Pods may behave
|
||||
in unpredictable ways too. Kubernetes will not stop you from making a mistake when
|
||||
specifying `.spec.selector`.
|
||||
|
||||
Here is an example of a case when you might want to use this feature.
|
||||
|
||||
Say job `old` is already running. You want existing pods
|
||||
to keep running, but you want the rest of the pods it creates
|
||||
to use a different pod template and for the job to have a new name.
|
||||
You cannot update the job because these fields are not updatable.
|
||||
Therefore, you delete job `old` but leave its pods
|
||||
running, using `kubectl delete jobs/old --cascade=false`.
|
||||
Say Job `old` is already running. You want existing Pods
|
||||
to keep running, but you want the rest of the Pods it creates
|
||||
to use a different pod template and for the Job to have a new name.
|
||||
You cannot update the Job because these fields are not updatable.
|
||||
Therefore, you delete Job `old` but _leave its pods
|
||||
running_, using `kubectl delete jobs/old --cascade=false`.
|
||||
Before deleting it, you make a note of what selector it uses:
|
||||
|
||||
```
|
||||
@@ -393,11 +397,11 @@ spec:
|
||||
...
|
||||
```
|
||||
|
||||
Then you create a new job with name `new` and you explicitly specify the same selector.
|
||||
Since the existing pods have label `job-uid=a8f3d00d-c6d2-11e5-9f87-42010af00002`,
|
||||
they are controlled by job `new` as well.
|
||||
Then you create a new Job with name `new` and you explicitly specify the same selector.
|
||||
Since the existing Pods have label `job-uid=a8f3d00d-c6d2-11e5-9f87-42010af00002`,
|
||||
they are controlled by Job `new` as well.
|
||||
|
||||
You need to specify `manualSelector: true` in the new job since you are not using
|
||||
You need to specify `manualSelector: true` in the new Job since you are not using
|
||||
the selector that the system normally generates for you automatically.
|
||||
|
||||
```
|
||||
@@ -421,25 +425,25 @@ mismatch.
|
||||
|
||||
### Bare Pods
|
||||
|
||||
When the node that a pod is running on reboots or fails, the pod is terminated
|
||||
and will not be restarted. However, a Job will create new pods to replace terminated ones.
|
||||
For this reason, we recommend that you use a job rather than a bare pod, even if your application
|
||||
requires only a single pod.
|
||||
When the node that a Pod is running on reboots or fails, the pod is terminated
|
||||
and will not be restarted. However, a Job will create new Pods to replace terminated ones.
|
||||
For this reason, we recommend that you use a Job rather than a bare Pod, even if your application
|
||||
requires only a single Pod.
|
||||
|
||||
### Replication Controller
|
||||
|
||||
Jobs are complementary to [Replication Controllers](/docs/user-guide/replication-controller).
|
||||
A Replication Controller manages pods which are not expected to terminate (e.g. web servers), and a Job
|
||||
manages pods that are expected to terminate (e.g. batch jobs).
|
||||
A Replication Controller manages Pods which are not expected to terminate (e.g. web servers), and a Job
|
||||
manages Pods that are expected to terminate (e.g. batch tasks).
|
||||
|
||||
As discussed in [Pod Lifecycle](/docs/concepts/workloads/pods/pod-lifecycle/), `Job` is *only* appropriate for pods with
|
||||
`RestartPolicy` equal to `OnFailure` or `Never`. (Note: If `RestartPolicy` is not set, the default
|
||||
value is `Always`.)
|
||||
As discussed in [Pod Lifecycle](/docs/concepts/workloads/pods/pod-lifecycle/), `Job` is *only* appropriate
|
||||
for pods with `RestartPolicy` equal to `OnFailure` or `Never`.
|
||||
(Note: If `RestartPolicy` is not set, the default value is `Always`.)
|
||||
|
||||
### Single Job starts Controller Pod
|
||||
|
||||
Another pattern is for a single Job to create a pod which then creates other pods, acting as a sort
|
||||
of custom controller for those pods. This allows the most flexibility, but may be somewhat
|
||||
Another pattern is for a single Job to create a Pod which then creates other Pods, acting as a sort
|
||||
of custom controller for those Pods. This allows the most flexibility, but may be somewhat
|
||||
complicated to get started with and offers less integration with Kubernetes.
|
||||
|
||||
One example of this pattern would be a Job which starts a Pod which runs a script that in turn
|
||||
@@ -447,10 +451,10 @@ starts a Spark master controller (see [spark example](https://github.com/kuberne
|
||||
driver, and then cleans up.
|
||||
|
||||
An advantage of this approach is that the overall process gets the completion guarantee of a Job
|
||||
object, but complete control over what pods are created and how work is assigned to them.
|
||||
object, but complete control over what Pods are created and how work is assigned to them.
|
||||
|
||||
## Cron Jobs
|
||||
## Cron Jobs {#cron-jobs}
|
||||
|
||||
Support for creating Jobs at specified times/dates (i.e. cron) is available in Kubernetes [1.4](https://github.com/kubernetes/kubernetes/pull/11980). More information is available in the [cron job documents](/docs/concepts/workloads/controllers/cron-jobs/)
|
||||
You can use a [`CronJob`](/docs/concepts/workloads/controllers/cron-jobs/) to create a Job that will run at specified times/dates, similar to the Unix tool `cron`.
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -10,39 +10,36 @@ weight: 10
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
ReplicaSet is the next-generation Replication Controller. The only difference
|
||||
between a _ReplicaSet_ and a
|
||||
[_Replication Controller_](/docs/concepts/workloads/controllers/replicationcontroller/) right now is
|
||||
the selector support. ReplicaSet supports the new set-based selector requirements
|
||||
as described in the [labels user guide](/docs/concepts/overview/working-with-objects/labels/#label-selectors)
|
||||
whereas a Replication Controller only supports equality-based selector requirements.
|
||||
A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often
|
||||
used to guarantee the availability of a specified number of identical Pods.
|
||||
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## How to use a ReplicaSet
|
||||
## How a ReplicaSet works
|
||||
|
||||
Most [`kubectl`](/docs/user-guide/kubectl/) commands that support
|
||||
Replication Controllers also support ReplicaSets. One exception is the
|
||||
[`rolling-update`](/docs/reference/generated/kubectl/kubectl-commands#rolling-update) command. If
|
||||
you want the rolling update functionality please consider using Deployments
|
||||
instead. Also, the
|
||||
[`rolling-update`](/docs/reference/generated/kubectl/kubectl-commands#rolling-update) command is
|
||||
imperative whereas Deployments are declarative, so we recommend using Deployments
|
||||
through the [`rollout`](/docs/reference/generated/kubectl/kubectl-commands#rollout) command.
|
||||
A ReplicaSet is defined with fields, including a selector that specifies how to identify Pods it can acquire, a number
|
||||
of replicas indicating how many Pods it should be maintaining, and a pod template specifying the data of new Pods
|
||||
it should create to meet the number of replicas criteria. A ReplicaSet then fulfills its purpose by creating
|
||||
and deleting Pods as needed to reach the desired number. When a ReplicaSet needs to create new Pods, it uses its Pod
|
||||
template.
|
||||
|
||||
While ReplicaSets can be used independently, today it's mainly used by
|
||||
[Deployments](/docs/concepts/workloads/controllers/deployment/) as a mechanism to orchestrate pod
|
||||
creation, deletion and updates. When you use Deployments you don't have to worry
|
||||
about managing the ReplicaSets that they create. Deployments own and manage
|
||||
their ReplicaSets.
|
||||
The link a ReplicaSet has to its Pods is via the Pods' [metadata.ownerReferences](/docs/concepts/workloads/controllers/garbage-collection/#owners-and-dependents)
|
||||
field, which specifies what resource the current object is owned by. All Pods acquired by a ReplicaSet have their owning
|
||||
ReplicaSet's identifying information within their ownerReferences field. It's through this link that the ReplicaSet
|
||||
knows of the state of the Pods it is maintaining and plans accordingly.
|
||||
|
||||
A ReplicaSet identifies new Pods to acquire by using its selector. If there is a Pod that has no OwnerReference or the
|
||||
OwnerReference is not a controller and it matches a ReplicaSet's selector, it will be immediately acquired by said
|
||||
ReplicaSet.
|
||||
|
||||
## When to use a ReplicaSet
|
||||
|
||||
A ReplicaSet ensures that a specified number of pod replicas are running at any given
|
||||
time. However, a Deployment is a higher-level concept that manages ReplicaSets and
|
||||
provides declarative updates to pods along with a lot of other useful features.
|
||||
provides declarative updates to Pods along with a lot of other useful features.
|
||||
Therefore, we recommend using Deployments instead of directly using ReplicaSets, unless
|
||||
you require custom update orchestration or don't require updates at all.
|
||||
|
||||
@@ -53,13 +50,31 @@ use a Deployment instead, and define your application in the spec section.
|
||||
|
||||
{{< codenew file="controllers/frontend.yaml" >}}
|
||||
|
||||
Saving this manifest into `frontend.yaml` and submitting it to a Kubernetes cluster should
|
||||
create the defined ReplicaSet and the pods that it manages.
|
||||
Saving this manifest into `frontend.yaml` and submitting it to a Kubernetes cluster will
|
||||
create the defined ReplicaSet and the Pods that it manages.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f http://k8s.io/examples/controllers/frontend.yaml
|
||||
replicaset.apps/frontend created
|
||||
$ kubectl describe rs/frontend
|
||||
kubectl create -f http://k8s.io/examples/controllers/frontend.yaml
|
||||
```
|
||||
|
||||
You can then get the current ReplicaSets deployed:
|
||||
```shell
|
||||
kubectl get rs
|
||||
```
|
||||
|
||||
And see the frontend one you created:
|
||||
```shell
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
frontend 3 3 3 6s
|
||||
```
|
||||
|
||||
You can also check on the state of the replicaset:
|
||||
```shell
|
||||
kubectl describe rs/frontend
|
||||
```
|
||||
|
||||
And you will see output similar to:
|
||||
```shell
|
||||
Name: frontend
|
||||
Namespace: default
|
||||
Selector: tier=frontend,tier in (frontend)
|
||||
@@ -88,66 +103,150 @@ Events:
|
||||
1m 1m 1 {replicaset-controller } Normal SuccessfulCreate Created pod: frontend-qhloh
|
||||
1m 1m 1 {replicaset-controller } Normal SuccessfulCreate Created pod: frontend-dnjpy
|
||||
1m 1m 1 {replicaset-controller } Normal SuccessfulCreate Created pod: frontend-9si5l
|
||||
$ kubectl get pods
|
||||
```
|
||||
|
||||
And lastly you can check for the Pods brought up:
|
||||
```shell
|
||||
kubectl get Pods
|
||||
```
|
||||
|
||||
You should see Pod information similar to:
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
frontend-9si5l 1/1 Running 0 1m
|
||||
frontend-dnjpy 1/1 Running 0 1m
|
||||
frontend-qhloh 1/1 Running 0 1m
|
||||
```
|
||||
|
||||
## Writing a ReplicaSet Spec
|
||||
You can also verify that the owner reference of these pods is set to the frontend ReplicaSet.
|
||||
To do this, get the yaml of one of the Pods running:
|
||||
```shell
|
||||
kubectl get pods frontend-9si5l -o yaml
|
||||
```
|
||||
|
||||
As with all other Kubernetes API objects, a ReplicaSet needs the `apiVersion`, `kind`, and `metadata` fields. For
|
||||
general information about working with manifests, see [object management using kubectl](/docs/concepts/overview/object-management-kubectl/overview/).
|
||||
The output will look similar to this, with the frontend ReplicaSet's info set in the metadata's ownerReferences field:
|
||||
```shell
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
creationTimestamp: 2019-01-31T17:20:41Z
|
||||
generateName: frontend-
|
||||
labels:
|
||||
tier: frontend
|
||||
name: frontend-9si5l
|
||||
namespace: default
|
||||
ownerReferences:
|
||||
- apiVersion: extensions/v1beta1
|
||||
blockOwnerDeletion: true
|
||||
controller: true
|
||||
kind: ReplicaSet
|
||||
name: frontend
|
||||
uid: 892a2330-257c-11e9-aecd-025000000001
|
||||
...
|
||||
```
|
||||
|
||||
## Non-Template Pod acquisitions
|
||||
|
||||
While you can create bare Pods with no problems, it is strongly recommended to make sure that the bare Pods do not have
|
||||
labels which match the selector of one of your ReplicaSets. The reason for this is because a ReplicaSet is not limited
|
||||
to owning Pods specified by its template-- it can acquire other Pods in the manner specified in the previous sections.
|
||||
|
||||
Take the previous frontend ReplicaSet example, and the Pods specified in the following manifest:
|
||||
|
||||
{{< codenew file="pods/pod-rs.yaml" >}}
|
||||
|
||||
As those Pods do not have a Controller (or any object) as their owner reference and match the selector of the frontend
|
||||
ReplicaSet, they will immediately be acquired by it.
|
||||
|
||||
Suppose you create the Pods after the frontend ReplicaSet has been deployed and has set up its initial Pod replicas to
|
||||
fulfill its replica count requirement:
|
||||
|
||||
```shell
|
||||
kubectl create -f http://k8s.io/examples/pods/pod-rs.yaml
|
||||
```
|
||||
|
||||
The new Pods will be acquired by the ReplicaSet, and then immediately terminated as the ReplicaSet would be over
|
||||
its desired count.
|
||||
|
||||
Fetching the Pods:
|
||||
```shell
|
||||
kubectl get Pods
|
||||
```
|
||||
|
||||
The output shows that the new Pods are either already terminated, or in the process of being terminated:
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
frontend-9si5l 1/1 Running 0 1m
|
||||
frontend-dnjpy 1/1 Running 0 1m
|
||||
frontend-qhloh 1/1 Running 0 1m
|
||||
pod2 0/1 Terminating 0 4s
|
||||
```
|
||||
|
||||
If you create the Pods first:
|
||||
```shell
|
||||
kubectl create -f http://k8s.io/examples/pods/pod-rs.yaml
|
||||
```
|
||||
|
||||
And then create the ReplicaSet however:
|
||||
```shell
|
||||
kubectl create -f http://k8s.io/examples/controllers/frontend.yaml
|
||||
```
|
||||
|
||||
You shall see that the ReplicaSet has acquired the Pods and has only created new ones according to its spec until the
|
||||
number of its new Pods and the original matches its desired count. As fetching the Pods:
|
||||
```shell
|
||||
kubectl get Pods
|
||||
```
|
||||
|
||||
Will reveal in its output:
|
||||
```shell
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
frontend-pxj4r 1/1 Running 0 5s
|
||||
pod1 1/1 Running 0 13s
|
||||
pod2 1/1 Running 0 13s
|
||||
```
|
||||
|
||||
In this manner, a ReplicaSet can own a non-homogenous set of Pods
|
||||
|
||||
## Writing a ReplicaSet manifest
|
||||
|
||||
As with all other Kubernetes API objects, a ReplicaSet needs the `apiVersion`, `kind`, and `metadata` fields.
|
||||
For ReplicaSets, the kind is always just ReplicaSet.
|
||||
In Kubernetes 1.9 the API version `apps/v1` on the ReplicaSet kind is the current version and is enabled by default. The API version `apps/v1beta2` is deprecated.
|
||||
Refer to the first lines of the `frontend.yaml` example for guidance.
|
||||
|
||||
A ReplicaSet also needs a [`.spec` section](https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status).
|
||||
|
||||
### Pod Template
|
||||
|
||||
The `.spec.template` is the only required field of the `.spec`. The `.spec.template` is a
|
||||
[pod template](/docs/concepts/workloads/pods/pod-overview/#pod-templates). It has exactly the same schema as a
|
||||
[pod](/docs/concepts/workloads/pods/pod/), except that it is nested and does not have an `apiVersion` or `kind`.
|
||||
The `.spec.template` is a [pod template](/docs/concepts/workloads/Pods/pod-overview/#pod-templates) which is also
|
||||
required to have labels in place. In our `frontend.yaml` example we had one label: `tier: frontend`.
|
||||
Be careful not to overlap with the selectors of other controllers, lest they try to adopt this Pod.
|
||||
|
||||
In addition to required fields of a pod, a pod template in a ReplicaSet must specify appropriate
|
||||
labels and an appropriate restart policy.
|
||||
|
||||
For labels, make sure to not overlap with other controllers. For more information, see [pod selector](#pod-selector).
|
||||
|
||||
For [restart policy](/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy), the only allowed value for `.spec.template.spec.restartPolicy` is `Always`, which is the default.
|
||||
|
||||
For local container restarts, ReplicaSet delegates to an agent on the node,
|
||||
for example the [Kubelet](/docs/admin/kubelet/) or Docker.
|
||||
For the template's [restart policy](/docs/concepts/workloads/Pods/pod-lifecycle/#restart-policy) field,
|
||||
`.spec.template.spec.restartPolicy`, the only allowed value is `Always`, which is the default.
|
||||
|
||||
### Pod Selector
|
||||
|
||||
The `.spec.selector` field is a [label selector](/docs/concepts/overview/working-with-objects/labels/). A ReplicaSet
|
||||
manages all the pods with labels that match the selector. It does not distinguish
|
||||
between pods that it created or deleted and pods that another person or process created or
|
||||
deleted. This allows the ReplicaSet to be replaced without affecting the running pods.
|
||||
The `.spec.selector` field is a [label selector](/docs/concepts/overview/working-with-objects/labels/). As discussed
|
||||
[earlier](#how-a-replicaset-works) these are the labels used to identify potential Pods to acquire. In our
|
||||
`frontend.yaml` example, the selector was:
|
||||
```shell
|
||||
matchLabels:
|
||||
tier: frontend
|
||||
```
|
||||
|
||||
The `.spec.template.metadata.labels` must match the `.spec.selector`, or it will
|
||||
In the ReplicaSet, `.spec.template.metadata.labels` must match `spec.selector`, or it will
|
||||
be rejected by the API.
|
||||
|
||||
In Kubernetes 1.9 the API version `apps/v1` on the ReplicaSet kind is the current version and is enabled by default. The API version `apps/v1beta2` is deprecated.
|
||||
|
||||
Also you should not normally create any pods whose labels match this selector, either directly, with
|
||||
another ReplicaSet, or with another controller such as a Deployment. If you do so, the ReplicaSet thinks that it
|
||||
created the other pods. Kubernetes does not stop you from doing this.
|
||||
|
||||
If you do end up with multiple controllers that have overlapping selectors, you
|
||||
will have to manage the deletion yourself.
|
||||
|
||||
### Labels on a ReplicaSet
|
||||
|
||||
The ReplicaSet can itself have labels (`.metadata.labels`). Typically, you
|
||||
would set these the same as the `.spec.template.metadata.labels`. However, they are allowed to be
|
||||
different, and the `.metadata.labels` do not affect the behavior of the ReplicaSet.
|
||||
{{< note >}}
|
||||
For 2 ReplicaSets specifying the same `.spec.selector` but different `.spec.template.metadata.labels` and `.spec.template.spec` fields, each ReplicaSet ignores the Pods created by the other ReplicaSet.
|
||||
{{< /note >}}
|
||||
|
||||
### Replicas
|
||||
|
||||
You can specify how many pods should run concurrently by setting `.spec.replicas`. The number running at any time may be higher
|
||||
or lower, such as if the replicas were just increased or decreased, or if a pod is gracefully
|
||||
shut down, and a replacement starts early.
|
||||
You can specify how many Pods should run concurrently by setting `.spec.replicas`. The ReplicaSet will create/delete
|
||||
its Pods to match this number.
|
||||
|
||||
If you do not specify `.spec.replicas`, then it defaults to 1.
|
||||
|
||||
@@ -157,7 +256,9 @@ If you do not specify `.spec.replicas`, then it defaults to 1.
|
||||
|
||||
To delete a ReplicaSet and all of its Pods, use [`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete). The [Garbage collector](/docs/concepts/workloads/controllers/garbage-collection/) automatically deletes all of the dependent Pods by default.
|
||||
|
||||
When using the REST API or the `client-go` library, you must set `propagationPolicy` to `Background` or `Foreground` in delete option. e.g. :
|
||||
When using the REST API or the `client-go` library, you must set `propagationPolicy` to `Background` or `Foreground` in
|
||||
the -d option.
|
||||
For example:
|
||||
```shell
|
||||
kubectl proxy --port=8080
|
||||
curl -X DELETE 'localhost:8080/apis/extensions/v1beta1/namespaces/default/replicasets/frontend' \
|
||||
@@ -167,8 +268,9 @@ curl -X DELETE 'localhost:8080/apis/extensions/v1beta1/namespaces/default/repli
|
||||
|
||||
### Deleting just a ReplicaSet
|
||||
|
||||
You can delete a ReplicaSet without affecting any of its pods using [`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete) with the `--cascade=false` option.
|
||||
When using the REST API or the `client-go` library, you must set `propagationPolicy` to `Orphan`, e.g. :
|
||||
You can delete a ReplicaSet without affecting any of its Pods using [`kubectl delete`](/docs/reference/generated/kubectl/kubectl-commands#delete) with the `--cascade=false` option.
|
||||
When using the REST API or the `client-go` library, you must set `propagationPolicy` to `Orphan`.
|
||||
For example:
|
||||
```shell
|
||||
kubectl proxy --port=8080
|
||||
curl -X DELETE 'localhost:8080/apis/extensions/v1beta1/namespaces/default/replicasets/frontend' \
|
||||
@@ -177,22 +279,22 @@ curl -X DELETE 'localhost:8080/apis/extensions/v1beta1/namespaces/default/repli
|
||||
```
|
||||
|
||||
Once the original is deleted, you can create a new ReplicaSet to replace it. As long
|
||||
as the old and new `.spec.selector` are the same, then the new one will adopt the old pods.
|
||||
However, it will not make any effort to make existing pods match a new, different pod template.
|
||||
To update pods to a new spec in a controlled way, use a [rolling update](#rolling-updates).
|
||||
as the old and new `.spec.selector` are the same, then the new one will adopt the old Pods.
|
||||
However, it will not make any effort to make existing Pods match a new, different pod template.
|
||||
To update Pods to a new spec in a controlled way, use a [rolling update](#rolling-updates).
|
||||
|
||||
### Isolating pods from a ReplicaSet
|
||||
### Isolating Pods from a ReplicaSet
|
||||
|
||||
Pods may be removed from a ReplicaSet's target set by changing their labels. This technique may be used to remove pods
|
||||
You can remove Pods from a ReplicaSet by changing their labels. This technique may be used to remove Pods
|
||||
from service for debugging, data recovery, etc. Pods that are removed in this way will be replaced automatically (
|
||||
assuming that the number of replicas is not also changed).
|
||||
assuming that the number of replicas is not also changed).
|
||||
|
||||
### Scaling a ReplicaSet
|
||||
|
||||
A ReplicaSet can be easily scaled up or down by simply updating the `.spec.replicas` field. The ReplicaSet controller
|
||||
ensures that a desired number of pods with a matching label selector are available and operational.
|
||||
ensures that a desired number of Pods with a matching label selector are available and operational.
|
||||
|
||||
### ReplicaSet as an Horizontal Pod Autoscaler Target
|
||||
### ReplicaSet as a Horizontal Pod Autoscaler Target
|
||||
|
||||
A ReplicaSet can also be a target for
|
||||
[Horizontal Pod Autoscalers (HPA)](/docs/tasks/run-application/horizontal-pod-autoscale/). That is,
|
||||
@@ -203,7 +305,7 @@ the ReplicaSet we created in the previous example.
|
||||
|
||||
Saving this manifest into `hpa-rs.yaml` and submitting it to a Kubernetes cluster should
|
||||
create the defined HPA that autoscales the target ReplicaSet depending on the CPU usage
|
||||
of the replicated pods.
|
||||
of the replicated Pods.
|
||||
|
||||
```shell
|
||||
kubectl create -f https://k8s.io/examples/controllers/hpa-rs.yaml
|
||||
@@ -213,34 +315,40 @@ Alternatively, you can use the `kubectl autoscale` command to accomplish the sam
|
||||
(and it's easier!)
|
||||
|
||||
```shell
|
||||
kubectl autoscale rs frontend
|
||||
kubectl autoscale rs frontend --max=10
|
||||
```
|
||||
|
||||
## Alternatives to ReplicaSet
|
||||
|
||||
### Deployment (Recommended)
|
||||
### Deployment (recommended)
|
||||
|
||||
[`Deployment`](/docs/concepts/workloads/controllers/deployment/) is a higher-level API object that updates its underlying ReplicaSets and their Pods
|
||||
in a similar fashion as `kubectl rolling-update`. Deployments are recommended if you want this rolling update functionality,
|
||||
because unlike `kubectl rolling-update`, they are declarative, server-side, and have additional features. For more information on running a stateless
|
||||
application using a Deployment, please read [Run a Stateless Application Using a Deployment](/docs/tasks/run-application/run-stateless-application-deployment/).
|
||||
[`Deployment`](/docs/concepts/workloads/controllers/deployment/) is an object which can own ReplicaSets and update
|
||||
them and their Pods via declarative, server-side rolling updates.
|
||||
While ReplicaSets can be used independently, today they're mainly used by Deployments as a mechanism to orchestrate Pod
|
||||
creation, deletion and updates. When you use Deployments you don’t have to worry about managing the ReplicaSets that
|
||||
they create. Deployments own and manage their ReplicaSets.
|
||||
As such, it is recommended to use Deployments when you want ReplicaSets.
|
||||
|
||||
### Bare Pods
|
||||
|
||||
Unlike the case where a user directly created pods, a ReplicaSet replaces pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your application requires only a single pod. Think of it similarly to a process supervisor, only it supervises multiple pods across multiple nodes instead of individual processes on a single node. A ReplicaSet delegates local container restarts to some agent on the node (for example, Kubelet or Docker).
|
||||
Unlike the case where a user directly created Pods, a ReplicaSet replaces Pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your application requires only a single Pod. Think of it similarly to a process supervisor, only it supervises multiple Pods across multiple nodes instead of individual processes on a single node. A ReplicaSet delegates local container restarts to some agent on the node (for example, Kubelet or Docker).
|
||||
|
||||
### Job
|
||||
|
||||
Use a [`Job`](/docs/concepts/jobs/run-to-completion-finite-workloads/) instead of a ReplicaSet for pods that are expected to terminate on their own
|
||||
Use a [`Job`](/docs/concepts/jobs/run-to-completion-finite-workloads/) instead of a ReplicaSet for Pods that are expected to terminate on their own
|
||||
(that is, batch jobs).
|
||||
|
||||
### DaemonSet
|
||||
|
||||
Use a [`DaemonSet`](/docs/concepts/workloads/controllers/daemonset/) instead of a ReplicaSet for pods that provide a
|
||||
machine-level function, such as machine monitoring or machine logging. These pods have a lifetime that is tied
|
||||
to a machine lifetime: the pod needs to be running on the machine before other pods start, and are
|
||||
Use a [`DaemonSet`](/docs/concepts/workloads/controllers/daemonset/) instead of a ReplicaSet for Pods that provide a
|
||||
machine-level function, such as machine monitoring or machine logging. These Pods have a lifetime that is tied
|
||||
to a machine lifetime: the Pod needs to be running on the machine before other Pods start, and are
|
||||
safe to terminate when the machine is otherwise ready to be rebooted/shutdown.
|
||||
|
||||
### ReplicationController
|
||||
ReplicaSets are the successors to [_ReplicationControllers_](/docs/concepts/workloads/controllers/replicationcontroller/).
|
||||
The two serve the same purpose, and behave similarly, except that a ReplicationController does not support set-based
|
||||
selector requirements as described in the [labels user guide](/docs/concepts/overview/working-with-objects/labels/#label-selectors).
|
||||
As such, ReplicaSets are preferred over ReplicationControllers
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -55,14 +55,18 @@ This example ReplicationController config runs three copies of the nginx web ser
|
||||
Run the example job by downloading the example file and then running this command:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f https://k8s.io/examples/controllers/replication.yaml
|
||||
kubectl create -f https://k8s.io/examples/controllers/replication.yaml
|
||||
```
|
||||
```
|
||||
replicationcontroller/nginx created
|
||||
```
|
||||
|
||||
Check on the status of the ReplicationController using this command:
|
||||
|
||||
```shell
|
||||
$ kubectl describe replicationcontrollers/nginx
|
||||
kubectl describe replicationcontrollers/nginx
|
||||
```
|
||||
```
|
||||
Name: nginx
|
||||
Namespace: default
|
||||
Selector: app=nginx
|
||||
@@ -97,8 +101,10 @@ Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
|
||||
To list all the pods that belong to the ReplicationController in a machine readable form, you can use a command like this:
|
||||
|
||||
```shell
|
||||
$ pods=$(kubectl get pods --selector=app=nginx --output=jsonpath={.items..metadata.name})
|
||||
pods=$(kubectl get pods --selector=app=nginx --output=jsonpath={.items..metadata.name})
|
||||
echo $pods
|
||||
```
|
||||
```
|
||||
nginx-3ntk0 nginx-4ok8v nginx-qrm3m
|
||||
```
|
||||
|
||||
|
||||
@@ -134,6 +134,10 @@ As each Pod is created, it gets a matching DNS subdomain, taking the form:
|
||||
`$(podname).$(governing service domain)`, where the governing service is defined
|
||||
by the `serviceName` field on the StatefulSet.
|
||||
|
||||
As mentioned in the [limitations](#limitations) section, you are responsible for
|
||||
creating the [Headless Service](/docs/concepts/services-networking/service/#headless-services)
|
||||
responsible for the network identity of the pods.
|
||||
|
||||
Here are some examples of choices for Cluster Domain, Service name,
|
||||
StatefulSet name, and how that affects the DNS names for the StatefulSet's Pods.
|
||||
|
||||
|
||||
@@ -63,6 +63,11 @@ Ask your cluster administrator or consult your cloud provider or distribution do
|
||||
to determine if any sources of voluntary disruptions are enabled for your cluster.
|
||||
If none are enabled, you can skip creating Pod Disruption Budgets.
|
||||
|
||||
{{< caution >}}
|
||||
Not all voluntary disruptions are constrained by Pod Disruption Budgets. For example,
|
||||
deleting deployments or pods bypasses Pod Disruption Budgets.
|
||||
{{< /caution >}}
|
||||
|
||||
## Dealing with Disruptions
|
||||
|
||||
Here are some ways to mitigate involuntary disruptions:
|
||||
@@ -102,7 +107,7 @@ percentage of the total.
|
||||
|
||||
Cluster managers and hosting providers should use tools which
|
||||
respect Pod Disruption Budgets by calling the [Eviction API](/docs/tasks/administer-cluster/safely-drain-node/#the-eviction-api)
|
||||
instead of directly deleting pods. Examples are the `kubectl drain` command
|
||||
instead of directly deleting pods or deployments. Examples are the `kubectl drain` command
|
||||
and the Kubernetes-on-GCE cluster upgrade script (`cluster/gce/upgrade.sh`).
|
||||
|
||||
When a cluster administrator wants to drain a node
|
||||
|
||||
@@ -83,7 +83,7 @@ Here are some ideas for how to use Init Containers:
|
||||
|
||||
* Register this Pod with a remote server from the downward API with a command like:
|
||||
|
||||
curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$(<POD_NAME>)&ip=$(<POD_IP>)'
|
||||
`curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d 'instance=$(<POD_NAME>)&ip=$(<POD_IP>)'`
|
||||
|
||||
* Wait for some time before starting the app Container with a command like `sleep 60`.
|
||||
* Clone a git repository into a volume.
|
||||
@@ -180,12 +180,24 @@ spec:
|
||||
This Pod can be started and debugged with the following commands:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f myapp.yaml
|
||||
kubectl create -f myapp.yaml
|
||||
```
|
||||
```
|
||||
pod/myapp-pod created
|
||||
$ kubectl get -f myapp.yaml
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl get -f myapp.yaml
|
||||
```
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
myapp-pod 0/1 Init:0/2 0 6m
|
||||
$ kubectl describe -f myapp.yaml
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl describe -f myapp.yaml
|
||||
```
|
||||
```
|
||||
Name: myapp-pod
|
||||
Namespace: default
|
||||
[...]
|
||||
@@ -218,18 +230,25 @@ Events:
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Pulled Successfully pulled image "busybox"
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Created Created container with docker id 5ced34a04634; Security:[seccomp=unconfined]
|
||||
13s 13s 1 {kubelet 172.17.4.201} spec.initContainers{init-myservice} Normal Started Started container with docker id 5ced34a04634
|
||||
$ kubectl logs myapp-pod -c init-myservice # Inspect the first init container
|
||||
$ kubectl logs myapp-pod -c init-mydb # Inspect the second init container
|
||||
```
|
||||
```shell
|
||||
kubectl logs myapp-pod -c init-myservice # Inspect the first init container
|
||||
kubectl logs myapp-pod -c init-mydb # Inspect the second init container
|
||||
```
|
||||
|
||||
Once we start the `mydb` and `myservice` services, we can see the Init Containers
|
||||
complete and the `myapp-pod` is created:
|
||||
|
||||
```shell
|
||||
$ kubectl create -f services.yaml
|
||||
kubectl create -f services.yaml
|
||||
```
|
||||
```
|
||||
service/myservice created
|
||||
service/mydb created
|
||||
$ kubectl get -f myapp.yaml
|
||||
```
|
||||
|
||||
```shell
|
||||
kubectl get -f myapp.yaml
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
myapp-pod 1/1 Running 0 9m
|
||||
```
|
||||
|
||||
@@ -39,6 +39,9 @@ Value | Description
|
||||
`Succeeded` | All Containers in the Pod have terminated in success, and will not be restarted.
|
||||
`Failed` | All Containers in the Pod have terminated, and at least one Container has terminated in failure. That is, the Container either exited with non-zero status or was terminated by the system.
|
||||
`Unknown` | For some reason the state of the Pod could not be obtained, typically due to an error in communicating with the host of the Pod.
|
||||
`Completed` | The pod has run to completion as there's nothing to keep it running eg. Completed Jobs.
|
||||
`CrashLoopBackOff` | This means that one of the containers in the pod has exited unexpectedly, and perhaps with a non-zero error code even after restarting due to [restart policy](#restart-policy).
|
||||
|
||||
|
||||
## Pod conditions
|
||||
|
||||
@@ -130,7 +133,6 @@ specify a readiness probe. In this case, the readiness probe might be the same
|
||||
as the liveness probe, but the existence of the readiness probe in the spec means
|
||||
that the Pod will start without receiving any traffic and only start receiving
|
||||
traffic after the probe starts succeeding.
|
||||
|
||||
If your Container needs to work on loading large data, configuration files, or migrations during startup, specify a readiness probe.
|
||||
|
||||
If you want your Container to be able to take itself down for maintenance, you
|
||||
@@ -155,13 +157,47 @@ and
|
||||
Note that the information reported as Pod status depends on the current
|
||||
[ContainerState](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#containerstatus-v1-core).
|
||||
|
||||
## Container States
|
||||
|
||||
Once Pod is assigned to a node by scheduler, kubelet starts creating containers using container runtime.There are three possible states of containers: Waiting, Running and Terminated. To check state of container, you can use `kubectl describe pod [POD_NAME]`. State is displayed for each container within that Pod.
|
||||
|
||||
* `Waiting`: Default state of container. If container is not in either Running or Terminated state, it is in Waiting state. A container in Waiting state still runs its required operations, like pulling images, applying Secrets, etc. Along with this state, a message and reason about the state are displayed to provide more information.
|
||||
|
||||
```yaml
|
||||
...
|
||||
State: Waiting
|
||||
Reason: ErrImagePull
|
||||
...
|
||||
```
|
||||
|
||||
* `Running`: Indicates that the container is executing without issues. Once a container enters into Running, `postStart` hook (if any) is executed. This state also displays the time when the container entered Running state.
|
||||
|
||||
```yaml
|
||||
...
|
||||
State: Running
|
||||
Started: Wed, 30 Jan 2019 16:46:38 +0530
|
||||
...
|
||||
```
|
||||
|
||||
* `Terminated`: Indicates that the container completed its execution and has stopped running.A container enters into this when it has successfully completed execution or when it has failed for some reason. Regardless, a reason and exit code is displayed, as well as the container's start and finish time. Before a container enters into Terminated, `preStop` hook (if any) is executed.
|
||||
|
||||
```yaml
|
||||
...
|
||||
State: Terminated
|
||||
Reason: Completed
|
||||
Exit Code: 0
|
||||
Started: Wed, 30 Jan 2019 11:45:26 +0530
|
||||
Finished: Wed, 30 Jan 2019 11:45:26 +0530
|
||||
...
|
||||
```
|
||||
|
||||
## Pod readiness gate
|
||||
|
||||
{{< feature-state for_k8s_version="v1.12" state="beta" >}}
|
||||
|
||||
In order to add extensibility to Pod readiness by enabling the injection of
|
||||
extra feedbacks or signals into `PodStatus`, Kubernetes 1.11 introduced a
|
||||
feature named [Pod ready++](https://github.com/kubernetes/community/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md).
|
||||
feature named [Pod ready++](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/0007-pod-ready%2B%2B.md).
|
||||
You can use the new field `ReadinessGate` in the `PodSpec` to specify additional
|
||||
conditions to be evaluated for Pod readiness. If Kubernetes cannot find such a
|
||||
condition in the `status.conditions` field of a Pod, the status of the condition
|
||||
|
||||
@@ -4,6 +4,9 @@ reviewers:
|
||||
title: Pod Overview
|
||||
content_template: templates/concept
|
||||
weight: 10
|
||||
card:
|
||||
name: concepts
|
||||
weight: 60
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -101,5 +104,5 @@ Rather than specifying the current desired state of all replicas, pod templates
|
||||
{{% capture whatsnext %}}
|
||||
* Learn more about Pod behavior:
|
||||
* [Pod Termination](/docs/concepts/workloads/pods/pod/#termination-of-pods)
|
||||
* Other Pod Topics
|
||||
* [Pod Lifecycle](../pod-lifecycle)
|
||||
{{% /capture %}}
|
||||
|
||||
@@ -173,8 +173,8 @@ An example flow:
|
||||
1. The Pod in the API server is updated with the time beyond which the Pod is considered "dead" along with the grace period.
|
||||
1. Pod shows up as "Terminating" when listed in client commands
|
||||
1. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
|
||||
1. If the pod has defined a [preStop hook](/docs/concepts/containers/container-lifecycle-hooks/#hook-details), it is invoked inside of the pod. If the `preStop` hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
|
||||
1. The processes in the Pod are sent the TERM signal.
|
||||
1. If one of the Pod's containers has defined a [preStop hook](/docs/concepts/containers/container-lifecycle-hooks/#hook-details), it is invoked inside of the container. If the `preStop` hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
|
||||
1. The container is sent the TERM signal. Note that not all containers in the Pod will receive the TERM signal at the same time and may each require a `preStop` hook if the order in which they shut down matters.
|
||||
1. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.
|
||||
1. When the grace period expires, any processes still running in the Pod are killed with SIGKILL.
|
||||
1. The Kubelet will finish deleting the Pod on the API server by setting grace period 0 (immediate deletion). The Pod disappears from the API and is no longer visible from the client.
|
||||
@@ -191,7 +191,7 @@ Force deletions can be potentially dangerous for some pods and should be perform
|
||||
|
||||
From Kubernetes v1.1, any container in a pod can enable privileged mode, using the `privileged` flag on the `SecurityContext` of the container spec. This is useful for containers that want to use linux capabilities like manipulating the network stack and accessing devices. Processes within the container get almost the same privileges that are available to processes outside a container. With privileged mode, it should be easier to write network and volume plugins as separate pods that don't need to be compiled into the kubelet.
|
||||
|
||||
If the master is running Kubernetes v1.1 or higher, and the nodes are running a version lower than v1.1, then new privileged pods will be accepted by api-server, but will not be launched. They will be pending state.
|
||||
If the master is running Kubernetes v1.1 or higher, and the nodes are running a version lower than v1.1, then new privileged pods will be accepted by api-server, but will not be launched. They will be in pending state.
|
||||
If user calls `kubectl describe pod FooPodName`, user can see the reason why the pod is in pending state. The events table in the describe command output will say:
|
||||
`Error validating pod "FooPodName"."FooPodNamespace" from api, ignoring: spec.containers[0].securityContext.privileged: forbidden '<*>(0xc2089d3248)true'`
|
||||
|
||||
|
||||
@@ -68,10 +68,14 @@ In order to use Pod Presets in your cluster you must ensure the following:
|
||||
|
||||
1. You have enabled the API type `settings.k8s.io/v1alpha1/podpreset`. For
|
||||
example, this can be done by including `settings.k8s.io/v1alpha1=true` in
|
||||
the `--runtime-config` option for the API server.
|
||||
the `--runtime-config` option for the API server. In minikube add this flag
|
||||
`--extra-config=apiserver.runtime-config=settings.k8s.io/v1alpha1=true` while
|
||||
starting the cluster.
|
||||
1. You have enabled the admission controller `PodPreset`. One way to doing this
|
||||
is to include `PodPreset` in the `--enable-admission-plugins` option value specified
|
||||
for the API server.
|
||||
for the API server. In minikube add this flag
|
||||
`--extra-config=apiserver.enable-admission-plugins=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PodPreset`
|
||||
while starting the cluster.
|
||||
1. You have defined your Pod Presets by creating `PodPreset` objects in the
|
||||
namespace you will use.
|
||||
|
||||
|
||||
@@ -38,6 +38,24 @@ for weekly rotations. The PR wrangler's duties include:
|
||||
[Intermediate contributing](/docs/contribute/intermediate/) for guidelines
|
||||
about how SIG Docs uses metadata.
|
||||
|
||||
### Helpful Github queries for wranglers
|
||||
|
||||
The following queries are helpful when wrangling. After working through these three queries, the remaining list of PRs to be
|
||||
reviewed is usually small. These queries specifically exclude localization PRs, and only include the `master` branch (except for the last one).
|
||||
|
||||
- [No CLA, not eligible to merge](https://github.com/kubernetes/website/pulls?q=is%3Aopen+is%3Apr+label%3A%22cncf-cla%3A+no%22+-label%3Ado-not-merge+label%3Alanguage%2Fen):
|
||||
Remind the contributor to sign the CLA. If they've already been reminded by both the bot and a human, close
|
||||
the PR and remind them that they can open it after signing the CLA.
|
||||
**We can't even review PRs whose authors have not signed the CLA!**
|
||||
- [Needs LGTM](https://github.com/kubernetes/website/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+-label%3Ado-not-merge+label%3Alanguage%2Fen+-label%3Algtm+):
|
||||
If it needs technical review, loop in one of the reviewers suggested by the bot. If it needs docs review
|
||||
or copy-editing, either suggest changes or add a copyedit commit to the PR to move it along.
|
||||
- [Has LGTM, needs docs approval](https://github.com/kubernetes/website/pulls?q=is%3Aopen+is%3Apr+-label%3Ado-not-merge+label%3Alanguage%2Fen+label%3Algtm):
|
||||
See if you can figure out what needs to happen for the PR to be merged.
|
||||
- [Not against master](https://github.com/kubernetes/website/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+-label%3Ado-not-merge+label%3Alanguage%2Fen+-base%3Amaster): If it's against a `dev-` branch, it's for an upcoming release.
|
||||
Make sure the [release meister](https://github.com/kubernetes/sig-release/tree/master/release-team) knows about it.
|
||||
If it's against an old branch, help the PR author figure out whether it's targeted against the best branch.
|
||||
|
||||
## Propose improvements
|
||||
|
||||
SIG Docs
|
||||
|
||||
@@ -3,6 +3,9 @@ title: Intermediate contributing
|
||||
slug: intermediate
|
||||
content_template: templates/concept
|
||||
weight: 20
|
||||
card:
|
||||
name: contribute
|
||||
weight: 50
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -487,23 +490,8 @@ Slack channel or the
|
||||
### View your changes locally
|
||||
|
||||
If you aren't ready to create a pull request but you want to see what your
|
||||
changes look like, you can use the `hugo` command to stage the changes locally.
|
||||
|
||||
1. Install Hugo version {{< hugoVersion >}} or later.
|
||||
|
||||
2. In a terminal, go to the root directory of your clone of the Kubernetes
|
||||
docs, and enter this command:
|
||||
|
||||
```bash
|
||||
hugo server
|
||||
```
|
||||
|
||||
3. In your browser’s address bar, enter `localhost:1313`.
|
||||
|
||||
4. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`
|
||||
or just close the terminal window.
|
||||
|
||||
Alternatively, you can build the Kubernetes docs using Docker.
|
||||
changes look like, you can build and run a docker image to generate all the documentation and
|
||||
serve it locally.
|
||||
|
||||
1. Build the image locally:
|
||||
|
||||
@@ -518,7 +506,26 @@ Alternatively, you can build the Kubernetes docs using Docker.
|
||||
```
|
||||
|
||||
3. In your browser's address bar, enter `localhost:1313`. Hugo will watch the
|
||||
filesystem for changes and rebuild the site as needed.
|
||||
filesystem for changes and rebuild the site as needed.
|
||||
|
||||
4. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`
|
||||
or just close the terminal window.
|
||||
|
||||
Alternatively, you can install and use the `hugo` command on your development machine:
|
||||
|
||||
1. [Install Hugo](https://gohugo.io/getting-started/installing/) version {{< hugoVersion >}} or later.
|
||||
|
||||
2. In a terminal, go to the root directory of your clone of the Kubernetes
|
||||
docs, and enter this command:
|
||||
|
||||
```bash
|
||||
hugo server
|
||||
```
|
||||
|
||||
3. In your browser’s address bar, enter `localhost:1313`.
|
||||
|
||||
4. To stop the local Hugo instance, go back to the terminal and type `Ctrl+C`
|
||||
or just close the terminal window.
|
||||
|
||||
## Triage and categorize issues
|
||||
|
||||
|
||||
@@ -5,29 +5,25 @@ approvers:
|
||||
- chenopis
|
||||
- zacharysarah
|
||||
- zparnold
|
||||
card:
|
||||
name: contribute
|
||||
weight: 30
|
||||
title: Translating the docs
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
Documentation for Kubernetes is available in multiple languages:
|
||||
|
||||
- English
|
||||
- Chinese
|
||||
- Japanese
|
||||
- Korean
|
||||
|
||||
We encourage you to add new [localizations](https://blog.mozilla.org/l10n/2011/12/14/i18n-vs-l10n-whats-the-diff/)!
|
||||
Documentation for Kubernetes is available in multiple languages. We encourage you to add new [localizations](https://blog.mozilla.org/l10n/2011/12/14/i18n-vs-l10n-whats-the-diff/)!
|
||||
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
{{% capture body %}}
|
||||
|
||||
## Getting started
|
||||
|
||||
Localizations must meet some requirements for workflow (*how* to localize) and output (*what* to localize).
|
||||
Localizations must meet some requirements for workflow (*how* to localize) and output (*what* to localize) before publishing.
|
||||
|
||||
To add a new localization of the Kubernetes documentation, you'll need to update the website by modifying the [site configuration](#modify-the-site-configuration) and [directory structure](#add-a-new-localization-directory). Then you can start [translating documents](#translating-documents)!
|
||||
To add a new localization of the Kubernetes documentation, you'll need to update the website by modifying the [site configuration](#modify-the-site-configuration) and [directory structure](#add-a-new-localization-directory). Then you can start [translating documents](#translating-documents)!
|
||||
|
||||
{{< note >}}
|
||||
For an example localization-related [pull request](../create-pull-request), see [this pull request](https://github.com/kubernetes/website/pull/8636) to the [Kubernetes website repo](https://github.com/kubernetes/website) adding Korean localization to the Kubernetes docs.
|
||||
@@ -209,7 +205,7 @@ SIG Docs welcomes [upstream contributions and corrections](/docs/contribute/inte
|
||||
|
||||
{{% capture whatsnext %}}
|
||||
|
||||
Once a l10n meets requirements for workflow and minimum output, SIG docs will:
|
||||
Once a localization meets requirements for workflow and minimum output, SIG docs will:
|
||||
|
||||
- Enable language selection on the website
|
||||
- Publicize the localization's availability through [Cloud Native Computing Foundation](https://www.cncf.io/) (CNCF) channels, including the [Kubernetes blog](https://kubernetes.io/blog/).
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
---
|
||||
title: Participating in SIG Docs
|
||||
content_template: templates/concept
|
||||
card:
|
||||
name: contribute
|
||||
weight: 40
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
|
||||
@@ -3,6 +3,9 @@ title: Start contributing
|
||||
slug: start
|
||||
content_template: templates/concept
|
||||
weight: 10
|
||||
card:
|
||||
name: contribute
|
||||
weight: 10
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -79,7 +82,7 @@ Anyone with a Github account can file an issue (bug report) against the
|
||||
Kubernetes documentation. If you see something wrong, even if you have no idea
|
||||
how to fix it, [file an issue](#how-to-file-an-issue). The exception to this
|
||||
rule is a tiny bug like a typo that you intend to fix yourself. In that case,
|
||||
you can instead [fix it](#fix-it) without filing a bug first.
|
||||
you can instead [fix it](#improve-existing-content) without filing a bug first.
|
||||
|
||||
### How to file an issue
|
||||
|
||||
@@ -133,10 +136,11 @@ The SIG Docs team communicates using the following mechanisms:
|
||||
introduce yourself!
|
||||
- [Join the `kubernetes-sig-docs` mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-docs),
|
||||
where broader discussions take place and official decisions are recorded.
|
||||
- Participate in the weekly SIG Docs video meeting, which is announced on the
|
||||
Slack channel and the mailing list. Currently, these meetings take place on
|
||||
Zoom, so you'll need to download the [Zoom client](https://zoom.us/download)
|
||||
or dial in using a phone.
|
||||
- Participate in the [weekly SIG Docs](https://github.com/kubernetes/community/tree/master/sig-docs) video meeting, which is announced on the Slack channel and the mailing list. Currently, these meetings take place on Zoom, so you'll need to download the [Zoom client](https://zoom.us/download) or dial in using a phone.
|
||||
|
||||
{{< note >}}
|
||||
You can also check the SIG Docs weekly meeting on the [Kubernetes community meetings calendar](https://calendar.google.com/calendar/embed?src=cgnt364vd8s86hr2phapfjc6uk%40group.calendar.google.com&ctz=America/Los_Angeles).
|
||||
{{< /note >}}
|
||||
|
||||
## Improve existing content
|
||||
|
||||
|
||||
@@ -108,7 +108,6 @@ Another widely used example is the `includes` bundle. It sets `headless: true` i
|
||||
en/includes
|
||||
├── default-storage-class-prereqs.md
|
||||
├── federated-task-tutorial-prereqs.md
|
||||
├── federation-content-moved.md
|
||||
├── index.md
|
||||
├── partner-script.js
|
||||
├── partner-style.css
|
||||
|
||||
@@ -2,6 +2,9 @@
|
||||
title: Using Page Templates
|
||||
content_template: templates/concept
|
||||
weight: 30
|
||||
card:
|
||||
name: contribute
|
||||
weight: 30
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -65,17 +68,13 @@ To write a new concept page, create a Markdown file in a subdirectory of the
|
||||
{{%/* /capture */%}}
|
||||
```
|
||||
|
||||
- Within each section, write your content. Use the following guidelines:
|
||||
- Use a minimum of H2 headings (with two leading `#` characters). The sections
|
||||
themselves are titled automatically by the template.
|
||||
- For `overview`, use a paragraph to set context for the entire topic.
|
||||
- For `body`, explain the concept using free-form Markdown.
|
||||
- For `whatsnext`, give a bullet list of up to 5 topics the reader might be
|
||||
interested in reading next.
|
||||
- Fill each section with content. Follow these guidelines:
|
||||
- Organize content with H2 and H3 headings.
|
||||
- For `overview`, set the topic's context with a single paragraph.
|
||||
- For `body`, explain the concept.
|
||||
- For `whatsnext`, provide a bulleted list of topics (5 maximum) to learn more about the concept.
|
||||
|
||||
An example of a published topic that uses the concept template is
|
||||
[Annotations](/docs/concepts/overview/working-with-objects/annotations/). The
|
||||
page you are currently reading also uses the concept template.
|
||||
[Annotations](/docs/concepts/overview/working-with-objects/annotations/) is a published example of the concept template. This page also uses the concept template.
|
||||
|
||||
## Task template
|
||||
|
||||
|
||||
@@ -3,6 +3,10 @@ title: Documentation Style Guide
|
||||
linktitle: Style guide
|
||||
content_template: templates/concept
|
||||
weight: 10
|
||||
card:
|
||||
name: contribute
|
||||
weight: 20
|
||||
title: Documentation Style Guide
|
||||
---
|
||||
|
||||
{{% capture overview %}}
|
||||
@@ -106,8 +110,13 @@ document, use the backtick (`).
|
||||
<tr><th>Do</th><th>Don't</th></tr>
|
||||
<tr><td>The <code>kubectl run</code> command creates a Deployment.</td><td>The "kubectl run" command creates a Deployment.</td></tr>
|
||||
<tr><td>For declarative management, use <code>kubectl apply</code>.</td><td>For declarative management, use "kubectl apply".</td></tr>
|
||||
<tr><td>Enclose code samples with triple backticks. <code>(```)</code></td><td>Enclose code samples with any other syntax.</td></tr>
|
||||
</table>
|
||||
|
||||
{{< note >}}
|
||||
The website supports syntax highlighting for code samples, but specifying a language is optional.
|
||||
{{< /note >}}
|
||||
|
||||
### Use code style for object field names
|
||||
|
||||
<table>
|
||||
@@ -318,7 +327,7 @@ Shortcodes inside include statements will break the build. You must insert them
|
||||
|
||||
```
|
||||
{{</* note */>}}
|
||||
{{</* include "federation-current-state.md" */>}}
|
||||
{{</* include "task-tutorial-prereqs.md" */>}}
|
||||
{{</* /note */>}}
|
||||
```
|
||||
|
||||
|
||||
@@ -116,7 +116,7 @@
|
||||
'body': '{{< toc >}}'
|
||||
'Insert code from file':
|
||||
'prefix': 'codefile'
|
||||
'body': '{{< code file="$1" >}}'
|
||||
'body': '{{< codenew file="$1" >}}'
|
||||
'Insert feature state':
|
||||
'prefix': 'fstate'
|
||||
'body': '{{< feature-state for_k8s_version="$1" state="$2" >}}'
|
||||
@@ -223,4 +223,4 @@
|
||||
${7:"next-steps-or-delete"}
|
||||
{{% /capture %}}
|
||||
"""
|
||||
|
||||
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# See the OWNERS docs at https://go.k8s.io/owners
|
||||
|
||||
reviewers:
|
||||
- errordeveloper
|
||||
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# See the OWNERS docs at https://go.k8s.io/owners
|
||||
|
||||
reviewers:
|
||||
- aveshagarwal
|
||||
- eparis
|
||||
|
||||
@@ -65,8 +65,7 @@ KUBE_MASTER="--master=http://fed-master:8080"
|
||||
systemctl mask firewalld.service
|
||||
systemctl stop firewalld.service
|
||||
|
||||
systemctl disable iptables.service
|
||||
systemctl stop iptables.service
|
||||
systemctl disable --now iptables.service
|
||||
```
|
||||
|
||||
**Configure the Kubernetes services on the master.**
|
||||
@@ -97,8 +96,7 @@ ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
|
||||
|
||||
```shell
|
||||
for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
|
||||
systemctl restart $SERVICES
|
||||
systemctl enable $SERVICES
|
||||
systemctl enable --now $SERVICES
|
||||
systemctl status $SERVICES
|
||||
done
|
||||
```
|
||||
@@ -146,8 +144,7 @@ current-context: kubelet-context
|
||||
|
||||
```shell
|
||||
for SERVICES in kube-proxy kubelet docker; do
|
||||
systemctl restart $SERVICES
|
||||
systemctl enable $SERVICES
|
||||
systemctl enable --now $SERVICES
|
||||
systemctl status $SERVICES
|
||||
done
|
||||
```
|
||||
|
||||
@@ -51,7 +51,7 @@ These are more in-depth guides for users choosing to run Kubernetes in productio
|
||||
- [Decommissioning](/docs/getting-started-guides/ubuntu/decommissioning/)
|
||||
- [Operational Considerations](/docs/getting-started-guides/ubuntu/operational-considerations/)
|
||||
- [Glossary](/docs/getting-started-guides/ubuntu/glossary/)
|
||||
|
||||
- [Authenticating with LDAP](https://www.ubuntu.com/kubernetes/docs/ldap)
|
||||
|
||||
## Third-party Product Integrations
|
||||
|
||||
@@ -73,5 +73,3 @@ We're normally following the following Slack channels:
|
||||
|
||||
and we monitor the Kubernetes mailing lists.
|
||||
{{% /capture %}}
|
||||
|
||||
|
||||
|
||||
@@ -18,6 +18,15 @@ sudo snap install conjure-up --classic
|
||||
sudo usermod -a -G lxd $(whoami)
|
||||
```
|
||||
|
||||
If you have never run or configured lxd before, you will need to run the following
|
||||
command to set up the default storage pool and the network bridge required:
|
||||
|
||||
```
|
||||
sudo lxd init
|
||||
```
|
||||
|
||||
If a bridge named `lxdbr0` already exists, you can tell the init config tool to use it.
|
||||
|
||||
Note: If conjure-up asks you to "Setup an IPv6 subnet" with LXD, answer NO. IPv6 with Juju/LXD is currently unsupported.
|
||||
If you already have a bridge configured, e.g. `lxdbr0`, [disable IPv6 on the bridge](https://docs.conjure-up.io/stable/en/troubleshoot#common-problems), otherwise you won't be able to choose it.
|
||||
{{% /capture %}}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user