From bb7abf59cd85901ce271c525b1c8746998d89b1b Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Fri, 22 Mar 2019 17:04:20 +0100 Subject: [PATCH] E2E testing blog post (#13280) * E2E testing blog post * E2E testing blog post: add information about optional command line flags * Update and rename 2019-03-30-e2e-testing-for-everyone.md to 2019-03-22-e2e-testing-for-everyone.md --- .../2019-03-22-e2e-testing-for-everyone.md | 366 ++++++++++++++++++ 1 file changed, 366 insertions(+) create mode 100644 content/en/blog/_posts/2019-03-22-e2e-testing-for-everyone.md diff --git a/content/en/blog/_posts/2019-03-22-e2e-testing-for-everyone.md b/content/en/blog/_posts/2019-03-22-e2e-testing-for-everyone.md new file mode 100644 index 0000000000..e23c4ce82f --- /dev/null +++ b/content/en/blog/_posts/2019-03-22-e2e-testing-for-everyone.md @@ -0,0 +1,366 @@ +--- +title: Kubernetes End-to-end Testing for Everyone +date: 2019-03-22 +--- + +**Author:** Patrick Ohly (Intel) + +More and more components that used to be part of Kubernetes are now +being developed outside of Kubernetes. For example, storage drivers +used to be compiled into Kubernetes binaries, then were moved into +[stand-alone Flexvolume +binaries](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-storage/flexvolume.md) +on the host, and now are delivered as [Container Storage Interface +(CSI) drivers](https://github.com/container-storage-interface/spec) +that get deployed in pods inside the Kubernetes cluster itself. + +This poses a challenge for developers who work on such components: how +can end-to-end (E2E) testing on a Kubernetes cluster be done for such +external components? The E2E framework that is used for testing +Kubernetes itself has all the necessary functionality. However, trying +to use it outside of Kubernetes was difficult and only possible by +carefully selecting the right versions of a large number of +dependencies. E2E testing has become a lot simpler in Kubernetes 1.13. + +This blog post summarizes the changes that went into Kubernetes +1.13. For CSI driver developers, it will cover the ongoing effort to +also make the storage tests available for testing of third-party CSI +drivers. How to use them will be shown based on two Intel CSI drivers: + +* [Open Infrastructure Manager (OIM)](https://github.com/intel/oim/) +* [PMEM-CSI](https://github.com/intel/pmem-csi) + +Testing those drivers was the main motivation behind most of these +enhancements. + +## E2E overview + +E2E testing consists of several phases: + +* Implementing a test suite. This is the main focus of this blog + post. The Kubernetes E2E framework is written in Go. It relies on + [Ginkgo](https://onsi.github.io/ginkgo/) for managing tests and + [Gomega](http://onsi.github.io/gomega/) for assertions. These tools + support “behavior driven development”, which describes expected + behavior in “specs”. In this blog post, “test” is used to reference + an individual `Ginkgo.It` spec. Tests interact with the Kubernetes + cluster using + [client-go](https://godoc.org/k8s.io/client-go/kubernetes). +* Bringing up a test cluster. Tools like + [kubetest](https://github.com/kubernetes/test-infra/blob/master/kubetest/README.md) + can help here. +* Running an E2E test suite against that cluster. Ginkgo test suites + can be run with the `ginkgo` tool or as a normal Go test with `go + test`. Without any parameters, a Kubernetes E2E test suite will + connect to the default cluster based on environment variables like + KUBECONFIG, exactly like kubectl. Kubetest also knows how to run the + Kubernetes E2E suite. + +## E2E framework enhancements in Kubernetes 1.13 + +All of the following enhancements follow the same basic pattern: they +make the E2E framework more useful and easier to use outside of +Kubernetes, without changing the behavior of the original Kubernetes +e2e.test binary. + +### Splitting out provider support + +The main reason why using the E2E framework from Kubernetes <= 1.12 +was difficult were the dependencies on provider-specific SDKs, which +pulled in a large number of packages. Just getting it compiled was +non-trivial. + +Many of these packages are only needed for certain tests. For example, +testing the mounting of a pre-provisioned volume must first provision +such a volume the same way as an administrator would, by talking +directly to a specific storage backend via some non-Kubernetes API. + +There is an effort to [remove cloud provider-specific +tests](https://github.com/kubernetes/kubernetes/issues/70194) from +core Kubernetes. The approach taken in [PR +#68483](https://github.com/kubernetes/kubernetes/pull/68483) can be +seen as an incremental step towards that goal: instead of ripping out +the code immediately and breaking all tests that depend on it, all +cloud provider-specific code was moved into optional packages under +[test/e2e/framework/providers](https://github.com/kubernetes/kubernetes/tree/release-1.13/test/e2e/framework/providers). The +E2E framework then accesses it via [an +interface](https://github.com/kubernetes/kubernetes/blob/6c1e64b94a3e111199c934c39a0c25bc219ed5f9/test/e2e/framework/provider.go#L79-L99) +that gets implemented separately by each vendor package. + +The author of a E2E test suite decides which of these packages get +imported into the test suite. The vendor support is then activated via +the `--provider` command line flag. The Kubernetes e2e.test binary in +1.13 and 1.14 still contains support for the same providers as in +1.12. It is also okay to include no packages, which means that only +the generic providers will be available: + +* “skeleton”: cluster is accessed via the Kubernetes API and nothing + else +* “local”: like “skeleton”, but in addition the scripts in + kubernetes/kubernetes/cluster can retrieve logs via ssh after a test + suite is run + +### External files + +Tests may have to read additional files at runtime, like .yaml +manifests. But the Kubernetes e2e.test binary is supposed to be usable +and entirely stand-alone because that simplifies shipping and running +it. The solution in the Kubernetes build system is to link all files +under `test/e2e/testing-manifests` into the binary with +[go-bindata](https://github.com/jteeuwen/go-bindata/go-bindata). The +E2E framework used to have a hard dependency on the output of +`go-bindata`, now [bindata support is +optional](https://github.com/kubernetes/kubernetes/pull/69103). When +accessing a file via the [testfiles +package](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/e2e/framework/testfiles/testfiles.go), +files will be retrieved from different sources: + +* relative to the directory specified with `--repo-root` parameter +* zero or more bindata chunks + +### Test parameters + +The e2e.test binary takes additional parameters which control test +execution. In 2016, an effort was started to replace all E2E command +line parameters with a Viper configuration file. But that effort +[stalled](https://github.com/kubernetes/kubernetes/blob/0ed33881dc4355495f623c6f22e7dd0b7632b7c0/test/e2e/framework/test_context.go#L318-L319 +), which left developers without clear guidance how they should handle +test-specific parameters. + +The approach in v1.12 was to add all flags to the central +[test/e2e/framework/test_context.go](https://github.com/kubernetes/kubernetes/blob/v1.12.0/test/e2e/framework/test_context.go), +which does not work for tests developed independently from the +framework. Since [PR +#69105](https://github.com/kubernetes/kubernetes/pull/69105) the +recommendation has been to use the normal `flag` package to +define its parameters, in its own source code. Flag names must be +hierarchical with dots separating different levels, for example +`my.test.parameter`, and must be unique. Uniqueness is enforced by the +`flag` package which panics when registering a flag a second time. The +new +[config](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/e2e/framework/config/config.go) +package simplifies the definition of multiple options, which are +stored in a single struct. + +To summarize, this is how parameters are handled now: + +* The init code in test packages defines tests and parameters. The + actual parameter *values* are not available yet, so test definitions + cannot use them. +* The init code of the test suite parses parameters and (optionally) + the configuration file. +* The tests run and now can use parameter values. + +However, recently it [was pointed +out](https://github.com/kubernetes/kubernetes/pull/69105#discussion_r267960062) +that it is desirable and was possible to not expose test settings as +command line flags and only set them via a configuration file. There +is an [open bug](https://github.com/kubernetes/kubernetes/issues/75590) and a +[pending PR](https://github.com/kubernetes/kubernetes/pull/75593) +about this. + +Viper support has been enhanced. Like the provider support, it is +completely optional. It gets pulled into a e2e.test binary by +importing the `viperconfig` package and [calling +it](https://github.com/kubernetes/kubernetes/blob/ddf47ac13c1a9483ea035a79cd7c10005ff21a6d/test/e2e/e2e_test.go#L49-L57) +after parsing the normal command line flags. This has been implemented +so that all variables which can be set via command line flags are also +set when the flag appears in a Viper config file. For example, the +Kubernetes v1.13 `e2e.test` binary accepts +`--viper-config=/tmp/my-config.yaml` and that file will set the +`my.test.parameter` to `value` when it has this content: my: test: +parameter: value + +In older Kubernetes releases, that option could only load a file from +the current directory, the suffix had to be left out, and only a few +parameters actually could be set this way. Beware that one limitation +of Viper still exists: it works by matching config file entries +against known flags, without warning about unknown config file entries +and thus leaving typos undetected. A [better config file +parser](https://github.com/kubernetes/kubeadm/issues/1040) for +Kubernetes is still work in progress. + +### Creating items from .yaml manifests + +In Kubernetes 1.12, there was some support for loading individual +items from a .yaml file, but then creating that item had to be done by +hand-written code. Now the framework has [new +methods](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/e2e/framework/create.go) +for loading a .yaml file that has multiple items, patching those items +(for example, setting the namespace created for the current test), and +creating them. This is currently [used to deploy CSI +drivers](https://github.com/kubernetes/kubernetes/blob/ddf47ac13c1a9483ea035a79cd7c10005ff21a6d/test/e2e/storage/drivers/csi.go#L192-L209 +) anew for each test from exactly the same .yaml files that are also +used for deployment via kubectl. If the CSI driver supports running +under different names, then tests are completely independent and can +run in parallel. + +However, redeploying a driver slows down test execution and it does +not cover concurrent operations against the driver. A more realistic +test scenario is to deploy a driver once when bringing up the test +cluster, then run all tests against that deployment. Eventually the +Kubernetes E2E testing will move to that model, once it is clearer how +test cluster bringup can be extended such that it also includes +installing additional entities like CSI drivers. + +## Upcoming enhancements in Kubernetes 1.14 + +### Reusing storage tests + +Being able to use the framework outside of Kubernetes enables building +a custom test suite. But a test suite without tests is still +useless. Several of the existing tests, in particular for storage, can +also be applied to out-of-tree components. Thanks to the work done by +Masaki Kimura, [storage +tests](https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/e2e/storage/testsuites) +in Kubernetes 1.13 are defined such that they can be instantiated +multiple times for different drivers. + +But history has a habit of repeating itself. As with providers, the +package defining these tests also pulled in driver definitions for all +in-tree storage backends, which in turn pulled in more additional +packages than were needed. This has been +[fixed](https://github.com/kubernetes/kubernetes/pull/70862) for the +upcoming Kubernetes 1.14. + +### Skipping unsupported tests + +Some of the storage tests depend on features of the cluster (like +running on a host that supports XFS) or of the driver (like supporting +block volumes). These conditions are checked while the test runs, +leading to skipped tests when they are not satisfied. The good thing +is that this records an explanation why the test did not run. + +Starting a test is slow, in particular when it must first deploy the +CSI driver, but also in other scenarios. Creating the namespace for a +test has been measured at 5 seconds on a fast cluster, and it produces +a lot of noisy test output. It would have been possible to address +that by [skipping the definition of unsupported +tests](https://github.com/kubernetes/kubernetes/pull/70992), but then +reporting why a test isn’t even part of the test suite becomes +tricky. This approach has been dropped in favor of reorganizing the +storage test suite such that it [first checks +conditions](https://github.com/kubernetes/kubernetes/pull/72434) +before doing the more expensive test setup steps. + +### More readable test definitions + +The same PR also rewrites the tests to operate like conventional +Ginkgo tests, with test cases and their local variables in [a single +function](https://github.com/pohly/kubernetes/blob/ec3655a1d40ced6b1873e627b736aae1cf242477/test/e2e/storage/testsuites/provisioning.go#L82). + +### Testing external drivers + +Building a custom E2E test suite is still quite a bit of work. The +e2e.test binary that will get distributed in the [Kubernetes 1.14 test +archive](https://dl.k8s.io/v1.14.0/kubernetes-test.tar.gz) will have +the [ability to +test](https://github.com/kubernetes/kubernetes/pull/72836) already +installed storage drivers without rebuilding the test suite. See this +[README](https://github.com/pohly/kubernetes/blob/6644db9914379a4a7b3d3487b41b2010f226e4dc/test/e2e/storage/external/README.md) +for further instructions. + +## E2E test suite HOWTO + +### Test suite initialization + +The first step is to set up the necessary boilerplate code that +defines the test suite. [In Kubernetes +E2E](https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/e2e), +this is done in the `e2e.go` and `e2e_test.go` files. It could also be +done in a single `e2e_test.go` file. Kubernetes imports all of the +various providers, in-tree tests, Viper configuration support, and +bindata file lookup in `e2e_test.go`. `e2e.go` controls the actual +execution, including some cluster preparations and metrics collection. + +A simpler starting point are the `e2e_[test].go` files [from +PMEM-CSI](https://github.com/intel/pmem-csi/tree/586ae281ac2810cb4da6f1e160cf165c7daf0d80/test/e2e). It +doesn’t use any providers, no Viper, no bindata, and imports just the +storage tests. + +Like PMEM-CSI, OIM drops all of the extra features, but is a bit more +complex because it integrates a custom cluster startup directly into +the [test +suite](https://github.com/intel/pmem-csi/blob/a7b0d66b59771bf615e07fcd3d4f0ba08cfdf90f/test/e2e/e2e.go), +which was useful in this case because some additional components have +to run on the host side. By running them directly in the E2E binary, +interactive debugging with `dlv` becomes easier. + +Both CSI drivers follow the Kubernetes example and use the `test/e2e` +directory for their test suites, but any other directory and other +file names would also work. + +### Adding E2E storage tests + +Tests are defined by packages that get imported into a test suite. The +only thing specific to E2E tests is that they instantiate a +`framework.Framework` pointer (usually called `f`) with +`framework.NewDefaultFramework`. This variable gets initialized anew +in a `BeforeEach` for each test and freed in an `AfterEach`. It has a +`f.ClientSet` and `f.Namespace` at runtime (and only at runtime!) +which can be used by a test. + +The [PMEM-CSI storage +test](https://github.com/intel/pmem-csi/blob/586ae281ac2810cb4da6f1e160cf165c7daf0d80/storage/csi_volumes.go#L51) +imports the Kubernetes storage test suite and sets up one instance of +the provisioning tests for a PMEM-CSI driver which must be already +installed in the test cluster. The storage test suite changes the +storage class to run tests with different filesystem types. Because of +this requirement, the storage class is created from a .yaml file. + +Explaining all the various utility methods available in the framework +is out of scope for this blog post. Reading existing tests and the +source code of the framework is a good way to get started. + +### Vendoring + +Vendoring Kubernetes code is still not trivial, even after eliminating +many of the unnecessary dependencies. `k8s.io/kubernetes` is not meant +to be included in other projects and does not define its dependencies +in a way that is understood by tools like `dep`. The other `k8s.io` +packages are meant to be included, but [don’t follow semantic +versioning +yet]([https://github.com/kubernetes/kubernetes/issues/72638) or don’t +tag any releases (`k8s.io/kube-openapi`, `k8s.io/utils`). + +PMEM-CSI uses [dep](https://golang.github.io/dep/). It’s +[Gopkg.toml](https://github.com/intel/pmem-csi/blob/0ad8251c064b1010c91e7fc1dd423b95d5594bba/Gopkg.toml) +file is a good starting point. It enables pruning (not enabled in dep +by default) and locks certain projects onto versions that are +compatible with the Kubernetes version that is used. When `dep` +doesn’t pick a compatible version, then checking Kubernetes’ +[Godeps.json](https://github.com/kubernetes/kubernetes/blob/master/Godeps/Godeps.json) +helps to determine which revision might be the right one. + +### Compiling and running the test suite + +`go test ./test/e2e -args -help` is the fastest way to test that the +test suite compiles. + +Once it does compile and a cluster has been set up, the command `go +test -timeout=0 -v ./test/e2e -ginkgo.v` runs all tests. In order to +run tests in parallel, use the `ginkgo -p ./test/e2e` command instead. + +## Getting involved + +The Kubernetes E2E framework is owned by the testing-commons +sub-project in +[SIG-testing](https://github.com/kubernetes/community/tree/master/sig-testing). See +that page for contact information. + +There are various tasks that could be worked on, including but not +limited to: + +* Moving test/e2e/framework into a staging repo and restructuring it + so that it is more modular + ([#74352](https://github.com/kubernetes/kubernetes/issues/74352)). +* Simplifying `e2e.go` by moving more of its code into + `test/e2e/framework` + ([#74353](https://github.com/kubernetes/kubernetes/issues/74353)). +* Removing provider-specific code from the Kubernetes E2E test suite + ([#70194](https://github.com/kubernetes/kubernetes/issues/70194)). + +Special thanks to the reviewers of this article: + +- Olev Kartau (https://github.com/okartau) +- Mary Camp (https://github.com/MCamp859)