diff --git a/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/index.md b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/index.md new file mode 100644 index 0000000000..4f41ba920c --- /dev/null +++ b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/index.md @@ -0,0 +1,190 @@ +--- +layout: blog +title: 'SIG Node CI Subproject Celebrates Two Years of Test Improvements' +date: 2022-02-16 +--- + +**Authors:** Sergey Kanzhelev (Google), Elana Hashman (Red Hat) + +Ensuring the reliability of SIG Node upstream code is a continuous effort +that takes a lot of behind-the-scenes effort from many contributors. +There are frequent releases of Kubernetes, base operating systems, +container runtimes, and test infrastructure that result in a complex matrix that +requires attention and steady investment to "keep the lights on." +In May 2020, the Kubernetes node special interest group ("SIG Node") organized a new +subproject for continuous integration (CI) for node-related code and tests. Since its +inauguration, the SIG Node CI subproject has run a weekly meeting, and even the full hour +is often not enough to complete triage of all bugs, test-related PRs and issues, and discuss all +related ongoing work within the subgroup. + +Over the past two years, we've fixed merge-blocking and release-blocking tests, reducing time to merge Kubernetes contributors' pull requests thanks to reduced test flakes. When we started, Node test jobs only passed 42% of the time, and through our efforts, we now ensure a consistent >90% job pass rate. We've closed 144 test failure issues and merged 176 pull requests just in kubernetes/kubernetes. And we've helped subproject participants ascend the Kubernetes contributor ladder, with 3 new org members, 6 new reviewers, and 2 new approvers. + +The Node CI subproject is an approachable first stop to help new contributors +get started with SIG Node. There is a low barrier to entry for new contributors +to address high-impact bugs and test fixes, although there is a long +road before contributors can climb the entire contributor ladder: +it took over a year to establish two new approvers for the group. +The complexity of all the different components that power Kubernetes nodes +and its test infrastructure requires a sustained investment over a long period +for developers to deeply understand the entire system, +both at high and low levels of detail. + +We have several regular contributors at our meetings, however; our reviewers +and approvers pool is still small. It is our goal to continue to grow +contributors to ensure a sustainable distribution of work +that does not just fall to a few key approvers. + +It's not always obvious how subprojects within SIGs are formed, operate, +and work. Each is unique to its sponsoring SIG and tailored to the projects +that the group is intended to support. As a group that has welcomed many +first-time SIG Node contributors, we'd like to share some of the details and +accomplishments over the past two years, +helping to demystify our inner workings and celebrate the hard work +of all our dedicated contributors! + +## Timeline + +***May 2020.*** SIG Node CI group was formed on May 11, 2020, with more than +[30 volunteers](https://docs.google.com/document/d/1fb-ugvgdSVIkkuJ388_nhp2pBTy_4HEVg5848Xy7n5U/edit#bookmark=id.vsb8pqnf4gib) +signed up, to improve SIG Node CI signal and overall observability. +Victor Pickard focused on getting +[testgrid jobs](https://testgrid.k8s.io/sig-node) passing +when Ning Liao suggested forming a group around this effort and came up with +the [original group charter document](https://docs.google.com/document/d/1yS-XoUl6GjZdjrwxInEZVHhxxLXlTIX2CeWOARmD8tY/edit#heading=h.te6sgum6s8uf). +The SIG Node chairs sponsored group creation with Victor as a subproject lead. +Sergey Kanzhelev joined Victor shortly after as a co-lead. + +At the kick-off meeting, we discussed which tests to concentrate on fixing first +and discussed merge-blocking and release-blocking tests, many of which were failing due +to infrastructure issues or buggy test code. + +The subproject launched weekly hour-long meetings to discuss ongoing work +discussion and triage. + +***June 2020.*** Morgan Bauer, Karan Goel, and Jorge Alarcon Ochoa were +recognized as reviewers for the SIG Node CI group for their contributions, +helping significantly with the early stages of the subproject. +David Porter and Roy Yang also joined the SIG test failures GitHub team. + +***August 2020.*** All merge-blocking and release-blocking tests were passing, +with some flakes. However, only 42% of all SIG Node test jobs were green, as there +were many flakes and failing tests. + +***October 2020.*** Amim Knabben becomes a Kubernetes org member for his +contributions to the subproject. + +***January 2021.*** With healthy presubmit and critical periodic jobs passing, +the subproject discussed its goal for cleaning up the rest of periodic tests +and ensuring they passed without flakes. + +Elana Hashman joined the subproject, stepping up to help lead it after +Victor's departure. + +***February 2021.*** Artyom Lukianov becomes a Kubernetes org member for his +contributions to the subproject. + +***August 2021.*** After SIG Node successfully ran a [bug scrub](https://groups.google.com/g/kubernetes-dev/c/w2ghO4ihje0/m/VeEql1LJBAAJ) +to clean up its bug backlog, the scope of the meeting was extended to +include bug triage to increase overall reliability, anticipating issues +before they affect the CI signal. + +Subproject leads Elana Hashman and Sergey Kanzhelev are both recognized as +approvers on all node test code, supported by SIG Node and SIG Testing. + +***September 2021.*** After significant deflaking progress with serial tests in +the 1.22 release spearheaded by Francesco Romani, the subproject set a goal +for getting the serial job fully passing by the 1.23 release date. + +Mike Miranda becomes a Kubernetes org member for his contributions +to the subproject. + +***November 2021.*** Throughout 2021, SIG Node had no merge or +release-blocking test failures. Many flaky tests from past releases are removed +from release-blocking dashboards as they had been fully cleaned up. + +Danielle Lancashire was recognized as a reviewer for SIG Node's subgroup, test code. + +The final node serial tests were completely fixed. The serial tests consist of +many disruptive and slow tests which tend to be flakey and are hard +to troubleshoot. By the 1.23 release freeze, the last serial tests were +fixed and the job was passing without flakes. + +[![Slack announcement that Serial tests are green](serial-tests-green.png)](https://kubernetes.slack.com/archives/C0BP8PW9G/p1638211041322900) + +The 1.23 release got a special shout out for the tests quality and CI signal. +The SIG Node CI subproject was proud to have helped contribute to such +a high-quality release, in part due to our efforts in identifying +and fixing flakes in Node and beyond. + +[![Slack shoutout that release was mostly green](release-mostly-green.png)](https://kubernetes.slack.com/archives/C92G08FGD/p1637175755023200) + +***December 2021.*** An estimated 90% of test jobs were passing at the time of +the 1.23 release (up from 42% in August 2020). + +Dockershim code was removed from Kubernetes. This affected nearly half of SIG Node's +test jobs and the SIG Node CI subproject reacted quickly and retargeted all the +tests. SIG Node was the first SIG to complete test migrations off dockershim, +providing examples for other affected SIGs. The vast majority of new jobs passed +at the time of introduction without further fixes required. The [effort of +removing dockershim](https://k8s.io/dockershim)) from Kubernetes is ongoing. +There are still some wrinkles from the dockershim removal as we uncover more +dependencies on dockershim, but we plan to stabilize all test jobs +by the 1.24 release. + +## Statistics + +Our regular meeting attendees and subproject participants for the past few months: + +- Aditi Sharma +- Artyom Lukianov +- Arnaud Meukam +- Danielle Lancashire +- David Porter +- Davanum Srinivas +- Elana Hashman +- Francesco Romani +- Matthias Bertschy +- Mike Miranda +- Paco Xu +- Peter Hunt +- Ruiwen Zhao +- Ryan Phillips +- Sergey Kanzhelev +- Skyler Clark +- Swati Sehgal +- Wenjun Wu + +The [kubernetes/test-infra](https://github.com/kubernetes/test-infra/) source code repository contains test definitions. The number of +Node PRs just in that repository: +- 2020 PRs (since May): [183](https://github.com/kubernetes/test-infra/pulls?q=is%3Apr+is%3Aclosed+label%3Asig%2Fnode+created%3A2020-05-01..2020-12-31+-author%3Ak8s-infra-ci-robot+) +- 2021 PRs: [264](https://github.com/kubernetes/test-infra/pulls?q=is%3Apr+is%3Aclosed+label%3Asig%2Fnode+created%3A2021-01-01..2021-12-31+-author%3Ak8s-infra-ci-robot+) + +Triaged issues and PRs on CI board (including triaging away from the subgroup scope): + +- 2020 (since May): [132](https://github.com/issues?q=project%3Akubernetes%2F43+created%3A2020-05-01..2020-12-31) +- 2021: [532](https://github.com/issues?q=project%3Akubernetes%2F43+created%3A2021-01-01..2021-12-31+) + +## Future + +Just "keeping the lights on" is a bold task and we are committed to improving this experience. +We are working to simplify the triage and review processes for SIG Node. + +Specifically, we are working on better test organization, naming, +and tracking: + +- https://github.com/kubernetes/enhancements/pull/3042 +- https://github.com/kubernetes/test-infra/issues/24641 +- [Kubernetes SIG-Node CI Testgrid Tracker](https://docs.google.com/spreadsheets/d/1IwONkeXSc2SG_EQMYGRSkfiSWNk8yWLpVhPm-LOTbGM/edit#gid=0) + +We are also constantly making progress on improved tests debuggability and de-flaking. + +If any of this interests you, we'd love for you to join us! +There's plenty to learn in debugging test failures, and it will help you gain +familiarity with the code that SIG Node maintains. + +You can always find information about the group on the +[SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) page. +We give group updates at our maintainer track sessions, such as +[KubeCon + CloudNativeCon Europe 2021](https://kccnceu2021.sched.com/event/iE8E/kubernetes-sig-node-intro-and-deep-dive-elana-hashman-red-hat-sergey-kanzhelev-google) and +[KubeCon + CloudNative North America 2021](https://kccncna2021.sched.com/event/lV9D/kubenetes-sig-node-intro-and-deep-dive-elana-hashman-derek-carr-red-hat-sergey-kanzhelev-dawn-chen-google?iframe=no&w=100%&sidebar=yes&bg=no). +Join us in our mission to keep the kubelet and other SIG Node components reliable and ensure smooth and uneventful releases! diff --git a/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/release-mostly-green.png b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/release-mostly-green.png new file mode 100644 index 0000000000..c88eb3b37e Binary files /dev/null and b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/release-mostly-green.png differ diff --git a/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/serial-tests-green.png b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/serial-tests-green.png new file mode 100644 index 0000000000..4ea64fedb7 Binary files /dev/null and b/content/en/blog/_posts/2022-02-16-sig-node-ci-subproject-celebrates/serial-tests-green.png differ