--- layout: blog title: "Kubernetes 1.24: gRPC container probes in beta" date: 2022-05-13 slug: grpc-probes-now-in-beta --- **Author**: Sergey Kanzhelev (Google) With Kubernetes 1.24 the gRPC probes functionality entered beta and is available by default. Now you can configure startup, liveness, and readiness probes for your gRPC app without exposing any HTTP endpoint, nor do you need an executable. Kubernetes can natively connect to your your workload via gRPC and query its status. ## Some history It's useful to let the system managing your workload check that the app is healthy, has started OK, and whether the app considers itself good to accept traffic. Before the gRPC support was added, Kubernetes already allowed you to check for health based on running an executable from inside the container image, by making an HTTP request, or by checking whether a TCP connection succeeded. For most apps, those checks are enough. If your app provides a gRPC endpoint for a health (or readiness) check, it is easy to repurpose the `exec` probe to use it for gRPC health checking. In the blog article [Health checking gRPC servers on Kubernetes](/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/), Ahmet Alp Balkan described how you can do that — a mechanism that still works today. There is a commonly used tool to enable this that was [created](https://github.com/grpc-ecosystem/grpc-health-probe/commit/2df4478982e95c9a57d5fe3f555667f4365c025d) on August 21, 2018, and with the first release at [Sep 19, 2018](https://github.com/grpc-ecosystem/grpc-health-probe/releases/tag/v0.1.0-alpha.1). This approach for gRPC apps health checking is very popular. There are [3,626 Dockerfiles](https://github.com/search?l=Dockerfile&q=grpc_health_probe&type=code) with the `grpc_health_probe` and [6,621 yaml](https://github.com/search?l=YAML&q=grpc_health_probe&type=Code) files that are discovered with the basic search on GitHub (at the moment of writing). This is good indication of the tool popularity and the need to support this natively. Kubernetes v1.23 introduced an alpha-quality implementation of native support for querying a workload status using gRPC. Because it was an alpha feature, this was disabled by default for the v1.23 release. ## Using the feature We built gRPC health checking in similar way with other probes and believe it will be [easy to use](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) if you are familiar with other probe types in Kubernetes. The natively supported health probe has many benefits over the workaround involving `grpc_health_probe` executable. With the native gRPC support you don't need to download and carry `10MB` of an additional executable with your image. Exec probes are generally slower than a gRPC call as they require instantiating a new process to run an executable. It also makes the checks less sensible for edge cases when the pod is running at maximum resources and has troubles instantiating new processes. There are a few limitations though. Since configuring a client certificate for probes is hard, services that require client authentication are not supported. The built-in probes are also not checking the server certificates and ignore related problems. Built-in checks also cannot be configured to ignore certain types of errors (`grpc_health_probe` returns different exit codes for different errors), and cannot be "chained" to run the health check on multiple services in a single probe. But all these limitations are quite standard for gRPC and there are easy workarounds for those. ## Try it for yourself ### Cluster-level setup You can try this feature today. To try native gRPC probes, you can spin up a Kubernetes cluster yourself with the `GRPCContainerProbe` feature gate enabled, there are many [tools available](/docs/tasks/tools/). Since the feature gate `GRPCContainerProbe` is enabled by default in 1.24, many vendors will have this functionality working out of the box. So you may just create an 1.24 cluster on platform of your choice. Some vendors allow to enable alpha features on 1.23 clusters. For example, at the moment of writing, you can spin up the test cluster on GKE for a quick test. Other vendors may also have similar capabilities, especially if you are reading this blog post long after the Kubernetes 1.24 release. On GKE use the following command (note, version is `1.23` and `enable-kubernetes-alpha` are specified). ```shell gcloud container clusters create test-grpc \ --enable-kubernetes-alpha \ --no-enable-autorepair \ --no-enable-autoupgrade \ --release-channel=rapid \ --cluster-version=1.23 ``` You will also need to configure `kubectl` to access the cluster: ```shell gcloud container clusters get-credentials test-grpc ``` ### Trying the feature out Let's create the pod to test how gRPC probes work. For this test we will use the `agnhost` image. This is a k8s maintained image with that can be used for all sorts of workload testing. For example, it has a useful [grpc-health-checking](https://github.com/kubernetes/kubernetes/blob/b2c5bd2a278288b5ef19e25bf7413ecb872577a4/test/images/agnhost/README.md#grpc-health-checking) module that exposes two ports - one is serving health checking service, another - http port to react on commands `make-serving` and `make-not-serving`. Here is an example pod definition. It starts the `grpc-health-checking` module, exposes ports `5000` and `8080`, and configures gRPC readiness probe: ``` yaml --- apiVersion: v1 kind: Pod metadata: name: test-grpc spec: containers: - name: agnhost image: k8s.gcr.io/e2e-test-images/agnhost:2.35 command: ["/agnhost", "grpc-health-checking"] ports: - containerPort: 5000 - containerPort: 8080 readinessProbe: grpc: port: 5000 ``` If the file called `test.yaml`, you can create the pod and check it's status. The pod will be in ready state as indicated by the snippet of the output. ```shell kubectl apply -f test.yaml kubectl describe test-grpc ``` The output will contain something like this: ``` Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True ``` Now let's change the health checking endpoint status to NOT_SERVING. In order to call the http port of the Pod, let's create a port forward: ```shell kubectl port-forward test-grpc 8080:8080 ``` You can `curl` to call the command... ```shell curl http://localhost:8080/make-not-serving ``` ... and in a few seconds the port status will switch to not ready. ```shell kubectl describe pod test-grpc ``` The output now will have: ``` Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True ... Warning Unhealthy 2s (x6 over 42s) kubelet Readiness probe failed: service unhealthy (responded with "NOT_SERVING") ``` Once it is switched back, in about one second the Pod will get back to ready status: ``` bsh curl http://localhost:8080/make-serving kubectl describe test-grpc ``` The output indicates that the Pod went back to being `Ready`: ``` Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True ``` This new built-in gRPC health probing on Kubernetes makes implementing a health-check via gRPC much easier than the older approach that relied on using a separate `exec` probe. Read through the official [documentation](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) to learn more and provide feedback before the feature will be promoted to GA. ## Summary Kubernetes is a popular workload orchestration platform and we add features based on feedback and demand. Features like gRPC probes support is a minor improvement that will make life of many app developers easier and apps more resilient. Try it today and give feedback, before the feature went into GA.