diff --git a/content/en/docs/concepts/workloads/pods/ephemeral-containers.md b/content/en/docs/concepts/workloads/pods/ephemeral-containers.md index 492027bd00..c6506df69c 100644 --- a/content/en/docs/concepts/workloads/pods/ephemeral-containers.md +++ b/content/en/docs/concepts/workloads/pods/ephemeral-containers.md @@ -12,16 +12,15 @@ weight: 80 {{< feature-state state="alpha" for_k8s_version="v1.16" >}} This page provides an overview of ephemeral containers: a special type of container -that runs temporarily in an existing {{< glossary_tooltip term_id="pod" >}} to accomplish user-initiated actions such -as troubleshooting. You use ephemeral containers to inspect services rather than -to build applications. +that runs temporarily in an existing {{< glossary_tooltip term_id="pod" >}} to +accomplish user-initiated actions such as troubleshooting. You use ephemeral +containers to inspect services rather than to build applications. {{< warning >}} Ephemeral containers are in early alpha state and are not suitable for production -clusters. You should expect the feature not to work in some situations, such as -when targeting the namespaces of a container. In accordance with the [Kubernetes -Deprecation Policy](/docs/reference/using-api/deprecation-policy/), this alpha -feature could change significantly in the future or be removed entirely. +clusters. In accordance with the [Kubernetes Deprecation Policy]( +/docs/reference/using-api/deprecation-policy/), this alpha feature could change +significantly in the future or be removed entirely. {{< /warning >}} {{% /capture %}} @@ -78,7 +77,11 @@ When using ephemeral containers, it's helpful to enable [process namespace sharing](/docs/tasks/configure-pod-container/share-process-namespace/) so you can view processes in other containers. -### Examples +See [Debugging with Ephemeral Debug Container]( +/docs/tasks/debug-application-cluster/debug-running-pod/#debugging-with-ephemeral-debug-container) +for examples of troubleshooting using ephemeral containers. + +## Ephemeral containers API {{< note >}} The examples in this section require the `EphemeralContainers` [feature @@ -87,8 +90,9 @@ enabled, and Kubernetes client and server version v1.16 or later. {{< /note >}} The examples in this section demonstrate how ephemeral containers appear in -the API. You would normally use a `kubectl` plugin for troubleshooting that -automates these steps. +the API. You would normally use `kubectl alpha debug` or another `kubectl` +[plugin](/docs/tasks/extend-kubectl/kubectl-plugins/) to automate these steps +rather than invoking the API directly. Ephemeral containers are created using the `ephemeralcontainers` subresource of Pod, which can be demonstrated using `kubectl --raw`. First describe @@ -180,35 +184,12 @@ Ephemeral Containers: ... ``` -You can attach to the new ephemeral container using `kubectl attach`: +You can interact with the new ephemeral container in the same way as other +containers using `kubectl attach`, `kubectl exec`, and `kubectl logs`, for +example: ```shell kubectl attach -it example-pod -c debugger ``` -If process namespace sharing is enabled, you can see processes from all the containers in that Pod. -For example, after attaching, you run `ps` in the debugger container: - -```shell -# Run this in a shell inside the "debugger" ephemeral container -ps auxww -``` -The output is similar to: -``` -PID USER TIME COMMAND - 1 root 0:00 /pause - 6 root 0:00 nginx: master process nginx -g daemon off; - 11 101 0:00 nginx: worker process - 12 101 0:00 nginx: worker process - 13 101 0:00 nginx: worker process - 14 101 0:00 nginx: worker process - 15 101 0:00 nginx: worker process - 16 101 0:00 nginx: worker process - 17 101 0:00 nginx: worker process - 18 101 0:00 nginx: worker process - 19 root 0:00 /pause - 24 root 0:00 sh - 29 root 0:00 ps auxww -``` - {{% /capture %}} diff --git a/content/en/docs/tasks/debug-application-cluster/debug-application.md b/content/en/docs/tasks/debug-application-cluster/debug-application.md index 053af8b654..f63173e334 100644 --- a/content/en/docs/tasks/debug-application-cluster/debug-application.md +++ b/content/en/docs/tasks/debug-application-cluster/debug-application.md @@ -64,38 +64,8 @@ Again, the information from `kubectl describe ...` should be informative. The m #### My pod is crashing or otherwise unhealthy -First, take a look at the logs of -the current container: - -```shell -kubectl logs ${POD_NAME} ${CONTAINER_NAME} -``` - -If your container has previously crashed, you can access the previous container's crash log with: - -```shell -kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME} -``` - -Alternately, you can run commands inside that container with `exec`: - -```shell -kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN} -``` - -{{< note >}} -`-c ${CONTAINER_NAME}` is optional. You can omit it for Pods that only contain a single container. -{{< /note >}} - -As an example, to look at the logs from a running Cassandra pod, you might run - -```shell -kubectl exec cassandra -- cat /var/log/cassandra/system.log -``` - -If none of these approaches work, you can find the host machine that the pod is running on and SSH into that host, -but this should generally not be necessary given tools in the Kubernetes API. Therefore, if you find yourself needing to ssh into a machine, please file a -feature request on GitHub describing your use case and why these tools are insufficient. +Once your pod has been scheduled, the methods described in [Debug Running Pods]( +/docs/tasks/debug-application-cluster/debug-running-pods/) are available for debugging. #### My pod is running but not doing what I told it to do diff --git a/content/en/docs/tasks/debug-application-cluster/debug-pod-replication-controller.md b/content/en/docs/tasks/debug-application-cluster/debug-pod-replication-controller.md index 56ba566bc6..ec84b82fd0 100644 --- a/content/en/docs/tasks/debug-application-cluster/debug-pod-replication-controller.md +++ b/content/en/docs/tasks/debug-application-cluster/debug-pod-replication-controller.md @@ -93,40 +93,9 @@ worker node, but it can't run on that machine. Again, the information from ### My pod is crashing or otherwise unhealthy -First, take a look at the logs of the current container: +Once your pod has been scheduled, the methods described in [Debug Running Pods]( +/docs/tasks/debug-application-cluster/debug-running-pods/) are available for debugging. -```shell -kubectl logs ${POD_NAME} ${CONTAINER_NAME} -``` - -If your container has previously crashed, you can access the previous -container's crash log with: - -```shell -kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME} -``` - -Alternately, you can run commands inside that container with `exec`: - -```shell -kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN} -``` - -{{< note >}} -`-c ${CONTAINER_NAME}` is optional. You can omit it for pods that -only contain a single container. -{{< /note >}} - -As an example, to look at the logs from a running Cassandra pod, you might run: - -```shell -kubectl exec cassandra -- cat /var/log/cassandra/system.log -``` - -If your cluster enabled it, you can also try adding an [ephemeral container](/docs/concepts/workloads/pods/ephemeral-containers/) into the existing pod. You can use the new temporary container to run arbitrary commands, for example, to diagnose problems inside the Pod. See the page about [ephemeral container](/docs/concepts/workloads/pods/ephemeral-containers/) for more details, including feature availability. - -If none of these approaches work, you can find the host machine that the pod is -running on and SSH into that host. ## Debugging ReplicationControllers diff --git a/content/en/docs/tasks/debug-application-cluster/debug-running-pod.md b/content/en/docs/tasks/debug-application-cluster/debug-running-pod.md new file mode 100644 index 0000000000..95065ca595 --- /dev/null +++ b/content/en/docs/tasks/debug-application-cluster/debug-running-pod.md @@ -0,0 +1,190 @@ +--- +reviewers: +- verb +- soltysh +title: Debug Running Pods +content_template: templates/task +--- + +{{% capture overview %}} + +This page explains how to debug Pods running (or crashing) on a Node. + +{{% /capture %}} + +{{% capture prerequisites %}} + +* Your {{< glossary_tooltip text="Pod" term_id="pod" >}} should already be + scheduled and running. If your Pod is not yet running, start with [Troubleshoot + Applications](/docs/tasks/debug-application-cluster/debug-application/). +* For some of the advanced debugging steps you need to know on which Node the + Pod is running and have shell access to run commands on that Node. You don't + need that access to run the standard debug steps that use `kubectl`. + +{{% /capture %}} + +{{% capture steps %}} + +## Examining pod logs {#examine-pod-logs} + +First, look at the logs of the affected container: + +```shell +kubectl logs ${POD_NAME} ${CONTAINER_NAME} +``` + +If your container has previously crashed, you can access the previous container's crash log with: + +```shell +kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME} +``` + +## Debugging with container exec {#container-exec} + +If the {{< glossary_tooltip text="container image" term_id="image" >}} includes +debugging utilities, as is the case with images built from Linux and Windows OS +base images, you can run commands inside a specific container with +`kubectl exec`: + +```shell +kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN} +``` + +{{< note >}} +`-c ${CONTAINER_NAME}` is optional. You can omit it for Pods that only contain a single container. +{{< /note >}} + +As an example, to look at the logs from a running Cassandra pod, you might run + +```shell +kubectl exec cassandra -- cat /var/log/cassandra/system.log +``` + +You can run a shell that's connected to your terminal using the `-i` and `-t` +arguments to `kubectl exec`, for example: + +```shell +kubectl exec -it cassandra -- sh +``` + +For more details, see [Get a Shell to a Running Container]( +/docs/tasks/debug-application-cluster/get-shell-running-container/). + +## Debugging with an ephemeral debug container {#ephemeral-container} + +{{< feature-state state="alpha" for_k8s_version="v1.18" >}} + +{{< glossary_tooltip text="Ephemeral containers" term_id="ephemeral-container" >}} +are useful for interactive troubleshooting when `kubectl exec` is insufficient +because a container has crashed or a container image doesn't include debugging +utilities, such as with [distroless images]( +https://github.com/GoogleContainerTools/distroless). `kubectl` has an alpha +command that can create ephemeral containers for debugging beginning with version +`v1.18`. + +### Example debugging using ephemeral containers {#ephemeral-container-example} + +{{< note >}} +The examples in this section require the `EphemeralContainers` [feature gate]( +/docs/reference/command-line-tools-reference/feature-gates/) enabled in your +cluster and `kubectl` version v1.18 or later. +{{< /note >}} + +You can use the `kubectl alpha debug` command to add ephemeral containers to a +running Pod. First, create a pod for the example: + +```shell +kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never +``` + +{{< note >}} +This section use the `pause` container image in examples because it does not +contain userland debugging utilities, but this method works with all container +images. +{{< /note >}} + +If you attempt to use `kubectl exec` to create a shell you will see an error +because there is no shell in this container image. + +```shell +kubectl exec -it pause -- sh +``` + +``` +OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown +``` + +You can instead add a debugging container using `kubectl alpha debug`. If you +specify the `-i`/`--interactive` argument, `kubectl` will automatically attach +to the console of the Ephemeral Container. + +```shell +kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo +``` + +``` +Defaulting debug container name to debugger-8xzrl. +If you don't see a command prompt, try pressing enter. +/ # +``` + +This command adds a new busybox container and attaches to it. The `--target` +parameter targets the process namespace of another container. It's necessary +here because `kubectl run` does not enable [process namespace sharing]( +/docs/tasks/configure-pod-container/share-process-namespace/) in the pod it +creates. + +{{< note >}} +The `--target` parameter must be supported by the {{< glossary_tooltip +text="Container Runtime" term_id="container-runtime" >}}. When not supported, +the Ephemeral Container may not be started, or it may be started with an +isolated process namespace. +{{< /note >}} + +You can view the state of the newly created ephemeral container using `kubectl describe`: + +```shell +kubectl describe pod ephemeral-demo +``` + +``` +... +Ephemeral Containers: + debugger-8xzrl: + Container ID: docker://b888f9adfd15bd5739fefaa39e1df4dd3c617b9902082b1cfdc29c4028ffb2eb + Image: busybox + Image ID: docker-pullable://busybox@sha256:1828edd60c5efd34b2bf5dd3282ec0cc04d47b2ff9caa0b6d4f07a21d1c08084 + Port: + Host Port: + State: Running + Started: Wed, 12 Feb 2020 14:25:42 +0100 + Ready: False + Restart Count: 0 + Environment: + Mounts: +... +``` + +Use `kubectl delete` to remove the Pod when you're finished: + +```shell +kubectl delete pod ephemeral-demo +``` + + + +## Debugging via a shell on the node {#node-shell-session} + +If none of these approaches work, you can find the host machine that the pod is +running on and SSH into that host, but this should generally not be necessary +given tools in the Kubernetes API. Therefore, if you find yourself needing to +ssh into a machine, please file a feature request on GitHub describing your use +case and why these tools are insufficient. + +{{% /capture %}}