Debug Running Pods
This page explains how to debug Pods running (or crashing) on a Node.
- Before you begin
- Examining pod logs
- Debugging with container exec
- Debugging with an ephemeral debug container
- Debugging via a shell on the node
Before you begin
- Your PodThe smallest and simplest Kubernetes object. A Pod represents a set of running containers on your cluster. should already be scheduled and running. If your Pod is not yet running, start with Troubleshoot Applications.
- For some of the advanced debugging steps you need to know on which Node the
Pod is running and have shell access to run commands on that Node. You don’t
need that access to run the standard debug steps that use
kubectl
.
Examining pod logs
First, look at the logs of the affected container:
kubectl logs ${POD_NAME} ${CONTAINER_NAME}
If your container has previously crashed, you can access the previous container’s crash log with:
kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
Debugging with container exec
If the container imageStored instance of a container that holds a set of software needed to run an application.
includes
debugging utilities, as is the case with images built from Linux and Windows OS
base images, you can run commands inside a specific container with
kubectl exec
:
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN}
Note:-c ${CONTAINER_NAME}
is optional. You can omit it for Pods that only contain a single container.
As an example, to look at the logs from a running Cassandra pod, you might run
kubectl exec cassandra -- cat /var/log/cassandra/system.log
You can run a shell that’s connected to your terminal using the -i
and -t
arguments to kubectl exec
, for example:
kubectl exec -it cassandra -- sh
For more details, see Get a Shell to a Running Container.
Debugging with an ephemeral debug container
Kubernetes v1.18
alpha
- The version names contain alpha (e.g. v1alpha1).
- Might be buggy. Enabling the feature may expose bugs. Disabled by default.
- Support for feature may be dropped at any time without notice.
- The API may change in incompatible ways in a later software release without notice.
- Recommended for use only in short-lived testing clusters, due to increased risk of bugs and lack of long-term support.
Ephemeral containersA type of container type that you can temporarily run inside a Pod
are useful for interactive troubleshooting when kubectl exec
is insufficient
because a container has crashed or a container image doesn’t include debugging
utilities, such as with distroless images. kubectl
has an alpha
command that can create ephemeral containers for debugging beginning with version
v1.18
.
Example debugging using ephemeral containers
Note: The examples in this section require theEphemeralContainers
feature gate enabled in your cluster andkubectl
version v1.18 or later.
You can use the kubectl alpha debug
command to add ephemeral containers to a
running Pod. First, create a pod for the example:
kubectl run ephemeral-demo --image=k8s.gcr.io/pause:3.1 --restart=Never
Note: This section use thepause
container image in examples because it does not contain userland debugging utilities, but this method works with all container images.
If you attempt to use kubectl exec
to create a shell you will see an error
because there is no shell in this container image.
kubectl exec -it pause -- sh
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
You can instead add a debugging container using kubectl alpha debug
. If you
specify the -i
/--interactive
argument, kubectl
will automatically attach
to the console of the Ephemeral Container.
kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
Defaulting debug container name to debugger-8xzrl.
If you don't see a command prompt, try pressing enter.
/ #
This command adds a new busybox container and attaches to it. The --target
parameter targets the process namespace of another container. It’s necessary
here because kubectl run
does not enable process namespace sharing in the pod it
creates.
Note: The--target
parameter must be supported by the Container RuntimeThe container runtime is the software that is responsible for running containers. . When not supported, the Ephemeral Container may not be started, or it may be started with an isolated process namespace.
You can view the state of the newly created ephemeral container using kubectl describe
:
kubectl describe pod ephemeral-demo
...
Ephemeral Containers:
debugger-8xzrl:
Container ID: docker://b888f9adfd15bd5739fefaa39e1df4dd3c617b9902082b1cfdc29c4028ffb2eb
Image: busybox
Image ID: docker-pullable://busybox@sha256:1828edd60c5efd34b2bf5dd3282ec0cc04d47b2ff9caa0b6d4f07a21d1c08084
Port: <none>
Host Port: <none>
State: Running
Started: Wed, 12 Feb 2020 14:25:42 +0100
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>
...
Use kubectl delete
to remove the Pod when you’re finished:
kubectl delete pod ephemeral-demo
Debugging via a shell on the node
If none of these approaches work, you can find the host machine that the pod is running on and SSH into that host, but this should generally not be necessary given tools in the Kubernetes API. Therefore, if you find yourself needing to ssh into a machine, please file a feature request on GitHub describing your use case and why these tools are insufficient.
Feedback
Was this page helpful?
Thanks for the feedback. If you have a specific, answerable question about how to use Kubernetes, ask it on Stack Overflow. Open an issue in the GitHub repo if you want to report a problem or suggest an improvement.