Debug container / pods

• Debug Running Pods
• kubectl debug
• Video: How the heck do I debug distroless containers?

Run a shell in distroless container, i.e. headscale:

kubectl -n headscale debug -it --image=alpine --target=headscale headscale-6f4b745765-tpfkc -- sh

Custom profile

• Applying Custom Profile
• Available in Kubernetes v1.32

Custom profile only supports the modification of the container spec, but modifications to name, image, command, lifecycle and volumeDevices fields of the container spec are not allowed. It does not support the modification of the Pod spec.

Debug pod and add volume mounts

• Debugging Kubernetes PVCs

Because custom profiles can only modify the container spec, a dedicated debug pod file spec is needed:

$ cat headscale-debug-pod.yaml
kind: Pod
apiVersion: v1
metadata:
  name: volume-debugger
  namespace: headscale
spec:
  volumes:
    - name: config
      persistentVolumeClaim:
        claimName: headscale
  containers:
    - name: debugger
      # You need to specify an image which has the `sleep` cmd included
      image: alpine
      command: ["sleep", "3600"]
      volumeMounts:
        - mountPath: /etc/headscale
          name: config

Start the debug pod and exec into it:

$ kubectl exec -it volume-debugger sh
$ kubectl -n headscale exec -it volume-debugger -- sh
  ls /etc/headscale/

Downsides:

• You still don't have access to the headscale binary