1.6. Troubleshooting

This lab helps you troubleshoot your application and shows you some tools to make troubleshooting easier.

Logging into a container

Running containers should be treated as immutable infrastructure and should therefore not be modified. However, there are some use cases in which you have to log into your running container. Debugging and analyzing is one example for this.

Task 1.6.1: Shell into Pod

With Kubernetes you can open a remote shell into a Pod without installing SSH by using the command kubectl exec. The command can also be used to execute any command in a Pod. If you want to get a shell to a running container, you will additionally need the parameters -it. These set up an interactive session where you can supply input to the process inside the container.

Choose a Pod with kubectl get pods --namespace <namespace> and execute the following command:

kubectl exec -it <pod> --namespace <namespace> -- /bin/bash

A neat trick is also that you can run the command on the first pod of a deployment, so you don’t need to lookup the pod name:

kubectl -n <namespace> exec -it deployments/example-frontend -- sh

You now have a running shell session inside the container in which you can execute every binary available, e.g.:

ls -l

total 12
-rw-r--r--    1 10020700 root          8192 Nov 27 15:12 hellos.db
-rwxrwsr-x    1 web      root          2454 Oct  5 08:55 run.py
drwxrwsr-x    1 web      root            17 Oct  5 08:55 static
drwxrwsr-x    1 web      root            63 Oct  5 08:55 templates

With exit or CTRL+d you can leave the container and close the connection:

exit

Task 1.6.2: Single commands

Single commands inside a container can also be executed with kubectl exec:

kubectl exec <pod> --namespace <namespace> -- env

Example:

$ kubectl exec example-frontend-69b658f647-xnm94 --namespace <namespace> -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=example-frontend-xnm94
KUBERNETES_SERVICE_PORT_DNS_TCP=53
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1
KUBERNETES_PORT_53_UDP_PROTO=udp
KUBERNETES_PORT_53_TCP=tcp://172.30.0.1:53
...

Watching log files

Log files of a Pod can be shown with the following command:

kubectl logs <pod> --namespace <namespace>

The parameter -f allows you to follow the log file (same as tail -f). With this, log files are streamed and new entries are shown immediately.

When a Pod is in state CrashLoopBackOff it means that although multiple attempts have been made, no container inside the Pod could be started successfully. Now even though no container might be running at the moment the kubectl logs command is executed, there is a way to view the logs the application might have generated. This is achieved using the -p or --previous parameter.

kubectl logs -p <pod> --namespace <namespace>

Task 1.6.3: Port forwarding

Kubernetes allows you to forward arbitrary ports to your development workstation. This allows you to access admin consoles, databases, etc., even when they are not exposed externally. Port forwarding is handled by the Kubernetes control plane nodes and therefore tunneled from the client via HTTPS. This allows you to access the Kubernetes platform even when there are restrictive firewalls or proxies between your workstation and Kubernetes.

Get the name of the Pod:

kubectl get pod --namespace <namespace>

Then execute the port forwarding command using the Pod’s name:

kubectl port-forward <pod> 5000:5000 --namespace <namespace>

Don’t forget to change the Pod name to your own installation. If configured, you can use auto-completion.

The output of the command should look like this:

Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000

The application is now available under the following url: http://localhost:5000/ , in the webshell try a curl command:

curl localhost:5000

Try to access our health probe at: http://localhost:5000/health. With port-forwarding it can be accessed now:

curl localhost:5000/health

With the same concept you can access databases from your local workstation or connect your local development environment via remote debugging to your application in the Pod.

This documentation page offers some more details about port forwarding.

Events

Kubernetes maintains an event log with high-level information on what’s going on in the cluster. It’s possible that everything looks okay at first but somehow something seems stuck. Make sure to have a look at the events because they can give you more information if something is not working as expected.

Use the following command to list the events in chronological order:

kubectl get events --sort-by=.metadata.creationTimestamp --namespace <namespace>

Dry-run

To help verify changes, you can use the optional kubectl flag --dry-run=client -o yaml to see the rendered YAML definition of your Kubernetes objects, without sending it to the API.

The following kubectl subcommands support this flag (non-final list):

  • apply
  • create
  • expose
  • patch
  • replace
  • run
  • set

For example, we can use the --dry-run=client flag to create a template for our Deployment:

kubectl create deployment example-frontend --image=quay.io/songlaa/example-web-python:latest --namespace songlaa-test --dry-run=client -o yaml

The result is the following YAML output:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: example-frontend
  name: example-frontend
  namespace: songlaa-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-frontend
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: example-frontend
    spec:
      containers:
        - image: quay.io/songlaa/example-web-python:latest
          name: example-frontend
          resources: {}
status: {}

kubectl API requests

If you want to see the HTTP requests kubectl sends to the Kubernetes API in detail, you can use the optional flag --v=10.

For example, to see the API request for creating a deployment:

kubectl create deployment test-deployment --image=quay.io/songlaa/example-web-python:latest --namespace <namespace> --replicas=0 --v=10

The resulting output looks like this:

I0404 14:05:12.906545     407 loader.go:395] Config loaded from file:  /home/theia/.kube/config
I0404 14:05:12.906733     407 merged_client_builder.go:121] Using in-cluster configuration
I0404 14:05:12.910729     407 request.go:1351] Request Body: {"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"test-deployment","namespace":"user2","creationTimestamp":null,"labels":{"app":"test-deployment"}},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"test-deployment"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"test-deployment"}},"spec":{"containers":[{"name":"example-web-python","image":"quay.io/songlaa/example-web-python:latest","resources":{}}]}},"strategy":{}},"status":{}}
I0404 14:05:12.910934     407 round_trippers.go:466] curl -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.31.2 (linux/amd64) kubernetes/5864a46" -H "Authorization: Bearer <masked>" 'https://10.43.0.1:443/apis/apps/v1/namespaces/user2/deployments?fieldManager=kubectl-create&fieldValidation=Strict'
I0404 14:05:12.912431     407 round_trippers.go:510] HTTP Trace: Dial to tcp:10.43.0.1:443 succeed
I0404 14:05:12.952159     407 round_trippers.go:553] POST https://10.43.0.1:443/apis/apps/v1/namespaces/user2/deployments?fieldManager=kubectl-create&fieldValidation=Strict 201 Created in 41 milliseconds
I0404 14:05:12.952210     407 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 1 ms TLSHandshake 2 ms ServerProcessing 36 ms Duration 41 ms
I0404 14:05:12.952229     407 round_trippers.go:577] Response Headers:
I0404 14:05:12.952247     407 round_trippers.go:580]     Content-Type: application/json
I0404 14:05:12.952254     407 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 4ad45830-5540-4832-b471-e4d5cceffc28
I0404 14:05:12.952263     407 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 44da248f-64ba-4706-88ef-f11bb767570e
I0404 14:05:12.952269     407 round_trippers.go:580]     Content-Length: 1744
I0404 14:05:12.952283     407 round_trippers.go:580]     Date: Fri, 04 Apr 2025 14:05:12 GMT
I0404 14:05:12.952288     407 round_trippers.go:580]     Audit-Id: 98b41ba6-0026-4436-bbb8-b26461aaa190
I0404 14:05:12.952294     407 round_trippers.go:580]     Cache-Control: no-cache, private
I0404 14:05:12.952336     407 request.go:1351] Response Body: {"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"test-deployment","namespace":"user2","uid":"f9c64003-4c0c-4108-bbe7-8938ce5f900e","resourceVersion":"35945","generation":1,"creationTimestamp":"2025-04-04T14:05:12Z","labels":{"app":"test-deployment"},"managedFields":[{"manager":"kubectl-create","operation":"Update","apiVersion":"apps/v1","time":"2025-04-04T14:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:progressDeadlineSeconds":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{},"f:strategy":{"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}},"f:type":{}},"f:template":{"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"example-web-python\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}}}}]},"spec":{"replicas":0,"selector":{"matchLabels":{"app":"test-deployment"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"test-deployment"}},"spec":{"containers":[{"name":"example-web-python","image":"quay.io/songlaa/example-web-python:latest","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","securityContext":{},"schedulerName":"default-scheduler"}},"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":"25%","maxSurge":"25%"}},"revisionHistoryLimit":10,"progressDeadlineSeconds":600},"status":{}}
deployment.apps/test-deployment created

As you can see, the output conveniently contains the corresponding curl commands which we could use in our own code, tools, pipelines etc.

You can delete it again as it’s not used anywhere else (which is also the reason why the replicas are set to 0):

kubectl delete deploy/test-deployment --namespace <namespace>

Kubernetes Dashboard

Until now you made everything using kubectl, so with the cli. But of course there are many GUI Frontends for Kubernetes out there. A popular one is the Kubernetes Dashboard , you can access it under https://kubedashboard.training.cluster.songlaa.com , for access please use the token provided by your teacher. Browse around in the dashboard and make sure to filter out your own namespace: <namespace>.

Previous
Next