How to get Pods history on Kubernetes Node?

Question

In our Kuberenetes cluster, we encounter sporadic situations where the cluster node runs out of memory and Linux calls the OOM killer. Looking at the logs, it seems that tabs planned on node require more memory than Node can allocate.

The problem is that when OOM killer is called, it prints a list of processes and their use in memory. However, since all of our Docker containers are Java services, the "process name" simply displays as "java", preventing us from tracking which specific Pod is causing the problems.

How can I get a history of what Pods were planned to run on a particular node and when?

Answer 1

We use Prometheus to monitor OOM events.

This expression should indicate the number of times that memory usage has reached its limits:

rate(container_memory_failcnt{pod_name!=""}[5m]) > 0

FYI: this is the best thing for the right documents, code

Answer 2

I think your modules have no queries or restrictions , or the values ​​are not perfect.

If you configure this correctly when the pod starts using too much bar, this module will be killed and you will be able to find out what causes the problems.

About viewing all containers on node you can go from kubectl get eventsor docker ps -ato node as indicated in other answers / comments.

Answer 3

- docker ps -a .