Why is my DataDog instance reporting Kubernetes no_pod error?

Question

We run the Kubernetes cluster in AWS, and we collect metrics in the DataDog using the dd-agent DaemonSet.

We have a Pod displayed in our metrics marked as "no_pod" and it uses a lot of resources, Memory / CPU / NetworkTx / NetworkRX.

Is there any explanation for what a pod is, how can I find it, kill it, restart it, etc.

I found the dd-agent source code , which seems to define the label "no_pod", but I can’t understand why it is, where it comes from and how I can find it through kubectl, etc.

Answer 1

After talking with the support group in DataDog, I was able to find out the following information regarding the fact that there were no_pod swaps.

Our Kubernetes check gets a list of containers from the Kubernetes API, which provides aggregated data. In the metric explorer configuration here, you can see a couple of containers named / docker and / that come together with other containers. Metrics called pod_name: no_pod, which come from container_name: / and container_name: / docker, are just metrics aggregated in several containers. (Thus, it makes sense that these are the highest values ​​in your charts.) If you do not want your charts to display these aggregated container metrics, you can clone the dashboard and then exclude these items from the query. To do this, on the cloned toolbar, simply edit the request in the JSON tab and add in the tag area! Pod_name: no_pod.

Thus, it seems that these containers are docker and root level containers that work outside the cluster and will always be displayed if you do not want to filter them specifically, which I am doing now.

Many thanks to the DataDog support guys for understanding this issue and giving me an excellent explanation as to what pods are and basically confirming that I can just filter them out and not worry about them.