How to avoid a docker crusher with too many containers?

Question

My history

I once called Docker from the larvel PHP queue to process thousands of media files. My code will create a new container for each task that I wanted to perform (for example, "process", "search", "slice", etc.). Then I went to bed, and disaster struck. I woke up to a 1 TB log file and several hundred thousand containers in different states.

Docker no longer works. As in the case when I print docker version (or docker anything ), it just sits and looks. I would like to avoid reinstalling, but I cannot remove the containers using the standard docker rm $(docker ps -a -q) , because Docker pressed very hard.

My questions

• How to remove docker containers if docker daemon is not responding?

• My code uses Docker as an illustrious executable using the docker API. What additional steps do I need to take to clear my code after myself in the future?

• In general, I noticed that when I try to deploy too many (i.e. 20) containers at the same time, Docker was prone to timeout. I would really like to be able to do this.

Some information

This is how I call the demon right now:

 $> sudo docker daemon INFO[0000] API listen on /var/run/docker.sock INFO[0000] [graphdriver] using prior storage driver "aufs" (Meanwhile in another terminal) $> docker version (insert infinite darkness here) 

Answer 1

I dug up something.

How to remove docker containers if docker daemon is not responding?

On Ubuntu, containers are stored in the /var/lib/docker/containers directory. Delete the contents of this directory and everything will be much better. This will live elsewhere depending on your system and installation configuration.

In addition, for someone curious there were about 250 thousand containers that needed to be removed. Given the number of files I needed to make with INCREDIBLE DANGEROUS: ls /var/lib/docker/containers | xargs -n200 rm -rf ls /var/lib/docker/containers | xargs -n200 rm -rf <- NOT THE TYPE THAT IS IN YOUR TEAM LINE IF YOU DO NOT KNOW WHAT YOU DO.

I use Docker as an illustrious executable called from code using their API. What additional steps do I need for my code to clear after myself in the future?

If the container will no longer be used, you must officially “remove” it after completion. On the command line, this will be done using docker rm $(containername) , and in the code it will completely depend on access to the docker. I heard you can also launch docker using docker run --rm , which will be cleared after closing.

I found that even after trying to uninstall programmatically, there will be times when the containers do not close. To prevent cron from being configured regularly docker ps -a | awk '{print $1}' | xargs docker rm docker ps -a | awk '{print $1}' | xargs docker rm

I noticed that when I try to deploy too many (i.e. 20) containers at a time, Docker was prone to timeout.

This stopped as soon as I cleared everything, but I still think I should use something more than Docker Swarm for this use case. In my case, I decided to just set up the production environment to remove the need for dockers there (where she used multiple threads).