Change the number of scrapyd spiders launched

Question

Hey, so I have about 50 spiders in my project, and I'm currently launching them through a scrapyd server. I run into a problem when some of the resources that I use are blocked and cause my spiders to fail or go very slowly. I was hoping that they could somehow be scrapyd, to have only one spider spider at a time and leave the rest in the waiting queue. I did not see the configuration parameter in the documentation. Any help would be greatly appreciated!

Answer 1

This can be controlled using scrapyd settings . Set max_proc to 1 :

max_proc

The maximum number of concurrent Scrapy processes to be launched.