Scrapyd: How to prioritize a scrapyd task?

Question

I have several scripting projects. I deploy them all with scrapyd. Some of the spiders are slow, while others are fast. Now I want to start the fast spiders first. How to do it?

Answer 1

I ran into the same problem. But I read the scrapyd source code, and it seems that scrapyd can handle this. However, this is not documented.

You need to add an argument named priority with the desired value. The default priority is 0.

For instance:

 curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d priority=1