Scrapy: catch responses using specific HTTP server codes

Question

We have a fairly standard Scrapy project (Scrapy 0.24).

I would like to catch certain HTTP response codes such as 200, 500, 502, 503, 504, etc.

Something like that:

class Spider(...): def parse(...): processes HTTP 200 def parse_500(...): processes HTTP 500 errors def parse_502(...): processes HTTP 502 errors ... 

How can we do this?

Answer 1

By default, Scrapy only processes responses with status codes 300 .

Let Scrapy process 500 and 502 :

 class Spider(...): handle_httpstatus_list = [500, 502] 

Then in the parse() callback, set the response.status flag:

 def parse(response): if response.status == 500: # logic here elif response.status == 502: # logic here