Python Scrapy - a straightforward spider for a particular pipeline

Question

I have a Scrapy project with several spiders along with several pipelines. Is there a way I can tell Spider A to use pipeline A, etc.

My pipelines.py has several pipeline classes, each of which does something different, and I want it to tell the spider to use a specific pipeline.

I see no obvious ways to look at the available scrapy commands for this ...

Answer 1

ITEM_PIPELINES setting is defined globally for all spiders in the project during engine startup. It cannot be changed to a spider on the fly.

Here is what you can do. Determine which spiders should be piped in the piping itself. Skip / continue processing elements returned by spiders in the process_item method of your pipeline, for example:

 def process_item(self, item, spider): if spider.name not in ['spider1', 'spider2']: return item # process item 

See also:

• Is there any method to use a separate pipeline for each spider?

Hope this helps.

Answer 2

You can specify the pipeline to use in the custom_settings property of your spider class:

 class BookSpider(BaseSpider): name = "book_spider" custom_settings = { 'ITEM_PIPELINES': { 'my_app.pipelines.BookPipeline': 300, } } def parse(self, response): return