Django and Scrapy hierarchy pros and cons

What is the best way to organize Django + Scrapy? My goal is to use Django to create screening tasks and have a Django database populating scrapy.

I previously created a scrapy and django project that sat in the root directory, but I had a lot of PATH and os.environ problems where I had scripts to configure PATH to make everything work. I want to avoid this.

I saw two solutions:

  • Launch the Django project and create a scrapy subproject using the SCRAPY_SETTINGS_MODULE environment (see "Answer to this question in the section " Accessing django models inside Scrapy ").
  • Run the Scrapy project, use DjangoItem and use DJANGO_SETTINGS_MODULE environment

What are the advantages and disadvantages of each solution?

+6
source share

Source: https://habr.com/ru/post/908684/


All Articles