What is the best way to organize Django + Scrapy? My goal is to use Django to create screening tasks and have a Django database populating scrapy.
I previously created a scrapy and django project that sat in the root directory, but I had a lot of PATH and os.environ problems where I had scripts to configure PATH to make everything work. I want to avoid this.
I saw two solutions:
- Launch the Django project and create a scrapy subproject using the SCRAPY_SETTINGS_MODULE environment (see "Answer to this question in the section " Accessing django models inside Scrapy ").
- Run the Scrapy project, use DjangoItem and use DJANGO_SETTINGS_MODULE environment
What are the advantages and disadvantages of each solution?
source share