How to get values ​​from conditional dropdown fields using Scrapy / Python?

Question

This tutorial helped me put together the first drop-down lists on a website. But I can not get the data of the second conditional / dependent drop-down list using Scrapy.

Assuming the following procedure is the way, how does step 2 work? Values ​​never appear until now ...

• Get the list of entries in the first drop-down list
• Select the first value of the first drop-down list.
• get all dependent outliers.
• iteration in the first dropdown ...

I think the code below needs to be adjusted, as the code for the submit form:

for i in range(len(brandCategoryList)): # Generate new request for each brand category page yield FormRequest("http://www.xxxxxxxxxx.com", method='POST', formdata={'BrandName':'','CatBrand':brandCategoryList[i],'submit1':'Find+Product'}, callback=self.parseBrandPage, meta={'brandCategoryId':i,'brandCategoryName':brandCategoryList[i]}) 

thanks

Answer 1

Scrapy will not run any javascript code on the page you are clearing, and dependent drop-down lists often rely on javascript to populate their options based on choices in the drop-down control list.

If you need to automate / clean up content that requires javascript, you may need to consider another library. Start here for some pointers.