Scrapy + Eclipse PyDev: how to configure the debugger?

Question

I have successfully installed Eclipse with my Scrapy project.

I did this by installing a new Run / Debug configuration:

• What is the main module associated with Scrapy / usr / local / bin / scrapy for me (I found a suggestion to use cmdline.py, but this failed on my computer (OSX Lion and scrapy installed via easy_install).
• Defining arguments to send a "ny traversal" in my case, as if I were using the Scrapy command line
• Setting the correct working directory ($ {workspace_loc: My project / src} in my case)

Eclipse can successfully run my project, but I don't have debbuger. I miss my checkpoints and variable checks, does anyone know how to configure debbugger with this environment?

Answer 1

• Save the entire scrapy project folder in PyDev.
• You need to install the main module on scrapy / cmdline.py
• Set crawl ny argument in your case

Answer 2

None of the above suggestions helped me. Everything will work, but there will be no breakpoints.

I added the main.py file to my local project and connected to the scrapy command line as follows:

 import scrapy.cmdline def main(): scrapy.cmdline.execute(argv=['scrapy', 'crawl', 'wiki']) if __name__ =='__main__': main() 

This can be easily improved to pass the name of the spider to have different debug configurations per spider.

Answer 3

Well, if you run it correctly from Eclipse, is it not easy to run it in debug mode?

That is: if you followed the instructions: http://pydev.org/manual_101_run.html (in relation to the option "Always run a previously launched application"), after starting your module in normal mode, just press F11 to start it in debug mode.

Answer 4

Here's how I solved it for my environment, with the Scrapy package in a virtual environment outside the project directory:

1) Create a simlink for your cmdline.py from within the project, for example:

 ln -s ../venv/lib/python3.4/site-packages/scrapy/cmdline.py cmdline.py 

2) Update the project so that it can be seen, and then set it as the main module in the debug configuration.

3) On the “Arguments” tab in the debugging configuration, where “Working directory” is indicated, select “Other”, and then enter or switch to using the “Scrapy” buttons in your project.