All geek questions in one place

ScrapyJS installation - new for python

I am trying to use this scrapy addon (or what it is): scrapyjs .

However, there are no installation instructions, and I'm new to Python. Is there anything basic that I am missing? How would I integrate this with the scrapy project.

Note. I would prefer to use the Scrapy boot handler rather than the middleware version, as it seems to run faster. (correct me if I am wrong).

+4
source share
2 answers

Since it is scrapyjsnot a regular python package and is not registered on PyPI- first you need to clone the repository and move the package scrapyjsunder PYTHONPATHor in the project project directory (make it "imported").

There are two options for integrating with Scrapy:

The latter is much simpler and cleaner, but will seriously affect performance, since each request will be processed in blocking mode.

+1
source

To add alecxe to the answer, for Ubuntu / Debian systems, first install the dependencies (webkit, gtk2 and jswebkit)

sudo apt-get install python-jswebkit libwebkitgtk-1.0-0 python-webkit
sudo apt-get install python-gtk2 python-gnome2 python-glade2 python-gobject

virtualenv,

mkdir your-venv/lib/python2.7/dist-packages
ln -s /usr/lib/python2.7/dist-packages/gtk-2.0* lib/python2.7/dist-packages/
ln -s /usr/lib/python2.7/dist-packages/pygtk.pth lib/python2.7/dist-packages/
ln -s /usr/lib/python2.7/dist-packages/gobject/ lib/python2.7/dist-packages/
ln -s /usr/lib/python2.7/dist-packages/glib/ lib/python2.7/dist-packages/
ln -s /usr/lib/python2.7/dist-packages/cairo lib/python2.7/dist-packages/
ln -s /usr/lib/python2.7/dist-packages/webkit lib/python2.7/dist-packages/
ln -s /usr/lib/python2.7/dist-packages/jswebkit.so lib/python2.7/dist-packages/

, , ( )

python -c "import scrapy; print scrapy.__file__"

- Scrapy __init__.py. __init__.py

from twisted.internet import gtk2reactor
gtk2reactor.install()
+1

Source: https://habr.com/ru/post/1569677/

More articles:


All Articles