GUI for debugging phantomjs script

Question

I use phantomjs to create a web crawler. My current crawler is a Python script using selenium, which requires the Firefox browser. Although Selenium is good for debugging scripts (since I can use firebug to validate a web page), it cannot be deployed to Linux servers without a GUI. So I am trying to translate my Python script to phantomjs.

To debug phantomjs scripts, I save the original HTML file and draw a png screenshot to disk and open it in Firefox to check the source page. I am wondering if there is a better way to do this, for example. plugin for phantomjs etc.

Answer 1

This little hack uses a simple technique: it captures the screen when PhantomJS or CasperJS sees it with captureBase64 ('png'), and then it sends the image to the receiving server, which then sends it via socket.io to the browser, which displays it as a built-in picture.

Source code is available on github:

https://github.com/maciejjankowski/flaming-octo-puss

I'm not sure if PhantomJS has this, but CasperJS does (and the latter adds some additional features to PhantomJS)

and use remote debugging as others suggest

Answer 2

As with most JS scripts, it is not that difficult to debug a phantomjs script, because there is no IDE / compiler behind it.

First, I highly recommend that you read the Troubleshooting section.

As torazaburo said, your best option is to use remote debugging: scripts can be run in an interface similar to the Web Inspector interface: step-by-step, shutdown, breakpoints, local variables ... many classic debugger functions are available. If you're familiar with the Chrome development tools, this is the same.

Answer 3

Although this is not a solution to your problems with Phantomjs, I think that Selenium with Python bindings can be used very effectively as a mute scraper in a Linux environment.

You can use PyVirtualDisplay, the Python wrapper for Xvfb, and Xephyr to fake the display. PvVirtualDisplay requires Xvfb as a dependency. On Ubuntu, first install Xvfb:

sudo apt-get install xvfb 

then install PyVirtualDisplay from Pypi:

 pip install pyvirtualdisplay 

Python Selenium script example in silent mode with PyVirtualDisplay:

 #!/usr/bin/env python from pyvirtualdisplay import Display from selenium import webdriver display = Display(visible=0, size=(800, 600)) display.start() # now Firefox will run in a virtual display. # you will not see the browser. browser = webdriver.Firefox() browser.get('http://www.google.com') print browser.title browser.quit() display.stop() 

Answer 4

I created a REPL for horseman that might interest you. It can be easily turned into a Phantom REPL, I threw it away pretty quickly, so the code is not the best, but it works. the link .