The best open source spider to cover a site

Question

I am interested in crawling through many websites. The most important consideration is that the spider is able to reach as much space as possible. One of the key features that most spiders lack is the ability to execute JavaScript. This is necessary to scan sites with ajax support. I really like Open Source, and I will need to change the code for my project.

Currently, I believe that Solr, which is different from Lucine, is a very good solution. http://lucene.apache.org/solr/features.html

Has anyone used Solr or Lucine? My biggest problem with Solr cannot run javascript, however, it has a rich feature set and scalability, which makes Solr attractive.

Answer 1

Solr is not a crawler, but a search engine (searches by index to return results).

However, I really like heritrix for its flexibility. Most crawlers will not run Javascript (but some, like Heritrix, will try to extract links from it), as that doesn't make much sense even today. The point is, Heritrix will let you hook your own classes to do anything you want with workarounds.

Answer 2

Try HTMLUnit. http://htmlunit.sourceforge.net/

Answer 3

Solr - , Lucene. . Apache Nutch. Cracking javascript , .

Answer 4

watir may be useful to you.

Answer 5

With pages that create dom based on the javascript template, you really need the full execution of javascript in your spider. Check out https://github.com/mikeal/spider for Node JS.