Need help finding web pages

Question

Hi, I am completing a small project of my hobby to create a small search engine.

I was wondering if anyone knows about the decent, reliable open source web crawler they used? It should be easy for noob to configure and use.

Thank you for not browsing search engines and inserting a list.

Answer 1

crawler4j is a pretty decent crawler, multi-threaded and easy to configure and use. It is written in Java.

You can find a list of open source crawlers on this wikipedia page.

Answer 2

I think you should read a similar experience.

http://infolab.stanford.edu/~backrub/google.html