Ruby code to check if a site has a search engine URL.

Question

I am developing an application on rails that requires checking for the presence or absence of an entered website with friendly search URLs. The solution I have in mind is using nokogiri to analyze the HTML site and view the link tag to find the URLs and see if they are search engine friendly. Is there any other way to do this? Any help would be really great.

Answer 1

You have two problems:

• How do you formally (programmatically) determine what a "frienldy search engine URL" is? I guess you have a way to do this already. So the leaves ...

• How to check all links on a website.

So, for (2) I would look at something like Anemone , which will make it easier for you to scan full websites:

Anemone is a Ruby library that allows you to quickly and painlessly write programs that distribute a website. It provides a simple DSL to perform actions on each page of the site, skips specific URLs and calculates the shortest path to this page on the site.

Multi-threaded design makes Anemone fast. The API makes it simple. And the expressiveness of Ruby makes it powerful.

For a simple workaround, Anemone will even give you an array of all the links on the page, so you don't necessarily need Nokogiri. For more complex things, you might want to combine Anemone with something like Mechanize and Nokogiri. It depends on your requirements.