How does the scanner provide maximum coverage?

Question

I read several articles about scanning on the Internet and learned the basics of scanning. According to them, web crawlers use only URLs received by other web pages and passing through a tree (actually a grid)

In this case, as a searcher provides maximum coverage. Obviously, there may be many sites that do not have links to other pages / sites. Does search engines support any other mechanisms besides crawling and manual registration? (i.e. retrieving information from domain registries)

If they are crawl-based, how do you choose a good set of Root sites to start crawling? (We are not able to predict the results. If we select 100 sites without links to links, only 100 sites + their internal pages will appear on the screen)

Answer 1

Obviously, there may be many sites that do not have referral links from other pages / sites.

I do not think that this is really the same big problem as you think.

Does search engines support any other mechanisms besides crawling and manual registration? (i.e. retrieving information from domain registries)

No, which I heard about.

If they are based only on crawling, How to choose a good set of "Root" sites will start crawling?

Any kind of universal web directory, such as an open directory project , would be an ideal candidate, as would social bookmarking sites like Digg or del.icio.us

Answer 2

One method used to help crawlers is a “sitemap”. A sitemap is basically a file that lists the contents of a website, so the crawler knows where to go, especially if your site has dynamic content. A more accurate site map will greatly increase the accuracy of the crawler.

Here is some information about the Google sitemap:

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40318

Answer 3

There is no magic mechanism that would allow the crawler to find a site that no other site has cited, already crawled or not manually added to the crawler.

The crawler only crosses the link schedule, starting with a set of manually registered - and therefore predefined - roots. Everything that is outside the schedule will be inaccessible to the searcher - he will not have the means to search for this content.