Google crawl indexing algorithms

Question

I am looking for some docs about how Google crawls and indexes content. I read a lot of “easy” articles and articles about what you need to do to improve your ranking and make sure your content is properly indexed, but I’m looking for some more complex technical documents on how Google crawls and indexes content.

What I would like to know more about:

• What Google elements are searched for when crawling: page content, URL format, keywords, description, etc.
• How is the index updated?

Basically, I'm trying to understand why some pages are indexed, but not others, even if the formats are similar. Why only 10% of the pages of my site appear when I search the entire domain, even if I can see on my server logs that Google crawls every link.

Answer 1

The answers to both things are carefully guarded trade secrets, supposedly to prevent games in the system.

Also keep in mind that Google makes over 400 algorithmic changes per year , making it impossible for an exact outsider to meet. If you don’t work for Google, you probably won’t find a detailed and accurate answer.

, , -, , Google , GoogleWebmasterHelp YouTube. , Google.

Answer 2

-, nutch.apache.org.

webcrawler : , , . webcrawler URL-, -, , 101 . , , and-or-the, , , .

, . , , - .

, Google - , , google, . Pagerank http://en.wikipedia.org/wiki/PageRank , -, , . , xml sitemap, - . . gsitecrawler.com/- .

- Google , Google , , , , , - google, .

, - , SEO, , seomoz.com ... , .

, !, .

Answer 3

"" Google, . - Google " " H1 H2 HTML- ....

. , , H1, H2, .

Rich snippets ..!

Answer 4

- . , javascript , , , . , , . . http://www.tutorialspoint.com/seo/, Google. 40 .

Answer 5

MapReduce:

Answer 6

,

Google provides more CONTENT rather than LINKS.

So, if your content is good enough with properly accessible tags, Google will automatically generate an index for you. I would suggest H1 - H6 everything you need to use in a good manner.