List of sitemaps and sitemap files in robots.txt file?

My site consists of 3 main sections: reviews, forum and blog. I have forum and blog plugins that automatically generate sitemaps for them. The forum plugin generates a sitemap INDEX file pointing to several indexes, and the blog plugin creates a regular sitemap containing all the contents of my blog. Here are their entries from the robots.txt file:

Sitemap: http://www.datesphere.com/forum/sitemap-index.xml Sitemap: http://www.datesphere.com/blog/sitemap.xml 

I just created a sitemap.xml Reviews file containing all the content in the Reviews section. I planned to just add a line to the robots.txt file, so it will all look like this:

 Sitemap: http://www.datesphere.com/forum/sitemap-index.xml Sitemap: http://www.datesphere.com/blog/sitemap.xml Sitemap: http://www.datesphere.com/reviews-sitemap.xml 

HERE MY QUESTION: I know that you can list several Sitemap files in a robots.txt file, but is it normal to have a sitemap index file as well as several files listed? Will Googlebot ignore other Sitemaps if it finds the sitemap-index.xml file in the robots.txt file? If so, do I need to post my blog and view sitemaps in another site index file and just list it in robots.txt?

I checked, but I can find answers to the question "can I list several Sitemaps?"

+6
source share
2 answers

Googlebot will not ignore any of the Sitemaps that you list in the robots.txt file, even if you also provide your parent Sitemap index. We follow every link that we find, and if we are allowed, we will crawl it. Personally, I would probably list only Sitemap indexes, but only for ease of management, but it is up to you, Googlebot will not mind if you specify both indexes and Sitemaps.

+14
source

If you have multiple Sitemaps, you can specify the URL of the Sitemap file in the robots.txt file, as shown in the following example:

 // robots.txt Sitemap: http://www.example.com/sitemap_index.xml User-agent:* Disallow: /some/disallowed/path 

Or you can specify separate URLs for your multiple Sitemaps, as shown in the following example:

 // robots.txt Sitemap: http://www.example.com/sitemap_host1.xml Sitemap: http://www.example.com/sitemap_host2.xml User-agent:* Disallow: /some/disallowed/path 

Finally, this is what you need to pay attention to when adding the Sitemap directive to the robots.txt file.

0
source

Source: https://habr.com/ru/post/890635/


All Articles