All geek questions in one place

Why doesn't Google Crawling CSS file exist?

Google Bot Crawler is consistently trying to bypass CSS files that do not exist on my site during production.

He asks:

http://www.mywebsite.com/assets/index-d45678283d4ab9905c3538184826e599.css

This exact file name does not exist in production (there is a slightly different file name in production).

However, the CSS file that it requests exists in development at:

 http://localhost:3000/assets/index-d45678283d4ab9905c3538184826e599.css

I am not sure why it is requesting this file.

I use Capistrano (load "deploy / assets") to pre-compose my assets before deploying to production.

Right now, I'm just blocking this file in the robots.txt file, but the css file that it requests changes after every deployment.

Why does GoogleBot want to bypass this file, which does not exist on the production site? How to stop him?

+4
source share
1 answer

GoogleBot probably sees one of two things:

  • He sees this file on your site somewhere where it was incorrectly specified - IE old code. I would look for your live site (included) for the file.
  • He remembers this file from the previous assembly and tries to check for updates.

I am puzzled that he will not ignore it after a 404 response from your server, however, the internal operation of Google software is a black box; there is no true way to say why he is doing what he is doing.

However, they do offer a webmaster toolbar that allows you to do some indexing adjustments, etc.

+3
source

Source: https://habr.com/ru/post/1487788/

More articles:


All Articles