We have a React application that downloads some data asynchronously from another domain. Requests are executed using the isomorphic-fetch mode in cors , and all requests and responses look great and work correctly when testing using my own browser.
We have monitoring responses and log errors in our analysis application.
While in most cases everything is fine (and everything seems to be correctly indexed and displayed well on Google), we still see a lot of crashes, only for Googlebot, where it reads the data incorrectly. Debugging the response object, I see that status is 200, but statusText empty. The answer has no body (and therefore no .json or .text methods), and no headers (which should not be), and the mode is correctly set to cors (not opaque , which may explain some of the other oddities).
From my understanding of CORS, this looks above the board in terms of sending and receiving headers, so why does the Googlebot have so many intermittent problems? Googlebot says it has an HTTP 200 response (successful, Promise is not rejected), but it skips everything that comes with the HTTP 200 responose - it has no body and no headers. Why doesn’t Googlebot return a response with headers and body (as described below)?
A typical pre-flight request looks like this (from Chome devtools) (an extra slash is added to */\* to stop SO thinking it is a comment tool)
Accept:*/\* Accept-Encoding:gzip, deflate, sdch, br Accept-Language:en-GB,en-US;q=0.8,en;q=0.6 Access-Control-Request-Headers:content-type, x-apikey Access-Control-Request-Method:POST Cache-Control:no-cache Connection:keep-alive DNT:1 Host:my.host.net Origin:http://my.origin.net Pragma:no-cache Referer:http://my.origin.net/ User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.100 Safari/537.36
And the answer to the preflight is as follows
Access-Control-Allow-Headers:content-type,x-apikey Access-Control-Allow-Origin:* Cache-Control:no-cache Connection:keep-alive Content-Length:0 Date:Mon, 05 Dec 2016 00:55:05 GMT Expires:-1 Pragma:no-cache Server:Microsoft-IIS/8.5 X-AspNet-Version:4.0.30319 X-Powered-By:ASP.NET
Then comes the actual request, which looks like this (sent as a POST with a JSON body)
accept:application/json Accept-Encoding:gzip, deflate, br Accept-Language:en-GB,en-US;q=0.8,en;q=0.6 Cache-Control:no-cache Connection:keep-alive Content-Length:62 content-type:application/json DNT:1 Host:someapi.net Origin:http://my.origin.net Pragma:no-cache Referer:http://my.origin.net/ User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.100 Safari/537.36 x-apikey:someapikey
Which returns an answer like this (with a JSON body)
Access-Control-Allow-Origin:* Cache-Control:no-cache Connection:keep-alive Content-Length:33576 Content-Type:application/json; charset=utf-8 Date:Mon, 05 Dec 2016 00:55:05 GMT Expires:-1 Pragma:no-cache Server:Microsoft-IIS/8.5 X-AspNet-Version:4.0.30319 X-Powered-By:ASP.NET