How to verify that a specific domain is accessible from a browser?

Question

Our one-page app includes videos from Youtube for end-user consumption. Everything works fine if the user has access to the Youtube domain and to the contents of these domain pages.

However, we often encounter users whose access to Youtube is blocked by a web filter on their network, for example https://us.smoothwall.com/web-filtering/ . The problem is that the filter does not actually kill the request, it simply returns a different page instead of the HTTP status of 200. This page usually says something like the line "hey, sorry, this content is blocked."

One option is to try to extract https://www.youtube.com/favicon.ico to prove that the domain is available. The problem is that these filters usually include a special SSL certificate that allows them to check the contents of HTTP (see https://us.smoothwall.com/ssl-filtering-white-paper/ ), so I cannot rely on TLS. forcing me to exchange content with the wrong certificate, and instead I get a completely authentic favicon.ico file, except for another site. Also, the whole CORS issue of releasing XHR from our domain in the youtube.com domain, which means that if I want to get this favicon.ico , I have to do it JSONP-style. However, even when using the plain old <img> I can’t check the contents of the image due to CORS, see Get image data in JavaScript? so I'm stuck with this approach.

Are there any proven and reliable ways to solve this situation and check the availability of the browser level for a specific domain?

Greetings.

Answer 1

In general, web proxies that want to play beautifully usually annotate an HTTP conversation with additional response headers that can be detected.

Thus, one of the approaches to creating a person-in-the-middle detector can be to check these response headers and compare the results from when MITM is and when not.

Many public websites will display headers for any request; redbot is one.

So maybe you could ask the member whose content will be modified to visit the url: youtube favicon via redbot .

Once you have collected enough samples, you can heuristically build the detector.

In addition, some CDNs (such as Akamai) will allow clients to visit the URL from remote proxy locations on their network. This can provide better coverage, although they are unlikely to be behind a blocking firewall.