Using CefSharp.Offscreen to retrieve a webpage that requires Javascript to render

Question

I have that, hopefully, a simple task, but in order to solve this problem, someone who understands CefSharp will be required.

I have a URL from which I want to extract HTML code. The problem is that this URL does not actually propagate the page to GET. Instead, it pops a piece of Javascript into the browser, which then executes and creates the actual page displayed. This means that conventional approaches, including HttpWebRequestand HttpWebResponse, will not work.

I looked at several different “headless” options, and the one that seems to best fit my needs for a number of reasons is CefSharp.Offscreen. But I don’t understand how it works. I see that there are several events that you can subscribe to, and some configuration options, but I do not need anything like a built-in browser.

All I really need is a way to do something like this (pseudocode):

string html = CefSharp.Get(url);

I have no problem subscribing to events, if necessary, in order to wait for Javascript to execute and create the displayed page.

Answer 1

Chromium, node.js jsdom. , . Github README, URL, javascript, javascript (: jQuery ), HTML , , , $ ('body'). Html() , . ( , SVG-, XML.)

#, , CefSharp.Offscreen . , CefSharp.WinForms CefSharp.WPF, , CefSharp.Offscreen, . JavaScript , body.innerHTML #, . , .

, CefSharp.MinimalExample , . webBrowser.Address #, , , webBrowser.EvaluateScriptAsync(".. JS code..") JavaScript ( ), -, ( bodyElement.innerHTML ).

Answer 2

, , 2- , - .

, Cefsharp.Offscreen .

, .

using System;
using System.IO;
using System.Threading;
using CefSharp;
using CefSharp.OffScreen;

namespace [whatever]
{
    public class Browser
    {

        /// <summary>
        /// The browser page
        /// </summary>
        public ChromiumWebBrowser Page { get; private set; }
        /// <summary>
        /// The request context
        /// </summary>
        public RequestContext RequestContext { get; private set; }

        // chromium does not manage timeouts, so we'll implement one
        private ManualResetEvent manualResetEvent = new ManualResetEvent(false);

        public Browser()
        {
            var settings = new CefSettings()
            {
                //By default CefSharp will use an in-memory cache, you need to     specify a Cache Folder to persist data
                CachePath =     Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "CefSharp\\Cache"),
            };

            //Autoshutdown when closing
            CefSharpSettings.ShutdownOnExit = true;

            //Perform dependency check to make sure all relevant resources are in our     output directory.
            Cef.Initialize(settings, performDependencyCheck: true, browserProcessHandler: null);

            RequestContext = new RequestContext();
            Page = new ChromiumWebBrowser("", null, RequestContext);
            PageInitialize();
        }

        /// <summary>
        /// Open the given url
        /// </summary>
        /// <param name="url">the url</param>
        /// <returns></returns>
        public void OpenUrl(string url)
        {
            try
            {
                Page.LoadingStateChanged += PageLoadingStateChanged;
                if (Page.IsBrowserInitialized)
                {
                    Page.Load(url);

                    //create a 60 sec timeout 
                    bool isSignalled = manualResetEvent.WaitOne(TimeSpan.FromSeconds(60));
                    manualResetEvent.Reset();

                    //As the request may actually get an answer, we'll force stop when the timeout is passed
                    if (!isSignalled)
                    {
                        Page.Stop();
                    }
                }
            }
            catch (ObjectDisposedException)
            {
                //happens on the manualResetEvent.Reset(); when a cancelation token has disposed the context
            }
            Page.LoadingStateChanged -= PageLoadingStateChanged;
        }

        /// <summary>
        /// Manage the IsLoading parameter
        /// </summary>
        /// <param name="sender"></param>
        /// <param name="e"></param>
        private void PageLoadingStateChanged(object sender, LoadingStateChangedEventArgs e)
        {
            // Check to see if loading is complete - this event is called twice, one when loading starts
            // second time when it finished
            if (!e.IsLoading)
            {
                manualResetEvent.Set();
            }
        }

        /// <summary>
        /// Wait until page initialization
        /// </summary>
        private void PageInitialize()
        {
            SpinWait.SpinUntil(() => Page.IsBrowserInitialized);
        }
    }
}

:

public MainWindow()
{
    InitializeComponent();
    _browser = new Browser();
}

private async void GetGoogleSource()
{
    _browser.OpenUrl("http://icanhazip.com/");
    string source = await _browser.Page.GetSourceAsync();
}

,

"<html><head></head><body><pre style=\"word-wrap: break-word; white-space: pre-wrap;\">NotGonnaGiveYouMyIP:)\n</pre></body></html>"