Rendering inside an OpenGL scene from an external application using a nested DLL

Question

I managed to inject DLLs into applications for things like keyboard actions, but how do I connect to the OpenGL program and draw it? I read a lot of “textbooks” and answers, but they all pose the idea that they are doing this, and not how to do it. If someone could point me in the right direction, I would really appreciate it.

What I intend to do is draw an overlay (or perhaps draw into the real scene? Is this possible?) From a third-party application only in RAM (injection dll). My only experience with OpenGL is through GLUT, but I want to find out if necessary.

After several hours of research, I found an open source application that can do what I want and looked at its code, but it seems that it can’t find which method is connected or how the ASM code changes to this particular method.

Is a method defined to determine the binding when dll is entered, or is dll executed when it joins? As far as I understand, you can do this too, but better than the other?

I am looking for a comprehensive tutorial or a simple example of such a thing. If someone could point me in the right direction, that would be great!

Answer 1

, OpenGL - API-, , . , , - , . , , , , , .

: wglSwapBuffers ( glXSwapBuffers X11, eglSwapBuffers EGL - Windows ).

, , SwapBuffers. .

OpenGL , , . , OpenGL, , . , SwapBuffers :

BOOL WINAPI (*host_SwapBuffers)(HDC);
HGLRC hook_last_rc_created = NULL; /* for context sharing */

/* C++ std::map for the lazy. */
std::map<int, HGLRC> hook_pixelformat_to_rc;

/* This function is called to setup per RC resources, i.e. things
 * that are not shared through the list namespace, like FBOs, VAOs
 * and such. */
void init_RC();

/* This function is called when the first hook OpenGL context is
 * created. This is to be used for one time initialization of OpenGL
 * objects that are shared among all contexts through the list namespace,
 * like textures, VBOs, PBOs and such. */
void init_GL();

/* The function that does the actual overlay drawing.
 * Must not call SwapBuffers when finished. The draw overlay has to set
 * up the viewport, projection and modelview transformations.
 *
 * rect gives the dimensions of the window we draw to, for
 * setting the viewport and stuff like that. */
void draw_overlay(RECT const *rect);

BOOL WINAPI hook_SwapBuffers(HDC dc)
{
    /* It illegal to call SwapBuffers with a NULL dc. To keep Windows
     * error behavior we bail out, forwarding the call to the system
     * SwapBuffers if dc == NULL */
    if( !dc ) {
        goto terminate;
    }

    /* OpenGL RCs can be shared between DCs of compatible pixelformat. So technically
     * any number of pixelformat IDs could be compatible. But we just assume that
     * compatibility is only between pixelformats of the same ID, which is safe. */
    int const pixelformat = GetPixelFormat(dc);

    HGLRC hook_rc;
    /* we need to create a RC for our hook only one time and can reuse that
     * later. It can even be used for different DCs or windows, as long as
     * they have been created with compatible pixel formats. To make this
     * hook really robust there is to be a mapping from DCs to pixelformat IDs.
     * If no RC can be found for the given pixelformat, create one. */
    if( 0 == hook_pixelformat_to_rc.count(pixelformat) ) {
        /* the DC is of a window that already got everything set up for OpenGL.
         * All we have to do is create our own HRC if we don't have one already. */

        hook_rc = wglCreateContext(dc);
        if( !hook_rc ) {
            /* bail out, but forward the SwapBuffers call on error. Don't use
             * exceptions in hook DLLs, or very bad things may happen, if you're
             * not very, very careful. */
            goto terminate;
        }

        /* This could throw an exception... */
        hook_pixelformat_to_rc[pixelformat] = hook_rc;
        init_RC();

        /* If there was a RC created before, share its list namespace, so
         * that textures, VBOs, PBOs and so on are available among all the
         * contexts our hook uses. */
        if( hook_last_rc_created ) {
            wglShareLists(hook_last_rc_created, hook_rc);
        }
        else {
            init_GL();
        }
        hook_last_rc_created = hook_rc;
    }
    else {
        hook_rc = hook_pixelformat_to_rc[pixelformat];
    }

    /* Preserve the RC bound to the window before our hook.
     * May be NULL, but that fine, this just means there was nothing
     * bound and we should return to that state at the end. */
    HGLRC const host_rc = wglGetCurrentContext();

    /* Find out which window the DC belongs to, so that we can query for its
     * dimension, which we need to properly setup a viewport region. */
    HWND const wnd = WindowFromDC(dc);
    RECT wnd_rect;
    GetClientRect(wnd, &wnd_rect);

    /* bind our own RC */
    wglMakeCurrent(dc, hook_rc);

    /* we can now draw the overlay */
    draw_overlay(&wnd_rect);

    /* Restore RC to DC binding */
    wglMakeCurrent(dc, host_rc);
terminate:
    /* host_SwapBuffers is a function pointer initialized to the address of the
     * system SwapBuffers function before hooking by the DLL injection code. */
    return host_SwapBuffers(hdc);
}