How to protect yourself from shell dll files loaded into your process?

Question

When you use the standard "Windows" file open dialog with GetOpenFileName (), the shell will load the various DLLs needed to display a list of files, including user-defined ones.

In my application, I found that the DLL used by TortoiseCVS to draw overlays on the icons called GdiPlusShutdown (), and so some time after the "open file" dialog box appears, the TortoiseCVS DLL will be unloaded, it will close down GDI +, and mine graphic functions will fail!

It seems pretty bad that in any case, any old DLL can be loaded by my application at any time and start doing random things in its state. The workaround in my case was pretty simple - just restart GDI + if I find that it is disabled. However, if this happened on a client machine, where I could not debug it, it would be much harder to figure out what was happening.

Can someone give an idea? What can I do to prevent this from happening?

Answer 1

I had to deal with the crap that Dell puts on its machines, in particular wxVault. My solution was to “just” fix the code. A bit complicated with DEP, but still doable. You could take a peek at Microsoft Detours, a slightly more structured way to do the same. You will still have the DLL loaded, but at least you can stop its calling functions, which it should not call.

For why Windows has such a shitty mechanism, read Raymond Chen's blog or book, "Old New Thing."