VCL Style from DLL affects TMenuItem in application

Question

I use the Delphi XE6 and VCL styles. I have a main application and a dll. My main application included runtime themes, and I use vcl style files. I am very similar to my dlls. I included runtime themes and added VCL.Themes, VCL.Styles in the usage area and a resource file with a VCL style file inside it. When the DLL loads, I load the VCL style from the resources and install it for the DLL-gui. The main application and DLLs are not built with runtime packages.

Now I have the main GUI of the application, designed in my own style, and the DLL-GUI created using my own style. It seems to be working fine for now ...

When I click the button in the main application, the event that TPopupMenu opens, it has the same style as the GUI DLL, and not the main style of the application. If I navigate the menu, I also get AV and a program crash. Take a look at the attached image.

What am I doing wrong? The only workaround that I can see right now would be to create my own custom TPopupMenu derived from some other control.

---

As I promised, I prepared a simple demo program similar to my application. It consists of a host application with its own style and a DLL with a style added to the resource. Launch it and click on the pop-up button, then try selecting something from the pop-up window. It will crash and stop at some kind of StdWindowProc or something like that. Also, if you go to the window system menu (upper left corner), when you try to select something from this menu, you will notice that the system menu is decorated as a gui DLL and it crashes. Link to the rar file: dropbox.com/sh/f2jmbsmw18akpyg/AAA6SWdBmVhf6n6K-mvYLLmua?dl=0

Thank you for your help.

Answer 1

This is a fundamental issue with VCL styles and how they style the menu. Styling is done with a wide-range hook. In particular, the CBT hook is set by calling SetWindowsHookEx from TCustomStyleEngine.CreateSysHook in the Vcl.Themes block. In fact, the hook only applies to the GUI thread, but it is a broad process in the sense that there is exactly one GUI thread in this process.

Since you have multiple VCL instances in your application (one in the DLL and one in the application), two hooks are installed. It's too much. A recently installed hook has been recently installed (a DLL, as it happens), and therefore the DLL menu style infects your executable. And why you are faced with a violation of access rights. The DLL is trying to work with a menu that belongs to an executable file. And so, in spite of all your efforts, you end up in a DLL code that accesses VCL objects from the host executable.

There is no easy way around this and fully support the styles in both modules. What we have here is a fundamental consequence of the design. The system was not designed to support multiple VCL instances. If you want to use VCL styles across multiple modules, designers expect you to use runtime packages.

I suggest that you could get some traction by controlling the DLL from a completely different thread. This involves loading the DLL from this other thread so that the VCL initializes in the thread. And all calls to the DLL must be from this thread. And you will need to start the message loop in this thread. You may be able to do this work, but I doubt it. Even with all the caveats mentioned, you still have to cope with the fact that you have two GUI threads that present all kinds of problems with processing the input queue.

Perhaps another approach is to remove the hook from the DLL. As long as your DLL does not display the menu, you can get away with removing this hook. This will disable the style for the menu displayed by the DLL, but perhaps this is acceptable.

This version of your DLL (after I simplified it also) removes the hook.

 library VCLStyleDLL; {$R 'Style.res' 'Style.rc'} uses VCL.Styles, VCL.Themes, VCL.SysStyles; // to gain access to TSysPopupStyleHook {$R *.res} begin TStyleManager.TrySetStyle('Glossy', false); TCustomStyleEngine.UnRegisterSysStyleHook('#32768', TSysPopupStyleHook); end. 

In this version of the DLL, the host executable does not suffer from the problems described in your question.

Answer 2

As David says, this is because each instance of VCL sets a hook to detect when a popup menu is created (# 32768). Thus, two instances of the hook work simultaneously.

As a workaround, you can disable the popupmenu style hook in the dll (or in the application) using the UnRegisterSysStyleHook function defined in the Vcl.SysStyles module.

 TCustomStyleEngine.UnRegisterSysStyleHook('#32768', TSysPopupStyleHook);