Since I do not have a specific answer, I would like to recommend you what I would do.
When there is a feature in an open source application, I recommend playing around with the sources.
If digging or reading is not enough, you can compile the application with full debugging information and run under the debugger to find the piece of code responsible for the interesting function.
If you have not been there, check the following places:
VirtualBox - source code organization
Frontend VirtuulBox GUI
Do some searches on chegesets, for example, search for changesets using mouse capture
In changeset 30448 I see a call to the "captureMouse" function, I think this or a similar place may follow you to the right part of the code.
What I found may interest you the most:
src / VBox / Frontends / VirtualBox / src / runtime / UIMouseHandler.cpp Description: "FE / Qt4-OSX: do not let the mouse leave the window in capture mode.", check these functions:
void UIMouseHandler::captureMouse(ulong uScreenId)
To be complete, here is the header file: UIMouseHandler.h
source share