JavaFX 2.2 does not have a framework to facilitate this.
You can track and vote on the JavaFX Docking Framework feature request using the JavaFX Problem Tracker.
You can create such a structure yourself using the open JavaFX API. Using Stages, it can be a little difficult to get exactly the behavior you want, but I think that it would be possible if you used customizable movable panels that are placed inside the main stage of the application.
However, until an official implementation of the docking infrastructure is provided, you may be better off using an existing application framework such as NetBeans and embedding your JavaFX components in JFXPanels as part of frames created by the existing framework .
To answer some of your specific questions.
not displayed / selected on the OS taskbar.
Before you show your new subtitle, call subwindow. initOwner to set the owner of the subtitle to the main stage of your application.
they cannot be moved outside the user window
You can add listeners to the x, y, width, and height properties for the subwindow and main window. The listener code reorders the subwindows if an attempt is made to move them outside the parent window. This strategy might turn out to be a little awkward if resizing and location notifications are handled by the operating system before your JavaFX listeners get notifications (which I think will be the case).