Here is a working example of adding some animation features to the jquery ui dialog.
This is not quite a pop-up window, but if you ran it with an absolute position and a smaller size in the window of the offsetbutton that was pressed, and then animated it to its original state, you could recreate the effect that you want.
Otherwise, I can suggest using a plugin that does this natively.
http://fancybox.net/ , .