I did the animation as you described earlier. However, this was not an easy task. Since your animation extends beyond the window itself, you need to display the animation in a transparent transparent window. When the animation is complete, you can order in a real window and remove the transparent one.
You will need a window image that will be used as the content of your animation, so I will place an order in the window (and probably make it also a key window so that it looks focused), but put it on the screen so that the user does not see it. Then use CGWindowListCreateImage to get a screenshot. Now you will have what you need to create the animation.
After the animation is complete, simply order the real window on top of the transparent one, then remove the transparent window. Using math correctly so that the image of the window in the animation and the real window is a little complicated, but it is definitely doable.