You must test using your own look! This is not a bug in the GUI per se.
Standard LAF uses a larger font than the one I use, and this forces one of the buttons to exit the layout. And I can not use pack () since this container requires a size requirement. if you did not call frame.pack () in this example, I think you will have the same question.
You have a size requirement for the container. This means that with the standard LAF, no LayoutManager could calculate any layout of the components in the container so that they all comply with the LAF standard, because, as you point out, they are simply too large. This means that you cannot have a standard LAF and size requirements.
While you use only your own LAF and test it with this custom LAF on all supported OS systems and no tests fail, everything is in order. So just change your tests and switch to your custom LAF before doing the tests.
If you ever want to run your application in standard LAF, you will need to cancel the size requirement until everything is correctly displayed using the appropriate LayoutManager. For example, with MigLayout, you can easily set all these size requirements and still use pack () in the JFrame at the end.
edit: I agree with Andrew that you should critically consider the need for size requirements. In many cases, these requirements are not required, and their drop significantly reduces the "failures" in the layout. For example, you can simply specify the minimum size for the container.