What version of Delphi are you using? Delphi 2007 and add overloaded TOpenDialog.Execute (ParentWnd: HWND); passing the handle to your modal dialog, since ParentWnd can help when calling the Execute () function.
Delphi. , Delphi 2009, , , Delphi 5, , .