You can completely disable swapfiles with set noswapfile or, if you still want this feature, but not a mess, tell Vim to put these files in a dedicated location:
set directory=c:\\tmp
Please note that you still have to delete existing swap files manually.
For more information, read the following help topics:
:help 'swapfile' :help swap-file :help 'directory'
source share