You can get a detailed trace of your ssh connection using a small script and specifying GIT_SSH so that this script is used when git connects to your server.
#!/bin/sh exec ssh -v $@
and export GIT_SSH=$HOME/ssh-debug or export GIT_SSH=$HOME/ssh-debug you call this file. The -v flag includes verbose messages from ssh, and you can see which stage allows the connection to be established. When testing this on my server, I see that it is trying to download various keys and then request a password to unlock the key file. If I give a bad password, it will continue to verify the password on the server. You may have allowed passwords with an empty password to be logged in to your git account, which then allows someone. However, I believe that verbose ssh logging should identify the problem.
The above script also works with Windows clients. Just use set GIT_SSH=c:\pathwithoutspace\ssh-debug if you are using cmd shell.
source share