Azure Key Vault uses Thales Hardware Security Modules (HSMs) . Especially about HSM is that they do not give you the keys. You create or import a key in HSM. Later you give HSM data, and HSM performs cryptographic operations on this data. For instance. encryption, decryption, hashing, etc. By the way, these hardware devices are really expensive. With Azure Key Vault, you can use this protection for a small price. This is one of the benefits of using Azure Key Vault.
To your question , why is it safer . Keys are more valuable than one vulnerable VM that has been compromised. If one of your virtual machines has been compromised and you have encryption keys on this virtual machine, the attacker will have your keys.
If your virtual machine has only your client ID and secret, then the attacker has only these credentials, but not your keys. And, as you know, even with these credentials, an attacker cannot get your HSM keys. This means that an attacker will be able to perform cryptographic operations for a certain time with your credentials. The moment you recognize the attack, you will cancel the credentials.
So, the difference in a nutshell:
Without HSM: the attacker has your keys and they can use these keys as long as they want (and they are valid).
With HSM: an attacker does not have your keys and can perform cryptographic operations with your credentials if they are valid.