If you are already rewriting your code to use smart pointers, you should go all the way and remove simple / raw pointers where possible.
Smart pointers do not use some magic to track their contents (e.g. garbage collector), but they use fairly simple heuristics to determine if the released object should be released or not. The misuse of smart pointers can easily break this heuristic.
For example, shared_ptr keeps track of all copies that are made (directly or indirectly, like copies from copies) and destroys the monitored object when the last copy is destroyed. This breaks down terribly if you manage to create two shared_ptr that manage the same object, but where it is not a copy of the other.
source share