pip and conda are two separate package managers. Only in very rare cases, package managers actually work together. In practical applications, cond and pip are usually not.
In fact, mixing conda and pip packages is usually inevitable. This often leads to confusing package management as you describe.
In my opinion, the best and currently the only way to solve this problem is to create a conda package for all (pypi-) packages and dependencies that you want to use in your conda environments.
conda-forge is a community that offers an easy way to contribute your own package to the conda infrastructure. You might want to check if your package is available, and if not, if input is an option for you.