The problems that are usually encountered are that there is some reference to a module in flight somewhere in the system. It can be anything from blocking, some kind of callback in flight - another module using a symbol, etc. Etc. Etc.
There are many reference counting systems in the kernel that should be used to help you ensure that this never happens. Things that your module will use, or aspects of your module, will contain relative reference counts. The kernel will prevent your form from doing "rmmod" when this counter is non-zero. You can see the number of links to all loaded modules in the output of "lsmod".
Explore them and use them wisely.
"SMP", . , , - , , , - - - () . - - . refcnt .