Unload a module in Python

Question

TL / DR:

import gc, sys print len(gc.get_objects()) # 4073 objects in memory # Attempt to unload the module import httplib del sys.modules["httplib"] httplib = None gc.collect() print len(gc.get_objects()) # 6745 objects in memory 

---

UPDATE I contacted Python developers on this issue, and indeed, I will not be able to completely unload the module in the next five years. (see link)

Accept that Python really does not support unloading modules for serious, fundamental, insurmountable technical problems in 2.x.

---

During my recent memleak hunt in my application, I narrowed it down to modules, namely: the inability of garbage to collect an unloaded module. Using any method listed below to unload a module, produces thousands of objects in memory. In other words - I cannot unload a module in Python ...

The rest of the question is an attempt to garbage collect a module somehow.

Try:

 import gc import sys sm = sys.modules.copy() # httplib, which we'll try to unload isn't yet # in sys.modules, so, this isn't the source of problem print len(gc.get_objects()) # 4074 objects in memory 

Save a copy of sys.modules to try to restore it later. So these are the basic 4074 objects. We should ideally return to this somehow.

You can import the module:

 import httplib print len(gc.get_objects()) # 7063 objects in memory 

We have up to 7K objects without garbage. Try removing httplib from sys.modules .

 sys.modules.pop('httplib') gc.collect() print len(gc.get_objects()) # 7063 objects in memory 

Well, that didn't work. Hmm, but is there a link in __main__ ? Oh yeah:

 del httplib gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

Hooray, down 300 objects. However, there is no cigar, thus more than 4000 original items. Try to restore sys.modules from the copy.

 sys.modules = sm gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

Hmmm, well, that was pointless, no change .. Maybe if we destroy the globals ...

 globals().clear() import gc # we need this since gc was in globals() too gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

local residents?

 locals().clear() import gc # we need this since gc was in globals() too gc.collect() print len(gc.get_objects()) # 6746 objects in memory 

What ... what if we imported module inside exec ?

 local_dict = {} exec 'import httplib' in local_dict del local_dict gc.collect() print len(gc.get_objects()) # back to 7063 objects in memory 

Now, dishonestly, he imported it into __main__ , why? He should never have left local_dict ... Argh! We will return to the fully imported httplib . Maybe if we replace it with a dummy object?

 from types import ModuleType import sys print len(gc.get_objects()) # 7064 objects in memory 

Bloody ..... !!

 sys.modules['httplib'] = ModuleType('httplib') print len(gc.get_objects()) # 7066 objects in memory 

Die cutting modules, die !!

 import httplib for attr in dir(httplib): setattr(httplib, attr, None) gc.collect() print len(gc.get_objects()) # 6749 objects in memory 

Well, after all the attempts, the best is +2675 (almost + 50%) from the starting point ... This is from only one module ... This one doesn't even have anything big inside ...

OK, now seriously, where is my mistake? How to unload a module and destroy its contents? Or are there Python modules for one giant memory leak?

Full source in a simpler form for copying: http://gist.github.com/450606

Answer 1

Python does not support unloading modules.

However, if your program does not load an unlimited number of modules over time, this is not a source of memory leak. Modules usually load once at startup and that they are. The likelihood of a memory leak is likely to lie elsewhere.

In the unlikely event that your program really loads an unlimited number of modules over time, you should probably redesign your program .; -)

Answer 2

I'm not sure about Python, but in other languages, calling the equivalent of gc.collect() does not free up unused memory - it will only free that memory if / when the memory is actually needed.

Otherwise, Python makes sense to keep modules in memory for a while, in case they need to be loaded again.

Answer 3

(You should try to write more concise questions, I just read the beginning and looked through everything else.) At first I see a simple problem:

 sm = sys.modules.copy() 

You made a copy of sys.modules, so now your copy has a link to the module - so, of course, it will not be compiled. You can see what applies to it with gc.get_referrers.

This works great:

 # module1.py class test(object): def __del__(self): print "unloaded module1" a = test() print "loaded module1" 

.

 # testing.py def run(): print "importing module1" import module1 print "finished importing module1" def main(): run() import sys del sys.modules["module1"] print "finished" if __name__ == '__main__': main() 

module1 is unloaded as soon as we remove it from sys.modules, because there are no remaining references to the module. (Doing module1 = None after import will work too - I just put the import in another function for clarity. All you have to do is drop the references to it.)

Now this is a little difficult to do in practice due to two problems:

• To assemble a module, all links to the module must be inaccessible (as in the collection of any object). This means that any other modules that imported it must also be dereferenced and reloaded.
• If you remove a module from sys.modules while it is still referencing somewhere else, you created an unusual situation: the module is still loaded and used by code, but the module loader no longer knows about it. The next time you import a module, you will not get a link to the existing one (since you deleted the entry), so it will download a second copy of the existing copy of the module. This can cause serious consistency issues. Thus, make sure that there are no remaining module references before permanently removing it from sys.modules.

There are several complex problems for using this as a whole: determining which modules depend on the paged module; Knowing whether it is normal to unload (to a large extent depends on your use case); processing threads while learning all of this (look at imp.acquire_lock), etc.

I could come up with a case where this might be useful, but most of the time I would recommend just restarting the application if its code changes. You probably just give yourself headaches.