Imported modules become None when function starts

Question

Update : another debugging information at the bottom of this post that shows something very sharp in python state.

I have a module that imports, among other things, a django user object.

Import works fine and the code loads. However, when you call a function in this module that uses the User object, it means that the user is NoneType.

There are also a number of other import data and some global module level variables, which were also missing by the time the function was called.

Oddly enough, this is only a problem in our intermediate environments (Ubuntu 12.04). It works fine locally, which is probably the most similar to setting up with additional python packages for working with dev. Also shallow in production.

Has anyone come across this before and had any ideas what might trigger it?

Here is the code:

import urllib import time import urlparse # Django imports from django.db.models.signals import post_delete from django.db import models from django.contrib.auth.models import User from backends.cache.dualcache import cache # Piston imports from managers import TokenManager, ConsumerManager from signals import consumer_post_delete KEY_SIZE = 18 SECRET_SIZE = 32 VERIFIER_SIZE = 10 CONSUMER_STATES = ( ('pending', 'Pending'), ('accepted', 'Accepted'), ('canceled', 'Canceled'), ('rejected', 'Rejected') ) def generate_random(length=SECRET_SIZE): return User.objects.make_random_password(length=length) class Consumer(models.Model): name = models.CharField(max_length=255) description = models.TextField() key = models.CharField(max_length=KEY_SIZE) secret = models.CharField(max_length=SECRET_SIZE) status = models.CharField(max_length=16, choices=CONSUMER_STATES, default='pending') objects = ConsumerManager() def __unicode__(self): return u"Consumer %s with key %s" % (self.name, self.key) def generate_random_codes(self): key = User.objects.make_random_password(length=KEY_SIZE) secret = generate_random(SECRET_SIZE) while Consumer.objects.filter(key__exact=key, secret__exact=secret).count(): secret = generate_random(SECRET_SIZE) self.key = key self.secret = secret self.save() 

and here is the work around, which means basically to import what you need again inside the function:

 import urllib import time import urlparse # Django imports from django.db.models.signals import post_delete from django.db import models from django.contrib.auth.models import User from backends.cache.dualcache import cache # Piston imports from managers import TokenManager, ConsumerManager from signals import consumer_post_delete KEY_SIZE = 18 SECRET_SIZE = 32 VERIFIER_SIZE = 10 CONSUMER_STATES = ( ('pending', 'Pending'), ('accepted', 'Accepted'), ('canceled', 'Canceled'), ('rejected', 'Rejected') ) def generate_random(length=SECRET_SIZE): return User.objects.make_random_password(length=length) class Consumer(models.Model): name = models.CharField(max_length=255) description = models.TextField() key = models.CharField(max_length=KEY_SIZE) secret = models.CharField(max_length=SECRET_SIZE) status = models.CharField(max_length=16, choices=CONSUMER_STATES, default='pending') objects = ConsumerManager() def __unicode__(self): return u"Consumer %s with key %s" % (self.name, self.key) def generate_random_codes(self): from piston.models import KEY_SIZE, SECRET_SIZE, Consumer from django.contrib.auth.models import User from piston.models import generate_random key = User.objects.make_random_password(length=KEY_SIZE) secret = generate_random(SECRET_SIZE) while Consumer.objects.filter(key__exact=key, secret__exact=secret).count(): secret = generate_random(SECRET_SIZE) self.key = key self.secret = secret self.save() 

Here's the stack trace. The error is caused by the line:

 key = User.objects.make_random_password(length=KEY_SIZE) 

in the generate_random_codes function.

 Traceback: File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response 111. response = callback(request, *callback_args, **callback_kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/contrib/admin/options.py" in wrapper 366. return self.admin_site.admin_view(view)(*args, **kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view 91. response = view_func(request, *args, **kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func 89. response = view_func(request, *args, **kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/contrib/admin/sites.py" in inner 196. return view(request, *args, **kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapper 25. return bound_func(*args, **kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view 91. response = view_func(request, *args, **kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/utils/decorators.py" in bound_func 21. return func(self, *args2, **kwargs2) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/db/transaction.py" in inner 224. return func(*args, **kwargs) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/contrib/admin/options.py" in add_view 970. form = ModelForm(initial=initial) File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/forms/models.py" in __init__ 234. self.instance = opts.model() File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/db/models/base.py" in __init__ 349. val = field.get_default() File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/db/models/fields/related.py" in get_default 983. field_default = super(ForeignKey, self).get_default() File "/sites/tellybug/shared/webserver/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py" in get_default 379. return self.default() File "/sites/tellybug/releases/b92109dd526607b2af92ad6b7f494f3f06e31bb2/webserver/tellybug/tbapp/models/tellybugapp.py" in generate_new_consumer 11. consumer.generate_random_codes() File "/sites/tellybug/releases/b92109dd526607b2af92ad6b7f494f3f06e31bb2/webserver/tellybug/piston/models.py" in generate_random_codes 57. key = User.objects.make_random_password(length=KEY_SIZE) Exception Type: AttributeError at /admin/tbapp/tellybugapp/add/ Exception Value: 'NoneType' object has no attribute 'objects' 

Refresh . This is not just deleting the User object, but something destroying the whole context in the function.

 def generate_random_codes(self): """ Used to generate random key/secret pairings. Use this after you've added the other data in place of save(). c = Consumer() c.name = "My consumer" c.description = "An app that makes ponies from the API." c.user = some_user_object c.generate_random_codes() """ import sys print "Globals", globals() print "Name ", __name__ print "Package ", __package__ print "Sys modules", sys.modules['piston.models'].__dict__ key = User.objects.make_random_password(length=KEY_SIZE) 

Using these print statements, the output is:

 Globals {'ColumnFamilyMap': None, 'datetime': None, 'KEY_SIZE': None, 'TokenManager': None, 'ConsistencyLevel': None, 'Nonce': None, 'uuid': None, 'cache': None, 'urllib': None, '__package__': None, 'models': None, 'User': None, .... } Name None Package None Sys modules {'ColumnFamilyMap': <class 'pycassa.columnfamilymap.ColumnFamilyMap'>, 'datetime': <type 'datetime.datetime'>, 'KEY_SIZE': 18, 'NonceType': <class 'piston.models.NonceType'>, 'OAuthToken': <class 'piston.models.OAuthToken'>, 'TokenManager': <class 'piston.managers.TokenManager'>, 'ConsistencyLevel': <class 'pycassa.cassandra.ttypes.ConsistencyLevel'>, 'Nonce': <class 'piston.models.Nonce'>, 'uuid': <module 'uuid' from '/usr/lib/python2.7/uuid.pyc'>, ...} 

Please note that both __package__ and __name__ are undefined, which I thought was almost impossible, and that although the sys.modules module version has the correct __dict__ , the return value from globals() nonsense.

Answer 1

This happens with a function in the imported module that is still executing after this module has garbage collected.

Since your code is not enough to reproduce the problem, here is a simplified example demonstrating the behavior. Create a file containing the following and import it from either Python or another file. This does not work if you just run it at the top level.

 import sys import threading x = "foo" def run(): while True: print "%s %s\n" % (sys, x) threading.Thread(target = run).start() sys.stdin.readline() 

Launch:

 $ python >>> import evil_threading <module 'sys' (built-in)> foo <module 'sys' (built-in)> foo ... press Ctrl-C None None None None ... press Ctrl-\ to kill the Python interpreter 

During shutdown, Python modules are installed on None . This is an obscure Python behavior that was removed in 3.4 . In this example, terminating the main thread causes a shutdown, but the other thread still works, so it sees the modules as None .

Below is a simpler example that does the same thing by removing the module reference directly from sys.modules.

 import sys print sys del sys.modules['__main__'] print sys 

Answer 2

Copy the following code and paste it after from django.contrib.auth.models import User . If any code changes User to None, this code will print it.

 import inspect import sys class DetectDeletion(object): def __init__(self, obj): self.obj = obj def __getattribute__(self, a): return getattr(object.__getattribute__(self, 'obj'), a) def __del__(self): print inspect.getframeinfo(sys._getframe(1)) User = DetectDeletion(User)