Import urllib.parse does not work when Python is launched from the command line

Question

I have observed the following behavior in python 3.4.2 and I cannot explain it. Hope someone can shed some light on this:

In IPython:

In [129]: import urllib In [130]: print(urllib.parse) <module 'urllib.parse' from '/Users/ashwin/.pyenv/versions/3.4.2/lib/python3.4/urllib/parse.py'> 

I imported the module and printed one of its attributes. Everything works as expected. Life is still good.

Now I do the same from the command line:

 $ python -c 'import urllib; print(urllib.parse)' Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: 'module' object has no attribute 'parse' 

Tell what?! this is not how it should work.
Well, perhaps this is python-level behavior; perhaps the modules are not immediately imported using the -c flag. Try another module:

 $ python -c 'import datetime; print(datetime.datetime)' <class 'datetime.datetime'> 

What?! How does this work for datetime , not urllib ? I use the same python version in both places (3.4.2)

Does anyone have any thoughts on this?

EDIT

According to one of the comments:

 $ which -a ipython /Users/ashwin/.pyenv/shims/ipython /Library/Frameworks/Python.framework/Versions/2.7/bin/ipython /usr/local/bin/ipython /usr/local/bin/ipython 

and

 $ which -a python /Users/ashwin/.pyenv/shims/python /Library/Frameworks/Python.framework/Versions/2.7/bin/python /usr/bin/python /usr/bin/python 

Answer 1

When import urllib it creates a module object of the urllib module (which is actually a package ) without importing its submodules (parsing, query, etc.).

You need the parent module object ( urllib ) to be in your namespace if you want to access its submodule using attribute access. In addition, this submodule should already be loaded (imported). From the documentation :

if the spam package has the foo submodule, after importing spam.foo , spam will have the foo attribute, which is associated with the submodule. [...] An invariant holding is that if you have sys.modules['spam'] and sys.modules['spam.foo'] (as you would after the specified import), the latter should be displayed as an attribute foo from first.

There is only one instance of each module, so any changes made to the urllib module urllib (stored in sys.modules['urllib'] ) are reflected everywhere.

You do not import urllib.parse , but IPython does. . To prove this, I'm going to create a startup file:

 import urllib print('Running the startup file: ', end='') try: # After importing 'urllib.parse' ANYWHERE, # 'urllib' will have the 'parse' attribute. # You could also do "import sys; sys.modules['urllib'].parse" urllib.parse except AttributeError: print("urllib.parse hasn't been imported yet") else: print('urllib.parse has already been imported') print('Exiting the startup file.') 

and run ipython

 vaultah@base :~$ ipython Running urllib/parse.py Running the startup file: urllib.parse has already been imported Exiting the startup file. Python 3.6.0a0 (default:089146b8ccc6, Sep 25 2015, 14:16:56) Type "copyright", "credits" or "license" for more information. IPython 4.0.0 -- An enhanced Interactive Python. 

This is a side effect of importing pydoc during IPython startup ( which ipython is / usr / local / bin / ipython):

 /usr/local/bin/ipython, line 7: from IPython import start_ipython /usr/local/lib/python3.6/site-packages/IPython/__init__.py, line 47: from .core.application import Application /usr/local/lib/python3.6/site-packages/IPython/core/application.py, line 24: from IPython.core import release, crashhandler /usr/local/lib/python3.6/site-packages/IPython/core/crashhandler.py, line 28: from IPython.core import ultratb /usr/local/lib/python3.6/site-packages/IPython/core/ultratb.py, line 90: import pydoc /usr/local/lib/python3.6/pydoc.py, line 68: import urllib.parse 

This explains why the code below does not work - you only import urllib and nothing will import urllib.parse :

 $ python -c 'import urllib; print(urllib.parse)' 

The following command, on the other hand, works because datetime.datetime not a module. This is the class that is imported during import datetime .

 $ python -c 'import datetime; print(datetime.datetime)' 

Answer 2

urllib.parse is available from Python 3 onwards. I think you might need import urllib.parse rather than import urllib . Not sure if (if) importing submodules is implicit.

I would suggest that IPython imports urllib.parse at startup, and that is why it is available.

parse is not an attribute, but

 Python 3.4.2 (default, Oct 15 2014, 22:01:37) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import urllib >>> urllib.parse Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute 'parse' >>> import urllib.parse >>> urllib.parse <module 'urllib.parse' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/parse.py'>