How to recover a freeze exception?

Question

I am installing python 2.7 packages on an ubuntu 14 system and run into a problem in which "pip freeze" produces output like this:

$ pip freeze Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags) File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist assert len(specs) == 1 and specs[0][0] == '==' AssertionError Storing debug log for failure in /home/ubuntu/.pip/pip.log $ cat /home/ubuntu/.pip/pip.log ------------------------------------------------------------ /usr/bin/pip run on Tue Feb 9 21:00:36 2016 Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags) File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist assert len(specs) == 1 and specs[0][0] == '==' AssertionError 

I'm not sure what caused this state of things.
My suspicion is that a crash of easy_install might cause damage.

I tried reinstalling pip.
It had no effect.

I suspect that I will need to do a full uninstall and reinstall all the python packages through pubon and the python related Ubuntu packages through apt-get. package for packing.

It will be quite a long time, and I'm not sure that he will achieve the desired result.

Is there a better approach to this problem?

Here are the packages that I installed before running easy_install:

 sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev protobuf-compiler gfortran libjpeg62 libfreeimage-dev libatlas-base-dev git python-dev python-pip libgoogle-glog-dev libbz2-dev libxml2-dev libxslt-dev libffi-dev libssl-dev libgflags-dev liblmdb-dev python-yaml python-numpy 

Here is the initial exception I received from easy_install:

 $ sudo easy_install pillow Searching for pillow Reading https://pypi.python.org/simple/pillow/ Best match: Pillow 3.1.1 Downloading https://pypi.python.org/packages/source/P/Pillow/Pillow-3.1.1.zip#md5=3868f54fd164e65f95fbcb32f62940ae Processing Pillow-3.1.1.zip Writing /tmp/easy_install-Bvu_2f/Pillow-3.1.1/setup.cfg Running Pillow-3.1.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Bvu_2f/Pillow-3.1.1/egg-dist-tmp-lMm5Tk warning: no previously-included files found matching '.editorconfig' Building using 4 processes _imaging.c: In function 'getink': _imaging.c:474:9: warning: 'r' may be used uninitialized in this function [-Wmaybe-uninitialized] int r, g, b, a; ^ libImaging/Resample.c:87:45: warning: always_inline function might not be inlinable [-Wattributes] static float __attribute__((always_inline)) i2f(int v) { ^ Building using 4 processes Building using 4 processes Building using 4 processes -------------------------------------------------------------------- PIL SETUP SUMMARY -------------------------------------------------------------------- version Pillow 3.1.1 platform linux2 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] -------------------------------------------------------------------- *** TKINTER support not available --- JPEG support available *** OPENJPEG (JPEG2000) support not available --- ZLIB (PNG/ZIP) support available --- LIBTIFF support available --- FREETYPE2 support available *** LITTLECMS2 support not available *** WEBP support not available *** WEBPMUX support not available -------------------------------------------------------------------- To add a missing option, make sure you have the required library, and set the corresponding ROOT variable in the setup.py script. To check the build, run the selftest.py script. Adding Pillow 3.1.1 to easy-install.pth file Installing pildriver.py script to /usr/local/bin Installing viewer.py script to /usr/local/bin Installing gifmaker.py script to /usr/local/bin Installing pilconvert.py script to /usr/local/bin Installing pilfont.py script to /usr/local/bin Installing pilfile.py script to /usr/local/bin Installing createfontdatachunk.py script to /usr/local/bin Installing explode.py script to /usr/local/bin Installing pilprint.py script to /usr/local/bin Installing player.py script to /usr/local/bin Installing thresholder.py script to /usr/local/bin Installing painter.py script to /usr/local/bin Installing enhancer.py script to /usr/local/bin Installed /usr/local/lib/python2.7/dist-packages/Pillow-3.1.1-py2.7-linux-x86_64.egg Processing dependencies for pillow Finished processing dependencies for pillow Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 330, in _handle_workers debug('worker handler exiting') TypeError: 'NoneType' object is not callable Exception Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 357, in _handle_tasks debug('task handler got sentinel') TypeError: 'NoneType' object is not callable TypeError: TypeError("'NoneType' object does not support item deletion",) in <Finalize object, dead> ignored ubuntu@ip-10-234-31-217 :~/nvidia_installers/cuda$ 

Subsequently, I was able to install the pillow through the pipeline.

I noticed that there are apparently 2 different dist packages in my path. This seems to be a little strange.

 >>> import sys >>> import pprint >>> pp = pprint.PrettyPrinter(indent=4) >>> pp.pprint(sys.path) [ '', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages'] 

Here is the complete procedure that I am following:
Install Caffe on EC2 from scratch

I understand that the pip version supported by apt-get is very old. I tried updating. I get OS Owners errors.
I am not sure how to deviate from using apt-get here and the consequences of this.

 $ pip install -U pip Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-8.0.2-py2.py3-none-any.whl#md5=2056f553d5b593d3a970296f229c1b79 Downloading pip-8.0.2-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded Installing collected packages: pip Found existing installation: pip 1.5.4 Not uninstalling pip at /usr/lib/python2.7/dist-packages, owned by OS Can't roll back pip; was not uninstalled Cleaning up... Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run requirement_set.install(install_options, global_options, root=options.root_path) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1436, in install requirement.install(install_options, global_options, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 672, in install self.move_wheel_files(self.source_dir, root=root) File "/usr/lib/python2.7/dist-packages/pip/req.py", line 902, in move_wheel_files pycompile=self.pycompile, File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 206, in move_wheel_files clobber(source, lib_dir, True) File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 193, in clobber os.makedirs(destsubdir) File "/usr/lib/python2.7/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip-8.0.2.dist-info' Storing debug log for failure in /home/ubuntu/.pip/pip.log $ sudo pip install -U pip Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-8.0.2-py2.py3-none-any.whl#md5=2056f553d5b593d3a970296f229c1b79 Downloading pip-8.0.2-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded Installing collected packages: pip Found existing installation: pip 1.5.4 Not uninstalling pip at /usr/lib/python2.7/dist-packages, owned by OS Successfully installed pip Cleaning up... $ pip --version pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7) ubuntu@ip-10-234-31-217 :~$ pip freeze Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags) File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist assert len(specs) == 1 and specs[0][0] == '==' AssertionError Storing debug log for failure in /home/ubuntu/.pip/pip.log 

Another update method failed:

 $ sudo pip install --upgrade pip The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo -H flag. The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo -H flag. /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning. SNIMissingWarning /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Requirement already up-to-date: pip in /usr/local/lib/python2.7/dist-packages /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning $ pip freeze Exception: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/freeze.py", line 74, in run req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags) File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 286, in from_dist assert len(specs) == 1 and specs[0][0] == '==' AssertionError Storing debug log for failure in /home/ubuntu/.pip/pip.log $ ls -lF /home/ubuntu/.cache/pip/http ls: cannot access /home/ubuntu/.cache/pip/http: No such file or directory $ ls -lF /home/ubuntu/.cache total 0 -rw-r--r-- 1 ubuntu ubuntu 0 Feb 6 00:39 motd.legal-displayed 

Answer 1

To install pip on debian based linux distributions (including ubuntu), I had good results using easy_install from the python-setuptools . So it usually looks like this:

 sudo apt-get install python-setuptools sudo easy_install pip 

It looks like you already have easy_install. In any case, before attempting to install pip this way, you should try to clear your current setup as much as possible. I removed the python-pip package, deleted any other traces of the updated pip, and also deleted the traces of the pillow module, since this is what you installed when it all started incorrectly. Maybe something like:

 sudo apt-get remove --purge python-pip sudo rm -rf /usr/local/lib/python2.7/{dist,site}-packages/pip* sudo rm -rf /usr/local/lib/python2.7/{dist,site}-packages/pillow* 

Finally, when you use easy_install or pip to install something that includes a command in addition to the imported python module, the command will be placed in /usr/local/bin , so you need to make sure /usr/local/bin is in your shell PATH . There will be a pip command. (I would use easy_install to install pip, and then use pip to install any other python package.)

EDIT: by the way, the "initial exception from the easy_install cushion" is no exception and an error. It shows some non-fatal compiler warnings, and some functions were not included, but it seems to have worked. These were probably later actions with pip that caused the problem for pip.

much later edit 2: I had a “brain” and wrote /usr/local/lib , where I meant /usr/local/bin in the previous initial paragraph above. Fixed.