Python and virtualenv madness

17 Mar 2017

Today I've been debugging an issue with virtualenv. The gist of the problem is that python -S's behaviours in virtualenv and in global python do not match.

In virtualenv, this causes an inexplicable problem like ImportError: no module named optparse, even though optparse seems to be in the standard library. subprocess and others have the same problem.

-S means “Disable the import of the module site and the site-dependent manipulations of sys.path that it entails.“. I think it means that only the built-in modules are imported automatically.

Let's see some examples. Without -S, here's the sys.path:

[tanin0] ~ 20:50:37 $ python -c ' import sys for p in sys.path: print p ' /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages/PILcompat /usr/lib/python2.7/dist-packages/gtk-2.0 /usr/lib/pymodules/python2.7 /usr/lib/python2.7/dist-packages/ubuntu-sso-client /usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode

With -S, the list of paths is a lot shorter:

[tanin0] ~ 20:50:48 $ python -S -c ' import sys for p in sys.path: print p ' /usr/lib/python2.7/ /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload

In virtualenv's env, the list of paths look equivalent:

(ENV) [tanin0] ~/projects/test-python (master) 20:52:35 $ python -S -c ' import sys for p in sys.path: print p ' /usr/local/google/home/tanin/projects/test-python/ENV/lib/python2.7/ /usr/local/google/home/tanin/projects/test-python/ENV/lib/python2.7/plat-x86_64-linux-gnu /usr/local/google/home/tanin/projects/test-python/ENV/lib/python2.7/lib-tk /usr/local/google/home/tanin/projects/test-python/ENV/lib/python2.7/lib-old /usr/local/google/home/tanin/projects/test-python/ENV/lib/python2.7/lib-dynload

Now here's the problem: Virtualenv defines the set of built-in modules differently from what python defines.

When we look at /usr/local/google/home/tanin/projects/test-python/ENV/lib/python2.7/, we see the list of modules that are considered as built-in by virtualenv.

[tanin0] ~/projects/test-python (master) 20:58:12 $ ls -l /usr/local/google/home/tanin/projects/test-python/ENV/lib/python2.7/ total 376 lrwxrwxrwx 1 tanin eng 29 Mar 16 20:58 _abcoll.py -> /usr/lib/python2.7/_abcoll.py -rw-r----- 1 tanin eng 29792 Mar 16 20:58 _abcoll.pyc lrwxrwxrwx 1 tanin eng 25 Mar 16 20:58 abc.py -> /usr/lib/python2.7/abc.py -rw-r----- 1 tanin eng 6660 Mar 16 20:58 abc.pyc lrwxrwxrwx 1 tanin eng 28 Mar 16 20:58 codecs.py -> /usr/lib/python2.7/codecs.py -rw-r----- 1 tanin eng 41003 Mar 16 20:58 codecs.pyc lrwxrwxrwx 1 tanin eng 30 Mar 16 20:58 copy_reg.py -> /usr/lib/python2.7/copy_reg.py -rw-r----- 1 tanin eng 5630 Mar 16 20:58 copy_reg.pyc drwxr-x--- 2 tanin eng 4096 Mar 16 20:58 distutils lrwxrwxrwx 1 tanin eng 28 Mar 16 20:58 encodings -> /usr/lib/python2.7/encodings lrwxrwxrwx 1 tanin eng 29 Mar 16 20:58 fnmatch.py -> /usr/lib/python2.7/fnmatch.py -rw-r----- 1 tanin eng 3816 Mar 16 20:58 fnmatch.pyc lrwxrwxrwx 1 tanin eng 33 Mar 16 20:58 genericpath.py -> /usr/lib/python2.7/genericpath.py -rw-r----- 1 tanin eng 3733 Mar 16 20:58 genericpath.pyc lrwxrwxrwx 1 tanin eng 30 Mar 16 20:58 lib-dynload -> /usr/lib/python2.7/lib-dynload lrwxrwxrwx 1 tanin eng 31 Mar 16 20:58 linecache.py -> /usr/lib/python2.7/linecache.py -rw-r----- 1 tanin eng 3534 Mar 16 20:58 linecache.pyc lrwxrwxrwx 1 tanin eng 28 Mar 16 20:58 locale.py -> /usr/lib/python2.7/locale.py -rw-r----- 1 tanin eng 52461 Mar 16 20:58 locale.pyc -rw-r----- 1 tanin eng 0 Mar 16 20:58 no-global-site-packages.txt lrwxrwxrwx 1 tanin eng 28 Mar 16 20:58 ntpath.py -> /usr/lib/python2.7/ntpath.py -rw-r----- 1 tanin eng 4 Mar 16 20:58 orig-prefix.txt lrwxrwxrwx 1 tanin eng 24 Mar 16 20:58 os.py -> /usr/lib/python2.7/os.py -rw-r----- 1 tanin eng 28048 Mar 16 20:58 os.pyc lrwxrwxrwx 1 tanin eng 31 Mar 16 20:58 posixpath.py -> /usr/lib/python2.7/posixpath.py -rw-r----- 1 tanin eng 12743 Mar 16 20:58 posixpath.pyc lrwxrwxrwx 1 tanin eng 24 Mar 16 20:58 re.py -> /usr/lib/python2.7/re.py -rw-r----- 1 tanin eng 14137 Mar 16 20:58 re.pyc drwxr-x--- 14 tanin eng 4096 Mar 16 20:58 site-packages -rw-r----- 1 tanin eng 27543 Mar 16 20:58 site.py -rw-r----- 1 tanin eng 25722 Mar 16 20:58 site.pyc lrwxrwxrwx 1 tanin eng 33 Mar 16 20:58 sre_compile.py -> /usr/lib/python2.7/sre_compile.py -rw-r----- 1 tanin eng 11585 Mar 16 20:58 sre_compile.pyc lrwxrwxrwx 1 tanin eng 35 Mar 16 20:58 sre_constants.py -> /usr/lib/python2.7/sre_constants.py -rw-r----- 1 tanin eng 6430 Mar 16 20:58 sre_constants.pyc lrwxrwxrwx 1 tanin eng 31 Mar 16 20:58 sre_parse.py -> /usr/lib/python2.7/sre_parse.py -rw-r----- 1 tanin eng 21083 Mar 16 20:58 sre_parse.pyc lrwxrwxrwx 1 tanin eng 25 Mar 16 20:58 sre.py -> /usr/lib/python2.7/sre.py lrwxrwxrwx 1 tanin eng 26 Mar 16 20:58 stat.py -> /usr/lib/python2.7/stat.py -rw-r----- 1 tanin eng 3221 Mar 16 20:58 stat.pyc lrwxrwxrwx 1 tanin eng 27 Mar 16 20:58 types.py -> /usr/lib/python2.7/types.py -rw-r----- 1 tanin eng 2788 Mar 16 20:58 types.pyc lrwxrwxrwx 1 tanin eng 30 Mar 16 20:58 UserDict.py -> /usr/lib/python2.7/UserDict.py -rw-r----- 1 tanin eng 10935 Mar 16 20:58 UserDict.pyc lrwxrwxrwx 1 tanin eng 30 Mar 16 20:58 warnings.py -> /usr/lib/python2.7/warnings.py -rw-r----- 1 tanin eng 14221 Mar 16 20:58 warnings.pyc lrwxrwxrwx 1 tanin eng 33 Mar 16 20:58 _weakrefset.py -> /usr/lib/python2.7/_weakrefset.py -rw-r----- 1 tanin eng 11934 Mar 16 20:58 _weakrefset.pyc

Now compare the list of what is inside /usr/lib/python2.7:

[tanin0] ~/projects/test-python (master) 20:58:17 $ ls -l /usr/lib/python2.7 total 8520 -rw-r--r-- 1 root root 17876 Oct 26 14:22 _abcoll.py -rw-r--r-- 1 root root 24794 Dec 5 05:14 _abcoll.pyc -rw-r--r-- 1 root root 24794 Jan 23 05:28 _abcoll.pyo -rw-r--r-- 1 root root 7145 Oct 26 14:22 abc.py -rw-r--r-- 1 root root 6121 Dec 5 05:14 abc.pyc -rw-r--r-- 1 root root 6065 Jan 23 05:28 abc.pyo -rw-r--r-- 1 root root 34231 Oct 26 14:22 aifc.py -rw-r--r-- 1 root root 30307 Dec 5 05:14 aifc.pyc -rw-r--r-- 1 root root 60 Oct 26 14:22 antigravity.py -rw-r--r-- 1 root root 201 Dec 5 05:14 antigravity.pyc -rw-r--r-- 1 root root 2663 Oct 26 14:22 anydbm.py -rw-r--r-- 1 root root 2794 Dec 5 05:14 anydbm.pyc -rw-r--r-- 1 root root 217 Oct 26 14:23 argparse.egg-info -rw-r--r-- 1 root root 88691 Oct 26 14:22 argparse.py -rw-r--r-- 1 root root 63859 Dec 5 05:14 argparse.pyc ... a lot of lines ... -rw-r--r-- 1 root root 18571 Oct 26 14:22 wave.py -rw-r--r-- 1 root root 19898 Dec 5 05:14 wave.pyc -rw-r--r-- 1 root root 12961 Oct 26 14:22 weakref.py -rw-r--r-- 1 root root 15254 Dec 5 05:14 weakref.pyc -rw-r--r-- 1 root root 5911 Oct 26 14:22 _weakrefset.py -rw-r--r-- 1 root root 9582 Dec 5 05:14 _weakrefset.pyc -rw-r--r-- 1 root root 9582 Jan 23 05:28 _weakrefset.pyo -rwxr-xr-x 1 root root 22723 Oct 26 14:22 webbrowser.py -rw-r--r-- 1 root root 19689 Dec 5 05:14 webbrowser.pyc -rw-r--r-- 1 root root 3379 Oct 26 14:22 whichdb.py -rw-r--r-- 1 root root 2237 Dec 5 05:14 whichdb.pyc drwxr-xr-x 2 root root 4096 Dec 5 05:14 wsgiref -rw-r--r-- 1 root root 187 Oct 26 14:22 wsgiref.egg-info -rw-r--r-- 1 root root 5563 Oct 26 14:22 xdrlib.py -rw-r--r-- 1 root root 9208 Dec 5 05:14 xdrlib.pyc drwxr-xr-x 6 root root 4096 Dec 5 05:14 xml -rw-r--r-- 1 root root 34865 Oct 26 14:22 xmllib.py -rw-r--r-- 1 root root 26739 Dec 5 05:14 xmllib.pyc -rw-r--r-- 1 root root 51546 Oct 26 14:22 xmlrpclib.py -rw-r--r-- 1 root root 43368 Dec 5 05:14 xmlrpclib.pyc -rw-r--r-- 1 root root 57986 Oct 26 14:22 zipfile.py -rw-r--r-- 1 root root 41351 Dec 5 05:14 zipfile.pyc

The number of files in /usr/lib/python2.7 is 8,520, which the number of files in ENV/lib/python2.7 is only 376.

Virtualenv's list of built-in modules is a lot smaller, and the inconsistency is the root cause of ImportError.

Give it a kudos