2

On a quite fresh Ubuntu 16.04, I'm installing ipython with pip, as a user installation. Pip itself was installed from the python-pip Ubuntu package (8.1.1), and so are some dependencies like pygments or setuptools (20.7.0).

My question: is it normal that when doing a user installation with pip, apt-get installed packages are not detected? Is is a known bug?

This is the command I run (as a user, not as root):

$ pip install ipython

→ I get the download of many packages from PyPI, including setuptool (27.3.0) and Pygments (2.1.3). I don't think it's a version issue, since ipython requires setuptools>=18.5 only. Incidentally, I also get a complaint that pip should be updated to latest version (8.1.2).

Even more intriguing, when I run the same command again, I get the same installation process (only difference: wheels are cached). Instead, I would expect pip telling me that ipython is already installed.

Notice that there is no doubt that ipython (version 5.1.0) is indeed installed in my ~/.local directory, and it runs well (the only adjustment I had to make was adding ~/.local/bin to the PATH variable in ~/.bashrc as mentioned in a dedicated question).

It seems to me that something is wrong with the way pip detects apt-get installed packages, but I cannot figure out what. Am I missing something obvious?

If it can help the diagnostic, this is the Python module path (modelica is the username):

python -c "import sys; print sys.path"
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/modelica/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/omniORB/COS', '/usr/lib/python2.7/dist-packages/gtk-2.0']
  • The latest version of pip is 8.1.2, so the python-pip package is indeed slightly outdated. You can ignore it if the pip you have still works. However, if you do plan to upgrade it, I would suggest using a virtualenv to do it. As for running pip install ipython a second time, I am surprised pip doesn't just tell you ipython is already installed; it definitely does say that for me in a virtualenv. Can you try again to make sure this particular behavior is reproducible? – edwinksl Sep 24 '16 at 8:00
  • To move things forward, I've reported this behavior as a bug report at bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1628432 I've expanded the title a bit because pip doesn't recognize any preinstalled packages, not just apt-get installed packages. – Pierre H. Sep 28 '16 at 9:27
  • Thanks @edwinksl for your feedback. The behavior of repeated installation of packages already installed is reproducible (on this VM!) and it's not just ipython. However, I'm not in a virtualenv. Finally, as I mentioned in the bug report, I don't feel comfortable updating pip with a broken pip. – Pierre H. Sep 28 '16 at 9:31
1

is it normal that when doing a user installation with pip, apt-get installed packages are not detected? Is is a known bug?

I just dived into pip's code and found that pip's list command filters a list of packages provided by pkg_resources.WorkingSet(), which in turn gets its entries from sys.path. The result is that pip --list will also list system packages installed by APT. But that doesn't mean pip should manage this packages.

Pip and APT work differently, starting with where they install to: pip installs globally to /usr/lib/pythonX.Y/site-packages or /usr/local/lib/pythonX.Y/dist-packages (depending on version) and locally to ~/.local/lib/pythonX.Y/site-packages, while APT based tools install globally to /usr/lib/pythonX.Y/dist-packages.

This oddity relates to a design decision by Debian developers in order to avoid conflicts between packages provided by them and packages obtained by other means (this Stack Overflow answer pointed me in the right direction).

So no, it isn't a bug that pip lets you install packages to ~/.local/lib/pythonX.Y/site-packages even if a package is already installed in /usr/lib/pythonX.Y/dist-packages.


Lets see with your ipython example. I also have installed it previously with APT:

DEB Package

$ dpkg --get-selections | grep ipython
ipython                     install

$ dpkg -s ipython | grep Version
Version: 2.4.1-1

$ dpkg-query -L ipython
(...)
/usr/lib/python2.7/dist-packages
(...)
/usr/bin/ipython
(...)

Pip Package

$ pip show -f ipython
(...)
Name: ipython
Version: 5.3.0
(...)
Installer: pip
(..)
Location: /usr/local/lib/python2.7/dist-packages
Files:
  ../../../bin/ipython

Note: ../../../bin/ipython ends up being /usr/local/bin/ipython.

The command whereis gives me both IPython instances:

$ whereis -b ipython
ipython: /usr/bin/ipython /usr/local/bin/ipython

And I could still get a third IPython on ~/.local if I wanted!


Even more intriguing, when I run the same command again, I get the same installation process (only difference: wheels are cached). Instead, I would expect pip telling me that ipython is already installed.

This seems odd indeed. Here's what I have observed so far: when a package has already been installed by pip locally, it appears to install every time (it always outputted "Successfully installed {package}"), but it really seems to just alter the packages dist-info. For example, I tried installing lxml locally (either with pip install lxml or pip install --user lxml) a couple of times:

Compare the timestamp from the first lxml installation with the one after the last lxml "installation":

1.

myusr@myhost:~$ ls -al .local/lib/python2.7/site-packages | grep lxml
drwxrwxr-x  5 myusr mygroup  4096 feb 20 15:01 lxml
drwxrwxr-x  2 myusr mygroup  4096 feb 20 15:01 lxml-3.7.3.dist-info

2.

myusr@myhost:~$ ls -al .local/lib/python2.7/site-packages | grep lxml
drwxrwxr-x  5 myusr mygroup  4096 feb 20 15:01 lxml
drwxrwxr-x  2 myusr mygroup  4096 feb 20 15:03 lxml-3.7.3.dist-info

However, when one tries to install a package globally, pip does show a message:

Requirement already satisfied (use --upgrade to upgrade): ipython in /usr/local/lib/python2.7/dist-packages

0

Try

python3 -m pip install ipython

For Me Working

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.