1

On a quite fresh Ubuntu 16.04, I'm installing ipython with pip, as a user installation. Pip itself was installed from the python-pip Ubuntu package (8.1.1), and so are some dependencies like pygments or setuptools (20.7.0).

My question: is it normal that when doing a user installation with pip, apt-get installed packages are not detected? Is is a known bug?

This is the command I run (as a user, not as root):

$ pip install ipython

→ I get the download of many packages from PyPI, including setuptool (27.3.0) and Pygments (2.1.3). I don't think it's a version issue, since ipython requires setuptools>=18.5 only. Incidentally, I also get a complaint that pip should be updated to latest version (8.1.2).

Even more intriguing, when I run the same command again, I get the same installation process (only difference: wheels are cached). Instead, I would expect pip telling me that ipython is already installed.

Notice that there is no doubt that ipython (version 5.1.0) is indeed installed in my ~/.local directory, and it runs well (the only adjustment I had to make was adding ~/.local/bin to the PATH variable in ~/.bashrc as mentioned in a dedicated question).

It seems to me that something is wrong with the way pip detects apt-get installed packages, but I cannot figure out what. Am I missing something obvious?

If it can help the diagnostic, this is the Python module path (modelica is the username):

python -c "import sys; print sys.path"
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/modelica/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/omniORB/COS', '/usr/lib/python2.7/dist-packages/gtk-2.0']
  • The latest version of pip is 8.1.2, so the python-pip package is indeed slightly outdated. You can ignore it if the pip you have still works. However, if you do plan to upgrade it, I would suggest using a virtualenv to do it. As for running pip install ipython a second time, I am surprised pip doesn't just tell you ipython is already installed; it definitely does say that for me in a virtualenv. Can you try again to make sure this particular behavior is reproducible? – edwinksl Sep 24 '16 at 8:00
  • To move things forward, I've reported this behavior as a bug report at bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1628432 I've expanded the title a bit because pip doesn't recognize any preinstalled packages, not just apt-get installed packages. – Pierre H. Sep 28 '16 at 9:27
  • Thanks @edwinksl for your feedback. The behavior of repeated installation of packages already installed is reproducible (on this VM!) and it's not just ipython. However, I'm not in a virtualenv. Finally, as I mentioned in the bug report, I don't feel comfortable updating pip with a broken pip. – Pierre H. Sep 28 '16 at 9:31
1

NOTE: Like OP, I'm using Ubuntu 16.04, Python 2.7 and Pip 8.1.1.

is it normal that when doing a user installation with pip, apt-get installed packages are not detected? Is is a known bug?

I just dived into pip's code and found that pip's list command filters a list of packages provided by pkg_resources.WorkingSet(), which in turn gets its entries from sys.path. The result is that pip --list (without options) will also list system packages installed by APT. So, in a way, pip does detect them. This, however, doesn't mean that pip can manage this packages, similar to how APT, RPM and other package management systems aren't interoperable.

Pip can't manage DEB packages installed with APT, even if they relate to Python, because there are enough differences regarding how each package manager works to prevent this; I don't know all of them, but one that is confusing is that pip installs globally to /usr/lib/pythonX.Y/site-packages or /usr/local/lib/pythonX.Y/dist-packages (depending on version) and locally to ~/.local/lib/pythonX.Y/site-packages, while DEB packages get installed globally to /usr/lib/pythonX.Y/dist-packages.

This oddity relates to a design decision by Debian developers in order to, precisely, avoid conflicts between packages provided by them and installed using their packaging system and packages obtained by other means (this Stack Overflow answer pointed me in the right direction).

So no, it isn't a bug that pip lets you install packages to ~/.local/lib/pythonX.Y/site-packages even if a package is already installed in /usr/lib/pythonX.Y/dist-packages.


Lets see with your ipython example. I also have installed it previously with APT:

DEB Package

$ dpkg --get-selections | grep ipython
ipython                     install

$ dpkg -s ipython | grep Version
Version: 2.4.1-1

$ dpkg-query -L ipython
(...)
/usr/lib/python2.7/dist-packages
(...)
/usr/bin/ipython
(...)

Pip Package

$ pip show -f ipython
(...)
Name: ipython
Version: 5.3.0
(...)
Installer: pip
(..)
Location: /usr/local/lib/python2.7/dist-packages
Files:
  ../../../bin/ipython

Note: ../../../bin/ipython ends up being /usr/local/bin/ipython.

As expected, whereis gives me both IPython instances:

$ whereis -b ipython
ipython: /usr/bin/ipython /usr/local/bin/ipython

And I could still get a third IPython on ~/.local!


Even more intriguing, when I run the same command again, I get the same installation process (only difference: wheels are cached). Instead, I would expect pip telling me that ipython is already installed.

This seems odd indeed and I'm still looking into it, but here's what I have observed so far: when a package has already been installed by pip locally, it appears to install every time (it always outputted "Successfully installed {package}"), but it really seems to just alter the packages dist-info. For example, I tried installing lxml locally (either with pip install lxml or pip install --user lxml) a couple of times:

Compare the timestamp of:

myusr@myhost:~$ ls -al .local/lib/python2.7/site-packages | grep lxml
drwxrwxr-x  5 myusr mygroup  4096 feb 20 15:01 lxml
drwxrwxr-x  2 myusr mygroup  4096 feb 20 15:01 lxml-3.7.3.dist-info

(This is after I first installed lxml.)

With:

myusr@myhost:~$ ls -al .local/lib/python2.7/site-packages | grep lxml
drwxrwxr-x  5 myusr mygroup  4096 feb 20 15:01 lxml
drwxrwxr-x  2 myusr mygroup  4096 feb 20 15:03 lxml-3.7.3.dist-info

(This is after the last "install" of lxml.)

However, when one tries to install a package globally, pip does show a message:

Requirement already satisfied (use --upgrade to upgrade): ipython in /usr/local/lib/python2.7/dist-packages

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.