pip doesn't know of apt-get installed packages

Question

On a quite fresh Ubuntu 16.04, I'm installing ipython with pip, as a user installation. Pip itself was installed from the python-pip Ubuntu package (8.1.1), and so are some dependencies like pygments or setuptools (20.7.0).

My question: is it normal that when doing a user installation with pip, apt-get installed packages are not detected? Is is a known bug?

This is the command I run (as a user, not as root):

$ pip install ipython

→ I get the download of many packages from PyPI, including setuptool (27.3.0) and Pygments (2.1.3). I don't think it's a version issue, since ipython requires setuptools>=18.5 only. Incidentally, I also get a complaint that pip should be updated to latest version (8.1.2).

Even more intriguing, when I run the same command again, I get the same installation process (only difference: wheels are cached). Instead, I would expect pip telling me that ipython is already installed.

Notice that there is no doubt that ipython (version 5.1.0) is indeed installed in my ~/.local directory, and it runs well (the only adjustment I had to make was adding ~/.local/bin to the PATH variable in ~/.bashrc as mentioned in a dedicated question).

It seems to me that something is wrong with the way pip detects apt-get installed packages, but I cannot figure out what. Am I missing something obvious?

If it can help the diagnostic, this is the Python module path (modelica is the username):

python -c "import sys; print sys.path"
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/modelica/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/omniORB/COS', '/usr/lib/python2.7/dist-packages/gtk-2.0']

Answer 1

is it normal that when doing a user installation with pip, apt-get installed packages are not detected? Is is a known bug?

I just dived into pip's code and found that pip's list command filters a list of packages provided by pkg_resources.WorkingSet(), which in turn gets its entries from sys.path. The result is that pip --list will also list system packages installed by APT. But that doesn't mean pip should manage this packages.

Pip and APT work differently, starting with where they install to: pip installs globally to /usr/lib/pythonX.Y/site-packages or /usr/local/lib/pythonX.Y/dist-packages (depending on version) and locally to ~/.local/lib/pythonX.Y/site-packages, while APT based tools install globally to /usr/lib/pythonX.Y/dist-packages.

This oddity relates to a design decision by Debian developers in order to avoid conflicts between packages provided by them and packages obtained by other means (this Stack Overflow answer pointed me in the right direction).

So no, it isn't a bug that pip lets you install packages to ~/.local/lib/pythonX.Y/site-packages even if a package is already installed in /usr/lib/pythonX.Y/dist-packages.

---

Lets see with your ipython example. I also have installed it previously with APT:

DEB Package

$ dpkg --get-selections | grep ipython
ipython                     install

$ dpkg -s ipython | grep Version
Version: 2.4.1-1

$ dpkg-query -L ipython
(...)
/usr/lib/python2.7/dist-packages
(...)
/usr/bin/ipython
(...)

Pip Package

$ pip show -f ipython
(...)
Name: ipython
Version: 5.3.0
(...)
Installer: pip
(..)
Location: /usr/local/lib/python2.7/dist-packages
Files:
  ../../../bin/ipython

Note: ../../../bin/ipython ends up being /usr/local/bin/ipython.

The command whereis gives me both IPython instances:

$ whereis -b ipython
ipython: /usr/bin/ipython /usr/local/bin/ipython

And I could still get a third IPython on ~/.local if I wanted!

---

Even more intriguing, when I run the same command again, I get the same installation process (only difference: wheels are cached). Instead, I would expect pip telling me that ipython is already installed.

This seems odd indeed. Here's what I have observed so far: when a package has already been installed by pip locally, it appears to install every time (it always outputted "Successfully installed {package}"), but it really seems to just alter the packages dist-info. For example, I tried installing lxml locally (either with pip install lxml or pip install --user lxml) a couple of times:

Compare the timestamp from the first lxml installation with the one after the last lxml "installation":

1.

myusr@myhost:~$ ls -al .local/lib/python2.7/site-packages | grep lxml
drwxrwxr-x  5 myusr mygroup  4096 feb 20 15:01 lxml
drwxrwxr-x  2 myusr mygroup  4096 feb 20 15:01 lxml-3.7.3.dist-info

2.

myusr@myhost:~$ ls -al .local/lib/python2.7/site-packages | grep lxml
drwxrwxr-x  5 myusr mygroup  4096 feb 20 15:01 lxml
drwxrwxr-x  2 myusr mygroup  4096 feb 20 15:03 lxml-3.7.3.dist-info

However, when one tries to install a package globally, pip does show a message:

Requirement already satisfied (use --upgrade to upgrade): ipython in /usr/local/lib/python2.7/dist-packages

Answer 2

Try

python3 -m pip install ipython

For Me Working