7

I'm still trying to understand this problem I'm having and have read a ton of posts regarding sudo apt-get install python-XXX vs pip install, and I understand the jist of it (apt-get is a package repo, managed by completely different group of people to pip, auto-updates along with everything else when there is an update available, it's binary etc). However there are a few particular things which I don't understand and haven't been able to find answers to:

  1. If I, for whatever reason, first install a package, e.g. numpy, with sudo apt-get install numpy, and then I later pip install numpy. What happens? is numpy installed again elsewhere with different version (e.g. in .local/lib), so now I have two independent versions? Which one is used when I import? (if I've understood correctly it is separate and apt-get is system wide, but I've seen conflicting replies to this question, e.g. here)

  2. I noticed a lot of the packages on apt-get are quite old compared to pip. e.g. at time of writing ipython 2.3.0 vs 4.0.3, numpy 1.10.2 vs 1.10.4, spyder 2.3.5 vs 2.3.8. Is there any advantage to first sudo apt-get install followed by a pip install? or is that just stupid?

  3. I noticed that my pip was 1.5.6 - it's the current latest if you do sudo apt-get install python-pip. whereas on pip it's 8.0.2! I believe this has been causing me problems, because with that old version you can install the same package over and over again and it doesn't warn. And somehow it doesn't write over the old one. I realized this because I can keep uninstalling the same package (e.g. numpy) and it's still always there when I do pip show numpy. And oddly, always showing the same folder (.local/), but with different versions :S So I have two questions related to this:

    a. what is the best way to install pip? I'm currently leaning towards sudo apt-get install python-pip, pip install pip, sudo apt-get purge python-pip. Is that wise? It seems a bit ridiculous to me.

    b. now that I know a lot of my packages are messed up with duplicate installs, probably because some were installed with apt-get and then later multiple times with pip 1.5.6, what's the best way to check for duplicates and clean it up? I'm guessing it's not safe to use a script to pip uninstall all and re-install?

P.S. I wasn't sure whether I should post this as a new question or as a response to my original problem. Since this question is a lot more concise and general I felt it would be better separate. Apologies if I was mistaken.

  • Possible duplicate of apt get install vs pip install – muru Jan 26 '16 at 13:59
  • I actually reference that post in my question, it was very helpful in clearing a lot of my confusion, but it doesn't answer my questions above (at least I couldn't see it) – memo Jan 26 '16 at 14:03
  • Oh, in your other question. I just looked at the list of linked posts and didn't see it there. – muru Jan 26 '16 at 14:04
7

For the archives, after much trial and error this is what I've found (in retrospect, some bits make sense, others quite confusing).

  • If you sudo apt-get install python-xyz, the package will appear in pip list. Of course you can import it in python, but it doesn't appear in pip.get_installed_distributions(). (Obviously the opposite is not true. i.e. if you do pip install xyz it will not appear in your apt/synaptic)

  • After sudo apt-get install python-xyz if you do pip install xyz what happens depends on the version of pip you have.

OLD v1.5.6 (the current version shipping on Ubuntu PPA):

  • This version of pip (1.5.6) will just install a new copy of xyz in a different location. You end up with multiple copies and this causes a huge mess. e.g:

    • numpy from apt-get is 1.8.2 at /usr/local/lib/python2.7/dist-packages
    • numpy from pip is 1.10.4 at ~/.local/lib/python2.7/site-packages
  • If I do pip install numpy yet again, it downloads and installs it again. So you could end up with many different versions which you can't really access. I can do pip install numpy 5x times, and then I can do pip uninstall numpy 5x times! Obviously you can pay attention to not do that, but sometimes other software's install scripts are a bit careless and can mess things up. Unbelievable that Ubuntu officially ships this version of pip.

NEW v8.0.2 (the current version on pip itself):

  • newer versions of pip (e.g. 8.0.2 which is on pip) will refuse to install the same package saying requirement is already met. So you cannot install a new version. This is good behaviour (more on getting this version of pip later).

  • In this case you can only upgrade, i.e. install with -U flag.

  • However when you try to pip install -U xyz on a package installed with sudo apt-get, you'll get permissions error because the apt-get was installed in /usr/ and you need root access to write there.

  • So AFAIK you have no choice but to sudo pip install -U xyz to be able to update it. In this case pip installs the latest package into the same place as sudo apt-get wrote it. e.g. in my case /usr/local/lib/python2.7/dist-packages. This is good.

  • It's also worth pointing out that the packages on apt are often quite older than those on pip (e.g. numpy v1.8.2 vs 1.10.4, scipy 0.14.1 vs 0.17.0, ipython 2.3 vs 4.0.3, spyder 2.3.5 vs 2.3.8)

So my current thoughts are to get the big things with sudo apt-get, e.g. sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose spyder

and then update them (or at least some of them) with sudo pip install -U.

NB It seems quite important to get the new pip from pip (very meta)

sudo apt-get install python-pip
sudo pip install -U pip

P.S. I am aware of virtualenv but I have no need for it right now. I need only one development environment.

And here is a little script to dump a list of packages, version and path (but only works on pip installed modules, not those from apt-get)

import pip
pp = pip.get_installed_distributions()
for p in sorted([p.location+"\t"+p.project_name+" ("+p.version+")" for p in pp]):
    print p

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.