Ubuntu 16.04: Software updater and pip stopped working

Question

Today I upgraded the Python library matplotlib with the following command:

sudo pip3 install --upgrade matplotlib

Everything looked normal during the installation process. But since then, I experience two problems. One is that pip doesn't work any more. Every pip command evokes the same error message:

    Traceback (most recent call last):
  File "/usr/local/bin/pip3", line 7, in <module>
    from pip import main
  File "/usr/local/lib/python3.5/dist-packages/pip/__init__.py", line 26, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "/usr/local/lib/python3.5/dist-packages/pip/utils/__init__.py", line 27, in <module>
    from pip._vendor import pkg_resources
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3018, in <module>
    @_call_aside
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3004, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 3046, in _initialize_master_working_set
    dist.activate(replace=False)
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2578, in activate
    declare_namespace(pkg)
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2152, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2092, in _handle_ns
    _rebuild_mod_path(path, packageName, module)
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2121, in _rebuild_mod_path
    orig_path.sort(key=position_in_sys_path)
AttributeError: '_NamespacePath' object has no attribute 'sort

Additionally, Ubuntu's Software Updater stopped working. It looks similar to this problem, only the solution won't work, because, well, pip doesn't work. But I guess, the Software Updater problem will disappear, when the pip problem is solved.

I've tried so far the following commands from threads with similar problems to no avail

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -f install
sudo apt-get install --reinstall aptdaemon
sudo rm -rvf /var/lib/apt/lists/*
sudo apt install --reinstall python3-pip

And obviously no pip command will work, e.g.

sudo pip3 install --upgrade pip setuptools wheel
pip3 install --upgrade pip
pip3 install --upgrade setuptools

Answer 1

I found, what looked like an answer that solved my problem, but it is rather a sledgehammer method that doesn't help in the long run. Deleting a whole directory is probably not the most elegant way:

sudo rm -rf /usr/local/lib/python3.5/dist-packages

Afterwards pip was not installed, so I reinstalled it with

sudo apt-get update
sudo apt-get install python3-pip 

pip was back and working and the problem with the Software Updater was as expected resolved as well. This "solution" works, though it removes many libraries like matplotlib, which have to be reinstalled afterwards. But when I reinstalled matplotlib 2.1.2, the same pip error was induced. I really can't recommend this approach.

tl; dr: Don't do this.

So back to square one. What resolved the problem was to reinstall pip and every following library with sudo -H, e.g.

sudo -H apt install --reinstall python3-pip

Without the -H flag, an installation of matplotlib evoked the same pip problem. But still I got the error message. The solution seems to be this modification originating here:

I edited line #2121~2122 of this file:
/usr/local/lib/python3.5/dist-packages/pip/_vendor/pkg_resources/__init__.py

#orig_path.sort(key=position_in_sys_path)
#module.__path__[:] = [_normalize_cached(p) for p in orig_path]
orig_path_t = list(orig_path)
orig_path_t.sort(key=position_in_sys_path)
module.__path__[:] = [_normalize_cached(p) for p in orig_path_t]

This revived pip/pip3 and I was able to install modules. So far, everything is working. Let's hope, this will be the last of it.

Edit: One week later, the problem hasn't resurfaced, so I mark this as the accepted answer.

Edit 2: A pip update revived the problem. Seemingly, the update had overwritten the modification. It was no problem to reintroduce it again and get rid of the aberrant behaviour.