I am providing an answer here not because N0rberts answer was bad. On the contrary, N0rberts answer was 1. inspirative and 2. very useful to me, hence I will accept is as an answer to a question because essentially it is that. Thank you N0rbert for support and patience.
My answer here is to shed a light on some side effects and edge cases.
I could do this in question but I am of opinion that question shall remain as question and answer is different thing. Since there is a lot of ground to cover and I have to dig some examples from my logs, this answer until it is complete will be edited several times until I come to the point of what has happened and how I overcome obstacles.
Review of what has happened
That I have learned on the journey of fixing...
In one moment I have realized that I cannot install or update anything on my Ubuntu14 (it could easily be a newer one)
I made a theory of what may be wrong because I was trying to build some new software and things were breaking so I was trying to install various fixes to that build. The things simply started breaking but not breaking critically.
lesson1: what it turned to be influential to the cause of the problem was coming from different angle, not from the theory I had. I thought that my recent installations of LLVM influenced python while culprit of the problem was PPA repository that I have installed to solve something else. The problem is that PPA in question I installed a while ago (1y+) and forgot that it is there. Silently Ubuntu was updating what's new (even when that is not really recommended). That bad decision of mine will be more elaborated later.
-- Moral of the lesson: do not theorize what it may be - try to investigate and ponder facts carefully.
- Tried various recipes until I came in the state of no recovery. In essence I have broken 2 essential things in my Linux: 1.
dpkg and 2. system python. In one moment whole system broke to the point that I lost graphical environment, first some apps start breaking (this was happening over period of couple of months) and finally at the lowest point I end-up that I could run my Ubuntu only from console terminals (Alt-F1 to Alt-F4). Being on linux for a long time I was originally dwelling on the command line. Of course things are way easier when you have graphics, mainly because you can run browser to research potential solutions. I am programmer and my specie is known to be a bit dogmatic (dogmatic = believe in automation of dogs :) ) this means I will cling to some thought ignoring potential facts.
lesson2: I am of belief that there are 2 ways of fixing thing when OS is in question. 1. Reinstall the whole thing 2. try to fix. I have tried both in the past. Lately, I am realizing that option 2 has more value. By trying to fix things I learned a lot more about OS. I was to the certain extent ignorant. Now, If I just reinstalled OS I would eventually come to more problematic situation which would eventually instigate clean installation from scratch which means all fine tuning and all sorts of other data will be lost.
-- Moral of the lesson: Learn, do not take easy way forward, be bold. Ultimately, it is rewarding.
- I have communicated problem here and talked to real people. Thank you JonathonF for valuable insights into PPA purpose and providing information about archive.
Lesson3: During investigation I learned that Jonathon does not really recommend his PPA to be used and if you do, do it on your own risk. In fact my Linux is not mission critical - it is my hobby computer (but it is valuable to me as I use it for all sort of things: from online banking via fun to evaluating software or programming languages that I would use professionally.)
-- Moral of the lesson: if you cannot figure everything yourself - talk to people. There are valuable hints in what they do and what they will tell you. But try to read carefully, especially purpose of PPA's.
The way to recovery
- the first thing that made difference was to repair
dpkg.
The beginning of NOrberts answer addresses that. Download .deb, extract files and do manually what all these package tools will do for you automatically.
sudo apt-get install dpkg --reinstall
vicious circle
did not work for me because apt looks to me to be reliant on scripts that are done in python and python modules did not want to get installed via apt (vicious circle). So I could not fix this step until I have fixed python first. The claim here that python is used is a feeling. In fact at the end I never tried to install/fix apt program to make system working - there is something to that. I cannot prove this but it remains strong automated dog to me.
Second part of N0rbert suggests to remove alternative python.
I have installed alternative python because I needed newer version to try to do something (do not remember exactly what was at the time).
In essence I have had a mess of different pythons installed and not very clear which python is using what. This goes partially to Ubuntu experience and partially to Python experience. Ubuntu runs its own version for the reasons known to them, in some cases this is not of help as documentation may mislead you until you learn the differences. The problem was I did not have an idea how to properly install system python. It comes pre-installed during OS installation and it is recommended to remain unchanged (in spite of facts that this policy may pose some other risk). On the other hand installation procedures generally tend not to talk to the user. Even if I did something wrong by different ways of configuring or installing I would not know. The bottom line, I did not find documentation how to fix system python anywhere. In fact there are plenty of recipes that would tell you to do:
sudo apt-get install python27
possibly this would fix a problem in normal circumstances, which means you have python considerably working well and apt is working for you.
first part of a problem with python was that my default python was alternative python. Whatever I did was actually not hitting system python.
Hence another hint from N0rbert was essential - remove any reference to alternative python. Since this python I have installed by compiling python source and installing from that, this python was not part of normal package system. Deleting it directly from the disk was what N0rbert suggested and this is what I did among first things (I could easily recover it if need be).
Some other hints suggested to de-install python. When I tried that it made a long list of items that would de-install. I was not ready for that - there were a lot of applications and other things and I thought - who would reinstall all of this and what it would do. About this a bit later.
Since de-installing system python was out of the question the only possibility was to try to force to install it on top.
I do not know how many different things I tried in order to do this. None of them replaced system python. In one moment I got desperate as python in the best case was 2.7.12 (in place of system python). Original ubuntu 14 distribution is hard fixed to 2.7.6. Which seemed to me too old. I would expect that ubuntu through the normal update will come to some 2.7.9 or 2.7.10 but this is not the case. If you look at ubuntu archives only python that is shown and has packages is called python27 and actually it is python 2.7.6.
This has lead me to the question - how come when I try to install pyton27 I get python 2.7.14 which is failing to install but also when I get version of /usr/bin/python (points to python2 which points to python27 which is binary) I regularly get python 2.7.12.
After some days, sleep thinking etc. I came to conclusion that I do not know where 2.7.12 and attempted 2.7.14 are coming from. In the meantime I was experimenting with installing 2.7.15 as alternative - which was good working and I thought - well perhaps I could do the thing with .15 better than previous versions. And this was a mistake.
Interestingly enough 2.7.12 in /usr/bin was quite resilient staying at same version no matter what I did.
This has lead me to suspicion and investigation showed that I had Jonathon's PPA and there in his PPA the most recent python27 is actually 2.7.14. Since 2.7.12 was still working for me with minimal side effects I have sent email to Jonathon to tell me if he has python27 in some earlier incarnation e.g. 2.7.10 (somewhere mentioned in installation) or perhaps 2.7.12 or even some earlier version I can try to install. Jonathon was quite quick to reply and pointed me to his archives (also available online) but in his archives I have found that earliest python27 build is actually 2.7.12 and this is why it was this version that was clinging. In fact Jonathon mentions that he is building currently for Ubuntu 16 and soon he will stop doing that and from the fall of this year will start building for Ubuntu 18. Python27 that was in his archive was not even built for my version of Ubuntu. WOW. I figured I have made bad decision a while a go.
I removed all PPAs from apt list. I even removed default ubuntu list (how bad was that decision will come later)
In one moment I thought I was done. There was no way back machine that would bring me to previous state. And for a while I pondered a lot of things including installing ubuntu16 to super impose with newer set up.
APT was still broken, pending python parts did not want to install.
Then I decided. I will try to install in hack or no hack way original python 2.7.6 as given from ubuntu. It was that or nothing.
But even this proved not to be easy task. Since APT was broken I needed a way to install and I started learning about dpkg. Apparently APT is nice front end for DPKG and actually it is dpkg that does the hard work.
It would be easier if I had Ubuntu CD where I installed this OS from but this CD is, after several moves I made in past couple of years, god knows where. Darn.
Luckily internet is vast library so help came in the form of a archive site. It was this one that came to be useful: https://pkgs.org/
I guess any package that you would need is there from several distributions including ubuntu.
Which .deb to download?
This posed as main question. APT was not working also because dependences may be broken. This is something any linux dist could improve - some sort of audit system which will tell you which file from which package is not what it is in original package.
I do not know what archives I need to install and I do not know how to install it but even in worst case since these .deb are archives I would eventually extract them and replace files with originals.
In fact it turned out that this is not really necessary. DPKG can do this job (provided it is recovered to working state considerably, and N0rbert task #1 was ensuring this. It was not ideal but DPKG seem to be binary program so it would work well if data is OK.
I discovered that dpkg has option -i to install.
So I downloaded .deb and:
dpkg -i dpkg_1.17.5ubuntu5.8_amd64.deb
actually went well. apt stopped complaining that dpkg is not configured well. This was already good wind.
It is worth mentioning that I did not exactly arrive at this point immediately, it was rather trial and error and realization that apt-get is not going to do the clean-up work for me for the installation. I am not sure where Ubuntu keeps meta info about packages (I suspect this is in /var/lib/dpkg/status file) because hacking into statuses of packages did not always render appropriate effect. I am still of belief that some sort of audit tool shall be developed as an aid to compare what is actually in files versus what the records of installation say that is in files.
APT system is modular, one thing may not be working but another may as well be. I learned that I can do:
apt-get download python
this will get you some version of python, but it may not be what you want. In Ubuntu 14 case above one will fetch you python 2.7.5. (if you have just system defined sources - which is the default)
Also this function makes sense as apt downloads stuff before installing. It is just I never saw this and I always see something like wget site bla bla. I believe apt-get download is safer option. Use it.
apt-get download python27
will get you some .deb archives to your local disk. I did not need all of them.
Investigation showed that I need the file /usr/bin/python27 but not just any /usr/bin/python27 file because my /usr/bin/python27 was version 2.7.12 and I was determined to bring back original one which is 2.7.6. Now there is about 2000+ packages in Ubuntu that are python and I needed archive that has this file in it. After looking into archives I have realized that there is only one package that holds this file is: python2.7-minimal_2.7.6-8_amd64.deb
So i have installed this with dpkg:
sudo dpkg -i libpython2.7-minimal_2.7.6-8_amd64.deb
dpkg: warning: downgrading libpython2.7-minimal:amd64 from 2.7.12-1~14.02 to 2.7.6-8
(Reading database ... 1548765 files and directories currently installed.)
Preparing to unpack libpython2.7-minimal_2.7.6-8_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.6-8) over (2.7.12-1~14.02) ...
Setting up libpython2.7-minimal:amd64 (2.7.6-8) ...
Subsequent step was a bit of a shock. I got into a habit of doing sudo apt-get -f install almost after every theory or trial command I made in the hope that this will fix the rest. In fact once I have put 2.7.6 in the original place (overwriting later versions) -f install option showed a long list of applications and libraries:
0 upgraded, 3 newly installed, 556 to remove and 8 not upgraded.
10 not fully installed or removed.
Need to get 0 B/354 kB of archives.
After this operation, 1,748 MB disk space will be freed.
Do you want to continue? [Y/n] y
This was a turning point. I either do this or I go for other option (installing Ubuntu from scratch). Before I have executed this one I took snapshot of all history in my terminal with the idea that even if I remove all these libs and packages that are dependent on later python version, perhaps if I need something of these that will be removed at least I would have a list of names I needed to install.
So I pressed Y.
Cooking took a while. And another problem started emerging with a shy message:
WARNING: The following packages cannot be authenticated!
libjack0 jackd1 jackd1-firewire
Install these packages without verification? [y/N] y
Essentially I did not have another option but to install it without verification.
Now, I have mentioned before that it was bad decision to remove all repositories to get software (was determined to switch them one by one). I selected a mirror source site closer to me instead of main Ubuntu site. Perhaps because things have not been properly installed secure keys have got messed up. I will never know what was the main reason for it.
At this point I did not realize how important is this so unauthenticated warning was following me in subsequent commands. But from another side installation was working so I did not find this critical for desperate situation. It was already all or nothing, so as long as it was working
Later on I switched to see if I can run 'Software Updater' with a hope that updater will resolve other dependences. But Software Updater was strict. It refused to install any updates or fixes at least until authentication is not setup properly. I discovered wonderful apt commands such as sudo apt-key update and sudo apt-key list not that they have been helpful immediately but at some later point they actually did become useful.
Among other interesting ways I find this useful:
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
Replace last one with key you need. Also there is 'Passwords and Keys' application that I did not find particularly friendly or useful (except to discover some old passwords for various logins)
Eventually this issue become fixed by removing all .gpg files /etc/apt/trusted.gpg.d/. Being bitten before I have moved them somewhere else for backup if I need some of them again.
What was helpful here was:
sudo apt-get install launchpad-getkeys
sudo launchpad-getkeys
sudo apt-get update
launchpad-getkeys is interesting piece of software, simple but effective. First it reads all your missing keys and then it tries to retrieve them. Saves a ton of manual work.
Now I could properly reinstall dpkg and python2.7 by using apt-get.
The End?
One may think that there is the end of this story. System recovered. Updates working. Software update working. A list of de-installed software with some manipulation and bringing back some ppa's are back in function.
But, it was too early to be proud on achievement in the same time relief was significant.
Not to make this endless story these are few pointers that I have learned by later fixing particular purpose items:
- mod_wsgi was not really working with 2.7.6 (so I longed to have some later version, perhaps this was reason in the first place why I looked for update of system python.
- Flask and Virtualenv have been behaving badly. Discovered that these will have some commands in
/usr/local/bin so I had to pip uninstall followed by removing commands and then reinstalling this properly to system python not by using pip but by using apt-get install python-flask ... on a general note /usr/local/ in relation to python kept appearing in all sort of places. I thought I removed all things python in /usr/local but had issues with pip installed python modules, so whenever I hit the issue I first do pip uninstall and then install it properly.
- My original feeling that something got wrong when I installed llvm (and updating C compiler) was not without facts. In fact apart from JonathonF's PPA there is another one that makes profound changes to system python and that is: ppa:ubuntu-toolchain-r/ppa. In fact once I brought that back into the picture suddenly system python got upgraded from 2.7.6 back to 2.7.12. But whole system is now working properly even mod_wsgi and virtualenv. This last one (virtualenv) is essential and I will make sure to use it more when I experiment even if it is by virtualenv-ing a version of python and them try to mess whatever. At least if I mess python I will mess the one in virtualenv ;)
Hi to all tldr; people :)
sudo which pythonalso pointed to /usr/local/bin/python I am not really sure where this comes from. Tried to find what could cause this, because i would guess sudo will not autoexecute ~/.bashrc from my user (or I am wrong in some basic understanding)