Sorry for the wall of text. I have observed some pretty strange behaviour in Juju and our Ubuntu MAAS OpenStack over the last couple of weeks, seemingly totally out of the blue (I was out of the office for a week):
- keystone reporting errno 111 (HTTPConnectionPool Max retries exceeded)
- keystone reporting Authorization Failed: Unable to establish connection to http://10.10.5.5:5000/v2.0/tokens)
- nova reporting errno 111 (ERROR: HTTPConnectionPool(host='10.10.5.5', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by : [Errno 111] Connection refused) )
- Horizon throwing a ConnectionError 111
- likewise with almost all the rest of the services
The instances currently up are still up, still SSH/VNC/RDPable and still networked, I should add.
Having read as much as I could on the error, it appears to be an issue with Keystone and the MySQL database it connects to. I promptly attempted to manually connect to the database directly, and did so successfully and, at least at first glance there appears to be no data corruption there.
And it gets even more intriguing (and vexing). Downgrading juju charms or enabling logging resulted in partial and random fixes of the situation.
Eg.:
- Enabling logging on keystone resulted in keystone working but nothing else.
- Enabling logging on glance resulted in nova working to some degree.
- Horizon continued to not work at all, but upgrading glance made some things work for a time.
Toggling logging on/off, saving and committing changes seems to result in unpredictable behaviour of the OpenStack network in general, same as with upgrading and downgrading juju charms.
Quite frankly, practically any change I make, save & commit in any of the juju charms, however small (eg. setting debugging to Enabled), results in different things working and different ones not working, seemingly randomly.
OpenStack log files are of not a lot of help as they display the same errors as previously mentioned + loads of information as to which part of the Python script in question failed.
Has anyone else seen/experienced this extremely strange behaviour before? Any suggestions on how to even begin to investigate this?
