3

I had working Landscape after installing Openstack with openstack-installer. After some time, and after rebooting everything, and after some struggling with juju (Landscape is not running after a reboot with juju-core 1.20.x), Landscape is running again, as well as Openstack.

The problem is that none computer is reporting to Landscape. I receive every 5 minutes an email with subject "[Landscape Alert] 18 computers haven't contacted Landscape within the last 5 minutes", even though all servers are up and running.

I'm facing serious problems on Openstack as well (cant launch new instances, backup volumes, etc) and my guess is the problem is related.

Improve this question
11
  • Please inspect and/or attach a /var/log/landscape/broker.log file from one of the computers that have this alert enabled. It's the log that contains information about the exchanges and will most likely have the reason why they are failing.
    – Andreas Hasenack
    Mar 13, 2015 at 20:53
  • Says something about a cert and stuff: paste.ubuntu.com/10593219
    – Sacha Yunusic
    Mar 13, 2015 at 21:11
  • Looks like your server certificate changed. There used to be a bug in the Apache charm where it would regenerate the certificate, if SELFSIGNED was in use, on every reboot (bugs.launchpad.net/charms/+source/apache2/+bug/1322759). It's marked "fix released", though. To validate this theory, can you get us the /var/log/juju/unit-apache2-*.log file from the apache2 unit(s) and the approximate reboot time? Then we can correlate that with the certificate regeneration, if this is indeed what happened.
    – Andreas Hasenack
    Mar 15, 2015 at 16:40
  • Reboot time: 2015-03-11 13:36:46. The log is here: paste.ubuntu.com/10610121. There are some other reboots: 2015-01-16 14:10:27, and 2015-02-24 18:24:04, but not totally sure.
    – Sacha Yunusic
    Mar 16, 2015 at 14:54
  • Hm, I don't a second certificate generation. Could you paste a copy of the apache2 certificate somewhere? Not the private key, just the public cert. You can obtain it with the command "openssl s_client -connect apache2-ip:443" and extract the BEGIN/END bits what will show up. Then go to a client and get the file pointed at by the ssl_public_key configuration in /etc/landscape/client.conf, it should be the same certificate (paste that one too).
    – Andreas Hasenack
    Mar 16, 2015 at 15:23

1 Answer 1

Reset to default
5

After the debugging session via comments to the original question, we found out what the issue is.

When using a self-signed certificate, the apache2 charm had a bug where it would generate such a certificate with only 30 days of validity.

What happened is that it expired. This was fixed in a newer version of the charm, so what we will do is upgrade to that and trigger a new certificate generation.

Please follow these steps to upgrade the charm and regenerate the certificate:

  1. Run juju ssh apache2/0 'sudo rm -f /etc/ssl/certs/apache2.cert /etc/ssl/private/apache2.key /etc/ssl/certs/landscape_server_ca.crt'.
  2. Run juju upgrade-charm apache2. This will upgrade the charm, and the new charm, once it sees there are no certificates (deleted in the first step), will generate a new pair.
  3. Watch the output of juju debug-log until it quiets down.
  4. Run juju ssh apache2/0 'sudo service apache2 restart' (I will file a bug about this step: it shouldn't be needed, but is right now)

Now we need to tell the clients about this new certificate. This is a bit trickier, because the clients are running in another juju environment which exists on the landscape/0 unit, inception-like. Follow these steps:

  1. Grab the new certificate: juju scp apache2/0:/etc/ssl/certs/apache2.cert .
  2. Make sure permissions are 0644: chmod 0644 apache2.cert
  3. Verify it expires 10 years from now: openssl x509 -in apache2.cert -noout -enddate
  4. Upload it to landscape/0: juju scp apache2.cert landscape/0:/tmp
  5. Login on landscape/0: juju ssh landscape/0
  6. Change the landscape user shell: sudo chsh -s /bin/bash landscape
  7. Become landscape: sudo -u landscape -i
  8. Point to the cloud juju environment: export JUJU_HOME=/var/lib/landscape/juju-homes/$(ls -1t /var/lib/landscape/juju-homes/|head -n 1)
  9. Run this to confirm juju is working in that environment: juju status. You should see a lot of OpenStack services
  10. Tell landscape-client about the new certificate: juju set landscape-client ssl-public-key="base64:$(cat /tmp/apache2.cert|base64)"
  11. Watch the output of juju debug-log until it quiets down
  12. Restart landscape-client everywhere: juju run --service landscape-client 'sudo service landscape-client restart'
  13. If the above command fails with something about subordinate services not being supported, run this instead: juju run --all 'sudo service landscape-client restart'

You can login on a client and tail -f the broker.log file, see if the SSL error persists.

Improve this answer
8
  • The certificate problem is gone, but there is still some issues. From your procedure, everything went fine, except the deletion of the file /etc/ssl/certs/landscape_server_ca.crt, because it didn't exists. The problems I'm facing now are here: paste.ubuntu.com/10612838. Your procedure it was applied at 2015-03-16 20:48:09,269.
    – Sacha Yunusic
    Mar 17, 2015 at 0:25
  • After some rebooting, troubleshooting, etc.. I have several "hook failed: "config-changed"" errors. broker.log looks pretty much the same than the on I pasted in prior comment. Landscape is stuck with 19 tasks like or so doing some synchronization. They are slowly finishing, so I guess it takes time. I'll check it out tomorrow, and work on the failed hooks.
    – Sacha Yunusic
    Mar 17, 2015 at 3:12
  • Those config-changed errors should be fixed. You will have to login to each unit that is in this state and inspect its /var/log/juju/unit-<name>.log file to see why it failed, fix it and run "juju resolved --retry <unitname>".
    – Andreas Hasenack
    Mar 17, 2015 at 12:25
  • ok, I'll do that. Now, in Lanscape there are some tasks runnung for 15 hours or more; some like "Resynchronize computer juju-machine-0-lxc-7 (scopes: disk)", but after that the same task is there again and it was successfull (same task applied to the same host). Should I cancel the first task? Or wait longer to finish the first, even the newer was successful already?
    – Sacha Yunusic
    Mar 17, 2015 at 14:06
  • Ok, I solved all of the errors but one. When I ran "juju resolved --retry mysql/0" the unit tries to download something from archive.ubuntu.com and gives this error: File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/config-changed", line 114, in <module> // with open('/var/lib/mysql/mysql.passwd','r') as rpw: // IOError: [Errno 2] No such file or directory: '/var/lib/mysql/mysql.passwd'
    – Sacha Yunusic
    Mar 17, 2015 at 16:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .