I have created Openstack based on MAAS using juju. I have created instances, projects etc. I have made backups of database and openstack config files. I am trying test recovery by destroying and removing service keystone, just to see if it was recoverable. I also drop the keystone database. (it is backed up) So with juju i destroy the old keystone which was broken, and deploy a new keystone. This actually gets me running again except for the fact that all projects (tenants) are gone, and the instances does not belong to any project. So i restore the keystone database but without the tokens but when i do that i get an error in horizon telling me that keystone will not authorize the client.

What would be the best way to restore a broken keystone node...? It seems when you redo keystone with juju you get new tokens. can juju somehow inject the old ones into the the new keystone?

  • Hi Julius, thanks for sharing your success here. I don't know the answer to your question, but would you mind sharing the version of Ubuntu you used for MAAS and Juju? I have tried 12.04 and 13.04 and have encountered obstacles with both. Thanks. – Ian D. Rossi May 3 '13 at 15:34
  • 13.04 is more stable for Maas. – Julius May 6 '13 at 6:56

The problem you are hitting is that the keystone charm stores some information outside of the keystone database; specifically usernames and passwords for services are stored locally on disk so that when additional service units related to keystone, the keystone service can hand out the correct password for the requested user to the related service.

You will also hit issues if you have not provided admin-password and admin-token configuration when deploying keystone; the keystone service units will randomly generate these if not explicitly configured - so they will change when you remove and then add a new keystone service unit.

What you end up with is a set of hashed passwords in the database (created by the first service unit) that don't match the generated passwords on disk in the second keystone service unit.

Ideally we would fix this by some sort of generic object storage feature in juju (think object-put/object-get) but this is not a supported feature.

All of the on disk storage of passwords and tokens takes place in /var/lib/keystone; you might be able to pick these out as part of a backup of the keystone service unit itself; however you will hit races when you add the new service unit as the service relations will fire prior to you being able to restore these files.

As one of the OpenStack charm maintainers, I'll give this a bit more thought and see if we can come up with something more elegant.

  • If backing up keystone when running as a charm is somewhat impossible for now, this would be nice if someone looked into it. So in other words nobody knows how to do it properly? so if I want to recover Keystone. I should do it manually injecting tokens and tenants? And then i would expect it not to function fully? – Julius May 7 '13 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.