Cloud instances in OpenStack can't import public SSH key

Question

does anybody know how to actually fix the problem of Ubuntu Cloud VM instances in OpenStack not being able to import public keys from Nova-api's meta-data server with the following message:

2012-07-18 11:05:45,409 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [113/120s]: url error [[Errno 111] Connection refused]
2012-07-18 11:05:52,419 - DataSourceEc2.py[CRITICAL]: giving up on md after 120 seconds

I've found numerous mentions of the problem (e.g. here or here) and tried turning this iptables rule on/off (using iptables-save and iptables-apply), but it doesn't work. The funny rule seems to be:

$ iptables -t nat -L -v | grep -n3 169.254.169.254
48-
49-Chain nova-network-PREROUTING (1 references)
50- pkts bytes target     prot opt in     out     source               destination         
51:   32  1920 DNAT       tcp  --  any    any     anywhere             169.254.169.254      tcp dpt:http to:128.131.172.155:8775
52-    0     0 DNAT       udp  --  any    any     anywhere             sneezy.infosys.tuwien.ac.at  udp dpt:1000 to:10.0.0.2:1194
53-
54-Chain nova-network-float-snat (1 references)

Is there any good way to manually debug this, by the way?

Answer 1

Fixing the issue on an all-in-one deployment

With the help of the good folks at the IRC channel #openstack (zynzel, livemoon) we solved this by simply restarting nova-api:

sudo restart nova-api

You can verify that the metadata server is indeed up by issuing:

lsof -i:8775

from your controller node or

nmap -sS -sV -O -P0 -p8773-8776 <controller_ip>

with your controller node's IP address and checking that port 8775/tcp is open and listened to.

It might be the case that this is a problem with Dodai-deploy, as nova-api has to be installed after nova-compute is installed for the meta-data server to be initialized correctly.

Fixing the issue on a multinode doployment

If you run a multihost deployment and want to use Ubuntu cloud instances that pull in public keys from a metadata server you need this in your controller's nova.conf:

multihost=True
enabled_apis=metadata,ec2,osapi_compute,osapi_volume

and this in your compute node's nova.conf:

multi_host=True
enabled_apis=metadata

Without this you'll get errors in your instance's console output (nova console-log test-instance) about it not being able to reach the metadata server and you won't be able to ssh into them.

Another solution is to tell your instance who you are on Launchpad through the user-data Dashboard form (or file in the terminal) so that it can pull in your public keys from there. The syntax is (not explained anywhere else but in the source code):

#cloud-config
ssh_import_id: [your-launchpad-username]