Trying to enable docker on nova-Kilo, running on top of 14.04, by following https://wiki.openstack.org/wiki/Docker. Added compute_driver = novadocker.virt.docker.DockerDriver to "config-flags" inside the nova-compute charm, commented the compute_driver in nova-cloud-controller's nova.conf, and restarted nova services on compute, controller and neutron nodes. Initially, got the following error, on the compute node:
2015-05-29 23:16:45.551 32936 ERROR nova.virt.libvirt.host [req-4aba1c34-9ac3-47bf-8b69-dd16d9a7f38c - - - - -] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host Traceback (most recent call last):
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 524, in get_connection
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host conn = self._get_connection()
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 511, in _get_connection
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host wrapped_conn = self._get_new_connection()
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 463, in _get_new_connection
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 314, in _connect
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host rv = execute(f, *args, **kwargs)
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host six.reraise(c, e, tb)
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host rv = meth(*args, **kwargs)
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host File "/usr/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2015-05-29 23:16:45.551 32936 TRACE nova.virt.libvirt.host
2015-05-29 23:16:45.559 32936 ERROR nova.openstack.common.threadgroup [req-50db7bae-f838-4d3a-a5f6-f70d5ce00a45 - - - - -] Connection to the hypervisor is broken on host: lab-cn01
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup x.wait()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup service.start()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/service.py", line 164, in start
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1261, in init_host
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup self.driver.init_host(host=self.host)
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 529, in init_host
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup self._do_quality_warnings()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 507, in _do_quality_warnings
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup caps = self._host.get_capabilities()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 745, in get_capabilities
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup xmlstr = self.get_connection().getCapabilities()
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 533, in get_connection
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup raise exception.HypervisorUnavailable(host=CONF.host)
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup HypervisorUnavailable: Connection to the hypervisor is broken on host: lab-cn01
2015-05-29 23:16:45.559 32936 TRACE nova.openstack.common.threadgroup
And walked around it by issuing chmod 777 /var/run/libvirt/libvirt-sock. The problem now is that I can't launch any image from Horizon.Instances exit due to absence of boot disk:
Booting from Hard Disk...
Boot failed: not a bootable disk
No bootable device.
However, I am able to spawn instances using docker cli, on the target compute node. Oddly enough, hypervisour type is still QEMU:
# nova hypervisor-show lab-cn01.maas17 | grep type
| hypervisor_type | QEMU
Any thoughts?
Cheers,
Edit 1: Followed the suggestion in https://ask.openstack.org/en/question/52485/nova-compute-does-not-work-with-docker-hypervisor/ and updated nova-compute.conf, accordingly. Nova compute node; however, is reporting the following error, upon restart:
2015-06-01 17:02:59.784 42703 ERROR nova.openstack.common.threadgroup [req-9f37298d-828c-4ac5-9834-320cc082f92b - - - - -] 'module' object has no attribute 'DOCKER'
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup x.wait()
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup service.start()
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1291, in pre_start_hook
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6240, in update_available_resource
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup rt.update_available_resource(context)
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 376, in update_available_resource
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup resources = self.driver.get_available_resource(self.nodename)
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup File "/home/ubuntu/dockerdrv/src/novadocker/novadocker/virt/docker/driver.py", line 312, in get_available_resource
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup (arch.I686, hv_type.DOCKER, vm_mode.EXE),
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup AttributeError: 'module' object has no attribute 'DOCKER'
2015-06-01 17:02:59.784 42703 TRACE nova.openstack.common.threadgroup
Edit 2: Added docker entries in /usr/lib/python2.7/dist-packages/nova/compute/hv_type.py and hypervisor type changed to docker:
# nova hypervisor-show lab-cn01 | grep type
| hypervisor_type | docker |
Problem now is that nova compute can't access docker images:
2015-06-01 23:06:22.641 7842 ERROR nova.compute.manager [-] [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] Instance failed to spawn
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] Traceback (most recent call last):
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2442, in _build_resources
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] yield resources
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2314, in _build_and_run_instance
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] block_device_info=block_device_info)
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/home/ubuntu/docker2/src/novadocker/novadocker/virt/docker/driver.py", line 436, in spawn
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] image = self._pull_missing_image(context, image_meta, instance)
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/home/ubuntu/docker2/src/novadocker/novadocker/virt/docker/driver.py", line 377, in _pull_missing_image
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] return self.docker.inspect_image(self._encode_utf8(image_meta['name']))
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/home/ubuntu/docker2/src/novadocker/novadocker/virt/docker/client.py", line 36, in wrapper
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] out = f(*args, **kwds)
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 15, in wrapped
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] return f(self, resource_id, *args, **kwargs)
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 752, in inspect_image
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] True
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 126, in _result
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] self._raise_for_status(response)
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 122, in _raise_for_status
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] raise errors.APIError(e, response, explanation=explanation)
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92] APIError: 404 Client Error: Not Found ("No such image: ubuntu-14.04.2-docker")
2015-06-01 23:06:22.641 7842 TRACE nova.compute.manager [instance: ac234f55-1dc5-416e-85a2-a5b5dff37a92]
Edit 3:
The above is caused by multiple problems:
- Bug report https://bugs.launchpad.net/nova/+bug/1461217
- Not all typical docker images, e.g., ubuntu latest 07f8e8c5e660 or cirros latest d8de71c04044 can be run. (So far, I can only run rastasheep/ubuntu-sshd 14.04 063f5e7d07b3.)
Edit 4:
Glance and docker image names have to be the same! That's it folks.
