Nested virtualization with KVM is not working in Ubuntu16.04

Question

[If there is a more suitable place to ask this question, let me know !]

1 What am I trying to do ?

I want to use libvirt KVM with nested virtualization like described in [1] and [2]. But it does not work. It has worked (~ 4 months ago), but not anymore.

I want 2 levels of virtualization like this:

• L0 – the bare metal host, running KVM on Ubuntu 16.04
• L1 – a Ubuntu 16.04 VM running on L0; also called the "guest hypervisor" — as it itself is capable of running KVM
• L2 – a Ubuntu 16.04 VM running on L1, also called the "nested guest"

[1] https://docs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/

[2] https://www.linux-kvm.org/page/Nested_Guests

My goal is to deploy an OpenStack environnement on top of VMs rather than on bare metal hosts for conveniance for a lab experiment. As a result, the OpenStack nodes are L1 VMs. Compute nodes are L1 VMs and the VMs created with OpenStack and running on the compute nodes are L2 VMs.

2 What is my problem ?

I can not run my 2nd levels of virtualization:

• L0 is just fine: running Ubuntu 16.04.5 LTS, installed with the .iso image
• L1: I install libvirt + KVM on L0. I can run VMs like the Ubuntu16.04 cloud image on L0.
• L2: I install libvirt + KVM on L1 as well. But I can not run VMs on L1: I get kernel panic or general protection fault.

But if I do the same with Ubuntu18.04 instead of Ubuntu16.04, it works without faults. I don't change the configuration or virt-install scripts (other than using the 18.04 .iso and cloud image).

3 Details

Details about L0 Ubuntu 16.04 bare metal host

L0 is running Ubuntu 16.04.5 LTS installed with the .iso.

kernel user@L0:~$ uname -a Linux L0 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

libvirt version running on L0 user@L0:~$ virsh version Compiled against library: libvirt 4.0.0 Using library: libvirt 4.0.0 Using API: QEMU 4.0.0 Running hypervisor: QEMU 2.11.1

KVM acceleration user@L0:~$ kvm-ok INFO: /dev/kvm exists KVM acceleration can be used

nested parameter user@L0:~$ cat /sys/module/kvm_intel/parameters/nested Y

number of CPU user@L0:~$ egrep -c '(vmx|svm)' /proc/cpuinfo 48

Details about a L1 Ubuntu 16.04 VM

A VM in L1 (which is running on L0) which is running Ubuntu 16.04.5 LTS installed by a cloud image.

kernel user@L0:~$ uname -a Linux L1 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

libvirt version running on the L1 VM user@L1-VM:~$ sudo virsh version Compiled against library: libvirt 4.0.0 Using library: libvirt 4.0.0 Using API: QEMU 4.0.0 Running hypervisor: QEMU 2.11.1

KVM acceleration user@L1-VM:~$ kvm-ok INFO: /dev/kvm exists KVM acceleration can be used

nested parameter user@L1-VM:~$ cat /sys/module/kvm_intel/parameters/nested Y

number of CPU, which are vCPU given by L0 to the L1 VM I give 20 vCPU. user@L1-VM:~$ egrep -c '(vmx|svm)' /proc/cpuinfo 20

Details about a L2 VM

I want to create a L2 Ubuntu 16.04.5 LTS VM installed by a cloud image VM within my L1 KVM VM. But whatever I do, my L2 VM crash before finishing to be instantiated. I get kernel panic or general protection fault.

However a Cirros OS image can run on a L1 VM !

4 Thoughts

I think this is a bug in eitheir Ubuntu16.04 or libvirt. I install libvir KVM in both L0 and L1 using a custom repository [3] from OpenStack team, because the version of libvirt in this repo is newer than on the one on 16.04 official repo and it match the version of libvirt in Ubuntu18.

[3] https://wiki.ubuntu.com/OpenStack/CloudArchive

If I do the same with Ubuntu 18.04 .iso and cloud image, it works.

It works if:

• L0 = Ubuntu18.04 (.iso)
• L1 = Ubuntu18.04 (cloud image)
• L2 = Ubuntu18.04 (cloud image)

It also works if:

• L0 = Ubuntu18.04 (.iso)
• L1 = Ubuntu18.04 (cloud image)
• L2 = Ubuntu16.04 (cloud image)

Thank you for your time reading !

nico