What are the different versions available as Ubuntu cloud-images?...General Guide for Ubuntu Cloud Image use in Virt-manager 2021

Question

I'm soooo thrilled to finally find prebuilt Ubuntu images. I was using osboxes, but the file size when converting is 500GBs and tedious to compact. Fresh installs are tedious, especially when 90% of people could use the default options or easily customize a from a central image!

Anyway, there are so many versions. What are the differences? Some are obvious: img, tar.gz, vmdk, ova. I'm assuming the azure vhd, vagrant, wsl are for those platforms. That squashfs might be a single file compacted version. And from the related question below, that kvm has a smaller kvm kernel. The lxd version is actually larger. What is the root version for?

Edit: I found what the root version is, see my self-reply below. I'm still having trouble setting one of these up in Virt-manager, discussion also below, so any help is appreciated. I can start a new question if someone wants to answer that separately.

EDIT2!: After much trial and searching, I was able to get the cloud-image VM working which was my primary goal. I even found an old answer to my primary question that helped me to that goal!

---

https://cloud-images.ubuntu.com/releases/focal/release/:

[   ] ubuntu-20.04-server-cloudimg-amd64-azure.vhd.manifest  2021-11-08 22:37   18K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-azure.vhd.zip       2021-11-09 00:17  549M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-disk-kvm.img        2021-11-08 22:37  523M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-disk-kvm.manifest   2021-11-08 22:37   17K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz          2021-11-08 22:42  832   Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-root.manifest       2021-11-08 22:40   17K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-root.tar.xz         2021-11-08 22:40  327M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-vagrant.box         2021-11-08 22:42  517M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-wsl.rootfs.manifest 2021-11-08 22:42   19K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-wsl.rootfs.tar.gz   2021-11-08 22:42  469M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64.img                 2021-11-08 22:39  542M  Cloud image for 64-bit computers (USB image)
[   ] ubuntu-20.04-server-cloudimg-amd64.manifest            2021-11-08 22:40   17K  Cloud image for 64-bit computers (contents of live filesystem)
[   ] ubuntu-20.04-server-cloudimg-amd64.ova                 2021-11-08 22:40  525M  Cloud image for 64-bit computers (Open Virtualization Appliance)
[   ] ubuntu-20.04-server-cloudimg-amd64.squashfs            2021-11-08 22:41  371M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64.squashfs.manifest   2021-11-08 22:41   17K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64.tar.gz              2021-11-08 22:43  498M  Cloud image for 64-bit computers (Cloud Image/EC2 tarball)
[   ] ubuntu-20.04-server-cloudimg-amd64.vmdk                2021-11-08 22:42  525M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds

Answer 1

I found solutions for some bugs. For some reason cloud-init doesn't start even though the cidata.iso is added.

To fix this up, the SMBIOS “serial” string parameter must be set. Source: https://paulgorman.org/technical/cloud-init.txt.html

This example do not work for me and i found some bugreports at RH.:

$ virt-install --sysinfo type=smbios,system_serial=ds=nocloud

But this example works:

$ virt-install --qemu-commandline="-smbios type=1,serial=ds=nocloud" ...

Then cloudinit takes over as desired and sets all settings.

Answer 2

EDIT2: there is a way to disable cloud-init with adding one file: touch /etc/cloud/cloud-init.disabled https://cloudinit.readthedocs.io/en/latest/howto/disable_cloud_init.html Original questions was answered here: What's the difference between ubuntu's amd64-disk-kvm.img and the regular amd64.img cloud images?

EDIT: see "Last Update" section below for the minimal things to get the image working. It looks like they are set up to use a VNC viewer to get the auto generated IP and login information shown at boot, unless bypassed.

Okay, here's the guide I found that worked after two days of trial and error. Ubuntu would do well by their community to post some instructions for local users. https://www.agiliq.com/blog/2012/07/using-ubuntu-cloud-images-in-kvm/

I was able to find the "vmlinuz-virtual-generic" version kernel in a buried "unpacked" folder in the cloud images releases link from above. https://cloud-images.ubuntu.com/focal/current/unpacked/. The main page has other versions than for Focal https://cloud-images.ubuntu.com. This is the most important part because the initrd-userve.img wont recognize the root disk. You can also use a normal ubuntu kernel.

The gist of it is to use qemu-img create -b ubuntu-*-server-cloudimg-amd64.img -f qcow2 samename.img.qcow2 command to create the VM (or just use the GUI create in virt-mgr). It will give the option to configure before install and then start the VM. You'll want to add the kernel and initrd from the "unpacked" downloads to the Direct Boot section under Boot Options plus these boot args: ro init=/usr/lib/cloud-init/uncloud-init root=/dev/vda ds=nocloud ubuntu-pass=initialpassword. You have to boot twice for the password to work. Actually, just rw root=/dev/vda1 init=/bin/bash will work to create a new user and password.

Notice these instructions say vda. I had to use vda1 because there are two boot partitions vda14 for GPT/bios_boot and vda15 for EFI/ESP/boot-flagged partition. The original raw .img file uses sdaX, fyi.. converting to qcow2 or just using qemu-img is what changes that apparently.

A couple more details:

Older help pages say the kernel files and floppy disk are bundled in the tar file. This isn't the case anymore apparently and led me to more trial and error. The common errors are hanging on "GRUB_FORCE_PARTUUID attempting initrdless boot", "Error unknown command hwmatch" and "random: fast init done". Actually, the first two are when not using Direct Boot. The Ova import in VirtualBox does boot on it's own, so it's figuring out the internal kernel files somehow. I would love to know how to do that in KVM. I looked at the XML files but didnt see an obvious clue beside it setting up a floppy disk, putting me on the search to find the floppy which led me to the main guide page above. The third error I quoted is when these internal kernel files are not found, even though the fstab has correct lables for the boot drives. It doesnt even get to grub, so I cant specify disk UUIDs there, though they match in the grub file (obviously because using Direct Boot doesnt change any of that).

Here is an answer to my original question. Again based on the older tar contents with floppy, etc. It did lead me to looking deeper for the "vmlinuz-virtual" files: Where is the usage of Ubuntu cloud guest image on OpenStack documented?

Other resources:

• Info on the -kvm version. Would like to know if this improves other things beside slightly smaller size: What's the difference between ubuntu's amd64-disk-kvm.img and the regular amd64.img cloud images?, especially if there is a separate vmlinuz-virtual version.
• A way to import .ova with virt-v2v (didnt work for me): https://www.redhat.com/en/blog/importing-vms-kvm-virt-v2v
• One way to set the password: Default username/password for Ubuntu Cloud image?
• The actual cloud-init docs page: https://cloudinit.readthedocs.io/en/latest/topics/network-config.html
• Two methods to remove cloud-init: https://blog.rylander.io/2020/12/23/how-to-remove-cloud-init-from-ubuntu-server-20-04/ touch /etc/cloud/cloud-init.disabled
• The official way to remove cloud-init, only the depreciated one worked for me: https://blog.wikichoon.com/2020/09/virt-install-cloud-init.html `virt-customize -a MY-CLOUD-IMAGE.qcow2 --root-password password:SUPER-SECRET-PASSWORD --uninstall cloud-init'
• How to fix an "Error unknown command hwmatch: Error unknown command hwmatch
• An old attempt to boot using extlinux which didnt work in the new images: How to set up cloud image in VirtualBox?
• How to config to use EFI, which didnt change anything and wasnt necessary because the working method boots using Bios/SeaBios: https://unix.stackexchange.com/questions/612813/virt-manager-change-firmware-after-installation
• A way to run cloud-init manually, which didnt help to get the password displayed: https://stackoverflow.com/questions/23151425/how-to-run-cloud-init-manually
• A page explaining the default password is created by cloud-init: Ubuntu 18.04 OVA default credentials
• Offical man page on GH for cloud-init params: https://github.com/virt-manager/virt-manager/blob/master/man/virt-install.rst#--cloud-init
• Info on the GRUB_FORCE_PARTUUID error: https://unix.stackexchange.com/questions/450946/grub2-using-wrong-boot-partition
• Question with link to outdated instructions from Ubuntu: How do I boot Ubuntu Cloud images in KVM?
• A help page from Ubuntu Community with instructions how to use it on major cloud providers but not on local hypervisor: https://help.ubuntu.com/community/CloudInit
• A method for Ovf (vdmk) that might help solve the non-Direct-Boot method. qemu-img makes a grub menu with 5 second timeout?: https://askubuntu.com/a/334546/795299

The kernel args uncloud-init tell cloud-init it is a local VM and the nocloud arg sets up local networking (instead of having to go in and change /etc/netplan/*yaml to remove the fixed mac address, and use netplan apply and then systemctl restart networking)

---

Here is some more info I've learned:

• There are docs in the images at /usr/share/doc/cloud-init/examples/.., possible the same as here: https://cloudinit.readthedocs.io/en/latest/topics/examples.html
• Logs are in: /var/log/cloud-init.log and /var/log/cloud-init-output.log
• There is a way to uses cloud-init locally by either creating and mounting a small iso or image, or by pointing to a 'seedfrom' directory or files.
  • info on using image: https://superuser.com/questions/827977/use-cloud-init-with-virtualbox
  • info on using iso: https://sumit-ghosh.com/articles/create-vm-using-libvirt-cloud-images-cloud-init/
  • info on using a dir: https://discourse.ubuntu.com/t/automated-server-install-quickstart/16614
  • info on 'seedfrom' as a local file: https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html#datasource-nocloud
• CentOS has images: https://cloud.centos.org/centos/7/images/

---

Trying to set the cloud-init yaml files in an easy way with a local file led me to this solution for using cloud-init with an Ubuntu server iso: https://discourse.ubuntu.com/t/automated-server-install-quickstart/16614. It has a few simple commands to create the files and then runs a neat little one-liner server to create the seedfrom net location. It installs and boots with Kvm. Unfortunately after using the raw image to create a VM in Virtmanager, it wouldnt boot because it couldnt find the /stillmounted/casper/initrd for some reason. So I copied the kernel files out of the mounted iso and tried using them (because I suspect the cloud image above doesnt have audio drivers in its kernel), but it also hung on the casper/initrd files.

So I went back to the VirtualBox ova. The my-seed.iso worked to change the server name. After a long search on how to pass kernel parameters (you cant beside a couple using vboxmanage setextradata), I rediscovered you can press shift while booting to edit the Grub args. After using init=/bin/bash to finally set the password and vga=792 to increase the display size, I found a bug in the image in the visudo file that the #include.. is missing a space which kills sudo. After getting apt to update, I found there is a VirtualBox bug that doesnt allow bidirectional clipboard in recent versions: https://superuser.com/questions/1293042/bidirectional-clipboard-not-working-in-virtualbox. But at least I finally have a way to create a basic Ubuntu VM without having to go through the whole install process. This method should work for many other distros. (Oh also, the touch /etc/cloud/cloud-init.disabled command above seems to work).

---

Followup tidbits:

• The my-config.iso only works the on the first run. (unless there is a way to trigger cloud-init).
• Hold shift to get into Grub. Use the kernel arg vga=792 for a larger terminal. Set grub timeout in /etc/default/grub and run update-grub.
• Audio doesnt work on these cloud images without installing apt install linux-generic (about 800Mbs). And set audio output enabled in Vbox.

---

Last Update: I finally figured out the key to getting the cloud-image to boot without using the Direct Boot files. Basically, either install the non-kvm, generic kernel version using apt install linux-image-$(uname -r) and change kvm to generic (~90Mb), or use the kvm kernel and ssh into the VM using systemctl enable [email protected] && systemctl start [email protected], then virtsh console <vm-name>. To get in initially to make a user use root=/dev/vda1 init=/bin/bash rw for the kernel args. Then delete or move two files /etc/grub.d/01.. and /etc/default/grub.d/50.., and comment out the first line of /etc/default/grub.d/40-force-partuuid.cfg. Change /etc/default/grub hidden to menu, adjust timeout, etc if desired. Then update-grub and it should work.

Answer 3

EDIT: See my other answer. The below is somewhat irrelevant now I have a working VM, beside the permissions issue I didnt solve when using virt-v2v. There is an Ubuntu specific version of that tool called virt-convert, but I ended up using qemu-img which succeeded without permissions issues.

The root version is literally the whole linux filesystem, ie boot, dev, etc..

/boot doesnt have any files, like grub. So I'm looking for the easiest version to create a Virt-manager VM from. Am I supposed to boot with direct boot and add rw init/=bin/bash to the root=/dev/sda1 kernel args to get a shell in initrd, and then somehow use grub-update to install grub? I could mount and copy over a /boot directory from my host linux, but it doesn't seem like this is the recommended way. I've also played with tagging the sda1 volume as boot on the img version using a Gparted live iso attached. I suppose I could use fdisk or cfdisk to do that. I wonder if it has an MBR.

I also tried to convert the ova version. I've tried the img version of Ubuntu Minimal and had to use direct kernel boot. I tried to get Virt-manager SeaBIOS to recognize the MBR, but there were 3 partitions (separate boot and esp/efi) and it errors with grub_force_partuuid attempting initrdless boot. Is it better to try connecting to the UEFI partition?

I just tried converting the ova using virt-v2v which succeeded, but I get this error Could not open '/var/lib/libvirt/images/ubuntu-focal-20.04-cloudimg-20211108-sda': Permission denied. So I tried chmodding it to 777 and tried running as root from the CLI virsh start ubuntu-focal-20.04-cloudimg-20211108 with the same error. The other images in that same directory run just fine, but I didnt use virt-v2v on them.