On a fresh Ubuntu install with Xen (learning Xen, proficient with Ubuntu), I try to create a domU PV VM with LUKS encryption, the whole setup procedure goes smoothly, including creating, unlocking the encrypted volumes and completing the full installation.
On reboot, however, it fails back to initramfs.

I would appreciate any help to get me on the right track to fixing this set-up, as I feel it must be pretty close, something silly may be in the way here that the right knowledge can unlock.

My set-up:

HW: 2011 i7 8GB RAM vt-x + vt-d, new SSD
Host OS (dom0): Ubuntu 18.04, no encryption
Xen: 4.9
Guest OS (domU #1): Ubuntu 18.04

Disk volumes in dom0 for domU

(dom0) /dev/vg0/lv-1 20GB setup as /dev/xvda (domU)
(dom0) /dev/vg0/lv-1swap 12GB setup as /dev/xvdb (domU)

Disk volumes on domU:

/dev/xvda partitioned as:
/dev/xvda1 ext4 1GB for /boot (bootable, noatime) /dev/xvda5 available space, LUKS volume /dev/xvdb1 available space, LUKS volume /dev/xvda5_crypt ext4 / (noatime) /dev/xvdb1_crypt swap

Starting domU 1 for the first time after install, console output (using screen):

# xl create -c /etc/xen/domu1.cfg
Parsing config from /etc/xen/domu1.cfg
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /grub/grub.cfg
WARNING:root:grub2's saved_entry/next_entry not supported
WARNING:root:Unknown directive else
WARNING:root:Unknown directive menuentry_id_option
WARNING:root:Unknown directive else
WARNING:root:Unknown directive menuentry_id_option
WARNING:root:Unknown directive export
WARNING:root:Unknown directive font
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive font
WARNING:root:Unknown directive load_video
WARNING:root:Unknown directive terminal_output
WARNING:root:Unknown directive else
WARNING:root:Unknown directive elif
WARNING:root:Unknown directive clear
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive else
WARNING:root:Unknown directive export
WARNING:root:Unknown image directive recordfail
WARNING:root:Unknown image directive load_video
WARNING:root:Unknown image directive gfxmode
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive recordfail
WARNING:root:Unknown image directive load_video   
WARNING:root:Unknown image directive gfxmode
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive recordfail
WARNING:root:Unknown image directive load_video
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown image directive knetbsd
WARNING:root:Unknown image directive if
WARNING:root:Unknown image directive else
WARNING:root:Unknown image directive fi
WARNING:root:Unknown directive source
WARNING:root:Unknown directive elif
WARNING:root:Unknown directive source


    pyGRUB  version 0.6
 ┌────────────────────────────────────────────────────────────────────────┐
 │ Ubuntu                                                                 │
 │ Ubuntu, with Linux 4.15.0-36-generic                                   │
 │ Ubuntu, with Linux 4.15.0-36-generic (recovery mode)                   │
 │ Memory test (memtest86+)                                               │
 │ Memory test (memtest86+, serial console 115200)                        │
 │                                                                        │
 │                                                                        │
 │                                                                        │
 └────────────────────────────────────────────────────────────────────────┘
     Use the ^ and ┴ keys to select which entry is highlighted.
     Press enter to boot the selected OS, 'e' to edit the
     commands before booting, 'a' to modify the kernel arguments
     before booting, or 'c' for a command line.







[    0.000000] e820: Cannot find an available gap in the 32-bit address range
[    0.000000] e820: PCI devices with unassigned 32-bit BARs may not work!
[    0.452369] dmi: Firmware registration failed.

Please unlock disk xvda5_crypt**********
cryptsetup (xvda5_crypt): set up successfully

Please unlock disk xvdb1_crypt**********
cryptsetup (xvdb1_crypt): set up successfully
/dev/xvda1: clean, 311/61056 files, 27841/243968 blocks
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /run on /root/run failed: No such file or directory
run-init: opening console: No such file or directory
Target filesystem doesn't have requested /sbin/init.
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
run-init: opening console: No such file or directory
No init found. Try passing init= bootarg.


BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

There seems to be something wrong out of the box with my grub configuration, however it still offers the boot options, proceeds to boot, requests the LUKS passwords, but then fails back to initramfs.


I don't know if this is significant, but mount shows xvda1 mounted as /root, instead of /boot & I don't see the encrypted volumes:

(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=944232k,nr_inodes=236058,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=195040k,mode=755)
/dev/xvda1 on /root type ext4 (ro,relatime,data=ordered)

(initramfs) lsblk
sh: lsblk: not found

(initramfs) ls -l /dev/xvd*
brw-------    1  202,  17 /dev/xvdb1
brw-------    1  202,  16 /dev/xvdb
brw-------    1  202,   5 /dev/xvda5
brw-------    1  202,   2 /dev/xvda2
brw-------    1  202,   1 /dev/xvda1
brw-------    1  202,   0 /dev/xvda

(domU) /etc/grub/grub.cfg:

(initramfs) cat grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi 

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function recordfail {
  set recordfail=1
  if [ -n "${have_grubenv}" ]; then if [ -z "${boot_once}" ]; then save_env recordfail; fi; fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_msdos
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root  94f039b9-bd0d-4938-bbdf-362f13ba99a1
else
  search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
fi  
    font="/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_GB
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=hidden
    set timeout=10
  # Fallback hidden-timeout code in case the timeout_style feature is
  # unavailable.
  elif sleep --interruptible 10 ; then
    set timeout=0
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
if background_color 44,0,30,0; then
  clear
fi
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
        set gfxpayload="${1}"
        if [ "${1}" = "keep" ]; then
                set vt_handoff=vt.handoff=1
        else
                set vt_handoff=
        fi
}
if [ "${recordfail}" != 1 ]; then
  if [ -e ${prefix}/gfxblacklist.txt ]; then
    if hwmatch ${prefix}/gfxblacklist.txt 3; then
      if [ ${match} = 0 ]; then
        set linux_gfx_mode=keep
      else
        set linux_gfx_mode=text
      fi
    else
      set linux_gfx_mode=text
    fi
  else
    set linux_gfx_mode=keep
  fi
else
  set linux_gfx_mode=text
fi
export linux_gfx_mode
menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
        recordfail
        load_video
        gfxmode $linux_gfx_mode
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_msdos
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  94f039b9-bd0d-4938-bbdf-362f13ba99a1
        else
          search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
        fi
        linux   /vmlinuz-4.15.0-36-generic root=UUID=3260e51a-3ccd-4af5-89a6-d201954bcacb ro  quiet splash $vt_handoff
        initrd  /initrd.img-4.15.0-36-generic
}
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
        menuentry 'Ubuntu, with Linux 4.15.0-36-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-36-generic-advanced-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
                recordfail
                load_video
                gfxmode $linux_gfx_mode
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_msdos
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  94f039b9-bd0d-4938-bbdf-362f13ba99a1
                else
                  search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
                fi
                echo    'Loading Linux 4.15.0-36-generic ...'
                linux   /vmlinuz-4.15.0-36-generic root=UUID=3260e51a-3ccd-4af5-89a6-d201954bcacb ro  quiet splash $vt_handoff
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.15.0-36-generic
        }
        menuentry 'Ubuntu, with Linux 4.15.0-36-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.15.0-36-generic-recovery-3260e51a-3ccd-4af5-89a6-d201954bcacb' {
                recordfail
                load_video
                insmod gzio
                if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
                insmod part_msdos
                insmod ext2
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root  94f039b9-bd0d-4938-bbdf-362f13ba99a1
                else
                  search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
                fi
                echo    'Loading Linux 4.15.0-36-generic ...'
                linux   /vmlinuz-4.15.0-36-generic root=UUID=3260e51a-3ccd-4af5-89a6-d201954bcacb ro recovery nomodeset
                echo    'Loading initial ramdisk ...'
                initrd  /initrd.img-4.15.0-36-generic
        }
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry 'Memory test (memtest86+)' {
        insmod part_msdos
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  94f039b9-bd0d-4938-bbdf-362f13ba99a1
        else
          search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
        fi
        knetbsd /memtest86+.elf
}
menuentry 'Memory test (memtest86+, serial console 115200)' {
        insmod part_msdos
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  94f039b9-bd0d-4938-bbdf-362f13ba99a1
        else
          search --no-floppy --fs-uuid --set=root 94f039b9-bd0d-4938-bbdf-362f13ba99a1
        fi
        linux16 /memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

(dom0) domu1.cfg:

Note: the initial kernel&ramdisk=/var/lib were for the network boot to perform the installation process as per instructions at Ubuntu Xen Guide

# cat domu1.cfg 
# Guest name
name = "domu1"

# 128-bit UUID for the domain as a hexadecimal number.
# Use "uuidgen" to generate one if required.
# The default behavior is to generate a new UUID each time the guest is started.
#uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"

# Kernel image to boot
#kernel = "/boot/vmlinuz"
#kernel = "/var/lib/xen/images/ubuntu-netboot/Ubuntu18.04-bionic-beaver/vmlinuz"
# Ramdisk (optional)
#ramdisk = "/boot/initrd.gz"
#ramdisk = "/var/lib/xen/images/ubuntu-netboot/Ubuntu18.04-bionic-beaver/initrd.gz"
bootloader = "/usr/lib/xen-4.9/bin/pygrub"

# Kernel command line options
extra = "root=/dev/xvda1"

# Initial memory allocation (MB)
memory = 2048

# Maximum memory (MB)
# If this is greater than `memory' then the slack will start ballooned
# (this assumes guest kernel support for ballooning)
maxmem = 6144

# Number of VCPUS
vcpus = 7

# Network devices
# A list of 'vifspec' entries as described in docs/misc/xl-network-configuration.markdown
vif = [ 'bridge=xenbr0' ]

# Disk Devices
# A list of `diskspec' entries as described in docs/misc/xl-disk-configuration.txt
disk = [ '/dev/vg0/lv-1,raw,xvda,rw','/dev/vg0/lv-1swap,raw,xvdb,rw' ]

EDIT: (on dom0) the filesystems are readable & valid:

# guestfish -a /dev/vg0/lv-1
><fs> run
><fs> luks-open /dev/sda5 sda5_crypt
><fs> mount /dev/mapper/sda5_crypt /
><fs> mount /dev/sda1 /boot

><fs> ll /boot/grub
total 2388
drwxr-xr-x 5 0 0    4096 Oct 12 11:22 .
drwxr-xr-x 4 0 0    4096 Oct 12 11:23 ..
drwxr-xr-x 2 0 0    4096 Oct 12 11:22 fonts
-rw-r--r-- 1 0 0     712 Oct 12 11:08 gfxblacklist.txt
-r--r--r-- 1 0 0    6857 Oct 12 11:22 grub.cfg
-rw-r--r-- 1 0 0    1024 Oct 12 11:22 grubenv
drwxr-xr-x 2 0 0   12288 Oct 12 11:22 i386-pc
drwxr-xr-x 2 0 0    4096 Oct 12 11:22 locale
-rw-r--r-- 1 0 0 2397557 Oct 12 11:08 unicode.pf2

><fs> ll /initrd.img
-rw-r--r-- 1 0 0 56938107 Oct 12 11:23 /sysroot/boot/initrd.img-4.15.0-36-generic

EDIT 2:

To try and recover the crypt setup, I did the following:

losetup -f
losetup /dev/loop15 /dev/vg0/lv-1
kpartx -av /dev/loop15
udisksctl unlock -b /dev/mapper/loop15p5
mount /dev/dm-9 /mnt/d1
mount /dev/mapper/loop15p1 /mnt/d1/boot
ln -s /dev/mapper/luks-ba3b60bf-f06c-48d8-819a-1c2a2620252a /dev/mapper/xvda5

losetup -f
losetup /dev/loop16 /dev/vg0/lv-1swap 
kpartx -av /dev/loop16
udisksctl unlock -b /dev/mapper/loop16p1 
ln -s /dev/mapper/luks-e3a84c9d-f9e8-424d-a473-48af7dc22500 /dev/mapper/xvdb1

mount --bind /dev /mnt/d1/dev
mount --bind /sys /mnt/d1/sys
mount --bind /proc /mnt/d1/proc
chroot /mnt/d1

ln -s /dev/mapper/loop15p5 /dev/mapper/xvda5_crypt
ln -s /dev/mapper/loop16p1 /dev/mapper/xvdb1_crypt
cat /etc/crypttab

    xvda5_crypt UUID=ba3b60bf-f06c-48d8-819a-1c2a2620252a none luks,discard
    xvdb1_crypt UUID=e3a84c9d-f9e8-424d-a473-48af7dc22500 none luks,swap,discard

update-initramfs -k all -c

This last command resulted in the following output:

update-initramfs: Generating /boot/initrd.img-4.15.0-36-generic
Warning: /sbin/fsck.crypto_LUKS doesn't exist, can't install to initramfs, ignoring.

Continuing with update-grub, exiting chroot (without installing grub!), unmounting links & rebooting: I could not even reboot my host system, I had to boot from USB, fix grub and reboot. And still my domU cannot start (same failure back to initramfs), so presumably my actions amounted to nothing.

  • I'm not familiar with Xen, but I'm assuming you partitioned everything manually (setting up LUKS and LVM) during install, yes? If so, are you aware that when you do so, the Ubuntu installer will not generate the necessary /etc/crypttab entries for you and that you will need to do this yourself and then update the initramfs after that? – b_laoshi Oct 15 at 0:21
  • @b_laoshi, thank you for your comment. Yes I partitioned manually & no I was not aware that the relevant entries were not generated. Is there a guide on how to do this? What's confusing is that during the boot process I get prompted for the luks passwords. So I assumed this part was set-up. Not sure I understand the different pieces of the puzzle. – asoundmove Oct 15 at 0:44
  • Check out my answer to another question and see if anything applies. – b_laoshi Oct 15 at 2:51
  • The last error message from update-initramfs in my chrooted environment is Warning: /sbin/fsck.crypto_LUKS doesn't exist, can't install to initramfs, ignoring. - see EDIT 2 for details in my main post. – asoundmove Oct 15 at 14:42
  • That's actually just a warning message which makes sense and wouldn't prevent the initramfs from being built. It looks like the update-initramfs tool tries to add filesystem checking applications for each partition. Since there is no such tool for a LUKS partition in its encrypted state, the non-existent /sbin/fsck.crypto_LUKS can't be added. Did you also run update-grub after running update-initramfs -k all -c while in the chroot? If so, the issues may reside with the Xen portion of all this which, unfortunately, I can't help with. – b_laoshi Oct 16 at 0:27

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.