2

TL;TR

Failed to boot with lvm2's new default cache-pool setting cache-policy-smq

The problem

I have created a lvm disk with lvmcache on SSD drive on ubuntu 16.04 Following this, but failed to mount my root volume after reboot server.

I can boot into 16.04 live CD and mount /dev/mapper/vg0-root successfully (with Boot-repair, mdadm, thin-provisioning-tools).

After some digging, It seems lvmcache's new policy dm-cache-smq is missing/unsupported in somewhere, but i have no idea what to do next.

How can i fix this?

Error messages on boot

The error message repeat showing until mount wait timeout, then console drops a (initramfs) shell.

Begin: Running /scripts/local-block . . . lumetad is not active yet, using direct activation during sysinit
/sbin/nodprobe failed: 1
  device-mapper: cache: You have created a cache device with a lot of individual cache blocks (7600000)
  All these mappings can consume a lot oF kernel memory, and take some time to read/write
  Please consider increasing the cache block size to reduce the overall cache block count
  device-mapper: cache-policy: unknown policy type
  device-mapper: table: 252:4: cache: Error creating cache's policy
  device-napper: ioctl: error adding target to table
  device-mapper: reload ioctl on (252:4) failed: Invalid argument
done.

Settings i have now

/etc/initramfs-tools/modules

dm_cache
dm_cache_mq
dm_cache_smq
dm_persistent_data
dm_bufio

/etc/initramfs-tools/hooks/lvmcache

#!/bin/sh
PREREQ="lvm2"
prereqs()
{
  echo "$PREREQ"
}
case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac
if [ ! -x /usr/sbin/cache_check ]; then
  exit 0
fi
. /usr/share/initramfs-tools/hook-functions
copy_exec /usr/sbin/cache_check
manual_add_modules dm_cache dm_cache_mq dm_cache_smq dm_persistent_data dm_bufio

Workaround

I change lvmcache back to mq policy to get my server online temporarily, but still looking for solutions for cache with smq policy.

lvremove vg0/cachedata
lvcreate -L 480M -n cachemeta vg0 /dev/sde
lvcreate -L 475000M -n cachedata vg0 /dev/sde
# Use mq here
lvconvert --type cache-pool --cachepolicy mq --chunksize 8192 --poolmetadata vg0/cachemeta --cachemode writeback vg0/cachedata --yes
lvconvert --type cache --cachepool vg0/cachedata vg0/root

References

Improve this question
1
  • If you solved your problem yourself, please answer your own question and accept it. Don’t put the answer in your question or the comments! Send me a message in the comments if you want my up-vote. :-)
    – David Foerster
    Feb 1, 2017 at 21:56

2 Answers 2

Reset to default
3

This is probably late as an answer, but I did the things in your post (but with a ubuntu 16.04.1 installation), and it worked with the smq-policy. It seems the issue you had has been resolved.

This is what I did, command by command.

First, I had a big harddisk on /dev/sda and an SSD on /dev/sdb.

I formatted the /dev/sda to have 10MB free in the beginning, then followed by a 1GB ext4 primary partition, which will be used for /boot in the installation (you cannot cache the kernel, so kernels can go in there). The rest of /dev/sda is cleared as sda2.

sudo pvcreate /dev/sda2
sudo pvcreate /dev/sdb
sudo vgcreate VG /dev/sda2 /dev/sdb
sudo lvcreate -L 3.5T -n lv VG /dev/sda2
sudo lvcreate -L 450G -n lv_cache VG /dev/sdb
sudo lvcreate -L 4.5G -n lv_cache_meta VG /dev/sdb
sudo lvconvert --type cache-pool --cachemode writeback --poolmetadata VG/lv_cache_meta VG/lv_cache
sudo lvs -a
sudo lvconvert --type cache --cachepool VG/lv_cache VG/lv
sudo lvs -a
sudo lvdisplay
sudo mkfs.ext4 /dev/VG/lv

On this point, I install ubuntu on the /dev/VG/lv partion as /, with /boot on /dev/sda1.

After installation, the kernel of this new installation needs to be recompiled with some modifications (in case of ubuntu 16.04.1) 

#### mount existing installation
sudo chmod 777 /var/cache/app-info/xapian/default -R
sudo add-apt-repository universe
sudo apt-get update
sudo apt-get install thin-provisioning-tools
sudo vgchange -a y VG
sudo mkdir /new
sudo nano /etc/fstab
###add this line to fstab: /dev/VG/lv /new ext4 defaults 0 0
sudo mount /dev/VG/lv

### We will need to install some new packages in the chroot, so fix internet
sudo mount --bind /dev /new/dev
sudo mount --bind /proc /new/proc
sudo mount --bind /sys /new/sys
sudo cp /etc/resolv.conf /new/etc/resolv.conf
sudo chroot /new
echo 'nameserver 8.8.4.4' | sudo tee -a /etc/resolv.conf
sudo add-apt-repository universe
sudo apt-get update
sudo apt-get install thin-provisioning-tools

#### recompile kernel with additional hook for dm_cache_smq
echo "dm_cache" >> /etc/initramfs-tools/modules
echo "dm_cache_mq"  >> /etc/initramfs-tools/modules
echo "dm_cache_smq"  >> /etc/initramfs-tools/modules
echo "dm_persistent_data"  >> /etc/initramfs-tools/modules
echo "dm_bufio"  >> /etc/initramfs-tools/modules

nano /etc/initramfs-tools/hooks/cache_hook

contents from here: http://forums.debian.net/viewtopic.php?f=5&t=119644 The last line has been modified for the smq to work

#!/bin/sh

PREREQ="lvm2"

prereqs()
{
    echo "$PREREQ"
}

case $1 in
prereqs)
    prereqs
    exit 0
    ;;
esac

if [ ! -x /usr/sbin/cache_check ]; then
    exit 0
fi

. /usr/share/initramfs-tools/hook-functions

copy_exec /usr/sbin/cache_check

manual_add_modules dm_cache dm_cache_mq dm_cache_smq dm_persistent_data dm_bufio

Then we continue with the following to compile the kernel and put it in the correct spot.

chmod +x /etc/initramfs-tools/hooks/cache_hook
mkdir /realboot
mount /dev/sda1 /realboot
update-initramfs -v -u -k all -b /realboot

Now reboot, and it worked. I tried this on 2 machines already, more to follow.

Improve this answer
0

To make the situation a bit more interesting, my root file system is on an lvm volume on an md raid 5 array. Writeback caching the root volume on an ssd disk rendered the system unbootable, stopping in busybox, where I need to read the above solution carefully to fix this. To restore booting, I started the system with a Ubuntu startup USB stick, installed mdadm to start the md array, added universe to /etc/apt/sources.list to be able to install thin-provisioning-tools and removed caching on the root volume:

Boot from Ubuntu USB startup stick
# sudo edit /etc/apt/sources.list # add "universe" to all deb lines
# sudo apt update
# sudo apt install -y mdadm
# sudo mdadm --assemble /dev/md0
# sudo apt install -y thin-provisioning-tools
# sudo lvm
lvm> lvconvert --splitcache vg/root
Logical volume vg/root is not cached and cache pool vg/CacheRoot is unused.
# exit
# reboot

Successful boot to uncached root and ready for next trial.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.