1

I was trying to take a hard drive out of a large LVM group consisting of 5 drives. It had a 5.5TB filesystem (ext4). The LV was 7.5TB. I wanted to reduce the LV by 1.82TiB so that I then could shrink the VG and pvmove the drive off the group.

I did the following:
fsck -f (passed)
lvreduce -L 1.82T /dev/mapper/my_storage_vg-storage_lv

this produces the warning:

Rounding up size to full physical extent 1.82 TiB
  WARNING: Reducing active logical volume to 1.82 TiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce storage_lv? [y/n]:

Which is what I expected, so I said yes, assuming I had the space to give.

HOWEVER, you may have noticed I forgot the minus sign on the 1.82T, so instead of reducing it by 1.82T, I instead reduced it to 1.82T!!
My next command was an fsck, which failed and I said no to any repairs, immediately seeing the error.
The next command I tried was:
lvextend -l +100%FREE /dev/mapper/my_storage_vg-storage_lv

This restored the size of the LV.

I then tried
fsck- f
but this time the fsck failed once it got to block bitmap differences.

I suspect the lvextend command put the blocks back in, in a different order then they were in when they were removed, essentially scrambling the filesystem.

Does this sound correct, and if so what can I do to unscramble the ext4 file system? Anything at all?

All help greatly appreciated...


Addendum: I did a pvscan at an earlier point in the process:

PV /dev/sdf1   VG my_storage_vg   lvm2 [1.82 TiB / 0    free]
PV /dev/sda1   VG my_storage_vg   lvm2 [1.82 TiB / 0    free]
PV /dev/sde1   VG my_storage_vg   lvm2 [1.82 TiB / 0    free]
PV /dev/sdc1   VG my_storage_vg   lvm2 [1.36 TiB / 0    free]
PV /dev/sdd1   VG my_storage_vg   lvm2 [698.63 GiB / 698.63 GiB free]

I suspect this is the order that the drives were added to the vg originally (maybe, not sure). I am considering taking the LV and VG down to 0 size, then adding in the drives one at a time

vgextend /dev/sda1
lvextend sda1
vg extend sde1
lvextend sde1
etc...

Is there a better way to do this?


Better yet:

lvreduce -l 0 /dev/mapper/my_storage_vg-storage_lv
lvextend /dev/mapper/my_storage_vg-storage_lv /dev/sdf1
lvextend /dev/mapper/my_storage_vg-storage_lv /dev/sda1
lvextend /dev/mapper/my_storage_vg-storage_lv /dev/sde1
lvextend /dev/mapper/my_storage_vg-storage_lv /dev/sdc1
fsck -f /dev/mapper/my_storage_vg-storage_lv

1 Answer 1

1

You can't extend the LV and hope to hit the right layout at random. Use vgcfgrestore (I'll let you read the manual) to restore the configuration from before your lvreduce. Do not mount or fsck anything.

7
  • Thank you so much, Gabriel. This is what I needed. However, it did not work entirely. I restored it to the point prior to the lvreduce, but an fsck still showed errors. The only command I invoked on the vg in it's reduced state was an fsck -f. I answered no to a few <fix?> prompts, then ctrl-c the process. Assuming the vgcfgrestore worked, that would imply that the fsck modified things (which I thought required user confirmation). At least I am a step closer....
    – Peter
    Sep 20, 2013 at 15:20
  • I am looking at one of 2 options now I guess: 1) figure out how to undo the fsck changes manually. 2) Allow fsck to fix the system and hope that it can figure out how to undo its changes itself. :/
    – Peter
    Sep 20, 2013 at 15:50
  • You can do lvchange --permission r to make the volume read-only, then use e2fsck -n or mount -o ro to get an idea of the filesystem state.
    – Gabriel
    Sep 20, 2013 at 16:14
  • 1
    I was suggesting those checks so that you'd know how much you've diverged from the original filesystem. There are ways to make snapshots, but that could also make things worse if you restored the wrong vg backup.
    – Gabriel
    Sep 20, 2013 at 23:26
  • 2
    Success! Filesystem restored. Even had the nerve to try the lvreduce again. This time correctly with lots of read only step checking. Sitting mid pvmove now. :)
    – Peter
    Sep 21, 2013 at 5:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .