Btrfs logical volume groups crash making the device unmountable

Question

I did an online upgrade from soft raid5 (3TB*8disk) to raid6 (3TB*10disk). This causes the logical volume groups to crash making the device unmountable. I try to recover the files using the method described here.

But even after completing the last step "btrfs restore -i -o -v -t ", nothing is restored.

Please help me to recover my files. Thanks in advance.

    bmxboy@bmxboy-virtual-machine:/$ sudo mount -t btrfs /dev/vg1/volume_1 /mnt/ds001 -o ro
        mount: /mnt/ds001: wrong fs type, bad option, bad superblock on /dev/mapper/vg1-volume_1, missing codepage or helper program, or other error.
    bmxboy@bmxboy-virtual-machine:/$ sudo mdadm -D /dev/md2
    /dev/md2:
               Version : 1.2
         Creation Time : Thu Mar 28 20:24:15 2019
            Raid Level : raid6
            Array Size : 23403556352 (22319.37 GiB 23965.24 GB)
         Used Dev Size : 2925444544 (2789.92 GiB 2995.66 GB)
          Raid Devices : 10
         Total Devices : 10
           Persistence : Superblock is persistent

           Update Time : Fri Apr 19 08:55:24 2019
                 State : clean 
        Active Devices : 10
       Working Devices : 10
        Failed Devices : 0
         Spare Devices : 0

                Layout : left-symmetric
            Chunk Size : 64K

    Consistency Policy : resync

                  Name : DS001:2
                  UUID : 356ff063:76450548:c57674b6:69223636
                Events : 29235

        Number   Major   Minor   RaidDevice State
           0       8      131        0      active sync   /dev/sdi3
           1       8      115        1      active sync   /dev/sdh3
           2       8       67        2      active sync   /dev/sde3
           3       8       83        3      active sync   /dev/sdf3
           4       8       51        4      active sync   /dev/sdd3
           8       8       99        5      active sync   /dev/sdg3
           6       8      147        6      active sync   /dev/sdj3
          12       8       19        7      active sync   /dev/sdb3
          11       8      163        8      active sync   /dev/sdk3
          10       8       35        9      active sync   /dev/sdc3

        bmxboy@bmxboy-virtual-machine:/$ dmesg | grep BTRFS
        [   17.026125] BTRFS: device label 2019.03.28-12:24:35 v23739 devid 1 transid 6011 /dev/dm-1
        [  168.013852] BTRFS info (device dm-1): using free space tree
        [  168.013855] BTRFS info (device dm-1): has skinny extents
        [  168.127156] BTRFS error (device dm-1): parent transid verify failed on 887020683264 wanted 6011 found 6270
        [  168.133110] BTRFS error (device dm-1): parent transid verify failed on 887020683264 wanted 6011 found 6270
        [  168.133118] BTRFS warning (device dm-1): failed to read tree root
        [  168.151448] BTRFS error (device dm-1): open_ctree failed
        [ 8686.922437] BTRFS warning (device dm-1): 'recovery' is deprecated, use 'usebackuproot' instead
        [ 8686.922439] BTRFS info (device dm-1): trying to use backup root at mount time
        [ 8686.922441] BTRFS info (device dm-1): disabling free space tree
        [ 8686.922444] BTRFS error (device dm-1): cannot disable free space tree
        [ 8686.923395] BTRFS error (device dm-1): open_ctree failed
        [ 9761.164452] BTRFS info (device dm-1): disabling free space tree
        [ 9761.164455] BTRFS error (device dm-1): cannot disable free space tree
        [ 9761.165254] BTRFS error (device dm-1): open_ctree failed
        [14237.853328] BTRFS info (device dm-1): unrecognized mount option '0'
        [14237.853528] BTRFS error (device dm-1): open_ctree failed


bmxboy@bmxboy-virtual-machine:/$ sudo btrfs fi  show
Label: '2019.03.28-12:24:35 v23739'  uuid: 90877621-a602-4915-914c-70eada4c3428
        Total devices 1 FS bytes used 14.68TiB
        devid    1 size 19.07TiB used 15.50TiB path /dev/mapper/vg1-volume_1

Answer 1

run "btrfs-find-root /dev/vg1/volume_1&> /mnt/iscsi/btrfs-find-root &"

parent transid verify failed on 887020683264 wanted 6011 found 6270
Ignoring transid failure
Couldn't setup extent tree
Couldn't setup device tree
Superblock thinks the generation is 6011
Superblock thinks the level is 1
Found tree root at 887014998016 gen 6011 level 1
Well block 448900530176(gen: 7903 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 777465511936(gen: 7844 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 503614685184(gen: 7412 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 470875013120(gen: 7343 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 470870278144(gen: 7341 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 471096754176(gen: 7063 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 471050649600(gen: 6927 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 558475149312(gen: 6774 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 886999793664(gen: 6567 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 886999777280(gen: 6567 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 470921674752(gen: 6291 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887014064128(gen: 6269 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887014031360(gen: 6269 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887014014976(gen: 6269 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887013965824(gen: 6269 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887013326848(gen: 6268 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887013294080(gen: 6268 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887013244928(gen: 6268 level: 0) seems good, but generation/level doesn't match, want gen: 6011 level: 1
Well block 887010803712(gen: 6267 level: 1) seems good, but generation/level doesn't match, want gen: 6011 level: 1
........

Answer 2

run 1."vim /mnt/iscsi/btrfs-restore-from-tree.sh" (I think this code may be wrong ???I am not sure)

#!/bin/bash
for i in `tac /mnt/iscsi/btrfs-find-root | grep 'Well block' | awk '{print $3}' | sed "s/(.*$//"`; do echo "--- Well block $i ---"; btrfs restore -F -D -i -v -t $i /dev/vg1/volume_1 /dev/null 2>&1 | tee /mnt/iscsi/rest-btrfs-restore-wb-$i.1; done
~

2."chmod +x /mnt/iscsi/btrfs-restore-from-tree.sh"

3."/mnt/iscsi/btrfs-restore-from-tree.sh &> /mnt/iscsi/restored-from-tree &"

After running step 1-3 ,directory"/mnt/iscsi/ "create 2000+++ files which name "rest-btrfs-restore-wb***"

examine one of this files，I think the code of setp 1 maybe wrong???

bmxboy@bmxboy-virtual-machine:~$ cat /mnt/iscsi/rest-btrfs-restore-wb-503745953792.1 
usage: btrfs restore [options] <device> <path> | -l <device>

    Try to restore files from a damaged filesystem (unmounted)

    -s|--snapshots       get snapshots
    -x|--xattr           restore extended attributes
    -m|--metadata        restore owner, mode and times
    -S|--symlink         restore symbolic links
    -v|--verbose         verbose
    -i|--ignore-errors   ignore errors
    -o|--overwrite       overwrite
    -t <bytenr>          tree location
    -f <bytenr>          filesystem location
    -u|--super <mirror>  super mirror
    -r|--root <rootid>   root objectid
    -d                   find dir
    -l|--list-roots      list tree roots
    -D|--dry-run         dry run (only list files that would be recovered)
    --path-regex <regex>
                         restore only filenames matching regex,
                         you have to use following syntax (possibly quoted):
                         ^/(|home(|/username(|/Desktop(|/.*))))$
    -c                   ignore case (--path-regex only)