4

I'm having a bizarre issue and all the pages that I've searched don't quite match my problem.

Basically, I can't access my small raid 1 array that comprises two 1TB WD Red disks (sdb and sdc in the fdisk check below).

Here are the usual checks (if I miss one please let me know):

fdisk You'll probably need to scroll the box below to see it all, also I have no idea what all the loops are...

$> sudo fdisk -l
Disk /dev/loop0: 140.7 MiB, 147496960 bytes, 288080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 13 MiB, 13619200 bytes, 26600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 3.7 MiB, 3878912 bytes, 7576 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 91 MiB, 95408128 bytes, 186344 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop4: 2.3 MiB, 2355200 bytes, 4600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop5: 14.5 MiB, 15208448 bytes, 29704 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop6: 34.6 MiB, 36216832 bytes, 70736 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop7: 88.5 MiB, 92778496 bytes, 181208 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 28352AE2-4322-4627-9BE2-DFBEDBAFF1BF

Device       Start       End   Sectors   Size Type
/dev/sda1     2048   1050623   1048576   512M EFI System
/dev/sda2  1050624 468860927 467810304 223.1G Linux filesystem


GPT PMBR size mismatch (1953519879 != 1953525167) will be corrected by w(rite).
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 84416481-C343-40E7-A8EB-3680B26FEF19

Device     Start        End    Sectors   Size Type
/dev/sdb1   2048 1953519615 1953517568 931.5G Linux filesystem


GPT PMBR size mismatch (1953519879 != 1953525167) will be corrected by w(rite).
Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 84416481-C343-40E7-A8EB-3680B26FEF19

Device     Start        End    Sectors   Size Type
/dev/sdc1   2048 1953519615 1953517568 931.5G Linux filesystem


Disk /dev/sdd: 119.2 GiB, 128035676160 bytes, 250069680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 4A8AA6CA-61E4-43A2-B616-EAD50214A106

Device      Start      End  Sectors  Size Type
/dev/sdd1    2048   999423   997376  487M EFI System
/dev/sdd2  999424 17000447 16001024  7.6G Linux swap


GPT PMBR size mismatch (1953519879 != 1953519615) will be corrected by w(rite).
Disk /dev/md126: 931.5 GiB, 1000202043392 bytes, 1953519616 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device       Boot Start        End    Sectors   Size Id Type
/dev/md126p1          1 1953519879 1953519879 931.5G ee GPT

Partition 1 does not start on physical sector boundary.

mdstat

$> cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md126 : active (auto-read-only) raid1 sdb[1] sdc[0]
      976759808 blocks super external:/md127/0 [2/2] [UU]

md127 : inactive sdc[1](S) sdb[0](S)
      5552 blocks super external:imsm

unused devices: <none>

mdadm.conf

$> sudo cat /etc/mdadm/mdadm.conf 
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY metadata=imsm UUID=fe0bb25b:d021df67:4d7fe09f:a30a6e08
ARRAY /dev/md/Volume1 container=fe0bb25b:d021df67:4d7fe09f:a30a6e08 member=0 UUID=3d2e36ef:e2314e97:11933fe5:f38135b1
ARRAY /dev/md/0  metadata=1.2 UUID=7d7acef8:cde50639:d9c04370:fbf727c6 name=chugster:0

# This configuration was auto-generated on Wed, 07 Aug 2019 00:10:23 +0100 by mkconf

mdadm -E /dev/sdb

$> sudo mdadm -E /dev/sdb
/dev/sdb:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : c1155891
         Family : c1155891
     Generation : 000000d2
     Attributes : All supported
           UUID : fe0bb25b:d021df67:4d7fe09f:a30a6e08
       Checksum : 03482b05 correct
    MPB Sectors : 1
          Disks : 2
   RAID Devices : 1

  Disk00 Serial : WD-WXV1E74D9L1F
          State : active
             Id : 00000002
    Usable Size : 1953519616 (931.51 GiB 1000.20 GB)

[Volume1]:
           UUID : 3d2e36ef:e2314e97:11933fe5:f38135b1
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 0
    Sector Size : 512
     Array Size : 1953519616 (931.51 GiB 1000.20 GB)
   Per Dev Size : 1953519880 (931.51 GiB 1000.20 GB)
  Sector Offset : 0
    Num Stripes : 7630936
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : normal
    Dirty State : clean
     RWH Policy : off

  Disk01 Serial : WD-WXV1E747PDZD
          State : active
             Id : 00000003
    Usable Size : 1953519616 (931.51 GiB 1000.20 GB)

mdadm -E /dev/sdc

$> sudo mdadm -E /dev/sdc
/dev/sdc:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : c1155891
         Family : c1155891
     Generation : 000000d2
     Attributes : All supported
           UUID : fe0bb25b:d021df67:4d7fe09f:a30a6e08
       Checksum : 03482b05 correct
    MPB Sectors : 1
          Disks : 2
   RAID Devices : 1

  Disk01 Serial : WD-WXV1E747PDZD
          State : active
             Id : 00000003
    Usable Size : 1953519616 (931.51 GiB 1000.20 GB)

[Volume1]:
           UUID : 3d2e36ef:e2314e97:11933fe5:f38135b1
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 1
    Sector Size : 512
     Array Size : 1953519616 (931.51 GiB 1000.20 GB)
   Per Dev Size : 1953519880 (931.51 GiB 1000.20 GB)
  Sector Offset : 0
    Num Stripes : 7630936
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : normal
    Dirty State : clean
     RWH Policy : off

  Disk00 Serial : WD-WXV1E74D9L1F
          State : active
             Id : 00000002
    Usable Size : 1953519616 (931.51 GiB 1000.20 GB)

mdadm detail scan

$> sudo mdadm --detail --scan
ARRAY /dev/md/imsm0 metadata=imsm UUID=fe0bb25b:d021df67:4d7fe09f:a30a6e08
ARRAY /dev/md/Volume1 container=/dev/md/imsm0 member=0 UUID=3d2e36ef:e2314e97:11933fe5:f38135b1

So as a little background, sdc failed with a missing superblock, but I read something somewhere that allowed me to "patch" sdc using the uuid of sdb. So, now "mdadm -E /dev/sdc" shows information rather than saying the superblock is missing. I'm not sure whether what I've done is the right thing to do.

If I try and assemble the raid it says that /dev/md127 doesn't exist in mdadm.conf. If I try to regenerate the mdadm.conf, then it doesn't add /dev/md127.

Basically, I have no idea how to reassemble the raid, or why it failed in the first place. Disk utility says that both of the disks are ok.

If all else fails, can I remove md127 from the array, mount the array with one disk (md126), remove all partitions on what is currently sdc and then add it back to the array?

Your help is very much appreciated.

Andrew

Edit 1 It may also help to know that all this happened when I reinstalled the OS - to go from 14.4 to 18.4.

Edit 2

I have just noticed that I can examine sdb1 but not sdc1:

$> sudo mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7d7acef8:cde50639:d9c04370:fbf727c6
           Name : chugster:0  (local to host chugster)
  Creation Time : Tue Aug  6 23:38:40 2019
     Raid Level : linear
   Raid Devices : 2

 Avail Dev Size : 1953253376 (931.38 GiB 1000.07 GB)
  Used Dev Size : 0
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=0 sectors
          State : clean
    Device UUID : beeda35f:a7c7f529:33e2c551:4bc87bfc

    Update Time : Tue Aug  6 23:38:40 2019
  Bad Block Log : 512 entries available at offset 8 sectors
       Checksum : f2302886 - correct
         Events : 0

       Rounding : 0K

   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)


$> sudo mdadm --examine /dev/sdc1
mdadm: cannot open /dev/sdc1: No such file or directory

I think something is seriously messed up with /dev/sdc. I'm not sure how to remove /dev/sdc from the array, given that /dev/sdc1 doesn't exist. Also I'm assuming that I remove it from md127, but that doesn't feel right, perhaps I should be trying to remove it from /dev/md/Volume1? The other thing that is concerning me is that /proc/mdstat suggests that the superblock for md126 is on md127, or am I reading that wrong?

Edit 3 Made a correction

  • By the way, I don't care if I need to drop a disk, my aim now is just to recover the data. I did make a backup of the disk just before my server went down after an update a few months back, and I don't think anything important has changed since that point, so if it really comes down to it, I'm not adverse to the idea of entirely wiping out the raid and starting again, but I'd rather not do that if I don't have to do so. – apmac Aug 7 at 14:26
  • This is starting to get a little more urgent, any help would be much appreciated. – apmac Aug 9 at 12:05
0

I really hate fake raid. It's a HW feature that users gravitate towards because they equate HW == better where it all it really does is complicate your storage setup and make it more brittle. The only time fake raid matters is when you want to dual boot and share the same volume between multiple operating systems. Otherwise run away from it like it's the plague.

The thing that really stands out to me is that you have partitions that are tagged with a filesystem which appear to span the entire disk, yet you assign the entire block device to the RAID. That's how you corrupt data. It could have gotten mounted at some point or fsck was run on it at boot which "repaired it" and that's when your superblock got corrupted.

It's fine to partition disks that are assigned to a RAID, just make sure you tag them as type FD (linux raid autodetect) so these sorts of collisions don't happen. The file system goes on the MD device.

At this point. I would boot from USB disk. Bring the array online. Forcibly remove "sdc", dd the entire thing with zeros, and then add it back to the array for a full resync.

Or just start over. You said you have a backup. disassemble the array, zero the superblocks or just dd if=/dev/zero of=/... and this time just use md, no fake raid. I advise that you create a single partition on each disk that spans all the space and tag it as FD so this doesn't happen again.

https://www.tecmint.com/create-raid1-in-linux/

Good luck.

A side note concerning fake raid. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/rst-linux-paper.pdf

"The recommended software RAID implementation in Linux* is the open source MD RAID package. Intel has enhanced MD RAID to support RST metadata and OROM and it is validated and supported by Intel for server platforms. There is a growing interest at OEMs in having Intel extend the validation and support for RST on mobile, desktop and workstation platforms in a Windows and Linux dual-boot environment"

Which reads "HW vendors are lazy and don't want to deal with operating systems so they want to prebuild systems 'with a RAID' and pretend they added value for the customer"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.