So I recently added a new hard drive to my existing (and working) LVM. Everything appeared to go smoothly and I received the outputs I expected through that whole process, but then after I tried to restart, the server is unable boot fully. Occasionally it will get to the log in, but within seconds it freezes and pops a bunch of these errors up on the screen.
This is a snippet I found in the kern.log
Jun 27 19:45:55 server kernel: [ 37.708137] ------------[ cut here ]------------
Jun 27 19:45:55 server kernel: [ 37.708735] kernel BUG at /build/linux-cs3yMe/linux-4.4.0/drivers/scsi/sd.c:844!
Jun 27 19:45:55 server kernel: [ 37.709289] invalid opcode: 0000 [#1] SMP
Jun 27 19:45:55 server kernel: [ 37.709863] Modules linked in: binfmt_misc joydev applesmc input_polldev coretemp kvm_intel kvm uvcvideo irqbypass videobuf2_vmalloc b43 videobuf2_m$
Jun 27 19:45:55 server kernel: [ 37.712044] CPU: 0 PID: 746 Comm: ext4lazyinit Not tainted 4.4.0-81-generic #104-Ubuntu
Jun 27 19:45:55 server kernel: [ 37.712044] Hardware name: Apple Inc. MacBookPro4,1/Mac-F42C86C8, BIOS MBP41.88Z.00C1.B03.0802271651 02/27/08
Jun 27 19:45:55 server kernel: [ 37.712044] task: ffff8800b7906600 ti: ffff8800b773c000 task.ti: ffff8800b773c000
Jun 27 19:45:55 server kernel: [ 37.712044] RIP: 0010:[<ffffffff815d42cd>] [<ffffffff815d42cd>] sd_setup_write_same_cmnd+0x14d/0x150
Jun 27 19:45:55 server kernel: [ 37.712044] RSP: 0018:ffff8800b773fac0 EFLAGS: 00010006
Jun 27 19:45:55 server kernel: [ 37.712044] RAX: 0000000000001000 RBX: ffff8800b5bf9870 RCX: 0000000000000000
Jun 27 19:45:55 server kernel: [ 37.712044] RDX: 0000000000200000 RSI: ffff88003522dc00 RDI: ffff8800b80ca480
Jun 27 19:45:55 server kernel: [ 37.712044] RBP: ffff8800b773fad0 R08: ffff8800352a8000 R09: 0000000000000000
Jun 27 19:45:55 server kernel: [ 37.712044] R10: ffff8800b5bf9870 R11: 0000000000000201 R12: ffff8800351d4000
Jun 27 19:45:55 server kernel: [ 37.712044] R13: ffff8800352a8000 R14: ffff8800352a8168 R15: ffff8800352a8000
Jun 27 19:45:55 server kernel: [ 37.712044] FS: 0000000000000000(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
Jun 27 19:45:55 server kernel: [ 37.712044] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jun 27 19:45:55 server kernel: [ 37.712044] CR2: 00007f0bc27da587 CR3: 00000000b0813000 CR4: 00000000000006f0
Jun 27 19:45:55 server kernel: [ 37.712044] Stack:
Jun 27 19:45:55 server kernel: [ 37.712044] ffff8800b80ca480 ffff8800351d4000 ffff8800b773fae0 ffffffff815d4596
Jun 27 19:45:55 server kernel: [ 37.712044] ffff8800b773fb00 ffffffff815c4ec0 ffff8800352a8168 ffff8800b5bf9870
Jun 27 19:45:55 server kernel: [ 37.712044] ffff8800b773fb38 ffffffff815c5071 ffff8800351d4000 ffff8800351d4000
Jun 27 19:45:55 server kernel: [ 37.712044] Call Trace:
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff815d4596>] sd_init_command+0x96/0xa0
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff815c4ec0>] scsi_setup_cmnd+0xf0/0x150
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff815c5071>] scsi_prep_fn+0xd1/0x170
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff813cad18>] blk_peek_request+0x168/0x290
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff815c683e>] scsi_request_fn+0x3e/0x610
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff813c4ce3>] __blk_run_queue+0x33/0x40
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff813cb570>] blk_queue_bio+0x3d0/0x3e0
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff813c95cf>] generic_make_request+0x10f/0x2c0
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff813c97f6>] submit_bio+0x76/0x170
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff813d2791>] blkdev_issue_write_same+0xb1/0x1a0
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff813d2931>] blkdev_issue_zeroout+0xb1/0x250
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff812ed65b>] ? jbd2_journal_get_write_access+0x5b/0x70
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff8129950d>] ext4_init_inode_table+0x18d/0x3a0
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff812b14d0>] ext4_lazyinit_thread+0x120/0x2b0
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff812b13b0>] ? ext4_chksum.isra.177+0x60/0x60
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff810a0c25>] kthread+0xe5/0x100
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff81840e0f>] ret_from_fork+0x3f/0x70
Jun 27 19:45:55 server kernel: [ 37.712044] [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
Jun 27 19:45:55 server kernel: [ 37.712044] Code: 00 00 00 89 87 04 01 00 00 41 8b 80 9c 00 00 00 89 43 5c e8 26 09 ff ff 44 89 63 5c 5b 41 5c 5d c3 5b b8 01 00 00 00 41 5c 5d c3 <$
Jun 27 19:45:55 server kernel: [ 37.712044] RIP [<ffffffff815d42cd>] sd_setup_write_same_cmnd+0x14d/0x150
Jun 27 19:45:55 server kernel: [ 37.712044] RSP <ffff8800b773fac0>
Jun 27 19:45:55 server kernel: [ 37.712044] ---[ end trace ec252a6392e51b4d ]---
I also found this from the dmesg:
wyth@server:/etc/lvm$ dmesg | tail
[11146.214607] sd 6:0:0:0: Attached scsi generic sg4 type 0
[11161.295791] sd 6:0:0:0: [sdd] 9767541167 512-byte logical blocks: (5.00 TB/4.55 TiB)
[11161.295798] sd 6:0:0:0: [sdd] 4096-byte physical blocks
[11161.296890] sd 6:0:0:0: [sdd] Write Protect is off
[11161.296897] sd 6:0:0:0: [sdd] Mode Sense: 53 00 00 08
[11161.297383] sd 6:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[11161.368195] sdd: sdd1
[11161.370774] sd 6:0:0:0: [sdd] Attached SCSI disk
[11161.686345] device-mapper: table: 252:2: adding target device sdd1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=1048576
[11161.686355] device-mapper: table: 252:2: adding target device sdd1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=1048576
wyth@server:/etc/lvm$
I ran e2fsck on the volume, and it appears to come back clean.
I tried updating the kernel to 4.11.7-041107-generic, which didn't make a difference.
If I edit out the line in the fstab I can get it to boot, but if I try to mount it, it crashes.
I went through the process of removing the new disk hoping going back to the original configuration would fix the issue, but that didn't help either.
I just finished running testdisk on it for the last 6 days, so I should at least have a backup of my data. I'm wondering what the next step I should try to do is? I recently just installed Ubuntu-gnome 17.04 on a laptop, should I try migrating the LVM to that to see if it will at least mount? Any suggestions are greatly appreciated, and if there's any other logs/info that would be helpful, I'll be happy to update the post.
Edit: Updating with lvdisplay, vgdisplay, pvdisplay, and fsck outputs
wyth@server:~$ sudo lvdisplay
--- Logical volume ---
LV Path /dev/media/volume
LV Name volume
VG Name media
LV UUID 0DVQoN-PTAy-jhya-eMlZ-yByZ-7tub-HLZwOr
LV Write Access read/write
LV Creation host, time ,
LV Status available
# open 0
LV Size 8.19 TiB
Current LE 2146187
Segments 4
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2
wyth@server:~$ sudo vgdisplay
--- Volume group ---
VG Name media
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 23
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 8.19 TiB
PE Size 4.00 MiB
Total PE 2146187
Alloc PE / Size 2146187 / 8.19 TiB
Free PE / Size 0 / 0
VG UUID k33MCv-X3Rv-d5xQ-rd4i-6iff-QYYX-3gNLqG
wyth@server:~$ sudo pvdisplay
--- Physical volume ---
PV Name /dev/sdb1
VG Name media
PV Size 3.64 TiB / not usable 2.83 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 953861
Free PE 0
Allocated PE 953861
PV UUID 6Z922e-jwLE-S5N8-nHLo-lQ9z-zqjq-Y5xEBs
--- Physical volume ---
PV Name /dev/sdc1
VG Name media
PV Size 4.55 TiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 1192326
Free PE 0
Allocated PE 1192326
PV UUID oXnkAi-dtur-Kl2I-vCUk-GMxu-Qbgq-pURY9n
wyth@server:~$ sudo fsck -f /dev/mapper/media-volume
fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
/dev/mapper/media-volume: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/media-volume: 138301/549429248 files (1.4% non-contiguous), 1929756762/2197695488 blocks
wyth@server:~$
fsckthe exiting LVM? Please describe your LVM? – heynnema Jul 5 '17 at 15:37