1

We just did a new release of our Ubuntu 16.04 systems in AWS. Besides apt-get updating and all, we added an explicit step in our puppet code to use tune2fs to add a UUID to the ext4 disks. (This is in preparation for moving to amazon's c5 instance types which use nvme device names, and we want to know which disk is which before and after.)

But then we needed to reboot a large number of those systems to instance-resize them in AWS (same instance family) and about 10% of them failed with filesystem corruption on their data drives (not the root drive).

grep -i ext4 /var/log/kern.log |grep xvdh
2019-03-14T14:07:39.954930+00:00 ip-10-2-219-30 kernel: [   26.059585] 
EXT4-fs (xvdh): ext4_check_descriptors: Checksum for group 0 failed                 (25645!=13919)
2019-03-14T14:07:39.961718+00:00 ip-10-2-219-30 kernel: [   26.064303]         EXT4-fs (xvdh): group descriptors corrupted!
2019-03-14T14:07:54.984741+00:00 ip-10-2-219-30 kernel: [   41.090302] EXT4-fs (xvdh): ext4_check_descriptors: Checksum for group 0 failed (25645!=13919)
2019-03-14T14:07:54.984757+00:00 ip-10-2-219-30 kernel: [   41.094897] EXT4-fs (xvdh): group descriptors corrupted!
2019-03-14T14:08:17.138117+00:00 ip-10-2-219-30 kernel: [   63.239655] EXT4-fs (xvdh): ext4_check_descriptors: Checksum for group 0 failed (25645!=13919)
2019-03-14T14:08:17.138141+00:00 ip-10-2-219-30 kernel: [   63.246723] EXT4-fs (xvdh): group descriptors corrupted!
2019-03-14T14:21:30.636962+00:00 redacted1 kernel: [    3.798075] EXT4-fs (xvdh): mounted filesystem with ordered data mode. Opts: (null)
2019-03-14T14:46:07.812220+00:00 redacted2 kernel: [    3.614731] EXT4-fs (xvdh): mounted filesystem with ordered data mode. Opts: (null)

We then have to fsck the drive to get the system back up.

The puppet code that makes this change follows. We're only using M4/C4 instance types so far so it should be all /dev/xvdh.

class our_storage::platforms::aws {

  # This shouldn't run during image generation.
  if $::packer_build != 'yes' {

    # If nvme0n1 is present this means we are using a M5 or C5 instance and then the data volume will be nvme1n1
    # We need to check the disk that are mounted in / because it might take time for the data volume to appear as totally mounted to the instance.
    # xvda    --> xvdh
    # nvme0n1 --> nvme1n1
    if $facts['disks']['nvme0n1'] {
      $st_volume = '/dev/nvme1n1'
    }
    elsif $facts['disks']['xvda'] {
      $st_volume = '/dev/xvdh'
    }
    else {
      fail("Invalid disk configuration ${facts['disks']}")
    }

    $fstype = 'ext4'
    $mount_opts = 'auto,noatime'

    # If /data is not mounted, go ahead and do it.
    if !$facts['mountpoints']['/data'] {

      # Get an unique, constant UUID for this volume.
      $ec2_userdata = parsejson($facts['ec2_userdata'])
      $domain = $ec2_userdata['domain']
      $subdomain = $ec2_userdata['subDomain']
      $st_volume_uuid = fqdn_uuid("${subdomain}.${domain}")

      # we may have to wait for the device to "appear"
      exec { 'Storage: waiting for data volume to be attached':
        path      => '/bin',
        command   => "lsblk -fn ${st_volume}",
        tries     => 60,
        try_sleep => 10,
        unless    => 'mountpoint -q -- "/data"',
        logoutput => true,
      } -> exec { 'Storage: formatting data volume': # WARNING: if we ever change from ext4, this will reformat volumes!
        path      => ['/sbin', '/bin'],
        command   => "mkfs.${fstype} -F ${st_volume}",
        unless    => "blkid ${st_volume} | grep -q 'TYPE=\"${fstype}\"'",
        logoutput => true,
      } -> exec { 'Storage: assign UUID to data volume':
        path      => ['/sbin', '/bin'],
        command   => "tune2fs ${st_volume} -U ${st_volume_uuid}",
        logoutput => true,
      } ~> mount { '/data':
        ensure  => mounted,
        device  => "UUID=${st_volume_uuid}",
        fstype  => $fstype,
        options => $mount_opts,
        require => File['/data'],
        before  => File[$our_storage::data_dirs],
      }
    } else {
      # Need to fetch the current UUID.
      # Cannot be changed if the volume is already mounted!
      $st_volume_uuid = $st_volume ? {
        '/dev/nvme1n1' => get_disk_uuid('/dev/nvme1n1'),
        '/dev/xvdh'    => get_disk_uuid('/dev/xvdh')
      }

      # If data is already mounted, just make sure that everything in fstab is in place.
      # e.g. it is using the UUID as disk identifier.
      mount { '/data':
        ensure  => mounted,
        device  => "UUID=${st_volume_uuid}",
        fstype  => $fstype,
        options => $mount_opts,
        require => File['/data'],
        before  => File[$our_storage::data_dirs],
      }
    }
  }
}

We haven't been able to figure out if this change is the culprit - it seems like the only relevant major change but we can't see how it would be corrupting things... The one thing we can correlate is that this seems somewhat biased towards busy systems where the EBS drive we're mounting is reasonably burst balance depleted so might be slow.

We've tried to reproduce this on a fleet of development systems but haven't been able to provoke the same failure.

I know we could automate the fsck but that's kinda spackling over whatever's doing the damage in the first place; what happens if it breaks things more than fsck can fix unattended? We run a big fleet.

Are there any known ways that doing a tune2fs on a slow or still-mounting system can corrupt an ext4 filesystem, or is there something else obvious we're doing that would lead to such corruption? And what can we do to determine if it is? Because this is intermittent non-reproducible and there were other changes (package updates and all) we can't be sure the UUID addition is the cause but it's sure suspicious timing wise.

0

You must log in to answer this question.

Browse other questions tagged .