- Ubuntu-14.04-LTS (Server) installed.
- It worked fine in the past year.
- Now, whenever I try to install a package file-system crashes and remounts as read-only.
Working on it
Trying to install a package (via
dpkg -i or
apt-get install) results in:
EXT4-fs error: mb_free_blocks:1433: group 3, block 130894:freeing already freed block (bit 32590); block bitmap corrupt.
Aborting journal on device vda1-8.
EXT4-fs error: ext4_journal_check_start:56: Detected aborted journal EXT4-fs: Remounting filesystem read-only EXT4-fs: Remounting filesystem read-only EXT4-fs error: ext4_mb_generate_buddy:756: group 3, 8181 clusters in bitmap, 8185 in gd; block bitmap corrupt.
fopen: Read-only file system debconf: DbDriver "config": could not write /var/cache/debconf/config.dat-new: Read-only file system dpkg: error processing package man-db (--install): subprocess installed post-installation script returned error exit status 1 dpkg: error: unable to create new file '/var/lib/dpkg/available-new': Read-only file system
The reason it mounts on read-only mode is because the default setting in
UUID=[MY-FS-UUID] / ext4 errors=remount-ro 0 1
/var/log/syslog I can find:
EXT4-fs: INFO: recovery required on readonly filesystem EXT4-fs: write access will be enabled during recovery EXT4-fs: orphan cleanup on readonly fs EXT4-fs: 3 orphan inodes deleted EXT4-fs: recovery complete EXT4-fs: mounted filesystem with ordered data mode. Opts: (null) Adding 1046524k swap. Priority:-1 extents:1 across:1046524k FS EXT4-fs: re-mounted. Opts: errors=remount-ro
I tried using some of the solutions in the following:
by doing so:
mount -o remount,rw -t ext4 MY_DISK /
mount: cannot remount block device MY_DISK read-write, is write-protected
blockdev -v --setrw MY_DISK
set read-write succeeded.
hdparm -r 0 MY_DISK
MY_DISK: setting readonly to 0 (off) readonly = 0 (off)
I can't access the disk. It seems that the system wrote to a place outside the disk and now it's corrupted. I made a QCOW image of the disk. I'm trying to run it as a VM, and analyze the corrupted image.
libguestfs and I'm trying some analysis tools such as
My main goal now is trying to access and read the system logs.
In order to access the image, I connected the guest image as a device on the host x86 (credits: http://www.randomhacks.co.uk/how-to-recover-fsck-a-qcow2-file/, https://www.scaleway.com/docs/connect-a-block-device-manually/):
# fsck /dev/nbd0p1
/dev/nbd0p1: clean, 62744/983040 files, 447645/3931904 blocks
partition 1 is the os one. On partitions 2,5 I get errors from
fsck - those are the extend and swap partitions, respectively.
# mount /dev/nbd0p1 /mnt/crpt
# cat /mnt/crpt/var/log/*
... currently searching through the logs ...
Any idea what might cause the system issue?