9

I stupidly decided to update from 14.04LTS to 14.10 and then 15.04.

Since doing that, my website has gone down and file system has become read only. I have no idea what has gone wrong, as the updates completed successfully.

This is what I have found so far:

    root@lew:/# service apache2 status
apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2)
   Active: failed (Result: exit-code) since Sun 2015-07-12 08:36:18 EDT; 31min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 901 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)

Jul 12 08:36:18 lew.im systemd[1]: Starting LSB: Apache2 web server...
Jul 12 08:36:18 lew.im apache2[901]: * Starting web server apache2
Jul 12 08:36:18 lew.im apache2[901]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: Read-only file system
Jul 12 08:36:18 lew.im apache2[901]: /etc/init.d/apache2: 91: /etc/init.d/apache2: cannot create : Directory nonexistent
Jul 12 08:36:18 lew.im apache2[901]: *
Jul 12 08:36:18 lew.im apache2[901]: * The apache2 configtest failed.
Jul 12 08:36:18 lew.im systemd[1]: apache2.service: control process exited, code=exited status=1
Jul 12 08:36:18 lew.im systemd[1]: Failed to start LSB: Apache2 web server.
Jul 12 08:36:18 lew.im systemd[1]: Unit apache2.service entered failed state.
Jul 12 08:36:18 lew.im systemd[1]: apache2.service failed.

then fdisk -l:

root@lew:/# fdisk -l

Disk /dev/vda: 20 GiB, 21476933632 bytes, 41947136 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 06F7B3C9-8E13-42CD-AD52-7A02301B6F16

Device     Start      End  Sectors Size Type
/dev/vda1   2048 41945087 41943040  20G Linux filesystem

and fsck /

root@lew:/# sudo fsck /
fsck from util-linux 2.25.2
fsck.ext4: Unable to resolve 'UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58'

I am a beginner at all of this, but from what I have read, I need to fix something in fstab? Why has updating broken this, what could have gone wrong?

I SSH in to this server, as it is hosted in DigitalOcean.

Edit:

Blkid

root@lew:~# blkid
/dev/vda1: LABEL="DOROOT" UUID="18254707-08e8-494e-b456-938592928a5e" TYPE="ext4" PTTYPE="dos" PARTLABEL="primary" PARTUUID="8c484e81-f919-4803-acc7-1447fdd81b45"

Mount

root@lew:~# mount
/dev/vda1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,nodev,noexec,nosuid)
sysfs on /sys type sysfs (rw,nodev,noexec,nosuid)
none on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755,size=1024)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,nodev,noexec,nosuid,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,nodev,noexec,nosuid,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,noexec,nodev,none,name=systemd)

Fstab

root@lew:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/vda1 during installation
#UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 /               ext4    errors=remount-ro 0       1
UUID=06F7B3C9-8E13-42CD-AD52-7A02301B6F16 /               ext4    errors=remount-rw 0       1

/swapfile       none    swap    sw      0       0
13
  • 1
    What would more interesting is the output of mount and the output of sudo blkid. May be it's only /tmp that is read-only ?
    – solsTiCe
    Jul 12, 2015 at 13:55
  • And maybe the content of /etc/fstab could be interesting ( = output of cat /etc/fstab) to edit to your question.
    – Byte Commander
    Jul 12, 2015 at 15:23
  • Just got locked out after trying to access the console via the web interface, it seems to have wiped my SSH keys. Just waiting on DigitalOcean to fix it. Then I'll be able to run the commands. I know that in etc/fstab the UUID is the 815063a9 one from the fsck / output. I have edited that to be the identifier from the fdisk -l output, could that fix it? @ByteCommander Jul 12, 2015 at 15:36
  • Yes, maybe. Make sure in fstab it is not selected to be mounted read-only too.
    – Byte Commander
    Jul 12, 2015 at 15:50
  • 1
    @MHS I did! If you email support, ask them to mount the recovery ISO, mount the filesystem and the open up /etc/fstab. Changed the UUID in there to the output of blkid, save and ask DO to remove the recovery disk. Restart and you should have access again! Jul 26, 2015 at 15:00

3 Answers 3

8

The solution was posted in the comments by @Lewis Lebentz Jul 26 at 15:00.

I'll paraphrase so anyone looking for the answer can find it here easily. But @Lewis should post the answer himself mark it as answered and you get due credit.

The solution: Open a support ticket, ask Digital Ocean to mount the recovery ISO (It's a special ISO that only they can mount).

  1. Choose 1 to mount the filesystem and the edit /etc/fstab. Note: Use the console and run nano or vi /mnt/etc/fstab. Alternatively you can enable SSH and networking (in the recovery options) to login with your terminal (see instruction) though I havn't tried that myself.
  2. Changed the UUID in there to the output of blkid, save.
  3. Ask DO to remove the recovery disk. Restart and you should have access again!
3
  • 11
    You don't need to use the recovery mode; mount -rw -o remount UUID=18254707-08e8-494e-b456-938592928a5e /, edit /etc/fstab, and then reboot.
    – ændrük
    Oct 2, 2015 at 19:59
  • mount -rw -o remount UUID=ID_RETRIEVED_FROM_BLKID worked for me! Thanks :) Oct 8, 2016 at 4:23
  • In case anyone is curious, the contents of /etc/fstab prior to this change are: UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 / ext4 errors=remount-ro 0 1 /swapfile none swap sw 0 0
    – Rob
    Mar 21, 2020 at 15:00
5

You can do as ændrük posted in comments:

$ mount -rw -o remount /dev/vda1 /
$ sed s/wrong_uuid/correct_uuid/ -i /etc/fstab

..and then boot your linux again! Make sure you change vda1 with your device name. And in the sed command, the right uuids, of course!

1
  • This worked for me except the device was just /dev/vda - obviously you need to put in what is specific to your server.
    – jfacemyer
    Feb 28, 2018 at 19:09
3

I found this happening to me aswell. The disk UUID in /etc/fstab couldn't be resolved. I fixed this by first finding the UUID of the disk by running

sudo blkid -c /dev/null -o list

And copying the disk UUID for mount point /

I then followed @ændrük comment and remounted the disk with

mount -rw -o remount UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx

I then edited /etc/fstab to change the disk UUID for the root disk.

1
  • This simple solution worked for me. After those 2 changes, I rebooted my system with reboot
    – AKKAweb
    Apr 13, 2020 at 19:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .