7

I stupidly decided to update from 14.04LTS to 14.10 and then 15.04.

Since doing that, my website has gone down and file system has become read only. I have no idea what has gone wrong, as the updates completed successfully.

This is what I have found so far:

    root@lew:/# service apache2 status
apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2)
   Active: failed (Result: exit-code) since Sun 2015-07-12 08:36:18 EDT; 31min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 901 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)

Jul 12 08:36:18 lew.im systemd[1]: Starting LSB: Apache2 web server...
Jul 12 08:36:18 lew.im apache2[901]: * Starting web server apache2
Jul 12 08:36:18 lew.im apache2[901]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: Read-only file system
Jul 12 08:36:18 lew.im apache2[901]: /etc/init.d/apache2: 91: /etc/init.d/apache2: cannot create : Directory nonexistent
Jul 12 08:36:18 lew.im apache2[901]: *
Jul 12 08:36:18 lew.im apache2[901]: * The apache2 configtest failed.
Jul 12 08:36:18 lew.im systemd[1]: apache2.service: control process exited, code=exited status=1
Jul 12 08:36:18 lew.im systemd[1]: Failed to start LSB: Apache2 web server.
Jul 12 08:36:18 lew.im systemd[1]: Unit apache2.service entered failed state.
Jul 12 08:36:18 lew.im systemd[1]: apache2.service failed.

then fdisk -l:

root@lew:/# fdisk -l

Disk /dev/vda: 20 GiB, 21476933632 bytes, 41947136 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 06F7B3C9-8E13-42CD-AD52-7A02301B6F16

Device     Start      End  Sectors Size Type
/dev/vda1   2048 41945087 41943040  20G Linux filesystem

and fsck /

root@lew:/# sudo fsck /
fsck from util-linux 2.25.2
fsck.ext4: Unable to resolve 'UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58'

I am a beginner at all of this, but from what I have read, I need to fix something in fstab? Why has updating broken this, what could have gone wrong?

I SSH in to this server, as it is hosted in DigitalOcean.

Edit:

Blkid

root@lew:~# blkid
/dev/vda1: LABEL="DOROOT" UUID="18254707-08e8-494e-b456-938592928a5e" TYPE="ext4" PTTYPE="dos" PARTLABEL="primary" PARTUUID="8c484e81-f919-4803-acc7-1447fdd81b45"

Mount

root@lew:~# mount
/dev/vda1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,nodev,noexec,nosuid)
sysfs on /sys type sysfs (rw,nodev,noexec,nosuid)
none on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755,size=1024)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,nodev,noexec,nosuid,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,nodev,noexec,nosuid,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,noexec,nodev,none,name=systemd)

Fstab

root@lew:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/vda1 during installation
#UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 /               ext4    errors=remount-ro 0       1
UUID=06F7B3C9-8E13-42CD-AD52-7A02301B6F16 /               ext4    errors=remount-rw 0       1

/swapfile       none    swap    sw      0       0
  • 1
    What would more interesting is the output of mount and the output of sudo blkid. May be it's only /tmp that is read-only ? – solsTiCe Jul 12 '15 at 13:55
  • And maybe the content of /etc/fstab could be interesting ( = output of cat /etc/fstab) to edit to your question. – Byte Commander Jul 12 '15 at 15:23
  • Just got locked out after trying to access the console via the web interface, it seems to have wiped my SSH keys. Just waiting on DigitalOcean to fix it. Then I'll be able to run the commands. I know that in etc/fstab the UUID is the 815063a9 one from the fsck / output. I have edited that to be the identifier from the fdisk -l output, could that fix it? @ByteCommander – Lewis Lebentz Jul 12 '15 at 15:36
  • Yes, maybe. Make sure in fstab it is not selected to be mounted read-only too. – Byte Commander Jul 12 '15 at 15:50
  • 1
    @MHS I did! If you email support, ask them to mount the recovery ISO, mount the filesystem and the open up /etc/fstab. Changed the UUID in there to the output of blkid, save and ask DO to remove the recovery disk. Restart and you should have access again! – Lewis Lebentz Jul 26 '15 at 15:00
7

The solution was posted in the comments by @Lewis Lebentz Jul 26 at 15:00.

I'll paraphrase so anyone looking for the answer can find it here easily. But @Lewis should post the answer himself mark it as answered and you get due credit.

The solution: Open a support ticket, ask Digital Ocean to mount the recovery ISO (It's a special ISO that only they can mount).

  1. Choose 1 to mount the filesystem and the edit /etc/fstab. Note: Use the console and run nano or vi /mnt/etc/fstab. Alternatively you can enable SSH and networking (in the recovery options) to login with your terminal (see instruction) though I havn't tried that myself.
  2. Changed the UUID in there to the output of blkid, save.
  3. Ask DO to remove the recovery disk. Restart and you should have access again!
| improve this answer | |
  • 11
    You don't need to use the recovery mode; mount -rw -o remount UUID=18254707-08e8-494e-b456-938592928a5e /, edit /etc/fstab, and then reboot. – ændrük Oct 2 '15 at 19:59
  • mount -rw -o remount UUID=ID_RETRIEVED_FROM_BLKID worked for me! Thanks :) – Steven Rosato Oct 8 '16 at 4:23
  • In case anyone is curious, the contents of /etc/fstab prior to this change are: UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 / ext4 errors=remount-ro 0 1 /swapfile none swap sw 0 0 – Rob Mar 21 at 15:00
5

You can do as ændrük posted in comments:

$ mount -rw -o remount /dev/vda1 /
$ sed s/wrong_uuid/correct_uuid/ -i /etc/fstab

..and then boot your linux again! Make sure you change vda1 with your device name. And in the sed command, the right uuids, of course!

| improve this answer | |
  • This worked for me except the device was just /dev/vda - obviously you need to put in what is specific to your server. – jfacemyer Feb 28 '18 at 19:09
1

I found this happening to me aswell. The disk UUID in /etc/fstab couldn't be resolved. I fixed this by first finding the UUID of the disk by running

sudo blkid -c /dev/null -o list

And copying the disk UUID for mount point /

I then followed @ændrük comment and remounted the disk with

mount -rw -o remount UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx

I then edited /etc/fstab to change the disk UUID for the root disk.

| improve this answer | |
  • This simple solution worked for me. After those 2 changes, I rebooted my system with reboot – Andre S Apr 13 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.