Server upgrade fails, boot partition too small

Question

The upgrade from Ubuntu 12.10 to 13.10 aborts right after starting, due to the /boot partition having only 54MB size. The server has 2 hard disks with 2 partitions each, being connected via RAID1. Since the server is located in a datacenter, it cannot be booted with a live system disk, to use gparted. The rescue system, loaded via network, does not access the hard disk (fdisk -l is empty there).

Question being: How can I make /boot bigger, so the initial RAM disks fit into (suggested: 300MB)?

do-release-upgrade shows this:

Not enough free disk space

The upgrade has aborted. The upgrade needs a total of 55.2 M free
space on disk '/boot'. Please free at least an additional 29.2 M of
disk space on '/boot'. Empty your trash and remove temporary packages
of former installations using 'sudo apt-get clean'.

uname -r shows this: 3.5.0-47-generic

dpkg -l linux-im* shows this:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                   Version                  Architecture             Description
+++-======================================-========================-========================-=================================================================================
un  linux-image                            <none>                                            (no description available)
un  linux-image-2.6                        <none>                                            (no description available)
un  linux-image-2.6.24-16-server           <none>                                            (no description available)
un  linux-image-2.6.24-23-server           <none>                                            (no description available)
un  linux-image-2.6.24-24-server           <none>                                            (no description available)
un  linux-image-2.6.24-25-server           <none>                                            (no description available)
un  linux-image-2.6.24-28-server           <none>                                            (no description available)
rc  linux-image-2.6.32-32-generic-pae      2.6.32-32.62             i386                     Linux kernel image for version 2.6.32 on x86
rc  linux-image-2.6.32-45-generic-pae      2.6.32-45.104            i386                     Linux kernel image for version 2.6.32 on x86
rc  linux-image-2.6.32-46-generic-pae      2.6.32-46.108            i386                     Linux kernel image for version 2.6.32 on x86
rc  linux-image-2.6.32-47-generic-pae      2.6.32-47.109            i386                     Linux kernel image for version 2.6.32 on x86
rc  linux-image-2.6.32-48-generic-pae      2.6.32-48.110            i386                     Linux kernel image for version 2.6.32 on x86
rc  linux-image-2.6.32-53-generic-pae      2.6.32-53.115            i386                     Linux kernel image for version 2.6.32 on x86
rc  linux-image-2.6.32-54-generic-pae      2.6.32-54.116            i386                     Linux kernel image for version 2.6.32 on x86
rc  linux-image-2.6.32-55-generic-pae      2.6.32-55.117            i386                     Linux kernel image for version 2.6.32 on x86
un  linux-image-3.0                        <none>                                            (no description available)
ii  linux-image-3.5.0-47-generic           3.5.0-47.71              i386                     Linux kernel image for version 3.5.0 on 32 bit x86 SMP
ii  linux-image-extra-3.5.0-47-generic     3.5.0-47.71              i386                     Linux kernel image for version 3.5.0 on 32 bit x86 SMP
ii  linux-image-generic                    3.5.0.47.63              i386                     Generic Linux kernel image
ii  linux-image-generic-pae                3.5.0.47.63              i386                     Transitional package

ls -la /boot shows this:

total 26028
drwxr-xr-x  4 root root     2048 Mar  1 14:24 .
drwxr-xr-x 26 root root     4096 Mar  1 14:23 ..
-rw-------  1 root root  2324217 Feb 19 01:28 System.map-3.5.0-47-generic
-rw-r--r--  1 root root   862198 Feb 19 01:28 abi-3.5.0-47-generic
-rw-r--r--  1 root root   154687 Feb 19 01:28 config-3.5.0-47-generic
drwxr-xr-x  2 root root     1024 Mar  1 14:23 grub
-rw-r--r--  1 root root 18016141 Mar  1 14:24 initrd.img-3.5.0-47-generic
drwx------  2 root root     1024 Jul 30  2011 lost+found
-rw-------  1 root root  5175536 Feb 19 01:28 vmlinuz-3.5.0-47-generic

df -h shows this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/md1        458G  161G  275G  37% /
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           809M  304K  809M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/md0         54M   26M   25M  51% /boot

fdisk -l shows this:

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00024aac

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      112454       56196   fd  Linux raid autodetect
/dev/sda2          112455     2216969     1052257+  82  Linux swap / Solaris
/dev/sda3         2216970   976768064   487275547+  fd  Linux raid autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002518c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *          63      112454       56196   fd  Linux raid autodetect
/dev/sdb2          112455     2216969     1052257+  82  Linux swap / Solaris
/dev/sdb3         2216970   976768064   487275547+  fd  Linux raid autodetect

Disk /dev/md1: 499.0 GB, 498970066944 bytes
2 heads, 4 sectors/track, 121818864 cylinders, total 974550912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md0: 57 MB, 57475072 bytes
2 heads, 4 sectors/track, 14032 cylinders, total 112256 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

cat /proc/mdstat shows this:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda1[0] sdb1[1]
      56128 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
      487275456 blocks [2/2] [UU]

unused devices: <none>

lspci -v | grep RAID shows this:

01:0e.0 RAID bus controller: Broadcom BCM5785 [HT1000] SATA (Native SATA Mode) (prog-if 05)

Answer 1

You could remove the current kernel (sudo apt-get purge linux-image-*) and then, without rebooting install the latest kernel (sudo apt-get install linux-image-generic). This should work^[1], however if it doesn't, you're left with no way to boot your system.

Another solution would be to use Ksplice Uptrack, which allows you to update your kernel without ever rebooting. The only problem here would be that the free version only supports Ubuntu Desktop, so you'd have to pay for the service but there might be ways to use the desktop version on a server.

_{1. “ubuntu just happily removed my (running!) kernel” Ubuntuforums.org}

Answer 2

I have a suggested solution, but I strongly recommend you try it on a local machine or in a virtual machine before futzing with your server, particularly if you can't get to it physically.

That caveat out of the way:

1. Unmount the /boot filesystem.
2. Disable swap space (sudo swapoff /dev/sda and likewise for /dev/sdb).
3. Using fdisk or parted, delete your two swap partitions and re-create them with smaller sizes and later start points. The "later start points" part is critical, because you want a gap between partitions 1 and 2. Also, be sure that the start points are identical on the two disks.
4. Using fdisk (not parted), delete partition 1 on /dev/sda and re-create it with the exact same start point and a later end point. This could be tricky. Your partition 1 on both disks has a start point of sector 63, which modern tools might be reluctant to create. Thus, you might need to dig up an old version of fdisk to get this to work.
5. Before saving your changes, verify that the start point of your new first partition is the same as it is now -- sector 63. If it's not, abort without saving your changes. (You can do this in fdisk by typing q or hitting Ctrl+c, but parted makes on-disk changes as soon as you type the command to make a change, which is why I said to do this with fdisk -- you can abort with fdisk but not with parted.)
6. If your new first partition starts on sector 63, save the changes by typing w in fdisk.
7. Repeat the expansion of the first partition on /dev/sdb.
8. Type sudo partprobe to get the kernel to recognize the changed partitions.
9. It may be necessary to do some RAID stuff here to get RAID to recognize the changed partitions, but I'm not positive of that.
10. If /boot uses ext2fs, ext3fs, or ext4fs, you can use resize2fs to resize the filesystem to fill the new size of the partitions on which it resides. Similar tools exist for other filesystems.
11. Create new swap space on both disks' second partitions, as in mkswap /dev/sda2.
12. Edit /etc/fstab to adjust the change to the swap partitions, if necessary. (They're often referenced via UUID values, which will have changed. Use blkid to learn the new UUID values; or refer to them as /dev/sda2 and /dev/sdb2.)
13. Use swapon to re-activate your swap space.
14. For an acid test, reboot.