Boot after 14.04 installation fails because of missing /tmp

Question

We have a an institution-wide installation of Ubuntu (previously 12.04) that is installed from network and uses a custom installer config for selecting the necessary packages and setting up LDAP authentication. Apart from that it is a normal Ubuntu.

We are now trying to setup this installation procedure for 14.04. We can successfully install with our configuration from network, but the first boot fails with the message that /tmp is not available. The machine does not respond to any key presses at that moment (so I cannot Skip or do Manual recovery). The boot process is also extremely slow (more than one minute until the message about /tmp is shown).

I found a work around that helps. After installation, I need to boot into recovery mode once. In the recovery menu, I need to select network (which mounts partitions and configures network) and then resume. Now the system boots fine, and any subsequent boot (also without recovery mode) is fine, too. So no manual fixing done, just a some how "special" boot once.

If I skip the network step in the recovery menu and go straight to resume, it does not work. If I reboot immediately after the network step (without going to resume), it also does not work.

I did it manually once and entered a root shell from the recovery menu, everything looked fine at that time (both before and after running the network steps).

I diff'ed /etc and /var before and after running the network step, and there were no big differences (just DHCP leases and logs in /var as expected).

Now my question is, what happens during this boot that is different from a normal boot? I guess, at the first boot there are some installation duties remaining, and these somehow do not work on a normal boot, but what could they be? Maybe the manual execution of the network step from recovery menu changes the order of some things, and that resolves the problem?

Can I fix these problems at the end of installation prior to first boot (we run a script at that stage anyway) so that everything just works?

How can I debug these problems and provide more information for you to help? What file might change during boot and could be interesting? I can reproduce both the problem and the workaround, so I can provide as much info as needed.

Our site-wide adjustments only affect LDAP authentication and NFS home directories, not the boot process or the partitioning. During installation a root partition (ext4) is created and a swap partition, no further partitions. This is a legacy boot (no UEFI).

Edit: I debugged it a little bit further. After installation I need to do the following steps once to:

1. boot into recovery mode and start a root shell
2. run mountall
3. run ifup -a

After that, everything works nicely. All three steps take much more time than expected (around one minute each). After finishing, booting takes only a few seconds (machine has an SSD).

Answer 1

After a careful examination of the differences between the /etc directories on a working and a broken system, I found the solution.

The problem is a missing nss_initgroups_ignoreusers line in /etc/ldap.conf. This line lists all local users, and prevents the system from accessing the LDAP directory when looking up these users. This line is created automatically on each boot by /usr/sbin/nssldap-update-ignoreusers, but with my installation method it was missing on the first boot. So during this first boot, the system was trying to lookup user information from LDAP before the network connection was brought up. Of course this failed.

I now added a call to /usr/sbin/nssldap-update-ignoreusers to our script that runs between installation and first boot, and now everything works fine.