It does not matter, there is no difference in kernels any more.
The only differences is the installer and the default packages.
Once installed you can have exactly the same OS configured, no matter if you start from the server or desktop installer.
What's the difference between desktop and server?
The first difference is in the CD contents. The "Server" CD avoids including what Ubuntu considers desktop packages (packages like X,
Gnome or KDE), but does include server related packages (Apache2,
Bind9 and so on). Using a Desktop CD with a minimal installation and
installing, for example, apache2 from the network, one can obtain the
exact same result that can be obtained by inserting the Server CD and
installing apache2 from the CD-ROM.
The Ubuntu Server Edition installation process is slightly different from the Desktop Edition. Since by default Ubuntu Server
doesn't have a GUI, the process is menu driven, very similar to the
Alternate CD installation process.
Before 12.04, Ubuntu server installs a server-optimized kernel by default. Since 12.04, there is no difference in kernel between Ubuntu
Desktop and Ubuntu Server since linux-image-server is merged into
For Ubuntu LTS releases before 12.04, the Ubuntu Desktop Edition only receives 3 years of support. This was increased to 5 years in
Ubuntu LTS 12.04 In contrast, all Ubuntu LTS Server Edition releases
are supported for 5 years.