First of all, yes you can (you know that from the existing answer). I would still like to add a few aspects. The desktop and server versions only differ by the packages installed by default or set to be installed by default (
tasksel), hence you can do it.
Running a production server on a "desktop version" of Ubuntu is therefore possible as there are no fundamental differences such as, say, between Windows XP Pro (32bit) and Windows 2003 Server R3 (32bit) - the latter would accept up to 64 GiB RAM in the more expensive editions. On Ubuntu you can install a kernel that speaks PAE and doesn't set arbitrary limits (like cheaper Server editions of Windows 2003) even on 32bit setups.
The bigger question should be whether you can run production servers on commodity hardware. And the answer is YES. Check out this article over at Wikipedia. Google runs most of their services on this kind of hardware, but redundantly. A few years ago they (Google) published some study about the reliability of desktop hard drives, too.
Therefore after we know commodity hardware isn't the problem your requirements about reliability and the known parameters concerning hardware failures to be expected etc should govern your decision making.
A "cheaper" (or rather: "more lightweight") alternative would be to use LXC (Linux Containers). Imagine them like an intermediate step between
chroot (which was never meant to be a jail like the respective concepts on Solaris or BSDs) and KVM. LXC does leverage
apparmor to achieve this, btw. The separation isn't perfect, but given the advantages, the tradeoff seems small.
About your rootkit worry, I am not sure because I haven't researched the topic enough. However, KVM guests run as user mode processes and you can even let each guest have its own account, if you like. Now parts of KVM must obviously operate in the kernel of the host, so there is a chance that attack vectors exist, I just wonder whether the attach vector allows code from the guests to break out ...
How you intend to make things more secure by only using the command line is beyond me, however. You have all the possibilities (usually rather more than less) that you have through a GUI (KDE, GNOME, Unity etc). So it doesn't make the job harder to install host or guest as command line only. The reason to leave out X11 should rather be governed by the use (you say "server", so it's not obvious) and the available resources. If X11 doesn't run it also won't take away resources (except for disk space).