You can start with removing
sudo apt-get purge ubuntu-desktop
sudo apt-get autoremove will remove the rest
If you have any other Desktop remove them as well. if you reboot after this you should have no GUI to log into. If you purge a program rather than just removing it you also remove any config files that may remain.
As Rinzwind suggests try
sudo apt-get remove gnome-*.
Ubuntu also provides a special kernel package for server installations, linux-image-server:
sudo apt-get install linux-image-server and reboot.
Then I suggest you install the server applications you want, like
But as always it's better to make a clean install. It gives less risk of errors and broken packages.