You can remove
ubuntu-desktop and simultaneously auto-remove all its orphaned dependants:
sudo apt-get autoremove ubuntu-desktop
--purge option if you also want to remove the configuration of the affected packages (and not keep it for possible later reinstallation).
If you have any other Desktop remove them as well. if you reboot after this you should have no GUI to log into. If you purge a program rather than just removing it you also remove any config files that may remain.
As Rinzwind suggests try
sudo apt-get remove gnome-*.
Before Ubuntu 16.04 it also provides a special kernel package for server installations,
sudo apt-get install linux-image-server and reboot.
Then I suggest you install the server applications you want, like
But as always it's better to make a clean install. It gives less risk of errors and broken packages.