The easy way
Look for a "meta package" that was installed as the "desktop", and by uninstalling this, apt should remove all packages that comprise the meta package. (I still need to confirm this, but just read it in a few posts here http://ubuntuforums.org/showthread.php?t=2110057)
The hard way
I went after the GNOME Desktop Manager
sudo apt-get purge gdm
and then to cleanup the leftovers
sudo apt-get autoremove
NOTE: autoremove removes unneeded files associated to most recent purge/remove, so it needs to be run each and every time.
These two commands may not remove everything though, so to look for remaining items, I did
dpkg-query -l *desktop*
dpkg-query -l *gnome*
and purged the remaining packages from these lists (also using autoremove aftwards).
To check your progress, after each purge/remove/uninstall, flip back to GUI login screen (ctl-alt-f7) and watch it be dismantled.