I'm supposing you want to remove all non-default packages.
First of all, the desktop version of Ubuntu has a manifest file listing all the packages contained on the cdrom (example). Unfortunately it seems that this manifest is not available online for server editions, but you can try to search it on the cdrom.
Also, if you installed ubuntu-desktop through apt-get you should inspect the log located in /var/log/apt/history.log. This log has entries like this (an example from my machine):
Start-Date: 2013-02-18 23:55:18
Commandline: apt-get install compizconfig-settings-manager
Install: compiz-plugins-main:amd64 (0.9.7.0~bzr19-0ubuntu10, automatic), compizconfig-settings-manager:amd64 (0.9.5.92-0ubuntu3), compiz-plugins:amd64 (0.9.7.12-0ubuntu1, automatic)
End-Date: 2013-02-18 23:55:35
So you can try to locate the command you issued to install ubuntu-desktop and get the listing of the installed packages.
If you opt to do a clean install, before try tasksel. It will probably damage your system because it doesn't deal well with dependencies, but you have nothing to loose :)
There you can select "Basic Ubuntu Server" and deselect "Ubuntu Desktop".
Good luck!