You can uninstall whatever you like, but you cannot remove things and expect them to still work. You can save a few megabytes of disk space by removing software that isn't in use, but you're also likely to cause serious issues if you don't know what you're doing.
If the goal is to explore a Linux system, then I'd rather recommend that you start with a basic system and try to build yourself something nice by adding bits and pieces to the base. That way, you'll see the effect the components have in your system. If instead you explore by taking things away, you won't really know what effect they have, because you'll break other components that depend on them.
Ubuntu is still based on Gnome, which is a fairly huge platform consisting of much more than you see. The file manager, Nautilus, for instance, is part of it. And libraries are shared between applications. So, for instance, if you replaced Nautilus with some other application, you wouldn't be able to remove all libraries that Nautilus uses, because you still have Totem.