Whilst waiting for more in depth technical answer as I don't have a clue about the internals of Ubuntu!
Tested this on apache2.2-common and mediawiki.
apt-get remove left some files, trying to perform the command again resulted in an error saying that the package was not installed, however, it was available on tab completion for
When I did
apt-get remove apache2.2-common --purge, this both removed the left over files and removed it from the tab completion.
Again, can only give the answer and not a technical "why", but, I look forward to reading someone else's answer!