As the man apt-get page says:
remove - Packages installed are removed (Does NOT include configuration files)
purge - Purge is identical to remove except that packages are removed and purged. Purge meaning that any configuration files are deleted too.
This of course, does not apply to packages that hold configuration files inside the user's home folder (eg: /home/SexyNoJutsuUser), this files will not be touched ( Why does "Purge" not remove everything related to an app? )
So for example, if you were to remove Chrome, Firefox, XBMC or any other that holds some configuration files inside your /home folder, this files will stay there.
On the other hand if you were to install apache, squid, mysql or any other services similar that save their files in /etc, this configuration files will be deleted if you use purge.