apt-get purge only removes system-level configuration files (e.g. those created by debconf). It doesn't touch anything in your home folder -- you have to remove those manually. That makes sense if you think about it: there might be multiple users, who probably wouldn't want administrators deleting data out of their home folders!
Applications usually store configuration data in one of the following hidden folders, which you can delete if you're sure you have no use for the data:
application with the name of the app or package. Most applications don't store user files in those folders, and if you accidentally delete something out of your home folder that the system needs, it usually just gets re-created. Of course, it doesn't hurt to look inside before you delete them, just to make sure there's nothing you want in there.
For gnome 2 apps you might also want to run:
gconftool-2 --recursive-unset /apps/application
application with the app name) which will get rid of your application preferences.
For gnome 3 apps, the gconftool-2 command above is replaced by:
gsettings reset-recursively [schema]
You can find the schema for the application with
gsettings list-schemas | grep application