I have the exact same problem with Xubuntu 12.04 LTS vanilla (Xfce 4.8) and also with Xubuntu 12.04 with Xfce upgraded to 4.10; both with home folder encrypted
This is a bugger since I was running away from Gnome and Unity resource consuming and Nautilus bug crashing Ubuntu 12.04 LTS
After reading about the bug Xfce not unmounting .gvfs at logout, i figured out it could be automatically unmounted at login, as i was doing mannually.
This is what I did and so far it works:
I created a file with leafpad within my home directory: .umountgvfs.sh
placed the following script to unmount .gvfs
/bin/fusermount -zu "$HOME/.gvfs"
in terminal gave it execution permision
chmod 777 .umountgvfs.sh
then created an autostart application, selecting the corresponding script at: configuration -> configuration administration -> session start -> autostart applications
Comment: script to unmount .gvfs on login
Since this solved the problems most of the times, but not always, i guessed the script was being called before time, so i added a sleep command (bash -c "sleep 5; command") where 5 is the seconds it delays the script at login, in terminal:
and edited in the Exec line adding: bash -c "sleep 5; "
Comment=script to unmount .gvfs on login
Exec=bash -c "sleep 5; /home/YOURUSER/.umountgvfs.sh"
Replace YOURUSER with your user name.
Delaying the script: http://ubuntuforums.org/showthread.php?t=1119945
hope this helps