I've just completed a fresh new installation of Ubuntu 11.04 Server (64 bit) including xfce. After the installation was complete (along with all the LAMP server components), I started X for the first time and all was good. After the initial reboot, I get the nice xfce login screen. I put in my credentials, it suddenly switches to command-line mode for less than half a second and returns me to the pretty xfce login prompt.
The login credentials are correct, as if I intentionally using the wrong username or password, it tells me so. I managed to capture an image of the terminal window that flashes up and disappears - all just an install script with all [ok] - no errors at all.
Now, this is running on a VBox virtual machine, but that shouldn't really matter - and I've successfully used the same VBox for dozens of other Linux installations - including Ubuntu server 10.04 LTS.