I was running in the 'vanilla' default mode, although I do have xfce
and kde installed.
A vanilla installation, to my understanding means a clean installation of a distribution. Try a clean install (as in format everything, create a new user and try again). Of course, backup your important files before doing this.
Also, instead of reinstalling, you could try creating a new user. This can pin-point the problem: If the new user works, the problem is located in your home user settings. If it doesn't, it's a system problem (and reinstalling would solve it probably)
Do not install anything else. Do not use PPAs. Only update the software through update manager or apt-get, without altering/adding the sources.list. That means vanilla installation. See how the system responds for 3-4 days. Then start adding software one-by-one in order to discover the problematic package.
After a reboot, the system usually appends a ".old" or ".0" in the filename.
I'd say these log files would be informative:
tar czf mylogs.tar.gz --ignore-failed-read ~/.xsession-errors.old /var/log/Xorg.0.log.old /var/log/dmesg.0 /var/log/syslog.1 /var/log/kern.log.1 /var/log/apport.log.1 /var/log/pm-powersave.log.1
...or if you prefer it separately: