TL;DR Try unplugging your USB hub (and potentially other USB devices)
My Xorg did not consume a lot of CPU but within about a day that the computer was up, it bloated up to 1.2 Gigabytes of memory. The only way to correct this was a hard reboot.
After trying disabling NVidia drivers, switching to cinnamon, switching to gdm and finally kicking the damn thing (that was just for me to feel better), I noticed that as I switched to a console tty ( ALT + F1 or F2, ..., F6 ) during some gdm/lightdm service restart, there were a lot of error messages scrolling down the screen. Note, that I had only switched to ALT + F1, I had not logged in yet.
The messages were reporting that my usb tv-tuner card was not properly initialised. These messages kept scrolling on and on as if the system was stuck in an infinite loop trying continuously to initialise the tv card (and of course failing every time).
As expected, as soon as I disconnected the usb tv-card, the messages stopped, but to my amazement Xorg stopped inflating too. Just to make sure, I reconnected the usb tv tuner and the error messages came up, and again Xorg started gorging memory like an SUV filling up at a gas station.
But, right before the villagers (a.k.a ubuntu users) are about to smash their dysfunctional usb equipment and stone the dysfunctional usb equipment vendor, there is a sudden turn in the scenario. It is NOT actually the usb tv-card that caused all this trouble. The tv-card was only the symptom and not the source of the problem. The plot thickens...
Having used this tv-card before without an incident, I took a better look at the error message complaining about the tv-card. There was another line complaining about the usb port of the usb-hub, where I plugged in my tv-card. Hmmm, I had just bought this hub. So then I tried plugging the tv-card in a usb port directly on the motherboard. And everything worked fine! No error messages, and no Xorg bloating.
So at the end the actual problem was this:
Xorg was consuming memory because of initialisation errors of a usb device (and in my book this is a serious bug). Since a lot of people report problems with Xorg in ubuntu 12.04, I suggest that you also take a look at your usb devices. My tv-card could not be initialised when plugged in my new usb-hub, and that caused the problem. Putting it in a separate usb port on the motherboard fixed the problem.
P.S. The usb-hub is most likely either linux-incompatible, or should be replaced by one that has its own power adapter instead of being powered by the computer's usb bus (see USB hub not working properly Ubuntu 12.04). My web-cam also does not work when I plug it in the hub. It does work if plugged in a usb port on the motherboard. However note that, when I plug it in the usb-hub,
dmesg reports that the web cam is properly recognised and initialised (although
cheese gives me just a black image), and since there are no error messages, Xorg does not fill up the computer's memory.
P.P.S. I invite the distraught villagers to gather outside one-eyed Tim's barn, to throw their dysfunctional usb-hubs in a big festive bonfire. When the fire has built up, we will throw in the usb-hub vendor too. Free beer for everyone! Mua ha ha ha ha (deep voice laughter)