I had a similar problem today. After upgrading from (K)Ubuntu 10.04 to 12.04 (I know it's a bit late) X did not really react anymore. It was extremely slow, absolutely unusable.
After de-installing the Nvidia drivers it ran smoothly, but the graphics resolution was intolerably small.
I tried all kinds of remedies suggested anywhere on the web, but nothing helped.
Only installing the 304 version of the Nvidia drivers actually helped. All other, like 96, 173, nvidia-current or 310 did not do the trick. Most of them did not work at all, so that the respective nvidia-settings program always told me I was not using Nvidia drivers at all, although they were the only installed ones and there was an X screen in front of me. The programs were just lying.
So, my recommendation is the 304 (experimental) driver.
And my recommendation for the Nvidia and Ubuntu people is to test the software just a little bit and refrain from releasing buggy drivers! It's a pain in the butt!