I reckon it is this nvidia bug that is causing your woes:
The 173 nvidia branch is not available as a package for 12.04 and nvidia current doesn't work well for old cards. My solution was to get a new 600 series card, as I hope to avoid this kind of issue come october.
However there are a couple of other solutions.
Solution A use nouveau until the 173 driver is packages (it looks on that bug report that the new driver could be packaged pretty soon, subscribe to the bug report for email updates)
This will disable the nvidia_current driver and enable nouveau:
sudo jockey-text -d xorg:nvidia_current
Download and execute this binary to build a 173 driver which works with the version of x.org precise ships with:
This solution will break whenever an updated kernel is released unfortunately and will force you to rebuild the driver