Have you tried a console alternative to setting it from the gnome-panel time applet?
ntpdate is a favorite of mine. i have it setup in a cron script across several Machines and VM's to ensure my system clock never gets skewed that far off. (once a week is fine by my standards. i find that I only migrate away from Atomic Time a few microseconds a week.)
However, with that said, give a try with
and see if it corrects your time issue, without a freeze.
If your system continues to freeze, see if you cant ssh in from another machine and have a look at /var/log/messages. On a Fedora system at one point i was 2 hours ahead of actual time, when i reset the date, it caused a panic because everything on the system was 2 hours ahead. Threw some ridiculous error about future time and locked up. I powered down the machine for a day and gave it another go in a recovery console without an issue.
Edit 1: Another thought occured to me to try and change your TimeZone settings. See if that will reconfigure your clock. See the article located at https://help.ubuntu.com/community/UbuntuTime
This may be a byproduct of messed up TZ settings.