I believe the main question when choosing between 32 and 64 bits is not "my processor support 64 bits - should I use this because 64 is "twice better" than 32", but "how much memory do I have".
64-bit programs use about 50% more memory then their 32-bit counterparts. The results of this are:
less code/data can fit into processor caches -> more cache misses -> your super-fast processor waits more for memory controller to fetch data from RAM. Processor also needs to read more data because it's more bloated.
Linux uses non-allocated RAM to cache disks. Less free memory available -> less space for disk caches -> slower disk access
Once you're out of RAM and the system starts swapping - disk access is thousands times slower than RAM, so any potential benefits of 64-bit code (see below) are flying out of window.
On the other hand, in 64-bit mode processor has more registers, so it doesn't need to access memory that often, also, some calculations (64-bit numbers etc.) are more efficient. So generally, code which is not memory-restricted runs slightly faster in 64-bit mode.
So, if your system has, say, 2 of RAM or less then I would definitely choose 32 bits. More than 4GiB - the benefits of 64 bits are becoming more prominent. Around 4GiB is a "grey zone" - you need to make a choice yourself.
Here are screenshots of a (rather old) vanilla Ubuntu running in a VirtualBox with 1GiB RAM:
32 bit: uses 388 MiB of RAM right after booting, no additional applications running (from the screenshot, 397780KiB, which is 388MiB)
64 bit: uses 491 MiB of RAM (from the screenshot, 503452KiB = 491MiB)
Additionally, in my testing, a web-application written in Python used up to 60% more memory on a 64-bit machine which resulted in a test suite running in 380 secs on a 32-bit machine but taking 523 seconds on a 64-bit one (both with 1GiB of RAM). If the machines were not RAM-limited the results would likely be different (as phoronix tests show).
Here are some more thorough and recent tests: http://kernel.ubuntu.com/~cking/x32/Quantal-x32-power-memory-comparisons.ods
Basically, it's like giving up 30% of your RAM in exchange of possible gain of 10% CPU speed.