You mention you're using 64-bit Ubuntu on a 2GB system. The main advantage of 64-bit is the ability to address more memory, but for 2GB you don't really need it. Another advantage is that 64-bit code is slightly faster to execute, but if you're actually feeling the system be sluggish you may be hitting the swap file due to memory exhaustion, which negates the speed advantage.
The downside is that 64-bit uses up more memory than 32-bit. This is a good tradeoff if you need a system to handle more than 4 GB RAM, but for systems with 3 GB and below, this is a non-issue as 32-bit can do that just fine.
Your question is whether to upgrade to 4 GB RAM. More memory is always good, but if you don't have a specific need for it, you could instead install Ubuntu 32-bit, which can handle 2 GB just fine, and should be noticeably faster on your system.