Virtualbox uses two separate applications, the actual virtual machine, and the Virtual Box Manager. We can not test this here, but it seems likely that the RAM usage you observed is from the Virtual Box Manager, the GUI to manage all your virtual machines, and not from the VM itself.
To speed up Ubuntu we need to make sure the following is met:
- enable VT-x or AMD-V in BIOS, and in the VM settings
- assign enough RAM (1024 is fine but leave at least 512 MB better more for the host)
- give enough Video RAM (choose 128 MB)
- enable 3D acceleration in the VM settings
- install Guest Additions for (limited) hardware 3D acceleration
- from Ubuntu 12.10 performance is better from 2 CPU cores onward (note that if you have only a dual-core system that this may affect the host operability)