Use a Linux Live USB creator that supports persistence - having files and settings carry over between reboots. You should note that this really sucks the life out of your USB. It won't last more than a few months of use.
For something a bit more long term, get 2 USBs. On one, do what you've been doing so so far. On the other, do whatever you want- it'll be wiped and Ubuntu will be installed onto it. Let's call this second one the 'other' USB.
Boot from your Live USB as you have been doing before. Plug in your other USB. On your live desktop, you should see a little icon called Install Ubuntu 12.04. Run this. It may take a while every time you click something but be patient. By the way, be sure to check both boxes about installing updates while downloading, and about using proprietary software.
When choosing installation type, hit Something Else and choose your other USB. You can probably get away with making one partition on it that fills the entire thing. Also, install the boot loader to your other USB. This is in a drop down box on the partitioning screen.
The rest of the installation is pretty self explanatory.
You clearly know how to boot from a USB, so if all goes well this should be perfect.