There are basically two ways to run Ubuntu from a USB drive.
The first is to do a regular install and select the USB drive as the target for installation. Be sure to select the right drive for writing GRUB to the Master Boot Record (MBR). If you select your internal drive by mistake then your computer will not boot if your USB drive is not connected, even though your intention is to boot into Windows.
The second is to use the program Startup Disk Creator on Ubuntu to create a Live USB install. To add persistence select the option "Stored in reserved extra space" under the question "When starting up from this disk, documents and settings will be:" and then use the slider to reserve as much space as you need to store your documents and extra applications.
If you want to remove the "install this" icon from your desktop all you have to do is uninstall the package called "ubiquity". Do this with:
$ sudo apt-get remove ubiquity
Note that this only works if you've enabled persistence when creating the live USB. If you use the first method then ubiquity will be uninstalled as part of the installation process.