Grub-bootloader will be installed in the first HDD (this is by default). So if you didn't change that manually , then grub is now installed in /dev/sda , which is your HDD and not USB.
As a solution I suggest
1) Boot from CD/DVD of Windows and fix MBR. (repair - option)
2) Reinstall Ubuntu and make sure that grub-bootloader will be installed in the USB (and not HDD). USB can be /dev/sdb or /dev/sdc ...etc (but not /dev/sda). You will see this option if you choose "something else" during installation.
3) If you want to achieve this goal
it loads Ubuntu when it is there and Windows when it is not
you have to "play" with BIOS all the time. If you have "one boot key" which allows you to select a device to boot (at my laptop is F12) then is easier. But if you don't have this key , anytime you want to boot in to Ubuntu you have to go in to BIOS configuration page and change the boot-device order and place fist the USB (you have already attached).