What you need to do is boot the LiveCD and chroot into the Ubuntu system on the USB drive, then install grub2
First thing to do is identify what the device name of the Live file system is. In most cases this will be /dev/sdb, if there is only one other drive in the system, but it could be something else depending the configuration of your hardware.
A little more complicated is to identify which partition Ubuntu lives on. This will depend on exactly how you installed Ubuntu so it's best to check and be sure. Once you've booted the LiveCD use either Disk Manager or Gparted to look at the hard drive and identify the Ubuntu root (/) partition that you set up. You would be looking for an ext4 partition of the correct size.
For the purposes of these instructions I'm going to assume the correct device (drive) is /dev/sdb and that the partition is /dev/sdb3. You should substitute the correct device names as appropriate.
Firstly, in a terminal window mount the Ubuntu root partition:
sudo mount /dev/sdb3 /mnt
Next, export the LiveCD filesystem to the root filesystem on the drive:
sudo mount ‐‐bind /dev /mnt/dev
sudo mount ‐‐bind /proc /mnt/proc
sudo mount ‐‐bind /sys /mnt/sys
Now change the root filesystem to the hard disk:
sudo chroot /mnt
Now install grub2 to /dev/sdb
sudo grub-install /dev/sdb
At this point, while grub2 is installed it might not know about all the filesystems available, such as a windows partition on the hard drive or another Linux partition on either drive.
To fix this run update-grub:
and it will find all OS's on your system and add them to the grub menu. You should now be able to boot off the external drive and select the Ubuntu partition or any others to boot from
I note that there is no mention of a swap partition in your configuration. This is not likely to be a problem if there is plenty of RAM in your system and you never want to hibernate the computer. You will still be able to suspend the computer but this does continue to use a small amount of power.
The correct way to do this at the time of installation is to specify the correct hard drive to install the grub bootloader to. This might require selecting the manual configuration instead of the default one, then selecting the correct drive to install grub to. It's also possible to install grub to the internal hard drive and have it configured to boot Ubuntu off the external USB drive.