You need to install "grub" bootloader to your harddrive, not to your USB pen drive.
After this step, you will be able to choose which operating system to boot, you won't need the usb stick anymore to use ubuntu.
Identify your hard disk device entry -> one of /dev/sda, /dev/sdb, etc.
Usually, it is /dev/sda.
If you are not sure, check at the command line (Terminal):
sudo fdisk -l
In the output, first line is important as you'll recognize the size of your hard disk.
If you have the usb stick too, you will have /dev/sda with any partitions like /dev/sda1, /dev/sda2, etc AND /dev/sdb -> your usb stick:
Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders, total 312581808 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e26f3
Device Boot Start End Blocks Id System
/dev/sda1 2048 40962047 20480000 83 Linux
/dev/sda2 40962048 275718143 117378048 5 Linux swap / Solaris
Disk /dev/sdb: 500.1 GB, 500107861504 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773167 sectors
......and other stuff ..........
sudo grub-install /dev/sda