There is a way to achieve similar behaviour if you are interested in booting Windows most of the time but occasionally booting into Ubuntu.
You are able to change the behaviour of the Grub bootloader to automatically select the last booted Operating System by default. So, for example, if you booted up Windows last time, then it will boot up Windows again.
When grub first loads, there is a timeout (usually 10 seconds but can be changed) and if you don't select an operating system before this time, it will boot up the default.
In order to get grub to default to the last selected Operating System, you need to adjust the file that is located in
sudo nano /etc/default/grub is probably the easiest way to edit this file.
You want to change the line
GRUB_DEFAULT=saved and also add the line
When you've exited and saved the file, run
sudo update-grub and on your next reboot the default choice for grub will be the last booted Operating System.
For Legacy boot, installing Ubuntu (and subsequently the grub bootloader) does technically replace the 'Windows bootloader' ("GRUB 2 normally writes its bootloader information to an area just beyond the Master Boot Record and before the first partition" which is where the 'Windows bootloader' would be) but grub will load Windows quite happily when selected.
With a UEFI boot of Windows, it doesn't even replace the Windows bootloader. Grub becomes the first UEFI boot option and if Windows is selected then Grub just points it to the Windows Boot Manager and away you go.
The 'Windows bootloader' (or Windows Boot Manager as they seem to call it) isn't replaced or changed, your machine just loads grub first.
Here is an example of what my grub bootloader shows me:
I mention all this because you say you 'don't want to replace the Windows bootloader' but unless there is a particularly complicated reason for why you don't wish to do this, then having the grub bootloader installed on your Hard Disk should not cause any problems.
By changing the grub default choice, you will have almost the same behaviour that you are after, but with a short delay of a few seconds as grub waits for you to choose.
This will also mean you don't have to go get the USB every time you want to boot Ubuntu.
(Something worth mentioning is that if your Windows installation is using UEFI, then when you are installing Ubuntu you will want to press
f12 to get the 'one-time boot menu' up and then select the USB under the UEFI Boot options, to ensure that Ubuntu is installed as UEFI also.)
Here is a picture to show you what I mean, with the 'UEFI: TDK' being the USB stick I am wanting to use: