Since I already have Windows installed, there is already an EFI partition on my disk. I want to install Ubuntu on that same disk. Do I have to create a new EFI partition for Ubuntu, or can the one from Windows be used for both OSes?
You can and should use one EFI partition for all OSes installed.
The only concern is if you have many OSes that there is enough space on it. But in most cases this is not an issue since efi files are small.
You do not have to create a new ESP (EFI system partition). In fact, you must leave the ESP alone as explained in the UEFI Ubuntu help page
Make sure you install Ubuntu in EFI mode so you will be able to boot both Ubuntu and Windows from GRUB.