The answers here all suggest modifying the file system on the live USB, which isn't really possible without rebuilding the entire ISO image as far as I have understood.
I think the reason my problem, and the OP's problem, occurred was that we checked the box for installing third-party software in the installer, and subsequently checked the box for managing secure boot through MOK as required, but were then unable to finish the installation and left the computer in a state where it was expecting to boot the MOK manager on the next boot with a path that exists in the Ubuntu system, but not on the Live USB.
To solve this I did a strange work around. I followed the instructions for installing Ubuntu through Wubi as provided here, and then, after restarting my computer, Wubi managed to launch the MOK manager and although I didn't perform any MOK management it seems like this stopped the computer from expecting to find a MOK manager on startup, and when I tried to boot from USB afterwards it suddenly worked. Afterwards I just deleted the Wubi install and all the Wubi files and installed Ubuntu the normal way using the Live USB.
I could be wrong about my assumptions in this answer, but after trying several USBs and many different Ubuntu ISO images, this was what finally did the trick.
Note: What I described in this answer requires that you have Windows installed, which you may not. If you don't, you might be able to solve it in a similar way by booting a system that has GRUB and the MOK manager on it.