The main difference to a pure live session is persistence of your modifications to the system across reboots no matter where you install Ubuntu, i.e. modifications are erased if you turn off the non-persistent live session (or someone pull the thumbdrive out of USB plug and asks politely if he_she might borrow that for a second). Adding a persistence mode to a live session is possible, though, see https://help.ubuntu.com/community/LiveCD/Persistence or get started with multisystem
, so that this difference can be compensated.
Note: not answering about differences to an installation on HDD as requested in the question :)