USB drives are easily lost and thus rather insecure. If security is what you're looking for, you should be using encryption.
As far as making a stand-alone installation goes, I don't know too much about doing that in Linux. However, if you've got a removable drive (hard drive, flash drive, etc.) that you can format to some Linux-native format (such as ext3 or ext4), and if you can have the drive accessible every time you install a new version, you can employ symlinks to suit your purposes:
- Check whether a particular software can be installed to an arbitrary location. If so, do so. Otherwise:
- Install the desired software via package manager such as Synaptic.
- Use the package manager's tools to get a list of installed files.
- Copy every installed file and directory to your external drive.
- Make symlinks from each original location to the external drive.
- Hope that the affected software doesn't freak out because of the symlinks.