On an installed Ubuntu install, cache of installed packages is stored at
/var/cache/apt/archives so they don't need to be downloaded again if they are needed to be re-installed.
On an install disk (Ubuntu 14.04, should be the same as the Xubuntu one), the compressed file-system appears to be at
casper/filesystem.squashfs. So you should be able to decompress the squashfs file (I wouldn't do it as root, odd things may happen then), and extract the cached deb archives. Then you could install from the debs the dependencies listed here.
The problem is that the packages may be installed on the compressed filesystem, but not cached, so installing from the disk becomes impossible. I tried this with a normal Ubuntu install disk (a 1GB 14.04 one downloaded from here - using the command
unsquashfs '/run/media/wilf/Ubuntu 14.04 LTS i386/casper/filesystem.squashfs in a empty directory), and there were no cached packages.
As cached packages use up an awful lot of space, when trying to fit a system installer to a ISO which has to fit on a CD or DVD, it is not surprising these are left out - especially on the Xubuntu disk, as it is more for low-spec systems. Unless you copy the installed data over, which is not a good idea, you can't really install from the iso. Cached packages may be included on the larger 4GB install ISOs available for some systems.
So you are probably stuck with installing
xubuntu-desktop - great idea though.