The only upgrade that requires a restart is the kernel. Even then, you don't have to restart as long as you're happy to still run the old kernel. However, you might be unable to load any more modules if the upgrade has changed binary compatibility of the kernel (which might have happened if the new kernel package has a different version, even if it's the minor version that indicates that only the Ubuntu patch has changed).
You do need to restart any programs you've upgraded if you want to run the new version. For your desktop environment, that might mean logging out and back in (system services, on the other hand, can just be restarted in the background). You also need to restart programs that use a library or data file that has been upgraded. Again, this is only if you want to be running the new version: if you're happy to keep running the old version, just don't reboot.