What do I need to install a Ubuntu Server on a USB stick? I don't want to create a bootable stick, but really install the system there.
You can make a LiveUSB from any distribution of Ubuntu (Server/Desktop). You can look on Ubuntu's Wiki, there is a step-by-step solution : LiveUSB.
I also found an article to literally install a portable Ubuntu on your flashdrive : How To: Install Portable Linux Ubuntu on a bootable USB Flash Drive.
With those two tutorials, you should be able to do what you want.