Could a hacker, with your IP, session hijack you and remote control your desktop on Linux?
Theoretically they could, if you have your computer set up to either allow Remote Desktop or SSH depending on which one you are asking about.
However they would need your IP address and your username. From this point they would have to brute force into your system if you have password authentication turned on. This can be mitigated by enabling Key Encryption instead of password authentication.
So yes they can but if your machine is running default settings and you are careful it will greatly mitigate the possibility of your machine being hijacked.
Basic Ubuntu Security Guide, Desktop Edition is a good read for information on Securing your Ubuntu machine.