Yes, it would absolutely be possible for someone to remotely access your computer with your user name and password. There are many ways to do this (ssh, vnc, etc).
The good news is that these backdoors are disabled by default, but if you have opened one of these backdoors in the past for your own use (I know I have), in many cases your username and password will get them in.
Or, if this person who knows your password has had physical access to your machine, I would assume they have already opened several backdoors into your machine.
Maybe I'm paranoid, but I would never trust another user with my password - and if someone stole my login information for a computer, I would no longer trust that computer.