No, you shouldn't expect that.
An uninvited connection merely means that someone sent packets your way. Anybody can do that, and it's a good thing. I suspect that krfb has opened a port on the firewall, which isn't a problem - again. You do want it to accept invited connections, so it has to be exposed to the public internet.
If you, for example, run a web-server, you might get hundreds of break-in attempts per day.
The important bit is that they are attempts. Security is only compromised when you have, for example, a weak password for your SSH login (and accept connections on the SSH port, and have SSH installed). Another possibility is that a specially crafted message to a program can trigger a Bug in the program that allows arbitrary code execution. This is very dangerous. But: on Ubuntu, it's also comparatively rare, these vulnerabilities get fixed very quickly(1, 2, 3)*.
If you follow this very basic advice, you should be absolutely fine.
Absolute security doesn't exist, and there is always some danger, but:
someone to connect to my computer and erase everything I got on disk?
This would be major headlines on all of the IT news sites. It's extremely unlikely.
If you really want to dig deep into the subject, there are two papers I found very interesting:
This is just a small snapshot of the security landscape, but you might find them valuable.
*: Note, Ubuntu, with the help of Canonical, takes great care to deal with security related bugs very quickly. However, there is some software that Ubuntu have no control over. This is software that is installed from the
multiverse repositories. There, the onus is on the authors. Some of this software is very secure indeed, some isn't. If you are extra-paranoid, use only software from the