Install x11vnc:
sudo apt-get install x11vnc
Create a password for your user:
x11vnc -storepasswd
If you have ssh setup you can use it to start x11vnc assuming you are logged in already, but remember to tell it to use your password file:
x11vnc -usepw
If you are not logged in you will get an error with the explanation:
If NO ONE is logged into an X session yet, but there is a greeter login
program like "gdm", "kdm", "xdm", or "dtlogin" running, you will need
to find and use the raw display manager MIT-MAGIC-COOKIE file.
Some examples for various display managers:
gdm: -auth /var/gdm/:0.Xauth
-auth /var/lib/gdm/:0.Xauth
kdm: -auth /var/lib/kdm/A:0-crWk72
-auth /var/run/xauth/A:0-crWk72
xdm: -auth /var/lib/xdm/authdir/authfiles/A:0-XQvaJk
dtlogin: -auth /var/dt/A:0-UgaaXa
Assuming you are using lightdm for the login you can fix this problem you can start x11vnc with the command:
sudo x11vnc -xkb -noxrecord -noxfixes -noxdamage -display :0 -auth /var/run/lightdm/root/:0 -usepw
I am not sure this is the best idea to run x11vnc as root. Maybe someone could edit with a way to access the login without using sudo.
Once this is running you should be able to connect using a vnc client such as krdc (for KDE). You might want to use GNU Screen to keep x11vnc running without needing the ssh session open
I was able to figure this out using http://ubuntuforums.org/showthread.php?t=2039022.
Here is a sample upstart job you can use to make it run at startup. It needs to be put in /etc/init/x11vnc.conf. (Note that newer versions of Ubuntu use systemd so see the other answer that has a sample systemd config):
# description "start and stop x11vnc"
description "x11vnc"
start on runlevel [2345]
stop on runlevel [^2345]
console log
#chdir /home/
#setuid 1000
#setgid 1000
respawn
respawn limit 20 5
exec x11vnc -xkb -noxrecord -noxfixes -noxdamage -display :0 -auth /var/run/lightdm/root/:0 -usepw
Once you have made this file you can start it by running:
sudo start x11vnc
You can check the log at: /var/log/upstart/x11vnc.log