This has become much easier in 11.10.
First you need to create a folder:
sudo mkdir -p /etc/guest-session/skel/. The -p switch creates /etc/guest-session so you can place /skel in it in one go. Now you just add the files you want to /etc/guest-session/skel and it will be copied onto the guest account whenever it's created.
The program that sets this up is /usr/sbin/guest-account in case you want to learn more about what's going on.
The tutorial CustomizeGuestSession elaborates on this topic. In particular it describes how you can create a "special purpose user" for a convenient way to customize the guest session appearance and behavior, without having to know which files to edit.