In Ubuntu 13.04 this was simple: go to System Setting → Keyboard Layout → Options → Key(s) to change layout and tick Caps Lock:
But in Ubuntu 13.10, Keyboard Layout, now called Text Entry, has undergone many changes, and for the moment, until a a correct fix for this bug will be released, is a little bit more complicated to switch input language with Caps Lock.
After I added the following PPA to my list of sources using the following command:
sudo add-apt-repository ppa:attente/1218322
and after I upgraded my system using:
sudo apt-get update
sudo apt-get upgrade
I was able to change the default Super+Space keyboard shortcut to Caps Lock:
Many thanks to William Hua (attente) for his Input source switching PPA. As he said, he will keep the PPA up-to-date until a correct fix for this bug is released.