11.10 and newer versions of Ubuntu
GRUB2 and its display of all kernels
The latest versions of Grub2 installed in Ubuntu automatically display the latest kernel and hides the older kernels that you may have installed.

If you do not see your grub - then remember to press SHIFT whilst booting.
As you can see, only the latest kernel is displayed.
If you select the option shown (press Enter) then all the old kernels become visible and available to boot from.
How to permanently delete ALL older kernels using the shell
This will remove ALL versions but the current:
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get purge
You can run this command as a cron job, if you add the -y option after "apt-get". However this is not wise, as you should ALWAYS have an old kernel or two to fall back to (just in case the new one doesn't work with your system). Read on for a saver, more manual way.
How to permanently delete older kernels
First boot with the latest available kernel.
There are a number of ways to delete old kernels. Personally, I wouldn't touch Computer Janitor since this is acknowledged to break your computer with its suggestions.
synaptic
An alternative is Synaptic (sudo apt-get install synaptic)
search for linux-image, right-click a kernel and choose complete removal and finally click the Apply button to delete the kernel.

Repeat the search but this time for linux-header - you can delete the associated headers for the kernel image chosen previously.
Synaptic though will not attempt to verify what you are trying to remove... you could inadvertently delete your newest kernel - or even delete all of your kernels via this tool leaving you with an unbootable Ubuntu!
Remember to check which kernel you are using type:
uname -r
The result would be similar to:

Remember the result and the number - make sure you don't delete the corresponding image or header.
Ubuntu-tweak
IMHO, the best GUI tool is Ubuntu-Tweak
It is not available from the standard repositories. To install you need to use the author's PPA:
sudo add-apt-repository ppa:tualatrix/ppa
sudo apt-get update
sudo apt-get install ubuntu-tweak
Choose the options shown by the arrows.
Select both the headers and image with the same version number.
It will not allow you to delete the current kernel you are booted with since the current kernel is not displayed.
Recommendation
My recommendation is to keep at least two or preferably three kernels including the latest. The reason for the recommendation is that you will have at least one/two other kernels to boot with, if for what-ever reason the latest kernel you are unable to boot with or introducing a regressed capability such as broken wireless.