All other connection speeds are normal, average download speed is about 2mbps but my updates are lucky to get to 100kbps. Updates and this website are the only things running any ideas what the problem could be?
Apart from what Jorge suggested (about choosing best servers), you can actually remove some extra third-party repositories (like launchpad ones which you might have installed for some extra themes etc) to make the process a LOT faster.
You can read about it here: How to remove a repository?
Let's say you want to download and install some extra plugin - say Gmail video chat one. Having just that repository selected in this list, before you do a
will make it drastically fast because you don't have to download all other unnecessary lists.
This list file is kinda big one (about 24 MB minimum -for default packages - and removing unwanted repositories will save you say one minute or something.). You can read about this more here: the size of apt-get update lists is too big
At the same place (where you select these repositories), you can "Select best server" to make download as fast as possible. Sometimes, when mirrors near you go down, this will automatically select the best server at that time. You can read the second answer here: How can I get apt to use a mirror close to me, or choose a faster mirror?
All those settings are under "Update Manager", so take a look at it, and happy tweaking!
Been using ubuntu a long time now and can tell you its the mirror. Back in the day the canadian mirrors were so slow it was almost impossible to install (with download updates while installing option). So most of us just got in the habit of selecting "New York" as our time zone during install so that it would default to us.archive.... (the american faster mirror at the time which is in same time zone so no harm no foul). Now days though I have noticed the same problems you are describing. During a week day you will be lucky to get 100K/s. On the weekend around 3am it seems to get up around 800K/s (as fast as my modem goes so might be faster for others). So what I did to test is open up the sources and change the country code from "us.archive...." to "ca.archive....". The first thing it did was educate me that the canadian mirrors have been upgraded and fly pretty good now (during the afore mentioned weekend time). Alas it still suffers during the week days, though in my experience is still faster then the US servers (around 100-200 K/s). Just keep changing the mirror until you find one that works. I havn't tested any but "us" and "ca" so I cannot offer any insight there.
Edit sources file: sudo gedit /etc/apt/sources.list
It might be your local mirror being slow from the 12.04 attention. You might want to switch mirrors temporarily: