Somewhere I read that I should remove my default programs in Ubuntu and then build them from source to speed up my system. Would this work?
The only time I would build myself is if you need a version of the software not in any repository, then consider downloading the source. Download the dependencies (development libraries etc) from Ubuntu repositories and build when you need to update.
I don't want to seem disrespectful, as I include myself in this, but the package maintainers on the whole will produce a far better build than you will. Unless your build options enable some sort of hardware acceleration, you won't see a lot of difference.
I've compiled complete systems from scratch using linux from scratch and gentoo and found I only felt performance improvements on really lower powered and specific hardware, such as the Playstation 2 and a 1ghz Atom cpu on a mini-ITX board. Bear in mind that I was required to do this on these systems as normal compilation flags would not work on the defaults.
The answer is yes.
By compiling the operating system just for your PC, by removing any and all modules that are not needed and compiling for your architecture and optimizing for your hardware and using the latest compiler you might get a 1-2% increase in performance.
It has been proven over time that a distribution like Gentoo offers a fraction of an improvement in performance, however the downside is really that it is not worth the effort. (Don't get me wrong Gentoo is a great distribution, just its not a mass market distribution)
Suddenly you'll have to track all software updates to your system and if there is a problem with the new version your needing to know the underlying application to debug why it wont build.
There are already people optimizing the operating system, only do it if you want to spend days doing this as a hobby or for research reasons.