It's not a good practice to upgrade Ubuntu packages manually, it's recommended to wait for the upgrade of the specific package that comes natively with the Ubuntu upgrades.
However, if you must use this approach, here's a walkthrough on how to upgrade your PHP to v5.4.x via PPA: http://www.zimbio.com/Ubuntu+Linux/articles/D_AsJR2qAL6/How+Upgrade+PHP+5+4+Ubuntu
Since you have reported "php -v" and phpinfo() version discrepancy, "php -v" gives you the version of the cli installation of php, while phpinfo() will give you the version which got installed as Apache module.
If you are not using cli php anywhere, you should be fine with running php scripts off Apache. But ideally you should fix this. I think the issue is this that your Apache and libapache2-php5-mod have been installed via ubuntu packages while you have compiled php from source. When you installed libapache2-php5-mod via apt-get install, it would have taken the relevant php package from the repo as a dependency, as seen in packages.ubuntu.com/hardy/libapache2-mod-php5 You will need to uninstall Apache, and then install apache from source instead of using the ubuntu repos to do what you wish to do.
I recommend you not to install PHP from source, but to use the PPA approach from above.