If you know of a specific fix or improvement in how the newer kernel handles your hardware, I say go for it.
I would not have upgraded to a mainline kernel due to lack of support, except that I was having some trouble with N64 game emulation running very sluggishly with lots of bugs. After trying every configuration I could find for the emulator as well as xorg, I read Here's Why Radeon Graphics Are Faster On Linux 3.12. This was enough to motivate me to test it and the results in appearance and performance were amazing with no new problems arising as of yet.
One thing to note is whether you have external modules (aka out-of-tree) installed (see Indentify out of tree modules to get an idea where these are). If you need these modules, consider whether they will build against the new kernel version. Do some research and test them yourself on the new kernel version. Worse case scenario, you can boot into the previous kernel version and uninstall the new one.
It is recommended to remove external or proprietary modules before attempting to upgrade to the mainline kernel. I did have to remove virtualbox to avoid the "Error! Bad return status for module build" during the kernel install.