I have noticed this mainly when playing flash games in my firefox or playing x-com in dosbox. Game just runs fine, but then suddenly it slows down to crawl. When i touch my touchpad and move the cursor around, it goes good again. But when i let go of the touchpad, it goes to crawl again.
It could be that you have set up a background process to run when the computer is idle. It could also be that your screensaver is not playing well with your configuration, and it's trying to load and slowing down the computer in the process.