The only real downside to using hardware acceleration is if you have a buggy driver, you can run into more glitches in the graphics (software rendering is more predictable).
For example, with Linux Mint 17, I see issues with Chromium not fully repainting the screen when re-sizing the windows (if I try to use hardware acceleration). I can't perceive any speed difference between the two, though I typically turn off all "eye candy effects" regardless.
If your driver is very buggy, I would imagine hardware acceleration could cause crashes, or your whole system could even run more slowly. Otherwise, I just try the acceleration first, and drop back to software rendering if I see any weirdness (crashes, lockups, repaint issues, etc). Hardware acceleration is great if the the drivers work.