The problem is I'm using Unity, the theme is Ubuntu Mono Dark, and I'm using non overlay scrollbars so I'm having a hard time actually seeing the scrollbar.
This bug has been plaguing me since at least 2011 and it doesn't look like anybody has looked at this. See here for an actual bug report: https://bugs.launchpad.net/ubuntu/+source/ubuntu-themes/+bug/721786