My guess is the video is stuttering because it's trying to decode it entirely on the CPU. Even a fairly beefy CPU will have to work very hard to play a full 1080p video stream. GPU-accelerated video playback is much less taxing.
I have an Nvidia card using VDPAU with the proprietary driver, so I'm not sure what options there are for hardware accelerated video on ATI cards. First thing you should do is confirm your video hardware is even capable of video playback acceleration... models older than 2 years may not. Second, make sure you're using a driver that supports it - it's possible the ATI open source driver supports it even if the (better performing?) proprietary one doesn't. Lastly, you'll need a build of your media player software that is confirmed to make use of the appropriate video acceleration API - I recall that the mplayer in the repos for releases of Ubuntu before 10.10 didn't have VDPAU support, and it was a pain to track down which features that binary actually supported.
I googled "ati linux hardware accelerated video" (minus the quotes) and came across a few things, but nothing conclusive. Maybe an ATI user can chime in here with some help.
I would suggest editing the title of your question to be more descriptive, something like "Slow Blu Ray video playback on Boxee hardware".