I think you can via Jack. Unfortunately I can't test it myself at this very moment so I can't give you a full answer.
Via Jack you can route your audio input/output from your applications to your desired output connections and this way you can send your application's sound in order to fit your needs. I have routed inputs and outputs this way, coming from several sources and to different sound cards, that's why I think that you can -in example- route both L&R from your games to the proper output connectors and the L&R from your music player to the right connectors.
Here is a screenshot of Jack's Patchbay (from the net):
Jack Download: http://jackaudio.org/download
The screenshot provided presents qjackctl, a graphical frontend for the JACK server -
sudo apt-get install qjackctl.