Xbox 360 is a tri-core and has 512MB of GDDR3 clocked at 700MHz. It is shared between the CPU and GPU via unified memory architecture It doesn't have to run a desktop environment or even an X server or many hardware drivers.
The GPU its self is called a Xenos. The GPU package contains two separate silicon dies, each on a 90 nm chip with a clock speed of 500 MHz; the GPU proper and a 10 MB eDRAM daughter-die. Thanks to the daughter die, the Xenos can do 4x FSAA, z-buffering, and alpha blending with no appreciable performance penalty on the GPU.
This is how it is able to play games the way it does.
So if you load the xbox up with Ubuntu, an X Server and a DE, there isn't going to be enough of that 512MB to run a game on a system that isn't using drivers designed for the hardware.
So I am afraid you have guessed right, it is a no go on the Xbox. You will have to invest on some upgrade for your machine. And even if you do, you will still have to run the games via wine, which may cause problems within its self.