There are 3 issues with this.
First, if you are using icedtea7/java7 then you will not be able to use OpenGL display mode unless you trick the Java binary to find its own library (which it for some reason is unable to find)
Secondly the 64 bit version of Java cannot use the client JVM which is designed for desktop Java applications. The 64bit version can only use the server JVM (which still runs the game but slow while textures are enabled) or other JVM that cannot run Runescape.
Lastly your graphic driver will cause issues because it will not be able to utilize OpenGL properly in Runescape. You are instead recommended to use the proprietary driver.
There is a community run wiki article that was created by us in the Runescape community tech support team to help Linux users play Runescape at the highest performance as possible.
There are even ways to run the Runescape client on Linux (either through Wine or native). The RSU client I made is an attempt to port the official client to Linux and apply as many fixes as possible
To finish off, the Linux users in Runescape have their own dedicated sticky thread in the Runescape forum where you will be able to get support for most issues.