You need to put a patch on the kernel to use it.
You may use this link to modify the files,
There are multiple ways to interface the SPI devices with Pandaboard, Mostly you will be using the SPI available on J3 connector.
Some parts from the link mentioned above:
You do not need to write any kernel code and study the kernel API for
SPI. The kernel already contains several sort of device drivers.
Pandaboard's SPI masters are fully supported by the Linux kernel's
mcspi driver. That being said, one will need to make only some minor
modifications to the PandaBoard's board file
(arch/arm/mach-omap2/board-omap4panda.c) in order to use them.
Long back I had a chance to work with RPi SPI, and as you said, there is no pain in using the SPI on RPi. On Panda, we need to perform some hacks.