If you'd rather not get too heavy about this, you can just stream over the raw sound, like you suggest in your question. You can also convert it to MP3 (or another codec - speex might be worth it, if it can be streamed) to save a bit of bandwidth.
Here's what I'm running successfully here:
ssh oli@bessy "arecord -q -t raw | lame -x -r -" | mpg321
That just uses
arecord to record from the default source (
-q is there to keep it quiet). That can, of course, be tuned. That recording data gets piped through
lame which is an MP3 encoder. That output is received back over SSH to your client machine, where it gets piped into
mpg123, a command line MP3 player.
Note I have SSH key authentication set up, so there's no password prompt. I'd suggest you do something similar.
You might have issues with
arecord. I tried it on our media centre first of all but that wasn't having any of it. I stopped trying to guess which device it wanted me to use and tried the laptop which "just worked".