There are four layers in this abridged explanation of the audio stack.
- The hardware.
ALSA has all the hardware support. It's great at talking to a whole load of different stuff but it's relatively rubbish at talking to more than one application or providing network-aware features like Pulseaudio.
In this sort of stack, PulseAudio is the only thing that interfaces with the main ALSA devices so if you want to make a sound, you either have to kick PulseAudio off, or interface with it.
PulseAudio can accept connections from clients that only speak ALSA (it pretends to be an ALSA device) for legacy reasons which allows things like Wine, Skype, et al to work (most of the time).
Theoretically you could remove PulseAudio. In practice this would probably completely knacker your system if you're using more than one application that wants to make a noise at once. Our media centre box here only uses ALSA for this reason (plus I need raw HD over HDMI access that old PulseAudio used to interfere with - it might not now).
I would suggest you keep both. You need ALSA and PulseAudio, for all its alleged sins, does a pretty good job these days.