I solved this by disabling xfce-volumed, Pulseaudio plugin was running along with it.
Open a Terminal window and type killall xfce4-volumed and try to change volume now. Only one notification will be shown.
To disable xfce4-volumed permanently open Session and Startup from settings manager. Go to Application Autostart tab. Look for XFCE Volume Daemon (pulseaudio) and untick it to disable.
Edit:
A more simple approach I found is to right-click on the volume icon in the panel and open properties. Uncheck the 'show notification when volume changes' box and log out/in.