When the user selects "Safely Remove Drive", the partition of the media (in this case the USB flash memory) is "unmounted" as soon as the drive has finished its current operation(s).
Then, it's safe to proceed and physically remove the USB stick, because the drive is "switched off".
If you do not "eject" the media, meaning that is still mounted (i.e. linked to a directory, usually in /media), and you remove the drive, you can corrupt the files on the USB stick, even if you think the system is not reading/writing anything on the drive itself. So my advice is to always eject it.
This is just a quick answer, you can also check this article: http://lifehacker.com/5863810/do-i-really-need-to-eject-usb-drives-before-removing-them