If you're referring to display drivers that come with xorg, for example
You can remove some of them with
apt-get remove <driver_name> if you want to but I'd advise against it. They are there in case you happen to change your display adapter and they don't take up very much space.
Removing one of them also removes
xserver-xorg-video-all which is the metapackage containing all the display drivers that come with xorg. This shouldn't cause any problems, but I'd still play it safe and leave it alone.