I suspect that what you are missing is the name of the package containing the driver (if I'm wrong, please provide more information). So, first of all, we have to find the name of the package. There are two ways to proceed, choose the one you prefer.
If you prefer to use a graphical application, you can open Synaptic package manager, then select Origin → Local from the left sidebar. From there, you can see the list of all the locally installed package (i.e. the packages that were not installed through a software repository). Once you have found your driver, you can right-click on it and
Mark (it) for removal.
If, instead, you prefer the terminal, you can get a list of the locally installed packages with this command:
aptitude search "~Ol"
Once you have found the name of your package, just use
sudo apt-get remove PACKAGE.