First some history. In the earlier days of Linux, device drivers were indeed compiled directly into the kernel. In fact, it's possible still to compile drivers directly in and many kernels may still have some very common drivers included this way. The way it worked was the generic kernel shipped with the early distributions like SLS or Slackware had virtually every driver the kernel supported compiled directly into it so it would work on the widest possible variety of hardware possible. Even then, it wasn't possible to put them all in because some of them are mutually incompatible. Then, after you got your system installed, you would build your own kernel, carefully going through the configuration to make sure you included the correct drivers for everything your computer had. Sometimes you had to manually edit header files in the driver to include support for oddball hardware - an Ethernet card that used a particular chipset used that chipset's drivers, but sometimes there were funny ways it was implemented. Then you'd compile, install, and hopefully after a reboot you then had a custom kernel built for just your computer. Lean and optimized. In reality, you generally repeated this process several times including things you missed, adding support for a filesystem you forgot, or tweaking the settings in some way. Rinse and repeat.
Those days are, thankfully, long past. The kernel has, for a very long time, supported loadable modules. These are kernel drivers compiled so as to become a type of shared library that can be loaded or unloaded on demand. Now the way a kernel boots is you have the kernel file itself and a small compressed filesystem (look at initramfs on Wikipedia) that has in it all the kernel modules that kernel supports. The kernel loads, it finds its initial filesystem and then it can start loading all the drivers it needs.
That little bit of history is skipping a lot of work and sweat along the way. In between using all compiled-in drivers in one huge monolithic kernel and having a fully automated driver loading system we have today were all the steps along that path where we had modules that had to be explicitly loaded, semi-automatic loading for some, etc etc.
So, since modern kernels demand load the vast majority of the drivers they need, there are no significant amount of redundant hardware drivers taking up resources in the kernel of any modern Linux distribution.