You don't need (and you can't) install your real hardware drivers inside VirtualBox. Windows, while running inside VirtualBox can't access your hardware, it can only interact with a Virtual (false) hardware that VirtualBox will set for it. Virtual Box will translate it to the real hardware. You can get a more complete explanation of how virtual machines work in wikipedia.
That it's the reason that you can't really play the latest games under Virtual Box. Nor the game, nor Windows can access your graphics card to get the most speed. This is also the reason why running windows under a virtual machine is slower than running it on bare metal.
But you will need to install the drivers for the Virtual Hardware to get the best experience possible. They are called Guest Additions in Virtual Box, check this question to install them.
Note that the virtual machine has no direct access to your card, so watching HD videos or playing 3D games can't be accelerated by it and it will be slower than doing it in the host. As you experienced, the speed will be similar to the one that you see while using windows without the drivers (i.e. using no video card acceleration).