I know its an old thread, but I recently had the same issue on an Ubuntu host+guest setup and figured I might be able to help someone in my same boat.
I had previously installed Guest Additions via sudo apt install virtualbox-guest-additions-iso
, and would get the issue where I can either have resizing without 3D acceleration through VBoxVGA, or 3D acceleration with no resizing with VMSVGA (and no imcompatible settings message).
I uninstalled the additions with sudo apt remove virtualbox-guest-additions-iso && sudo apt autoremove
, then rebooted. I then installed Guest Additions the intended way, by inserting the ISO and installing the .run file, and viola, I now have auto resizing display with VMSVGA and 3D acceleration.