This question already has an answer here:
After installing Ubuntu 14.04 guest on VirtualBox running on a Max OSX host, I am stuck with 640x480 screen resolution. Adding guest additions again doesn't help. Previous Ubuntu installations are fine with bigger resolution. The 'Detect Displays' button on the 'Display Settings' Settings screen is not reachable, because 640x480 is too small. I could not find a
/etc/X11/xorg.conf file to configure manually.