I understand that Ubuntu has switched from Unity to GNOME. So how do I scale display in Ubuntu 17.10?
This OMG Ubuntu article refers to this as "fractional scaling" and notes it has been excluded from the Gnome shell v3.26 used in Ubuntu 17.10. However, it can be enabled "experimentally" by using the following command in a terminal:
gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer']"
This will allow you to open 'Displays' (Settings > Devices > Displays) and see 'Scale' values other than 100% and 200%.
Your other option is to add the Untiy desktop back to Ubuntu 17.10 which retains the old scale factor options.