Most articles mention disabling "require encryption" but you also need to change the "security mode", otherwise it never connects. There are two changes you must make in dconf-editor.
While the article is far more detailed, below is an overview, go to linked article for a more detailed explanation and other things you might want to do. Even though it is for an older version of Ubuntu some of it may still be useful and most of the settings for this specific issue are the same. This was tested on 16.04.
1) Install dconf tools
sudo apt-get install dconf-tools
2) run dconf-editor
3) edit org > gnome > desktop > remote-access disable require-encryption (do not press set-default)
4) Change authentication-method from 'none' to 'vnc' (do not press set default)
5) exit dconf editor
6) In terminal type
and select "Require the user to enter this password:" then enter a password.
Go to your Mac and connect via the Connect in the finder vnc://yourhostname.local it should work rather than show a spinner.