I have done some investigation into this issue.
I don't believe there is a way around this.
The dmesg command directly prints the contents of the Kernel Ring Buffer. This contains all the ufw log entries that you are seeing.
The /etc/rsyslog.d/20-ufw.conf file is telling rsyslog which of the ufw entries in the Kernel Ring Buffer to log to either the /var/log/ufw.log or /var/log/kern.log.
You can prevent the ufw entries from being logged to /var/log/kern.log (to remove duplication) by uncommenting the line in /etc/rsyslog.d/20-ufw.conf that contains & ~.
Unfortunately there is no way to prevent the dmesg command from displaying these messages. Your work around is the best I can come up with.