I actually had this problem since at least 10.04.
Empathy ignores gnome-appearance-properties, and settings in /etc/fonts/fonts.conf
It's not just a matter of font size, but also it renders Unicode incorrectly due to none configurable font settings that also doesn't obey system settings.
Is there a work around for this? because the empathy guys haven't even confirmed these bugs yet.
The reason why Empathy doesn't have same fonts as the rest of your desktop UI is because in the "client area", it uses UI settings as specified by the chat theme (Since version 2.27.3, Empathy uses Adium Message Styles which define how fonts and overall UI should look like for your chat conversations, visit this page for more details), so until something's wrong with the chat theme you're using, it shouldn't look ugly.
As far as you've used default chat theme ("Ubuntu" in any case since 9.10), there shouldn't be any trouble what you've mentioned as long as the font the theme uses are still installed in the system. But as you've mentioned that you're facing it since 10.04, I wonder if you're using default settings or have customized the system so dramatically that the problem has surfaced in every version of Ubuntu since 10.04.
As a quick work-around I recommend you to download chat theme for Empathy called Renkoo (It is the theme from which the Ubuntu chat theme is created, so you'll hardly notice any difference). Once you download it, the theme folder has .AdiumMessageStyle extension, keep the folder in your home directory in .local/share/adium/message-styles/ (if directory doesn't exist, create it). Once you put the folder in there, it should show up in Edit->Preferences->Themes[Tab]. Change the theme and see if fonts now look properly (You'll need to reopen the chat window for changes to take effect).
