I suspect this is the undesired font:
EDIT: I think is this font: AR PL UMing TW.
Is there a way of removing it or giving preference to another Chinese font?
You can change the default UI font in System » Preferences » Appearence » Fonts.
If the default Chinese font when installing the Chinese Language Support is not good enough, you can report it so that the font gets replaced.