I suspect this is the undesired font:
EDIT: I think is this font: AR PL UMing TW.
Is there a way of removing it or giving preference to another Chinese font?
You can change the default UI font in System » Preferences » Appearence » Fonts.
If the default Chinese font when installing the Chinese Language Support is not good enough, you can report it so that the font gets replaced.
The best fonts for Traditional Chinese is "Source Han Sans"(Noto Sans CJK), I suggest you to download the latest version in the official website, the system fonts should be remain as default, because it is way too ugly to use a Chinese font for English.
How to install: