I often met people stating they want to install XFCE, Gnome or other desktop manager in order to get better performance. But as the conversation goes on, it appears they use plethora of apps from other desktop. Which means they have to install and run related librairies. So is it interesting to install a « light » desktop manager if you use apps from different desktop ?
If you only "sometimes" use an application, which uses GTK or QT libraries, but usually don't use them, then it is worth it as the libraries are only loaded if one application uses it. Furthermore, most applications only use "some" of the libraries, not as many as the window manager itself might require to be loaded. And in addition to the libraries there are more things which a window manager loads, which are not loaded when using another window manager even with the libraries of this manager.
Finally you will for sure notice a faster bootup of your system when using a lightweight window manager.
You should try to avoid using, for example, KDE apps loading kde-modules (try, i mean; there's nothing wrong with them but the point of keeping the system light) There's nothing wrong with using Qt apps; for example, comparing Amarok and Clementine, they are similar apps but Amarok is based around KDE while Clementine is just plain Qt; and so on with Gnome apps, although as they are Gtk apps usually you will have less dependencies and libraies to load if you are in Xfce or Lxde.