I have never used other than Latin subtitles, but even for that some supplementary setting is needed in some cases to properly show French or Romanian.
That setting is to be found in the image you posted under Subtitle Language - Default encoding (for example, Romanian requires eastern European, Latin).
I see on the internet that other South Asian and Indian characters like Hindi work with Universal UTF-8, like you have already tried for Malayalam, and possibly VLC lacks a more specific setting. You could try other choices and see if it helps.
Also, the Font setting might count too. Have you tried different font types? (remember to restart the player)
I have also found a somewhat older post saying that VLC now renders subtitles in South Asian scripts specifically mentioning Malayalam, and recommending the use of VLC version 3 nightly.
(That can be installed instead of the stable version from the master-dayly PPA, or, in order to avoid uninstalling your stable version, you can try the "portable" 3.0.0 appimage from here, by downloading it and making it executable. At the first execution you have a choice to create the launcher ~/.local/share/applications/appimagekit-vlc.desktop. If you want a launcher, avoid name confusion with the stable version by editing that launcher and changing the Name line to Name=VLC media player 3.0.0 Nightly - Appimage.)
