Check and ajust content process limit in Firefox preferences.
In my case I unchecked "Use recommended performance settings" and reduced the limit from 8 (which was default and is still "recommended") to 1. Firefox stopped eating RAM. But twitch and youtube streaming in multiple tabs had noticeably degraded and almost became unusable. Later I increased the number to 4 and I didn't notice increase in RAM usage but browser was able to properly render heavy content again.
After the entire day in Pycharm and Firefox my RAM is only 46% full instead of 90+% that I experienced during previous couple of months.
For reference, specs of my laptop are intel i5-8265U (4 core, 8 threads) and 16 GB of RAM.
My case was really similar because I also use Pycharm (maximum heap size set to 4gb), latest Firefox with 8-10 addons and 10-100 tabs (when I closed all but 5-6 tabs, Firefox didn't free RAM) and a couple of more lightweight apps on my Ubuntu 20.04. So I hope my answer qualifies as proper answer.
The downside is I was not able to use any tools to inspect and resolve the problem. Almost pure random.
about:memory didn't reduce the RAM usage from ~10GB level even if I left only one tab with some lightweight html page (Hacker News, for example).
about:performance was useless as well, it didn't show high RAM usage.