Well this isn't really an answer because I haven't fully solved the problem myself. My current "solution" is to mostly use Brave but it can't handle all website so it is not a complete solution.
Another solution could be Facebooks OOMD or something similar that keeps track of running processes and automatically kills them if the threaten system stability. That implementation is rather server-centered and probably not very suitable for a desktop (or terminal server)
However the real solution is to download the source code and patch it to become less memory hungry. I am working on that but the irony here is that the build process are killed by the OOM killer. At first it was freezing the system but then I added RPM_BUILD_NCPUS=1 (I'm doing this on fedora) which made it run almost to the end. But LD chokes on merging all those .o files to a binary and runs out of memory.
I think that the main problem is that Firefox treats available memory (including swap) as free memory and use it for caching and garbage. The first thing that I will try is to find out where Firefox checks/calculates available memory and see if I can plug the memory hunger by correctly assessing free memory. That could be about changing one line of code if they have been kind enough to put such code in one location.
For now my answer is to use Brave when it works and switch to Firefox or Chrome when it doesn't. But NEVER use Firefox or Chrome when another browser is running. Especially not together. They will eat each others memory and often lock into a loop of releasing and grabbing memory.
Finally if you put more than 24 GB per browser you intent to run that should be enough according to testing done by others. Neither Firefox nor Chrome seem to use more than that specific amount.
Oh and another solution man be to use a 32 bit version of Firefox with would be limited to 4GB per process. Although Firefox spawn many processes in recent versions so that may not really solve the problem.
I also tried creating a virtual machine where I tried both NetBSD and OpenBSD with various RAM sizes. In NetBSD I ran Firefox without much problems with 1 GB of RAM. and in OpenBSD I needed 2 GB. A VM like VirtualBox and without an internal swapfile will physically limit how much memory an application can use. But you will lose all hardware acceleration of graphics. I used VirtualBox and not libvirt because VirtualBox allow constraining both memory and CPU for the guest OS. I did not try this with Linux because my main objective was to try out *BSD OS:es and Firefox was just one of the things I tested in them.
When I triggered heavy swapping in NetBSD the interesting result is that Firefox choked and was frozen for about 15 minutes but the rest of the OS was completely fluent. I seems like swapping is constrained to the individual process somehow in NetBSD. But doing that in Linux would probably require heavy kernel changes and it still doesn't really solve the problem. It solves the problems for the other applications that you are running but Firefox still freezes. I did not try this in OpenBSD but it is a fork of NetBSD so I would suspect similar results.
The goal here would be to get a solution where both the application and the operating system works without freezing or entering a swap loop. Killing the application doesn't really reach that goal. Nor would a different scheduler. The only solution is to fix the application. And frankly Firefox are in need of heavy modularization so that building it doesn't take hours and doesn't require rebuilding all components, including all the bundled third party components.
prlimit- see the man page.ulimitfunction to call the program (e.g.,ulimit -m <size> firefox)- seeman bash