If you believe that the swap partition is the culprit, then you should try to disable or unmount it first (
swapoff -a) to see if your guess was correct. Or look at the System Monitor and see how much of the swap partition is currently in use. Note that you can also configure the swappiness, which means that you can have a active swap partition for system functionality such as hibernation, but that is otherwise only utilized if there is really not enough RAM available (like when an application has a memory leak).
I doubt that more RAM will do any good. 8 to 16 GB should be sufficient current games. Depending on how the game preemptively fetches data from the drive, a SSD will either only have an impact when loading new maps or levels or will minimize lags when new events are triggered, but that mostly depends on the game and how it is written.
It is also possible that you have set your graphics settings too high or you have interferences in your input devices.