If you frequently experience slowdown due to page swapping you likely need more memory. Shutting down unused or low priority services and making other memory reducing adjustments may be used as a temporary stopgap.
What is the reason behind this and how does the OS decide which part of data should go to swap?
While your memory heavy process was running the memory management swapped out pages to make room for your process. This is usually done on a Least Recently Used basis. (Programs often have pages of memory which were required at one point, but no longer are.)
If you have multiple devices, you can break the page mapping by removing that swap partition. This will force the pages into memory (although they may be swapped out fairly quickly).
How long is this data supposed to be there and how is it "freed"?
Once mapped to swap, pages may remain mapped to swap until the process is terminated at which point the pages are "freed". Pages mapped to the swap space may be located in memory. The page manager may maintain the mapping to prevent the need to remap the page if it needs to be swapped out again.
Some slowdown while the pages are swapped into memory is to be expected. Depending on memory demand and program scheduling, use of swap in known to significantly decrease performance.
munin to monitor page in/out activity should enable you do see how long it takes to map active pages into memory. Once they have been swapped back into memory they should stay there as long as memory is available.
Some things which may increase swap performance.
- A dedicated swap partition located next to actively accessed partitions. This reduces head latency when swapping is required.
- Using a swap file located on the most heavily used partition. This may reduce head latency.
- A dedicated drive for the swap partition. A number of I/O performance improvements apply.
- Use of an SSD for swap space. I/O is much faster than disk. (A fast USB drive may also perform better than your disk, but measure I/O rates first.)
Swap files and partitions can be added and removed from a running system. Swap devices can be given different priorities to control which device is used first.