There are multiple possibilities for degrading performance, but the most likely cause is a shortage on RAM, causing the system to swap.
You can use free -m to get the memory usage, an example output:
total used free shared buffers cached
Mem: 3632 2512 1120 0 233 1456
-/+ buffers/cache: 822 2810
Swap: 8192 0 8192
The actual free space that can be allocated is visible on the row with -/+ buffers/cache. If you find that your server is swapping too fast, you might want to lower the vm.swappiness setting. You can find your current swappiness setting by running sysctl vm.swappiness. It ranges from 0 - 100, a lower value will use the swap later, a higher value make the kernel use the swap earlier. If you decide to set this value to 10, you should run the next command to change it at run-time:
sudo sysctl -w sysctl vm.swappiness=10
To make it persistent (i.e. after a reboot), add the line sysctl vm.swappiness=10 to /etc/sysctl.conf.