Once data goes into swap, it is normal for it to stay there even once your memory starts to free up again. It is a good thing and there is no need for concern.
Due to hard disk access being significantly slower than memory, your system will avoid swapping data in or out of swap when it doesn't think it is necessary for system performance or stability. So, data will only go into swap when the system is running out of free memory or there is a lot of pressure on the disk cache, and your system decides the time-consuming process of placing data into swap will pay off with better or more reliable system performance afterwards.
Transferring data out of swap is (for traditional hard disks, at least) just as time-consuming as putting it in there. So your system will be reluctant to remove data from swap if it's not actually being used. If you have data in swap and it's not being used, then it's actually a good thing that it remains in swap, since it leaves more free memory for other things that are being used, potentially speeding up your system.
The best way to completely avoid swapping is to buy more physical RAM if that's possible. When it swaps, your system is giving you the best performance it can with the amount of physical memory it has.
All that said, if you are sure that the low-memory/high cache event that caused the swap in the first place won't happen again and you don't mind waiting, you can force the system to release all of its swap data with the following commands.
sudo swapoff -a
sudo swapon -a
This will disable then re-enable swap, forcing data out of it in the process. As explained, there is probably no real reason to do this - beyond curiosity or seeing how your swap is working.