First of all, I don't know if Krusader uses a different password store in XFCE or any non-KDE environment, but in KDE it saves passwords in the KDE Wallet. Anyway, you could try to see if you find your password there.
You can start the Waller Manager by running
kwalletmanager at a terminal or using Alt+F2. A window should open that will show your wallet; click it to open it in a new window.
In the wallet window, search for your ssh connection (or something like that) in the top text box, or expand Passwords -> Maps in the left tree. If you find your entry, just right-click it and choose Delete.
If the Waller Manager shows more than one wallet, you can find in the KDE System Settings which one is the default. Run
systemsettings and go to "Account Details" -> "KDE Wallet"; the default wallet should be selected in a combo box labeled "Select wallet to use as default".
Of course, if there is no
kwalletmanager in your system or you can't find your password in there, then Krusader must be storing them elsewhere.