You can use Unetbootin to create a persistent USB drive as you can see here.
So what you should do next:
Update: ... or you can use the binaries from the author's site
To run these binaries, download them and run the command chmod +x
./unetbootin-linux, or go to Properties->Permissions and check "Execute"), then
start the application by running ./unetbootin-linux
Set it up as you can see on the image below
Once everything is done, the final step is to delete the small casper-rw file created on the vfat partition you installed the LiveCD on, so that the large ext4 casper-rw partition is used instead. Mount this partition (here, sdb1), and delete /casper-rw.
Persistence (via the persistent kernel option) should already be set up if you used Unetbootin, so safely eject the USB and you're ready to go!