Ask Ubuntu is a question and answer site for Ubuntu users and developers. It only takes a minute to sign up.
Sign up to join this community
There are some web sites describing
that can create a persistent live drive from an iso file with Ubuntu 19.10 and the corresponding Ubuntu family flavours.
Yes, there are some web sites describing new tools and new features in [new versions of] old tools that can create a persistent live drive from an iso file with Ubuntu 19.10 and the corresponding Ubuntu family flavours.
This makes it much easier to create a live drive with a partition for persistence with Ubuntu 19.10.
In previous versions of Ubuntu it was possible to use a file in a FAT32 file system for persistence. Its size is limited to 4 GiB. It was possible to boot via grub in a separate partition (from the content of the iso file) and use a partition for persistence. But this is more complicated so not suitable for manual methods.
In 19.10 and future versions the size of a partition for persistence is only limited by the size of the drive (USB pendrive, SSD, HDD, memory card).
Debian 10 live iso files have also this feature, and can be used to create persistent live drives with a partition for persistence.
Newest new: In the developing Focal Fossa, to be released as 20.04 LTS, the default label of the partition for persistence is changed from casper-rw to writable. This is implemented in mkusb-plug version 2.5.5, (which is bundled with mkusb version 12.4.3 in the PPA).
If you run standard Ubuntu live, you need an extra instruction to get the repository Universe. (Kubuntu, Lubuntu ... Xubuntu have the repository Universe activated automatically.)
sudo add-apt-repository universe # only for standard Ubuntu
sudo add-apt-repository ppa:mkusb/ppa # and press Enter
sudo apt-get update
sudo apt-get install mkusb mkusb-nox usb-pack-efi
Please read the warnings when you try older versions or re-spins (distros based on Ubuntu)
Beginning of dialogue:
End of dialogue:
Update: Rufus - a well-known tool that can use this new feature in Ubuntu 19.10. Be sure to use the current version (Rufus 3.9 or newer),
works well to create live-only drives with all current Ubuntu versions,
can create persistent live drives with Ubuntu 19.10 and future versions. Please upgrade to version 3.9 (or newer), which creates a good ext3 file system and can make good persistent live drives.
Please read the changelog at the web site, and the warnings (pop-up windows), they can help you avoid serious mistakes.
Edit: Rufus 3.9 has arrived :-)
Simple method where the iso file is edited before cloning
This new feature in Ubuntu 19.10 can be used also from Windows,
Use HxD (a binary editor) to edit the iso file in order to replace two cosmetic boot options 'quiet splash' with 'persistent ' (replace 12 characters with 12 characters)
Use Win32 Disk Imager to clone the edited iso file
The first time the cloned drive is booted, the Ubuntu system will create a casper-rw partition with an ext4 file system automatically. Simple and robust!
There are detailed descriptions at these links:
casper-rw
partition for persistence manually and remove or rename the casper-rw fileSee this link describing how to add a persistent partition to a UNetbootin live/persistent USB (Ubuntu 19.10+)
Please notice 'Space used to preserve files across reboots ...'
[![enter image description here][10]][10]
You can extract and clone a compressed image file directly according to the following link,
Xubuntu Core 20.04 LTS with mkusb 12.4.5 (mkusb-dus and mkusb-plug)
This works in all operating systems where you have
xz.
You need only a few manual steps to create a persistent live drive with a partition for persistence, when you are running Ubuntu.
Edit the iso file to replace quiet splash with persistent . Yes, you can
edit the binary iso file and replace 12 characters with 12 other characters and
flash the output to the target device (usually a USB pendrive). sed can do it.
Create a partition 'behind' the flashed copy of the edited iso file. fdisk can do it.
Create an ext2 file system in this partition and put the label casper-rw
on this partition. mkfs.ext2 can do it.
Flush the buffers. sync can do it.
This is what mkusb-minp is doing plus a lot of checking, that wrap a safety
belt around the process. In other words, you get help to write to the correct
target device and get warnings when there might be problems.
You can do it yourself and feel every step, or you can read the code of the
shellscript mkusb-minp and understand the details.
Please notice that I use dus and select 'Cloning' in the following example.
You can use another cloning tool, but if you use a simple tool like dd, please
double-check that you are cloning to the currect device, and that all partitions
on that device are unmounted.
strings lubuntu-19.10-desktop-amd64.iso |grep 'quiet splash' # check that 'quiet splash' is there to be replaced by 'persistent ' (12 characters)
sed 's/quiet splash/persistent /' lubuntu-19.10-desktop-amd64.iso > persistent-lubuntu-19.10-desktop-amd64.iso # yes, sed works with binary files
ls -l *19.10* # check that the size is the same
strings persistent-lubuntu-19.10-desktop-amd64.iso |grep 'persistent ' # check that 'persistent' is there now
dus persistent-lubuntu-19.10-desktop-amd64.iso # I use mkusb-dus, you can use the Ubuntu Startup Disk Creator or another cloning tool
sudo lsblk -fm # It is important to check the device letter of the target drive (the USB drive, that you want persistent live
sudo fdisk /dev/sdx # x is the device letter of the target drive, please double-check that you have the correct letter
n # new partition
p # primary
<Return> # default: 3
<Return> # default: next free sector
<Return> # default: last addressable sector
w # write and quit
sudo lsblk -fm # check that things look good and verify that partition #3 is the correct partition to be used to store the persistent data
sudo mkfs.ext2 -L casper-rw /dev/sdx3 # put label and file system into the partition of persistence
sudo lsblk -fm # check that things look good
sync # flush the buffers and wait for prompt
There are general manual methods that can be easily modified from using a file for persistence to a partition for persistence. See the following link,
Isn't live the default for a cloned drive?
Well, it used to be, and it is still live but not live-only in 19.10.
If you want to make a purely live-only drive, where nothing will be preserved after shutdown and reboot, you can
use mkusb-minp with the option -n or mkusb-plug and select 'No-persistent live drive'.
or manually modify the built-in boot options to replace 'quiet splash' with 'nopersistent'
mkusb-minp maybe I should reduce the screenshots and/or remove the code list with the whole dialogue.
I wrote a bash script (one liner) to edit live USB disk to change "quiet splash" to "persistent " based on the answer by @sudodus.
Disclaimer: it worked on my system, use at your own risk. It is for bash, may not work on other shells w/out edit.
After boot from USB extra partition writable was created w/out my intervention for overlay for persistence, if not fdisk can be used.
First variant of code is not efficient as it starts search with grep all over after each edit; but it can be stopped via ctrl-c when some changes are made.
device_to_edit=/dev/sdb; string_to_edit="quiet splas\x68"; new_string="persistent "; block_size=512; ed_count=0; to_end=""; if [ ! ${#string_to_edit} -eq ${#new_string} ]; then echo -e "\n Suspect a danger as strings lengths not same"; elif [ $block_size -lt ${#new_string} ]; then echo -e "\n\ Suspect a danger as block size too small for string length"; elif file_size=$(lsblk -b --output PATH,SIZE | grep "$device_to_edit " | sed -E 's/.* ([0-9]{1,})/\1/'); [ $((file_size/1024/1024/1024)) -gt 20 ]; then echo -e "\n Suspect a danger as device size is large"; else while [ -z $to_end ]; do offset_to_edit=$(sudo grep --only-matching --byte-offset --max-count=1 --text --perl-regexp "$string_to_edit" $device_to_edit | awk --field-separator ":" '{ FS = "0" ; print $1 ; exit }'); if [ -z "$offset_to_edit" ]; then echo -e "\n end of file is reached" ; to_end="yes" ; else block_to_edit=$(($offset_to_edit/$block_size)) ; sudo dd if=/$device_to_edit count=2 skip=$block_to_edit bs=$block_size | sed "s/$string_to_edit/$new_string/" | sudo dd conv=notrunc of=$device_to_edit count=2 seek=$block_to_edit bs=$block_size; echo "blocks $block_to_edit and next edited (blocks start with 0)"; ((ed_count++)) ; fi ; done ; echo -e "\n$ed_count editions made"; fi
In second variant of code grep is done only one time, but grep goes through complete file (e.g. block device, USB) which can take significant time before any editing (on my system it took ~ 15 minutes for 8Gb drive, I added date for the sake of that curiosity):
date; device_to_edit=/dev/sdb; string_to_edit="quiet splash"; new_string="persistent "; block_size=512; ed_count=0; to_end=""; if [ ! ${#string_to_edit} -eq ${#new_string} ]; then echo -e "\n Suspect a danger as strings lengths not same"; elif [ $block_size -lt ${#new_string} ]; then echo -e "\n\ Suspect a danger as block size too small for string length"; elif file_size=$(lsblk -b --output PATH,SIZE | grep "$device_to_edit " | sed -E 's/.* ([0-9]{1,})/\1/'); [ $((file_size/1024/1024/1024)) -gt 20 ]; then echo -e "\n Suspect a danger as device size is large"; else sudo grep --only-matching --byte-offset --text --perl-regexp "$string_to_edit" $device_to_edit | awk --field-separator ":" '{ FS = ":" ; print $1 }' | { IFS=$'\n' read -d '' -a offsets_to_edit ; for offset in ${offsets_to_edit[@]}; do block_to_edit=$(($offset/$block_size)) ; sudo dd if=/$device_to_edit count=2 skip=$block_to_edit bs=$block_size | sed "s/$string_to_edit/$new_string/" | sudo dd conv=notrunc of=$device_to_edit count=2 seek=$block_to_edit bs=$block_size; echo "blocks $block_to_edit and next edited (blocks start with 0)"; ((ed_count++)); done; echo -e "\n$ed_count editions made"; } ; fi; date
Notes:
Meaning of the scripts parameters and syntax could be understood by reading man bash, man awk, man sed, info '(coreutils) dd invocation', help read. Why (list) / { list; } needed in pipe is explained in https://stackoverflow.com/questions/2746553/read-values-into-a-shell-variable-from-a-pipe ("A pipeline may spawn a subshell, where the environment is inherited by value, not by reference"), working with two blocks is needed in case string starts in one block and ends in next one.
The scrips include three safety checks: lengths of original and replacement strings are same, block size is sufficiently large, device size is not larger then 20Gb (against accidental edit of main drive, need to be edited for larger flash storage, code is $((file_size/1024/1024/1024)) -gt 20).
There are some redundancies in the scripts, i.e. FS = ":" ; in awk, for some reason field separator did not work as I expected and I had to add --field-separator ":", also in grep --perl-regexp is needed to use hex e.g. \xaa values, but this edit does not have them.
sed, and maybe you can make it shorter with awk. See this link: sed 's/quiet splash/persistent /' eoan-desktop-amd64.iso > persistent-eoan-desktop-amd64.iso # 12 char
(But not the necessarily the best).
For making a Persistent drive not limited to 4GB in Ubuntu, mkusb is easiest, (and best), to use. (What else is there)?
For making a Persistent drive not limited to 4GB in Windows 10, Rufus seems to currently be the easiest tool to use.
That does not make it the best tool though.
I did some simple experiments comparing the latest version of Rufus and mkusb:
First I wiped a 4GB USB and then ran Rufus in Windows 10, rebooted, and had a look ar the /casper-rw/upper/ folder with Disk Usage Analyzer.
Then I deleted the casper-rw partition and made a new one.
I rebooted the drive a couple times, without making any changes or saving anything, and again had a look at the repopulated Persistent partition with DUA.
I again wiped the drive and created another Persistent USB with mkusb, rebooted and had another look with DUA.
Again I deleted the casper-rw partition and made a new one.
I rebooted the drive a couple times, without making any changes, and again had a look at the repopulated Persistent partition with DUA.
As can be seen from the attached charts, mkusb seems to be superior in the use of disk space, and grub 2 is more straight forward than syslinux.
Persistent USB made with Rufus
Persistent USB made with Rufus with Repopulated casper-rw partition
Persistent USB made with mkusb
Persistent USB made with mkusb with Repopulated casper-rw partition
Edit: following the trail of persistent disk space usage.
I took a look at disk space usage in the casper-rw partition of Rufus and mkusb freshly made persistent USB's
Rufus var folder
Contents of Rufus snaps folder
mkusb var folder
Contents of mkusb snaps folder
Data not yet confirmed
Edit: Rufus 3.9.1624
Ran Disk Usage Analyzer with the latest version of Rufus, Results were close to Rufus 3.8. It apears that mkusb still makes a more space efficient Persistent drive.
Persistent USB made with Rufus 3.9
Rufus 3.9 var folder
Again, Data and methodology has not been confirmed.
ext2fs_r_blocks_count_set()), so I would surmise that, if mkusb went for less, then of course it will look like mkusb makes better use of space, even as what might happen in the long run is that you're going to be short of inodes to create new files as the drive fills... For the record, Rufus uses the default values from mke2fs.conf.
e2fsck -f the drive is almost as good as a fresh drive made by mkusb. So there is a real problem with the file system. Please take a second look at it.