I am trying to copy a 6GB file to a USB flash drive but it won't copy.
If you don't want to reformat your USB drive or need it to be FAT32 you can simply split your big file into parts. Most archive managers come with the split option and for the command line there's
split, e.g. for your case:
split -b4294967295 /path/to/input.file /path/to/pen/drive/output.file.
man split for the full documentation.
This will create the following files:
4,0G output.file.aa 1,6G output.file.ab
The filesize of
output.file.aa matches exactly the maximum file size of your FAT32 formatted USB drive, which is 4 Gibibyte (GiB, that's not the same as Gigabyte GB) minus 1 Byte.
Thanks to Gilles for this important addition.
Before you can access the file again you need to merge its parts first. On Linux systems you can do so with:
cat output.file.* > input.file
If you're afraid that this could sort the files incorrectly read In Bash, are wildcard expansions guaranteed to be in order?
The corresponding command on Windows systems is:
copy /b output.file.aa+output.file.ab input.file
Along with many other useful GNU utilities
split can be installed in Windows too, see GNU utilities for Win32.
Problem: FAT32 has a 4GiB limit for file size
Different scenarios and files systems are examined looking for alternatives taking into account
- the file size problem
- which operating systems that should read/write the USB drive
If you intend to use the drive only with Ubuntu (and other linux distros), it is a good idea to use a linux file system, for example ext4. This way you might get higher read/write speed (depending on which process is the bottleneck), and you will get higher flexibility concerning ownership and permissions.
- You can use the GUI program
gpartedto create the ext4 file system.
Full compatibility with Linux, Windows and MacOS
Windows has problems with linux file systems, and I think MacOS has problems both with linux file systems and NTFS. So if you want 'full compatibility' for reading and writing, only FAT32, UDF and exFAT remain.
FAT32 has a 4GiB (gibibyte, base 2) limit for file size.
Can be created in all three operating systems.
Maintain (repair) in Windows, if you have access to Windows.
Can be maintained in Ubuntu with
dosfsck, that comes with the package
sudo apt-get install dosfstools sudo dosfsck -a /dev/sdxn # least destructive option sudo dosfsck -r /dev/sdxn # more powerful option
where x is the drive letter and n is the partition number, for example
/dev/sdb1for the first partition in drive b.
exFAT is another option. It is new compared to FAT32 and UDF and claimed to work well
natively with Windows
More about exFAT
Edit August 2021:
Since this answer was written, the support for exFAT is bundled in the new versions of standard Ubuntu, so you need no extra packages to run it.
end of edit
I have started to test exFAT in Ubuntu. I intend to edit this answer, when I have more experience of using it. Let us start with the following links,
ExFAT is a proprietary file system of Microsoft optimized for
> embedded systems because it is lightweight and is better suited for > solutions that have low memory and low power requirements, and can be > implemented in firmware.
and this command line to install [read/write] support for exFAT in Ubuntu
sudo apt-get install exfat-utils exfat-fuse
Can be created in Windows and MacOS and in Linux with the following command line
sudo mkfs.exfat -n YOUR-LABEL /dev/sdxi
x is the drive letter and
i is the partition number, for example
- Can be maintained in Windows.
My first test results:
- Creating the exFAT file system worked well in Ubuntu (16.04 LTS (64-bit), installed system with the Xenial kernel series (4.4.0-93-generic), up to date).
- Writing and reading a file greater than 4 GiB were successful. Ubuntu and exFAT co-operated correctly and fast (I checked with
md5sumand the speed was limited by the USB system).
- Ubuntu and Windows could read what was created in the other operating system. (I have no MacOS for testing, and have to rely on the reports from other people.)
So if you want full read/write access from Ubuntu and Windows, I would suggest that you use NTFS, which has journaling and is very debugged and polished as the [proprietary] file system for Windows. (It is also possible to use exFAT.)
In Ubuntu you can use the GUI program
gpartedto create an NTFS file system.
In Windows it is easy to create NTFS and exFAT file systems (they are native).
UDF probably lacks tools to repair the file system,
Maybe it is possible to find repair tools in Windows via this link: fsck tools for UDF, and there is some tool available as source code
Can be created in Ubuntu
Compatible with linux style links.
Compatible with linux style permissions. You can create and modify permissions of individual files (which is not possible with FAT and NTFS).
A UDF partition will not be prompted for formatting by Windows 10 (while the linux ext4 file system is affected, and can be destroyed by mistake).
How to create and use UDF: Using the UDF as a successor of FAT for USB sticks
So, to use it, assuming your USB stick is
Install the package
sudo apt-get install udftools
Create a partition table and one partition with
Wipe the first mibibyte of the target partition with the risky
dd(double-check the command line!)
sudo dd if=/dev/zero of=/dev/sdx1 bs=1M count=1
sudo mkudffs -b 512 --media-type=hd --lvid=my-label /dev/sdx1
Wipe the first mibibyte of the partition to erase the previous file system information (or other remaining data), to prevent you USB stick from being detected as a FAT after it has been formatted with UDF.
-b 512is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with a more appropriate block size.
After that, your USB stick will be usable for reading and writing with GNU/Linux but also with current versions of Windows (read-only with the outdated version XP). Unfortunately this version of UDF created in Linux cannot be managed by MacOS (tested in several versions of MacOS in November 2020).
Edit August 2021: HFS+ (journaled)
Since this answer was written, I discovered that Linux can read and write HFS+ (journaled), so use Disk Utility.app on your Mac to format the partition with HFS+ (journaled) in order to have a drive (for example an external drive) that can transfer or share data between Ubuntu and MacOS.
end of edit
In MacOS it is possible to use FAT32 and exFAT.
You can also use ext4 with a workaround, Ubuntu Server with an SSH server in a virtual machine. (I think the same workaround would work also with Windows.) This may be worthwhile when you intend to access a lot of files via the drive and its file system, but probably not with a small USB2 pendrive.
See this link: Using ext4 on OS X Yosemite, the long but safe way
Having had a simular issue and unwilling to reformat the USB stick, I just used an archiver. More or less all modern GUIs (such as File Roller, 7Zip, etc.) or CLIs allow for splitting the archive file. Set the split limit below the FAT boundary (that somewhat 4GB), for speed you can choose a low compression rate, even "store", i.e. do not compress at all, an voila!
If you have access to a Windows machine you can convert you USB-stick file system without formatting.
You should use Windows command line as follows:
Win + R, type
OK, in the open window enter command
convert X: /FS:ntfs where
X: is the letter for your mounted USB-stick. In less than a minute it is ready. Enjoy.