45

I am trying to copy a 6GB file to a USB flash drive but it won't copy.

  • 23
    Is the USB drive formatted with FAT32? That file system does not support files larger than 4GB. You have to format it differently then, e.g. as NTFS. Beware that formatting deletes all data on the drive. – Byte Commander Sep 4 '17 at 15:32
  • 14
    Can you please put more detail in the question about the "it won't copy"? What commands did you try? What error message do you get? (These are very helpful in case someone has the same problem in future and is searching for an answer...) – psmears Sep 5 '17 at 16:34
  • 2
    google for "cannot copy files larger than 4GB" and you'll get tons of answers on stackexchange – phuclv Sep 6 '17 at 8:47
  • 5
    Also might be helpful to include the size of the USB – George Tian Sep 7 '17 at 9:40
  • 3
    @Nukeface Can you include a answer with instructions on how to do that? Leaving a half-answer as a comment can often cause more harm than good. Thanks. – Eliah Kagan Sep 8 '17 at 17:56
100

This is due to FAT32 limitation. Files larger than 4GB cannot be stored on a FAT32 volume. Formatting the flash drive as exFAT or NTFS will resolve this issue.

WARNING: Back up your data. Formatting will delete all the data on your device.

  • 20
    BTW, formatting the flash drive as ext3 is also an option – Basile Starynkevitch Sep 5 '17 at 10:33
  • 18
    @BasileStarynkevitch yes, but it's not very usable on some proprietary OSes ;) – Ruslan Sep 5 '17 at 10:52
  • 11
    @Ruslan: I second this. Using ext3/4 on Windows was more painful than using exFat and NTFS in Linux. – JIV Sep 5 '17 at 13:02
  • 13
    Even though it is a Microsoft creation, there's a lot to be said for exFAT. As others have noted it is handled by all three desktop OSs, therefore you will tend to maximize your interoperability. In addition it was designed for use on large flash drives. Citation: en.wikipedia.org/wiki/ExFAT In particular, because it can mark a file as contiguous, and thereby avoid having to touch the FAT, it works extremely efficiently in the OPs use case: handling a very large file. – dgnuff Sep 5 '17 at 18:44
  • 8
    @dgnuff you mean, all three major desktop OSs :) – dn3s Sep 6 '17 at 3:38
89

If you don't want to reformat your USB drive or need it to be FAT32 you can simply split your big file into parts. Most archive managers come with the split option and for the command line there's split, e.g. for your case:

split -b4294967295 /path/to/input.file /path/to/pen/drive/output.file.

See man split for the full documentation. This will create the following files:

4,0G output.file.aa
1,6G output.file.ab

The filesize of output.file.aa matches exactly the maximum file size of your FAT32 formatted USB drive, which is 4 Gibibyte (GiB, that's not the same as Gigabyte GB) minus 1 Byte.
Thanks to Gilles for this important addition.


Before you can access the file again you need to merge its parts first. On Linux systems you can do so with:

cat output.file.* > input.file

If you're afraid that this could sort the files incorrectly read In Bash, are wildcard expansions guaranteed to be in order?
The corresponding command on Windows systems is:

copy /b output.file.aa+output.file.ab input.file

Along with many other useful GNU utilities split can be installed in Windows too, see GNU utilities for Win32.

  • 2
    I bet half of the upvotes I get here are just because of GiB vs. GB. ;) – dessert Sep 5 '17 at 13:04
  • 2
    I'd have upvoted for split even without the distinction between GiB and GB. But I'll send an attaboy for that... – Zeiss Ikon Sep 5 '17 at 13:16
  • 1
    @dessert I would expect that to get you only about 7.4% more upvotes. – Eliah Kagan Sep 6 '17 at 8:21
  • 1
    I upvoted because reformatting the drive just to be able to move one large file is a horrible answer when split exists precisely for this purpose and is not required on the receiving end. FAT32 is the lingua franca of USB drives for a reason. – Monty Harder Sep 6 '17 at 18:06
  • I think -b 4095M would be far simpler and achieve the same as the bc command. In Bash one could also use shell arithmetic: $(((4<<30)-1)). – David Foerster Dec 30 '17 at 20:23
38

Problem: FAT32 has a 4GiB limit for file size

Different scenarios and files systems are examined looking for alternatives taking into account

  • the file size problem
  • which operating systems that should read/write the USB drive

Linux only

If you intend to use the drive only with Ubuntu (and other linux distros), it is a good idea to use a linux file system, for example ext4. This way you might get higher read/write speed (depending on which process is the bottleneck), and you will get higher flexibility concerning ownership and permissions.

  • You can use the GUI program gparted to create the ext4 file system.

Full compatibility with Linux, Windows and MacOS

Windows has problems with linux file systems, and I think MacOS has problems both with linux file systems and NTFS. So if you want 'full compatibility' for reading and writing, only FAT32, UDF and exFAT remain.

  • FAT32 has a 4GiB (gibibyte, base 2) limit for file size.

    • Can be created in all three operating systems.
    • Maintain (repair) in Windows, if you have access to Windows.
    • Can be maintained in Ubuntu with dosfsck, that comes with the package dosfstools,

      sudo apt-get install dosfstools
      
      sudo dosfsck -a /dev/sdxn  # least destructive option
      sudo dosfsck -r /dev/sdxn  # more powerful option
      

      where x is the drive letter and n is the partition number, for example /dev/sdb1 for the first partition in drive b.

  • UDF probably lacks tools to repair the file system,

    • FOSS
    • Maybe it is possible to find repair tools in Windows via this link: fsck tools for UDF, and there is some tool available as source code
    • Can be created in Ubuntu
    • Compatible with linux style links.
    • Compatible with linux style permissions. You can create and modify permissions of individual files (which is not possible with FAT and NTFS).
    • A UDF partition will not be prompted for formatting by Windows 10 (while the linux ext4 file system is affected, and can be destroyed by mistake).
    • How to create and use UDF: Using the UDF as a successor of FAT for USB sticks

      So, to use it, assuming your USB stick is /dev/sdx:

      1. Install the package udftools

        sudo apt-get install udftools
        
      2. Create a partition table and one partition with gparted or gnome-disks

      3. Wipe the first mibibyte of the target partition with the risky dd (double-check the command line!)

        sudo dd if=/dev/zero of=/dev/sdx1 bs=1M count=1
        
      4. Run mkudffs,

        sudo mkudffs -b 512 --media-type=hd --lvid=my-label /dev/sdx1
        

      Wipe the first mibibyte of the partition to erase the previous file system information (or other remaining data), to prevent you USB stick from being detected as a FAT after it has been formatted with UDF.

      The -b 512 is to force a file system block size equal to the USB stick's physical block size, as required by the UDF specification. Adapt it if you have the luck of having a USB stick with a more appropriate block size.

      After that, your USB stick will be usable for reading and writing with GNU/Linux and the other free operating systems of course, but also with current versions of Windows (read-only with the outdated version XP) and with MacOS.

  • exFAT is another option. It is new compared to FAT32 and UDF and claimed to work well

    • natively with Windows
    • with MacOS
    • with Linux

More about exFAT

I have started to test exFAT in Ubuntu. I intend to edit this answer, when I have more experience of using it. Let us start with the following links,

and this command line to install [read/write] support for exFAT in Ubuntu

sudo apt-get install exfat-utils exfat-fuse
  • Can be created in Windows and MacOS and in Linux with the following command line

    sudo mkfs.exfat -n YOUR-LABEL /dev/sdxi
    

    where x is the drive letter and i is the partition number, for example /dev/sdb1.

  • Can be maintained in Windows.

My first test results:

  • Creating the exFAT file system worked well in Ubuntu (16.04 LTS (64-bit), installed system with the Xenial kernel series (4.4.0-93-generic), up to date).
  • Writing and reading a file greater than 4 GiB were successful. Ubuntu and exFAT co-operated correctly and fast (I checked with md5sum and the speed was limited by the USB system).
  • Ubuntu and Windows could read what was created in the other operating system. (I have no MacOS for testing, and have to rely on the reports from other people.)

Windows

So if you want full read/write access from Ubuntu and Windows, I would suggest that you use NTFS, which has journaling and is very debugged and polished as the [proprietary] file system for Windows. (It is also possible to use exFAT.)

  • In Ubuntu you can use the GUI program gparted to create an NTFS file system.
  • In Windows it is easy to create NTFS and exFAT file systems (they are native).

MacOS

In MacOS it is possible to use FAT32, UDF and exFAT.

You can also use ext4 with a workaround, Ubuntu Server with an SSH server in a virtual machine. (I think the same workaround would work also with Windows.) This may be worthwhile when you intend to access a lot of files via the drive and its file system, but probably not with a small USB2 pendrive.

See this link,

Using ext4 on OS X Yosemite, the long but safe way

  • 3
    AFAIR Windows XP can't write to UDF volumes. It's not much of a problem nowadays but still worth noting, just in case. – gronostaj Sep 4 '17 at 18:12
  • 2
    macOS mounts everything as read-only NTFS by default (I presume Apple don't believe the NTFS write support is sufficiently reliable.) – gsnedders Sep 5 '17 at 2:55
  • 1
    @gsnedders I expect they just don't believe in Windows. – wizzwizz4 Sep 5 '17 at 6:22
  • 2
    @sudodus Personally I've had no problems with everything from heavy NTFS partition management to manipulating Streams from Debian, but I appreciate being safe with data. Also, I have a backup, so it doesn't matter if I lose old data (I can just copy it from the backup). – wizzwizz4 Sep 5 '17 at 6:33
  • 2
    Note: you can use exfat on MacOS without problems (it can even format the drive as exfat). exfat is also becoming the default for larger flash disks / memory cards as well. – SztupY Sep 5 '17 at 12:12
11

Having had a simular issue and unwilling to reformat the USB stick, I just used an archiver. More or less all modern GUIs (such as File Roller, 7Zip, etc.) or CLIs allow for splitting the archive file. Set the split limit below the FAT boundary (that somewhat 4GB), for speed you can choose a low compression rate, even "store", i.e. do not compress at all, an voila!

5

If you have access to a Windows machine you can convert you USB-stick file system without formatting. You should use Windows command line as follows: press Win + R, type cmd -> OK, in the open window enter command convert X: /FS:ntfs where X: is the letter for your mounted USB-stick. In less than a minute it is ready. Enjoy.

  • 2
    This method can be useful :-) But converting a file system might be risky, so I would recommend to backup everything important on the USB-stick before converting. By the way, have you used this method? Does it keep the files? Does it work also to/from exfat? – sudodus Sep 5 '17 at 12:55
  • 1
    I used this method several times, all the files were readable after this procedure. This utility works only to convert to NTFS, not vice versa. – M. Dm. Sep 5 '17 at 14:34
2

Reformatting or using split are great options, or you could try using zip to compress the file, just know that some files are already compressed, so zipping them will make them larger. (Also you may lose file ownership, timestamps etc.)

protected by Community Sep 8 '17 at 7:00

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.