31

I want to create a large file ~10G filled with zeros and random values. I have tried using:

dd if=/dev/urandom of=10Gfile bs=5G count=10

It creates a file of about 2Gb and exits with a exit status '0'. I fail to understand why?

I also tried creating file using:

head -c 10G </dev/urandom >myfile

It takes about 28-30 mins to create it. But I want it created faster. Anyone has a solution?

Also i wish to create multiple files with same (pseudo) random pattern for comparison. Does anyone know a way to do that?

Improve this question
2

5 Answers 5

Reset to default
26

How about using fallocate, this tool allows us to preallocate space for a file (if the filesystem supports this feature). For example, allocating 5GB of data to a file called 'example', one can do:

fallocate -l 5G example

This is much faster than dd, and will allocate the space very rapidly.

Improve this answer
10
  • Does this file contain random data or does it contain whatever happened to be on the allocated disk space?
    – cprn
    Jul 26, 2016 at 16:37
  • It will contain all zeros. Basically, space is preallocated, and if you don't modify the data it will be presumed to be zero.
    – Colin Ian King
    Jul 26, 2016 at 16:38
  • 1
    It's very fast because it's one system call which does block preallocation (e.g. it reserves the space but does minimal I/O), where as dd'ing from /dev/zero to a file involves a load of read/writes.
    – Colin Ian King
    Jul 26, 2016 at 16:42
  • 1
    see man fallocate. It says: "fallocate is used to preallocate blocks to a file. For filesystems which support the fallocate system call, this is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros." For me "no IO to the data blocks" means that data is NOT initialized at all. Fallocate just reserves the space, nothing more.
    – user4955663
    Feb 21, 2017 at 10:43
  • 2
    The blocks are marked as uninitialized at the filesystem level, but when you read them the blocks returned to userspace will be zero filled. If you write the data then it will go to the space that is allocated. This is how sparse files work and I expect you are using non allocated zero'd file blocks without knowing it all the time. The fiemap() IOCTL will show you that a lot of files are sparse and have holes in them that you never know. The latest versions of cp even use this mechanism so they speed up copies of spare data blocks.
    – Colin Ian King
    Feb 21, 2017 at 12:57
17

You can use dd to create a file consisting solely of zeros. Example: 

dd if=/dev/zero of=zeros.img count=1 bs=1 seek=$((10 * 1024 * 1024 * 1024 - 1))

This is very fast because only one byte is really written to the physical disc. However, some file systems do not support this.

If you want to create a file containing pseudo-random contents, run:

dd if=/dev/urandom of=random.img count=1024 bs=10M

I suggest that you use 10M as buffer size (bs). This is because 10M is not too large, but it still gives you a good buffer size. It should be pretty fast, but it always depends on your disk speed and processing power.

Improve this answer
1
  • ahaha that's awesome. Just did a 12GB file on EXT4 in 0.052s. Thank you kindly
    – J-Cake
    Nov 15, 2022 at 4:18
8

Using dd, this should create a 10 GB file filled with random data:

dd if=/dev/urandom of=test1 bs=1M count=10240

count is in megabytes.

Source: stackoverflow - How to create a file with a given size in Linux?

Improve this answer
2
  • 2
    I just tried dd if=/dev/urandom of=10Gfile bs=500M count=20, which gave me 10237226010 bytes in just under 20 minutes.
    – Jos
    Aug 4, 2014 at 21:55
  • Valuable link, thanks!
    – Manuel Jordan
    Apr 1, 2022 at 15:53
3

This question was opened 5 years ago. I just stumbled across this and wanted to add my findings.

If you simply use

dd if=/dev/urandom of=random.img count=1024 bs=10M

it will work significantly faster as explained by xiaodongjie. But, you can make it even faster by using eatmydata like

eatmydata dd if=/dev/urandom of=random.img count=1024 bs=10M

What eatmydata does is it disables fsync making the disc write faster.

You can read more about it at https://flamingspork.com/projects/libeatmydata/.

Improve this answer
1
  • 1
    The way I look at it dd is fast enough to begin with, and it's called libEAT-MY-DATA for a reason.
    – karel
    Aug 9, 2019 at 11:18
1

Answering the first part of your question:

Trying to write a buffer of 5GB at a time is not a good idea as your kernel probably doesn't support that. It won't give you any performance benefit in any case. Writing 1M at a time is a good maximum.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .