19

How to create a file from terminal repeating a set of words infinitely? I need it to create a huge file for parsing purposes like 2-4GB in size. Currently I am manually copying pasting lines into the same file to increase the size.

  • 1
    I'd like to see an answer working with special unix files, so it wouldn't actually occupy that space. Is that possible? – Délisson Junio Sep 27 '16 at 20:59
  • 1
    You mean something truly infinite like mkfifo huge.tmp; while true; do yes "a dummy line" > huge.tmp; done? – Boldewyn Sep 28 '16 at 14:14
50

There's an easy way to repeat a line lots of times:

yes we have no bananas | head -n 10000 > out.txt

will result in out.txt containing 10,000 lines all saying "we have no bananas".


To limit the output to an exact number of bytes, use head's -c option instead of -n. For example, this generates exactly 10 kB of text:

yes we have no bananas | head -c 10000 > out.txt
  • 2
    OP wants to deal with bytes, not lines. – heemayl Sep 27 '16 at 9:37
  • 4
    To specify a limit in bytes, simply use head -c 10000 for 10 kB instead of head -n 10000 for 10k lines. – Byte Commander Sep 27 '16 at 10:01
  • @ByteCommander yes, but that won't prevent the output from being cut off in the middle of a line. Since the size doesn't have to be precise, I would just figure out the number of lines to get the right size, and round up :) – hobbs Sep 27 '16 at 15:36
  • 1
    I agree, but I am not sure whether that would be a problem either. The OP did not specify which method he wants, but your answer still contains both. Oh, and congrats for doubling your reputation score today :) – Byte Commander Sep 27 '16 at 15:46
  • @ByteCommander yep, fair. – hobbs Sep 27 '16 at 17:07
10

I can't recommend infinitely repeating text, but you could make a ~2GB file of repeated text with python like so...

python3 -c 'with open("bigfile", "w") as f: f.write(("hello world "*10+"\n")*2*10**7)'

That will print "hello world " 10 times and make a new line, and repeat that 20,000,000 times, writing the result to the file bigfile. If all your chars are ASCII, then each one is one byte, so calculate appropriately depending on what you want to write...

Your cpu may be owned. I run out of RAM if I try doing more than 10,000,000 lines...

I'm running a toaster though

  • OP wants to deal with bytes, not lines. – heemayl Sep 27 '16 at 9:38
  • @heemayl of course your answer is better, but I have (vaguely) explained how to calculate how many lines to use to get desired bytes so I don't think my answer is utterly useless – Zanna Sep 27 '16 at 9:40
  • 4
    @heemayl what makes you so sure the OP wants bytes? The question essentially states that the OP wants a large file. The specific size is very vague (2-4GB), so I really doubt there's a specific byte limit in mind. – terdon Sep 27 '16 at 10:23
  • 1
    @heemayl yes, but that is very, very vague. My understanding is that the OP just wants a big file and doesn't care about an exact size. Otherwise, they'd have given a size instead of such a huge range of sizes. – terdon Sep 27 '16 at 10:33
  • 1
    @cat ikr! <3python<3 – Zanna Sep 27 '16 at 12:43
9

Perl has the nifty x operator:

$ perl -e 'print "foo\n" x 5'
foo
foo
foo
foo
foo

So, as a simple solution, you could just write your line a few million times. For example, this command created a 3G file:

perl -e 'print "This is my line\n" x 200000000' > file

If you need to specify an exact size (2 GiB in this case), you can do:

perl -e 'use bytes; while(length($str)<2<<20){ $str.="This is my line\n"} print "$str\n"' > file
  • If you are patient, you can use cool Perl 6 operators except Perl 6 is much, much, much, much slower :D – cat Sep 27 '16 at 12:42
  • @cat is it really? I haven't touched 6 at all, but I had assumed it'd just have all the perly goodness plus OO extras. Any idea why it's slower? – terdon Sep 27 '16 at 12:56
  • 1
    My comment was mostly glib, but I found at the beginning of this year that Perl 6 is quite slow, compared to Python 3 which is canonically much slower than Perl 5 (which I didn't test). Work is focused on features and correctness, not performance yet, but it was listed as a goal for 2015. Also, Is Perl 6 fast enough for me?. – cat Sep 27 '16 at 13:07
  • (On the other hand, the list of features is impressive to say the least.) – cat Sep 27 '16 at 13:09
7
  • Put the set of words to be repeated in a file e.g. source.txt. Get the size of the source.txt, in bytes e.g. by:

     stat -c '%s' source.txt
    
  • Decide the size of the destination file e.g. destination.txt, 2 GB or 4 GB or whatever. Convert the size in bytes.

  • Divide the destination file size by source file size. bash can't do floating point arithmetic, but it's not needed in this case.

  • Use a for construct to repeat a cat source.txt operation the division result times. This would be closest approximate of the destination file size you can get by repetition. The output of the operation is saved in destination.txt.

For example, assuming the source.txt is of 30 bytes, and we want to create a 2 GB file, we need:

for ((i=0; i<=((16777216/30)); i++)); do cat source.txt; done >destination.txt

Here I am setting upper limit by ((16777216/30)) at initialization time; you can get the result and put it here too.

The operation would take some time; the larger the source.txt, the less time will be needed.

  • 1
    Doesn't this open and close destination.txtonce for every iteration of the loop? – ζ-- Sep 27 '16 at 15:02
  • @hexafraction Duh, fixed. – heemayl Sep 27 '16 at 15:07
6

You can also use a while-loop.

Example: Content of foo.txt (This is your source):

foo
bar
foobar

bar.txt is empty (This is your target file). You can now rn the following loop to write the content of foo.txt multiple times into bar.txt:

while [ $(stat --format "%s" bar.txt) -lt 150 ] 
do 
    cat foo.txt >> bar.txt
done

Explanation:

  • stat --format "%s" bar.txt displays the size of bar.txt in bytes.
  • while [ $(stat --format "%s" bar.txt) -lt 150 ] the following actions will be repeated until the target size (in this case 150 bytes) is reached.
  • cat foo.txt >> bar.txt append the content of foo.txt to bar.txt
4

first of fire the command :

dd if=/dev/urandom of=file.txt bs=2048 count=10

will create a file on path of size bs*count random bytes, in our case 2048*10 = 20Kb. that can be changed as per requirement .

cat - > file.txt

This commands redirects STDIN to a file, so you will need to enter two lines and then press Ctrl+D. Then you will need to run the following command:

for i in {1..n}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done

Where n is an integer. This will create a file with 2^(n+1) lines in it, by duplicating your original two lines. So to create a file with 16 lines you would do:

for i in {1..3}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done

Here are some more numbers to get you started:

n=15 will give you 65536 lines (if the original two lines were 'hello' and 'world' the file will be 384Kb)
n=20 will give you 2097152 lines (12Mb file with 'hello' and 'world' as the two starting lines)
n=25 will give you 67108864 lines (384Mb file with 'hello' and 'world' as the two starting lines)
  • 2
    OP wants to deal with bytes, not lines. – heemayl Sep 27 '16 at 9:22
  • OP is also keep coping line for filling the file . and my first command already created file as per required bytes of memory . – Avani badheka Sep 27 '16 at 9:53
  • @heemayl the newline character still occupies a byte, same as my earlier comment. It's a legitimate character. However, the OP did specify words, Avani, so I don't think your /dev/urandom technique answers their question. – Mike S Sep 28 '16 at 21:41
  • It depends on /dev/urandom , whether you are trying some random bytes . Even you can choose your own some files that contains that much bytes of data. – Avani badheka Sep 29 '16 at 5:24
4

FIFOs are probably what you're looking for. Instead of calling your program with a given file, you can tie the result of a shell command to it via process subtitution and the program will see its output as a plaintext file. The advantage here is that you are no longer limited by your disk space, so you can reach filesizes that would be impossible otherwise, so long as your program doesn't need to buffer the entire file first and can just parse it line by line. For instance, using @hobbs' reply to generate content :

wc -c <(yes we have no bananas | head -n 5000000000)

This lends me a 95 gigabytes file (according to wc) at no cost in HDD space and barely any RAM, just enough to buffer what the command returns before it gets read. This is about as close to "infinitely" as you're going to get.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.