How to create a file from terminal repeating a set of words infinitely? I need it to create a huge file for parsing purposes like 2-4GB in size. Currently I am manually copying pasting lines into the same file to increase the size.
7 Answers
There's an easy way to repeat a line lots of times using the yes
command:
yes we have no bananas | head -n 10000 > out.txt
will result in out.txt containing 10,000 lines all saying "we have no bananas".
To limit the output to an exact number of bytes, use head
's -c
option instead of -n
. For example, this generates exactly 10 kB of text:
yes we have no bananas | head -c 10000 > out.txt
-
2
-
4To specify a limit in bytes, simply use
head -c 10000
for 10 kB instead ofhead -n 10000
for 10k lines.– Byte Commander ♦Sep 27, 2016 at 10:01 -
@ByteCommander yes, but that won't prevent the output from being cut off in the middle of a line. Since the size doesn't have to be precise, I would just figure out the number of lines to get the right size, and round up :)– hobbsSep 27, 2016 at 15:36
-
1I agree, but I am not sure whether that would be a problem either. The OP did not specify which method he wants, but your answer still contains both. Oh, and congrats for doubling your reputation score today :)– Byte Commander ♦Sep 27, 2016 at 15:46
-
Perl has the nifty x
operator:
$ perl -e 'print "foo\n" x 5'
foo
foo
foo
foo
foo
So, as a simple solution, you could just write your line a few million times. For example, this command created a 3G file:
perl -e 'print "This is my line\n" x 200000000' > file
If you need to specify an exact size (2 GiB in this case), you can do:
perl -e 'use bytes; while(length($str)<2<<20){ $str.="This is my line\n"} print "$str\n"' > file
-
If you are patient, you can use cool Perl 6 operators except Perl 6 is much, much, much, much slower :D– catSep 27, 2016 at 12:42
-
@cat is it really? I haven't touched 6 at all, but I had assumed it'd just have all the perly goodness plus OO extras. Any idea why it's slower?– terdonSep 27, 2016 at 12:56
-
1My comment was mostly glib, but I found at the beginning of this year that Perl 6 is quite slow, compared to Python 3 which is canonically much slower than Perl 5 (which I didn't test). Work is focused on features and correctness, not performance yet, but it was listed as a goal for 2015. Also, Is Perl 6 fast enough for me?.– catSep 27, 2016 at 13:07
-
(On the other hand, the list of features is impressive to say the least.)– catSep 27, 2016 at 13:09
I can't recommend infinitely repeating text, but you could make a ~2GB file of repeated text with python like so...
python3 -c 'with open("bigfile", "w") as f: f.write(("hello world "*10+"\n")*2*10**7)'
That will print "hello world " 10 times and make a new line, and repeat that 20,000,000 times, writing the result to the file bigfile
. If all your chars are ASCII, then each one is one byte, so calculate appropriately depending on what you want to write...
Your cpu may be owned. I run out of RAM if I try doing more than 10,000,000 lines...
I'm running a toaster though
-
-
@heemayl of course your answer is better, but I have (vaguely) explained how to calculate how many lines to use to get desired bytes so I don't think my answer is utterly useless– Zanna ♦Sep 27, 2016 at 9:40
-
4@heemayl what makes you so sure the OP wants bytes? The question essentially states that the OP wants a large file. The specific size is very vague (2-4GB), so I really doubt there's a specific byte limit in mind.– terdonSep 27, 2016 at 10:23
-
1@heemayl yes, but that is very, very vague. My understanding is that the OP just wants a big file and doesn't care about an exact size. Otherwise, they'd have given a size instead of such a huge range of sizes.– terdonSep 27, 2016 at 10:33
-
1
Put the set of words to be repeated in a file e.g.
source.txt
. Get the size of thesource.txt
, in bytes e.g. by:stat -c '%s' source.txt
Decide the size of the destination file e.g.
destination.txt
, 2 GB or 4 GB or whatever. Convert the size in bytes.Divide the destination file size by source file size.
bash
can't do floating point arithmetic, but it's not needed in this case.Use a
for
construct to repeat acat source.txt
operation the division result times. This would be closest approximate of the destination file size you can get by repetition. The output of the operation is saved indestination.txt
.
For example, assuming the source.txt
is of 30 bytes, and we want to create a 2 GB file, we need:
for ((i=0; i<=((16777216/30)); i++)); do cat source.txt; done >destination.txt
Here I am setting upper limit by ((16777216/30))
at initialization time; you can get the result and put it here too.
The operation would take some time; the larger the source.txt
, the less time will be needed.
-
1Doesn't this open and close
destination.txt
once for every iteration of the loop? Sep 27, 2016 at 15:02 -
You can also use a while
-loop.
Example:
Content of foo.txt
(This is your source):
foo
bar
foobar
bar.txt
is empty (This is your target file). You can now rn the following loop to write the content of foo.txt
multiple times into bar.txt
:
while [ $(stat --format "%s" bar.txt) -lt 150 ]
do
cat foo.txt >> bar.txt
done
Explanation:
stat --format "%s" bar.txt
displays the size ofbar.txt
in bytes.while [ $(stat --format "%s" bar.txt) -lt 150 ]
the following actions will be repeated until the target size (in this case 150 bytes) is reached.cat foo.txt >> bar.txt
append the content offoo.txt
tobar.txt
first of fire the command :
dd if=/dev/urandom of=file.txt bs=2048 count=10
will create a file on path of size bs*count random bytes, in our case 2048*10 = 20Kb. that can be changed as per requirement .
cat - > file.txt
This commands redirects STDIN to a file, so you will need to enter two lines and then press Ctrl+D. Then you will need to run the following command:
for i in {1..n}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done
Where n is an integer. This will create a file with 2^(n+1) lines in it, by duplicating your original two lines. So to create a file with 16 lines you would do:
for i in {1..3}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done
Here are some more numbers to get you started:
n=15 will give you 65536 lines (if the original two lines were 'hello' and 'world' the file will be 384Kb)
n=20 will give you 2097152 lines (12Mb file with 'hello' and 'world' as the two starting lines)
n=25 will give you 67108864 lines (384Mb file with 'hello' and 'world' as the two starting lines)
-
2
-
OP is also keep coping line for filling the file . and my first command already created file as per required bytes of memory . Sep 27, 2016 at 9:53
-
@heemayl the newline character still occupies a byte, same as my earlier comment. It's a legitimate character. However, the OP did specify words, Avani, so I don't think your /dev/urandom technique answers their question.– Mike SSep 28, 2016 at 21:41
-
It depends on /dev/urandom , whether you are trying some random bytes . Even you can choose your own some files that contains that much bytes of data. Sep 29, 2016 at 5:24
FIFOs are probably what you're looking for. Instead of calling your program with a given file, you can tie the result of a shell command to it via process subtitution and the program will see its output as a plaintext file. The advantage here is that you are no longer limited by your disk space, so you can reach filesizes that would be impossible otherwise, so long as your program doesn't need to buffer the entire file first and can just parse it line by line. For instance, using @hobbs' reply to generate content :
wc -c <(yes we have no bananas | head -n 5000000000)
This lends me a 95 gigabytes file (according to wc) at no cost in HDD space and barely any RAM, just enough to buffer what the command returns before it gets read. This is about as close to "infinitely" as you're going to get.
mkfifo huge.tmp; while true; do yes "a dummy line" > huge.tmp; done
?