7

I want to make a file that is very large in size for testing purposes that should contains unique words (strictly not repeated). It can in GB, TB, etc. May I do this with the terminal?

I am using Ubuntu 16.04 (Xenial Xerus).

  • Comments are not for extended discussion; the conversation that was taking place here has been moved to chat. – Thomas Ward Nov 5 '16 at 14:34
  • Define what you mean by word? An English word? Or is 'k345j#_' a word for you as well? – Grzegorz Oledzki Nov 5 '16 at 17:43
12

Creating an infinite number of words, guaranteed unique

The script below will generate guaranteed unique words from characters from the alphabet. The issue with any fixed length of characters is that it will produce a limited set of possibilities, limiting the size of your file.

I therefore used python's permutations, which produces a (finite) number of unique words. However After using all the combinations, we simply start over, printing the words 2, then 3, four, n times etc., every value for n will create a new unique word. Thus we have a generator to produce 100% certainly unique words.

The script:

import itertools
import string

ab = [c for c in string.ascii_lowercase]

t = 1
while True:   
    for n in range(1, len(ab)+1): 
        words = itertools.permutations(ab, n)
        for word in words:
            print(t*("".join(word)))
    t += 1

How to use

  • Simply copy the script into an empty file, save it as unique_generator.py
  • Run it by the command:

    python3 /path/to/unique_generator.py > /path/to/bigfile.txt
    

Note

The script produces unique words of various lenght. If you want, start- or max length can be set, by changing the lines:

for n in range(1, len(ab)+1)

(replace the start of the range), and changing:

while True:  

into (for example):

while t < 10:   

In the last case, the length of the words is max 10 times the alphabet.

Ending the process

  • When running it from terminal, simply press Ctrl+C
  • Otherwise:

    kill $(pgrep -f /path/to/unique_generator.py)
    

    should do the job.

  • Very nice ! I've used permutations once. Quite a useful tool – Sergiy Kolodyazhnyy Nov 5 '16 at 10:58
  • @Serg I love itertools :) Th loop thing is stuborn btw, still trying... permissions are fine. – Jacob Vlijm Nov 5 '16 at 11:00
  • while running "python3 /path/to/unique_generator.py > /path/to/bigfile.txt " command how to stop it when we don't want further . – Avani badheka Nov 5 '16 at 11:40
  • @JacobVlijm : thank you for your kind effort . how to stop command "python3 /path/to/unique_generator.py > /path/to/bigfile.txt " when we reached at desired size ? – Avani badheka Nov 5 '16 at 11:48
  • Hi, @Avanibadheka edited my answer to include stopping the process. – Jacob Vlijm Nov 5 '16 at 12:22
6

To get a large file full on random words use the following command:

cat /dev/urandom | head -c 1000000 | tr -dc "A-Za-z0-9\n" | sort | uniq

This will create a file with a unqiue word on each line and strings of random text. You can increase the size of the file by make 1000 larger or smaller. Each count is equal to roughly one byte.

To make the words space seperated, simply pass them back through tr "\n" " ".

cat /dev/urandom | head -c 1000000 | tr -dc "A-Za-z0-9\n" | sort | uniq | tr "\n" " "

This also avoids the performance problems associated with loops on the shell.

  • what are the performance problems with loops on the shell? I hope this is not off-topic – Ciprian Tomoiagă Nov 5 '16 at 22:13
  • 2
    @CiprianTomoiaga: short answer: spawning new processes is expensive. Loops like in Serg's answer do so in each iteration, potentially mutliple times. This answer reuses the same processes to do all work. For longer answer, please submit a site question ;-) – liori Nov 6 '16 at 0:08
6

Simplest one-liner to make random lines of characters:

while true; do echo $RANDOM | base64 >> BIGFILE.txt  ; done

or alternatively:

while true; do echo $RANDOM | sha512sum  >> BIGFILE.txt ; done   

For better uniqueness , you can use /dev/urandom:

cat /dev/urandom | base64 

Terminate the command with Ctrl+C, when your file reaches desired size

Consider also random string/password generators :

Thus,one of the answers there, can be adapted to have this:

while true; do openssl rand -base64 20 ; done

In case you don't want to use punctuation characters and numbers, but use only letters, then we can use tr to correct that:

while true; do openssl rand -base64 20 | tr -d '[[:digit:]][[:punct:]]'  ; done 

For extra randomness, you can shuffle the characters of generated strings using shuf.

while true; do openssl rand -base64 20 | fold -w1 | shuf | tr -d '\n' ; done

And so on and so forth. You could even pass the output to another set of base64 or sha256sum commands to make them even more random.

For those who like alternative languages other than shell, here's a python one-liner:

python -c $'import string,random;i = [i for i in string.uppercase + string.lowercase + string.digits];\nwhile not random.shuffle(i): print "".join(i)'
  • yes , it can generate problem . It can be random so no guarantee for unique words. – Avani badheka Nov 5 '16 at 9:05
  • I've addressed the uniqueness problems from several angles already, guys, so consider removing the comments – Sergiy Kolodyazhnyy Nov 5 '16 at 9:20
  • 1
    Technically it's still possible for something to be not unique, but it's pretty unlikely – DonyorM Nov 5 '16 at 9:24
  • 1
    @DonyorM if you're pointing at the example with $RANDOM variable, then yes, but /dev/urandom is quite low in random probability, especially in combination with shasum. – Sergiy Kolodyazhnyy Nov 5 '16 at 9:26
  • "quite low in random probability" . . . em,my sleep deprived brain means that there's low probability to have non-unique data generated. But you get the idea – Sergiy Kolodyazhnyy Nov 5 '16 at 9:41
3

If a limited alphabet and very predictable output are fine, the following is probably the fastest (and most golfed :P) answer:

seq 100

(if words may consist of just numbers), or:

seq 100 | tr 0-9 A-I

(if there is a desire to use actual letters).

3

You can also use /proc/sys/kernel/random/uuid in Linux to generate UUIDs, which should be guaranteed to be unique:

[~]$ for i in {1..10}; do cat /proc/sys/kernel/random/uuid; done
c8072c40-32f5-4f14-8794-c3ab68e1a0f5
2f2630d8-0e17-4cba-8e62-586ee23f0ebb
97606886-f227-46f6-827a-141b0db57c59
5ffea57c-c3bf-4ba6-8c08-8a1b29ee8f6c
2b90f797-2def-4433-ae71-6f404db944fc
fcb793e9-6102-472d-a7a0-7bf5204dbee5
d84e2877-6804-4bed-85f0-0a551234425a
3d9445ca-335c-4960-83d5-6cb1bef8b9eb
913bce71-5c20-47f7-a22e-277be6856a57
8f232541-f8c1-46ba-b57a-0d11314c3483

You can remove - character with tr -d and then redirect this to a file:

[~]$ for i in {1..10000}; do cat /proc/sys/kernel/random/uuid | tr -d '-' ; done > /tmp/words
[~]$ sort /tmp/words | uniq | wc -l
10000

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.