How can I randomly replace specific strings in one text file with strings from another file? For example:

file1.txt(file has more than 200 lines):
moonwalker@address.com
hansolo@address.com
anakinskywalker@address.com
obiwankenobi@address.com
darthvader@address.com

file2.txt(file has 10-20 lines):
@adress1.com
@adress2.com
@adress3.com
@adress4.com
@adress5.com

output.txt:
moonwalker@address4.com
hansolo@address1.com
anakinskywalker@address5.com
obiwankenobi@address2.com
darthvader@address3.com
  • 4
    That isn't random, it looks like you don't want anything repeated. Do you want it to be actually random, or should each line of the second text file only be used once? Also, does it need to be bash, or are you open to other tools? – terdon Dec 17 '17 at 15:58
  • 1
    @terdon It looks like he wants a random permutation (all 5 elements but in a random order). A random permutation is actually random, you just need to eliminate the already-chosen elements when randomly selecting the next element. Sometimes called a "random sort" – thomasrutter Dec 17 '17 at 22:39
  • 1
    @thomasrutter yes, I know that and that's what my answer does. But that's why I was asking the OP to clarify since both a random permutation and a random pick would be reasonable depending on what they need. – terdon Dec 17 '17 at 23:17
up vote 9 down vote accepted

If you really want a random selection, then here's one way using awk:

awk '
  BEGIN{FS="@"; OFS=""} 
  NR==FNR{a[NR]=$0; n++; next} 
  {$2=a[int(1 + n * rand())]; print}
' file2.txt file1.txt
moonwalker@adress2.com
hansolo@adress2.com
anakinskywalker@adress5.com
obiwankenobi@adress1.com
darthvader@adress3.com

OTOH if you want a random permutation of the addresses, I'd suggest something like

paste -d '' <(cut -d'@' -f1 file1.txt) <(sort -R file2.txt)
moonwalker@adress2.com
hansolo@adress1.com
anakinskywalker@adress5.com
obiwankenobi@adress4.com
darthvader@adress3.com
  • 1
    Nice! I was looking into doing it with paste but it didn't occur to me to use cut to remove the non-matching field. – terdon Dec 17 '17 at 16:09
  • 2
    One downside to the paste solution is when file1 has more lines than file2. Instead of <(sort -R file2.txt) we can use something like <(yes "$(<file2.txt)" | head -n $(wc -l < file1.txt) | sort -R) -- that may skew the randomness in favour of lines closer to the top of file2. – glenn jackman Dec 17 '17 at 16:27

You could implement this algorithm:

  • Load the content of file2.txt to an array
  • For each line in file1.txt:
    • Extract the name part
    • Get a random address
    • Print the output correctly formatted

Like this:

mapfile -t addresses < file2.txt
while IFS='' read -r orig || [[ -n "$orig" ]]; do
    ((index = RANDOM % ${#addresses[@]}))
    name=${orig%%@*}
    echo "$name${addresses[index]}"
done < file1.txt

(Special thanks to @GlennJackman and @dessert for the improvements.)

You could use shuf (you might need to sudo apt install shuf) to shuffle the lines of the second file and then use them to replace:

$ awk -F'@' 'NR==FNR{a[NR]=$1;next}{print a[FNR]"@"$2} ' file1 <(shuf file2)
moonwalker@adress3.com
hansolo@adress1.com
anakinskywalker@adress5.com
obiwankenobi@adress4.com
darthvader@adress2.com

shuf simply randomizes the order of its input lines. The awk command there will first read all of file1 (NR==FNR will only be true while the first file is being read), and saves the second field (fields are defined by @, so this is the domain) in the associative array a whose values are the domains and whose keys are the line numbers. Then, when we get to the next file, it will simply print whatever was stored in a for this line number, along with what's in file 2 for the same line number.

Note that this assumes both files have exactly the same number of lines and isn't actually being "random", since it will not allow anything to be repeated. But that looks like what you wanted to ask for.

Python 2.7 and 3 solution

This solution replaces the first occurrence of a single arbitrary given string (the “needle”) in every line of the input file with a string each time chosen randomly from the set of lines of the replacements string list.

#!/usr/bin/python
from __future__ import print_function
import sys, random

needle = sys.argv[1]

if sys.argv[2] == '-':
    f_replacements = sys.stdin
else:
    f_replacements = open(sys.argv[2])
with f_replacements:
    replacements = [l.rstrip('\n') for l in f_replacements]
if not replacements:
    raise ValueError('No replacement strings given')

if len(sys.argv) <= 3 or sys.argv[3] == '-':
    f_in = sys.stdin
else:
    f_in = open(sys.argv[3])
with f_in:
    for s in f_in:
        rep = replacements[random.randrange(len(replacements))]
        print(s.rstrip('\n').replace(needle, rep, 1))

It should be almost trivial to anchor the needle to the beginning or end of the string or to use regular expressions altogether.

Usage

python replace-random.py NEEDLE REPLACEMENTS-FILE [INPUT-FILE]

Example:

python replace-random.py '@address.com' file2.txt file1.txt

or

python replace-random.py '@address.com' file2.txt < file1.txt

Here's a perl way:

#!/usr/bin/perl
use warnings;
use strict;
use Tie::File;

tie my @file1,'Tie::File','file1.txt' or die "Can't open file1.txt\n";
tie my @file2,'Tie::File','file2.txt' or die "Can't open file2.txt\n";

for my $file_index (0..$#file1) {
   my $suffix = $file2[int(rand($#file2+1))];
   $file1[$file_index] =~ s/@.*$/$suffix/;
}

untie @file1;
untie @file2;

Another bash solution. It uses bash built-in string replacement feature. It also assumes file2.txt contains the replacement strings only. If not they can be first filtered using grep -o <replace> file2.txt

With shuf

#search string
Search="@address.com"
for lines in $(grep $Search file1.txt)
do 
    echo ${lines/$Search/$(shuf file2.txt -n 1)} 
done

Without shuf (almost pure bash)

Here we have to create a function first that mimics shuf like so

bshuf () 
{ 
    nlines=$(( $(wc -l < $1) + 1))
    rand=0
    while [ "$rand" -eq 0 ]; do
        rand=$(( $RANDOM % nlines ))
    done
    echo $(head -n $rand $1 | tail -1)
}

Then it is similar

for lines in $(grep $Search file1.txt) 
do 
    echo ${lines/$Search/$(bshuf file2.txt)}
done

Test:

$ for lines in $(grep $Search file1.txt); do echo ${lines/$Search/$(bshuf file2.txt)} ; done
moonwalker@adress4.com
hansolo@adress2.com
anakinskywalker@adress2.com
obiwankenobi@adress3.com
darthvader@adress5.com
$ 

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.