2

Let's say I have the following files:

/etc/dir1/file.txt
/etc/dir2/file.txt
/etc/dir3/file.txt

... all the way up to dir100 (100 directories), every directory has file.txt.

And I have the following text file in /root/list.txt. In list.txt, I have 100 lines, each line with a different string of text.

In each file.txt, there is the string of text, word1.

How would I use sed (or something similar) to replace the word word1 in every file.txt, with one line from list.txt? Each line in list.txt, is only to be used once.

So for example, replace word1 in /etc/dir1/file.txt with the first line in /root/list.txt, and replace word1 /etc/dir2/file.txt with the second line in /root/list.txt and so on, all the way up to 100.

I greatly appreciate any help and assistance here as sed is not my strong point.

3

You can do this with sed in a loop if the lines of list.txt are well-behaved.

How would I use sed (or something similar) to replace the word word1 in every file.txt, with one line from list.txt? Each line in list.txt, is only to be used once.

Ubuntu has GNU sed, which makes it easy to replace only the first occurrence of a pattern in a file, then stop. To use separate replacement strings for each input file, you can use a loop. The code below is just complex enough that I suggest making it a script and running the script. There are three major caveats:

  1. The pattern word1--which I presume you may change to something else--must not contain /, unless you use a different delimiter in the sed command. Nor may it contain characters sed treats specially, such as regular expression metacharacters (\, *, ., and so forth), unless that's what you intend.
  2. The lines in list.txt must not contain / or most special characters either.
  3. Your dir1, dir2, ..., are in /etc and your list.txt is in /root, but I've written my script to assume those directories--and that file--are in the current directory instead. I did this because files stored in those locations are often important, and I assume you will want to test this script--and perhaps make your own modifications--before using it for real. You can change the script to use the locations you gave, or any other locations you need.

I've bolded #2 becuase it's the one I expect might cause you trouble, depending on what list.txt might contain. Now that you've been warned, here's the script:

#!/bin/bash

mapfile -t <list.txt

for ((i=1; i<=${#MAPFILE[@]}; ++i))
do sed -i.bak "0,/word1/ s//${MAPFILE[i-1]}/" "dir$i/file.txt"
done

That's all it takes. In case you're interested, here's how it works:

  • mapfile is a Bash builtin that reads lines into an array. I use it to read from list.txt. I didn't specify the array's name so the default name of MAPFILE is used.
  • Bash offers an alternate (C-style) for loop, which is useful when one wants to loop from or to a value obtained by parameter expansion, since brace expansion in Bash won't expand things like {1..$var}. I use it to loop from 1 to the length of the MAPFILE array.
  • sed -i replaces the original file. You lose the old version unless you provide a backup suffix. You can remove .bak from the command if you don't want to keep the old version but I recommend you consider keeping it. Please at least test with it before removing it.
  • With GNU sed, 0,/word1/ s//REPLACEMENT/ operates only in the range from the beginning of the file to the first match with word1 (0,/word1/), replacing text matching that same pattern with REPLACEMENT (s//*REPLACEMENT*/). This is to say that it replaces only the first match in the input. The rest of the input is unchanged regardless of whether it would match the pattern.
  • Bash arrays are indexed starting with 0, but your files are named starting from 1, which is where I decided the loop variable $i should start. Fortunately this is easy to deal with because Bash arrays accept arithmetic expressions as indices. ${MAPFILE[i-1]} expands to element i - 1 (that is, the ith element) of the array of lines from list.txt.

To replace using arbitrary text read from a file, consider alternatives to sed.

If you can't abide caveat #2--that is, the lines in list.txt could be just about anything--then I don't know of a good way to do this with sed. But there are a number of alternatives. Bash can actually do this itself, without any external commands, and I'll show an almost pure-Bash solution. (Perhaps more answers will be posted to show methods using awk or other utilities.)

In sed, a pattern is a regular expressio. But this method treats the pattern as a glob. See also this FOLDOC entry and man 7 glob. Thus special characters including *, ?, [, and ]--and some others like \--have special meanings in word1 (just not usually the same meanings as in sed). If word1 is literally word1 or any other text with no globbing characters, no problem. Otherwise you must modify it accordingly. This method removes caveat #2 but not caveat #1--nor #3, but you can deal with that yourself easily.

#!/bin/bash

pattern='word1' # text to search for
suffix='.bak'   # suffix to append for backup files

mapfile -t <list.txt

for ((i=1; i<=${#MAPFILE[@]}; ++i)); do
    name="dir$i/file.txt"
    mv "$name" "$name$suffix" || exit  # quit with an error if we can't rename

    {
        while read -r; do  # output up to and including the replacement
            case "$REPLY" in
            *"$pattern"*)
                printf '%s\n' "${REPLY/$pattern/${MAPFILE[i-1]}}"
                break ;;
            *)
                printf '%s\n' "$REPLY" ;;
            esac
        done

        while read -r; do  # output the rest
            printf '%s\n' "$REPLY"
        done

    } <"$name$suffix" >"$name"
done

In case you're (still) interested, here's how that works:

  • As before, I read list.txt into an array. I could also read each file.txt into an array, but for all I know these files could be enormous, so I'm reading them one line at a time with read -r instead.
  • I move each file aside by renaming it with a .bak suffix with mv. mv is the only external utility this script uses. I haven't bothered to pass -- before the path because, in this case, there's no way the path can start with -. If the move operation fails, mv will output an error and || exit terminates the script, preventing accidental data loss. The only data that should be possible to lose by running this script is data in pre-existing .bak files.
  • The two loops that read input and write output are grouped together with { }. The whole group has both its input and output redirected to use the .bak file as input and a file named the same as the original as output (<"$name$suffix" >"$name").
  • Each loop may read many lines, and the only way read -r itself modifies them is to remove the newline characters at the ends (which printf will put back with \n later). In Bash, read -r with no variable name reads a line to $REPLY and doesn't strip leading and trailing whitespace; it is equivalent to IFS= read -r REPLY.
  • The first loop reads until a line appears consisting of any or no characters (*), followed by word1 ("$pattern"), followed again by any or no characters (*). When it finds such a line it prints it but replaces the part that matches "$pattern" with the ith line from list.txt (${MAPFILE[i-1]}) and then breaks the loop. All the lines before that one are just printed verbatim.
  • The second loop prints all the remaining lines verbatim.

By using two loops grouped together, I've achieved the same basic logic as the sed way detailed above--text up to, including, but not beyond the first match is processed first so the match is substituted, then subsequent text is not searched at all, just copied. However, unlike in that sed method, special characters in what ${MAPFILE[i-1]} expands to are not treated as part of the command.

For example, observe that a replacement string that tried to cause trouble by closing the inner parameter expansion and injecting additional substitutions would not succeed:

$ s=foobarbaz t=bar u='}$s$s$s'; echo "${s/$t/${u}}"
foo}$s$s$sbaz
1

Let's create test environment placed in the user's $HOME directory.

  • First execute next line as single command:

    path="${HOME}/etc/dir"; for i in {1..100}; do mkdir -p "$path$i" ; echo -e "$path$i/file.txt:\nline1 some text here\nline2 word1 some text here word1\nline3 word1 some text here" > "$path$i/file.txt"; done
    

    This will create hundred directories - ~/etc/dir{1..100}. In each directory will be created also a file, called file.txt, that contains the string word1 few times:

    $ cat ~/etc/dir{1..100}/file.txt
    /home/<user>/etc/dir1/file.txt:
    line1 some text here
    line2 word1 some text here word1
    line3 word1 some text here
    /home/<user>/etc/dir2 file.txt: 
    ...
    
  • Then execute this line:

    path="${HOME}/root" && mkdir "$path"; for i in {1..100}; do echo '*{string line ['"$i"']}*' >> "$path/list.txt"; done
    

    This will create a directory, called ~/root. In the directory will be created also a file, called list.txt, that contains hundred lines:

    $ cat ~/root/list.txt
    *{string line [1]}*
    *{string line [2]}*
    ...
    

Let's solve the task. According to the circumstances, create into the above step, because the string word1 occurs several times, we have several cases. Example solutions:

  • To replace only the first occurrence of word1 in each file.txt, execute this line:

    i=""; while read line; do i=$((i+1)); sed "0,/word1/ s|\word1|${line}|1" "$HOME/etc/dir$i/file.txt"; done < "$HOME/root/list.txt"
    

    The output should be:

    /home/<user>/etc/dir1/file.txt:
    line1 some text here
    line2 *{string line [1]}* some text here word1
    line3 word1 some text here
    /home/<user>/etc/dir2 file.txt:
    ...
    
  • To replace only the first occurrence of word1 on each line in each file.txt, execute this line:

    i=""; while read line; do i=$((i+1)); sed "s|\word1|${line}|1" "$HOME/etc/dir$i/file.txt"; done < "$HOME/root/list.txt"
    

    The output should be:

    /home/<user>/etc/dir1/file.txt:
    line1 some text here
    line2 *{string line [1]}* some text here word1
    line3 *{string line [1]}* some text here
    /home/<user>/etc/dir2 file.txt:
    ...
    
  • To replace all occurrences of word1 in each file.txt, execute this line:

    i=""; while read line; do i=$((i+1)); sed "s|\word1|${line}|g" "$HOME/etc/dir$i/file.txt"; done < "$HOME/root/list.txt"
    

    The output should be:

    /home/<user>/etc/dir1/file.txt:
    line1 some text here
    line2 *{string line [1]}* some text here *{string line [1]}*
    line3 *{string line [1]}* some text here
    /home/<user>/etc/dir2 file.txt:
    ...
    

Notes. Into the above examples:

  • Change sed with sed -i for actual replacement of the strings or use sed -i.bak to do the replacements and leave also a backup file.

  • Remove $HOME according to the circumstances, described into the question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.