Is there a command-line utility app which can find a specific block of lines in a text file, and replace it?

Question

UPDATE (see end of question)

The text "search and replace" utility programs I've seen, seem to only search on a line-by-line basis...

Is there a command-line tool which can locate one block of lines (in a text file), and replace it with another block of lines.?

For example: Does the test file file contain this exact group of lines:

'Twas brillig, and the slithy toves
Did gyre and gimble in the wabe:
All mimsy were the borogoves,  
And the mome raths outgrabe. 

'Beware the Jabberwock, my son!
The jaws that bite, the claws that catch!
Beware the Jubjub bird, and shun
The frumious Bandersnatch!'

I want this, so that I can replace multiple lines of text in a file and know I'm not overwriting the wrong lines.

I would never replace "The Jabberwocky" (Lewis Carroll), but it makes a novel example :)

UPDATE:
..(sub-update) My following comment about reasons when not use sed are only in the context of; don't push any tool too far beyond its design intent (I use sed quite often, and consider it to be invaluable.)

I just now found an interesting web page about sed and when not to use it.
So, because of all the sed answers, I"ll post the link.. it is part of the sed FAQ on sourceforge

Also, I'm pretty sure there is some way diff can do the job of locating the block of text (once it's located, the replacement is quite straight foward; using head and tail) ... 'diff' dumps all the necessary data, but I haven't yet worked out how to filter it , ... (I'm still working on it)

Answer 1

This simple python script should do the task:

#!/usr/bin/env python

# Syntax: multiline-replace.py input.txt search.txt replacement.txt

import sys

inp = open(sys.argv[1]).read()
needle = open(sys.argv[2]).read()
replacement = open(sys.argv[3]).read()

sys.stdout.write(inp.replace(needle,replacement))

Like most other solutions, it has the disadvantage that the whole file is slurped into memory at once. For small text files, it should work well enough, however.

Answer 2

Approach 1: temporarily change newlines into something else

The following snippet swaps newlines with pipes, performs the replacement, and swaps separators back. The utility may choke if the line it sees it extremely long. You can choose any character to swap with as long as it's not in your search string.

<old.txt tr '\n' '|' |
sed 's/\(|\|^\)'\''Twas … toves|Did … Bandersnatch!'\''|/new line 1|new line 2|/g' |
tr '|' '\n' >new.txt

Approach 2: change the utility's record separator

Awk and perl support setting two or more blank lines as the record separator. With awk, pass -vRS= (empty RS variable). With Perl, pass -000 (“paragraph mode”) or set $,="". This is not helpful here though since you have a multi-paragraph search string.

Awk and perl also support setting any string as the record separator. Set RS or $, to any string that is not in your search string.

<old.txt perl -pe '
    BEGIN {$, = "|"}
    s/^'\''Twas … toves\nDid … Bandersnatch!'\''$/new line 1\nnew line 2/mg
' >new.txt

Approach 3: work on the whole file

Some utilities easily let you read the whole file into memory and work on it.

<old.txt perl -0777 -pe '
    s/^'\''Twas … toves\nDid … Bandersnatch!'\''$/new line 1\nnew line 2/mg
' >new.txt

Approach 4: program

Read the lines one by one. Start with an empty buffer. If you see the “'Twas” line and the buffer is empty, put it in the buffer. If you see the “Did gyre” and there's one line in the buffer, append the current line to the buffer, and so on. If you've just appended the “Bandersnatch line”, output the replacement text. If the current line didn't go into the buffer, print the buffer contents, print the current line and empty the buffer.

psusi shows a sed implementation. In sed, the buffer concept is built-in; it's called the hold space. In awk or perl, you'd just use a variable (perhaps two, one for the buffer contents and one for the number of lines).

Answer 3

I was sure there had to be a way to do this with sed. After some googling I came across this:

http://austinmatzko.com/2008/04/26/sed-multi-line-search-and-replace/

Based on that I ended up writing:

sed -n '1h;1!H;${;g;s/foo\nbar/jar\nhead/g;p;}' < x

Which correctly took the contents of x:

foo bar

And spit out:

jar head

Answer 4

Even if you dislike hoary sed and perl, you might still find a liking in grey-templed awk. This answer seems to be what you're looking for. I reproduce it here. Say you have three files and want to replace needle with replacement in haystack:

awk ' BEGIN { RS="" }
      FILENAME==ARGV[1] { s=$0 }
      FILENAME==ARGV[2] { r=$0 }
      FILENAME==ARGV[3] { sub(s,r) ; print }
    ' needle replacement haystack > output

This doesn't involve regular expressions and supports newline characters. It seems to work with reasonably large files. It does involve slurping the whole file into memory, so it will not work with files of arbitrary size. If you want it more elegant, you can enclose the whole shebang in a bash script, or turn it into a awk script.

Answer 5

UPDATE: loevborg's python script is certainly the simplest and best solution (there's no doubt about that) and I'm very happy with it, but I'd like to point out that the bash script I presented (at the end of the question) is nowhere near as complicated as it looks.. I trimmed out all the debugging dross which I used to test it.. and here it is again without the overburden (for anyone visiting this page).. It's basically a sed one-liner, with pre and post hex-conversions :

F=("$haystack"  "$needle"  "$replacement")
for f in "${F[@]}" ; do cat "$f" | hexdump -v -e '1/1 "%02x"' > "$f.hex" ; done
sed -i "s/$(cat "${F[1])}.hex")/$(cat "${F[2])}.hex")/p" "${F[0])}.hex"
cat "${F[0])}.hex" | xxd -r -p > "${F[0])}"
# delete the temp *.hex files.

---

Just to throw my hat into the ring, I've come up with a 'sed' solution which won't run into any problems with special regex characters, because it uses not even one! .. instead it works on Hexdumped versions of the files...

I think it is too "top heavy", but it works, and is apparently not restricted by any size limitations.. GNU sed has an unlimited pattern buffer size, and that is where the Hexdumped block of search-lines ends up.. So it's okay in that respect...

I am still looking for a diff solution, because it will be more flexible regarding white-space (and I would expect; faster)... but until then.. It's the famous Mr Sed. :)

This script is fully running as is, and is reasonably commented...
It looks bigger that it is; I has only 7 lines of essential code.
For a semi-realistic test, it downloads the book "Alice Through the Looking Glass" from Project Gutenberg (363.1 KB) ... and replaces the original Jabberwocky poem with a line-reversed version of itself.. (Interestingly, it's not much different reading it backwards :)

PS. I just realized that a weakness in this method is if your original uses \r\n (0xODOA) as it's newline, and your "text to match" is saved with \n (0x0A).. then this matching process is dead in the water... ('diff' doesn't have such a issues) ...

---

# In a text file, replace one block of lines with another block
#
# Keeping with the 'Jabberwocky' theme, 
#  and using 'sed' with 'hexdump', so 
#  there is no possible *special* char clash.
# 
# The current setup will replace only the first instance.
#   Using sed's 'g' command, it cah change all instances. 
#

  lookinglass="$HOME/Through the Looking-Glass by Lewis Carroll"
  jabberwocky="$lookinglass (jabberwocky)"
  ykcowrebbaj="$lookinglass (ykcowrebbaj)"

  ##### This section if FOR TEST PREPARATION ONLY
        fromURL="http://www.gutenberg.org/ebooks/12.txt.utf8"
        wget $fromURL -O "$lookinglass"
        if (($?==0))
        then  echo "Download OK"
        else  exit 1
        fi
        # Make a backup of the original (while testing)
        cp "$lookinglass" "$lookinglass(fromURL)"
        #
        # Extact the poem and write it to a file. (It runs from line 322-359)
        sed -n 322,359p "$lookinglass" > "$jabberwocky"
        cat "$jabberwocky"; read -p "This is the original.. (press Enter to continue)"
        #
        # Make a file containing a replacement block of lines
        tac "$jabberwocky" > "$ykcowrebbaj"
        cat "$ykcowrebbaj"; read -p "This is the REPLACEMENT.. (press Enter to continue)"
  ##### End TEST PREPARATION

# The main process
#
# Make 'hexdump' versions of the 3 files... source, expected, replacement 
  cat "$lookinglass" | hexdump -v -e '1/1 "%02x"' > "$lookinglass.xdig"
  cat "$jabberwocky" | hexdump -v -e '1/1 "%02x"' > "$jabberwocky.xdig"
  cat "$ykcowrebbaj" | hexdump -v -e '1/1 "%02x"' > "$ykcowrebbaj.xdig"
# Now use 'sed' in a safe (no special chrs) way.
# Note, all files are now each, a single line  ('\n' is now '0A')
  sed -i "s/$(cat "$jabberwocky.xdig")/$(cat "$ykcowrebbaj.xdig")/p" "$lookinglass.xdig"

  ##### This section if FOR CHECKING THE RESULTS ONLY
        # Check result 1
        read -p "About to test for the presence of  'jabberwocky.xdig'  within itself (Enter) "
        sed -n "/$(cat "$jabberwocky.xdig")/p"     "$jabberwocky.xdig"
        echo -e "\n\nA dump above this line, means: 'jabberwocky' is as expected\n" 
        # Check result 2
        read -p "About to test for the presence of  'ykcowrebbaj.xdig'  within itself (Enter) "
        sed -n "/$(cat "$ykcowrebbaj.xdig")/p"     "$ykcowrebbaj.xdig"
        echo -e "\n\nA dump above this line, means: 'ykcowrebbaj' is as expected\n" 
        # Check result 3
        read -p "About to test for the presence of  'lookinglass.xdig'  within itself (Enter) "
        sed -n "/$(cat "$ykcowrebbaj.xdig")/p"     "$lookinglass.xdig"
        echo -e "\n\nA dump above this line, means: 'lookinglass' is as expected\n" 
        # Check result 4
        read -p "About to test for the presence of  'lookinglass.xdig'  within itself (Enter) "
        sed -n "/$(cat "$jabberwocky.xdig")/p"     "$lookinglass.xdig"
        echo -e "\n\nNo dump above this line means: 'lookinglass' is as expected\n"
  ##### End of CHECKING THE RESULTS

# Now convert the hexdump to binary, and overwrite the original
  cat "$lookinglass.xdig" | xxd -r -p > "$lookinglass"
# Echo the "modified" poem to the screen
  sed -n 322,359p "$lookinglass"
  echo -e "\n\nYou are now looking at the REPLACEMENT text (dumped directly from the source 'book'"

Answer 6

The rpl utility, available in Ubuntu, allows doing this. rpl replaces text in files, and you can use the --files option to specify that the text to be replaced, and its replacement, come from files. You can also use -F to specify that they should be treated as literal text rather than regexs.

Putting that all together:

$ rpl --files -F from.txt to.txt file-to-modify1.txt file-to-modify2.txt…

(Disclaimer: I'm the maintainer of rpl, and I had forgotten it could do this.)