Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

UPDATE (see end of question)

The text "search and replace" utility programs I've seen, seem to only search on a line-by-line basis...

Is there a command-line tool which can locate one block of lines (in a text file), and replace it with another block of lines.?

For example: Does the test file file contain this exact group of lines:

'Twas brillig, and the slithy toves
Did gyre and gimble in the wabe:
All mimsy were the borogoves,  
And the mome raths outgrabe. 

'Beware the Jabberwock, my son!
The jaws that bite, the claws that catch!
Beware the Jubjub bird, and shun
The frumious Bandersnatch!'

I want this, so that I can replace multiple lines of text in a file and know I'm not overwriting the wrong lines.

I would never replace "The Jabberwocky" (Lewis Carroll), but it makes a novel example :)

UPDATE:
..(sub-update) My following comment about reasons when not use sed are only in the context of; don't push any tool too far beyond its design intent (I use sed quite often, and consider it to be invaluable.)

I just now found an interesting web page about sed and when not to use it.
So, because of all the sed answers, I"ll post the link.. it is part of the sed FAQ on sourceforge

Also, I'm pretty sure there is some way diff can do the job of locating the block of text (once it's located, the replacement is quite straight foward; using head and tail) ... 'diff' dumps all the necessary data, but I haven't yet worked out how to filter it , ... (I'm still working on it)

share|improve this question
add comment

5 Answers

up vote 4 down vote accepted

This simple python script should do the task:


#!/usr/bin/env python

# Syntax: multiline-replace.py input.txt search.txt replacement.txt

import sys

inp = open(sys.argv[1]).read()
needle = open(sys.argv[2]).read()
replacement = open(sys.argv[3]).read()

sys.stdout.write(inp.replace(needle,replacement))

Like most other solutions, it has the disadvantage that the whole file is slurped into memory at once. For small text files, it should work well enough, however.

share|improve this answer
1  
Well now! That's the way to go! ... I've thrown everything at it, and it stood the test.. (and why shouldn't it.. Python has a "rep" for good reason.;).... I'll need to look further into the "replace" method (or whatever Python calles them).. but.. she's a goer!... Thanks.loevborg.. –  Peter.O Jan 9 '11 at 20:04
add comment

I was sure there had to be a way to do this with sed. After some googling I came across this:

http://austinmatzko.com/2008/04/26/sed-multi-line-search-and-replace/

Based on that I ended up writing:

sed -n '1h;1!H;${;g;s/foo\nbar/jar\nhead/g;p;}' < x

Which correctly took the contents of x:

foo bar

And spit out:

jar head

share|improve this answer
    
I wouldn't recommend sed here. While it can do it, using the hold space gets complicated quickly (here, with three lines it's a already a mess). –  Gilles Jan 8 '11 at 20:05
    
This has certainly got me thingking. I use sed (and like it), but as Gilles mentioned, working with the hold space can quickly become a pain (so I try to keep away from it).. However the biggest problem with any regex search is when your regex pattern is "unknown"; the likelyhood of a clash is high... So I have rule out anything which uses regex. The main point of this, is to avoid manual tweaking... Quite frankly, if sed has some way of turning off regex, then its address range feature could be very useful, otherwise it's not the tool for the job.. thanks though; sed talk is good talk –  Peter.O Jan 9 '11 at 2:26
add comment

Approach 1: temporarily change newlines into something else

The following snippet swaps newlines with pipes, performs the replacement, and swaps separators back. The utility may choke if the line it sees it extremely long. You can choose any character to swap with as long as it's not in your search string.

<old.txt tr '\n' '|' |
sed 's/\(|\|^\)'\''Twas … toves|Did … Bandersnatch!'\''|/new line 1|new line 2|/g' |
tr '|' '\n' >new.txt

Approach 2: change the utility's record separator

Awk and perl support setting two or more blank lines as the record separator. With awk, pass -vRS= (empty RS variable). With Perl, pass -000 (“paragraph mode”) or set $,="". This is not helpful here though since you have a multi-paragraph search string.

Awk and perl also support setting any string as the record separator. Set RS or $, to any string that is not in your search string.

<old.txt perl -pe '
    BEGIN {$, = "|"}
    s/^'\''Twas … toves\nDid … Bandersnatch!'\''$/new line 1\nnew line 2/mg
' >new.txt

Approach 3: work on the whole file

Some utilities easily let you read the whole file into memory and work on it.

<old.txt perl -0777 -pe '
    s/^'\''Twas … toves\nDid … Bandersnatch!'\''$/new line 1\nnew line 2/mg
' >new.txt

Approach 4: program

Read the lines one by one. Start with an empty buffer. If you see the “'Twas” line and the buffer is empty, put it in the buffer. If you see the “Did gyre” and there's one line in the buffer, append the current line to the buffer, and so on. If you've just appended the “Bandersnatch line”, output the replacement text. If the current line didn't go into the buffer, print the buffer contents, print the current line and empty the buffer.

psusi shows a sed implementation. In sed, the buffer concept is built-in; it's called the hold space. In awk or perl, you'd just use a variable (perhaps two, one for the buffer contents and one for the number of lines).

share|improve this answer
    
Gilles, thanks for the options. As I've mentioned in other comments, I've ruled out anything that uses a regex. Because my search-text will, at some time, clash with regex special chars.. I thought something like this might be well known in the coder's realms (source-code mods), but maybe not... I'll look into awk a bit further, but I suspect that read -r (I just discovered the -r today) may be the way to go... I wrote a working script last week that doesn't use regex, but I already knew where to find the text (via its line-number). read -r or awk may work for locating in an unknown file. –  Peter.O Jan 9 '11 at 3:22
    
@fred.bear: Searching for text containing regexp special characters is a separate issue. One way to solve it is to preprocess the search string to quote special characters. A better way, if the utility supports it, is to do a literal string search. Awk: index function. Perl: index function, \Q…\E regexp escape. –  Gilles Jan 9 '11 at 11:51
    
Gilles, the tool would need to support not only literal string search but also literal string substitution. –  loevborg Jan 9 '11 at 17:48
    
@loevborg: In Perl, it's just s/\Q$needle/$haystack/g. Awk is less straightforward but doable. –  Gilles Jan 9 '11 at 17:59
    
thanks, that's useful, –  loevborg Jan 10 '11 at 13:10
add comment

Even if you dislike hoary sed and perl, you might still find a liking in grey-templed awk. This answer seems to be what you're looking for. I reproduce it here. Say you have three files and want to replace needle with replacement in haystack:


awk ' BEGIN { RS="" }
      FILENAME==ARGV[1] { s=$0 }
      FILENAME==ARGV[2] { r=$0 }
      FILENAME==ARGV[3] { sub(s,r) ; print }
    ' needle replacement haystack > output

This doesn't involve regular expressions and supports newline characters. It seems to work with reasonably large files. It does involve slurping the whole file into memory, so it will not work with files of arbitrary size. If you want it more elegant, you can enclose the whole shebang in a bash script, or turn it into a awk script.

share|improve this answer
    
I must have given the wrong message.. I LOVE 'sed'! and use it for all things under the sun... I just think it isn't the best tool for this job.... I'm just about to post a 'sed' solution.. which works, and will NOT have problems with clashing regex special characters, but I feel it creates too many files and is therefore "top heavy" for the job ...... After I've posted it and had a break, I'll take a good look at your solution.. Thanks... –  Peter.O Jan 9 '11 at 13:37
    
.. Unfortunately, in relation to regex special characters, if failed equally as well as all regex utils. When they encounter a special character, they treat it in a special way.. Not good when it isn't intended to be special. It crashed on [,( and false results on $,^ (didn't find the data, even though it was identical)... I didn't check any further... (But the command looks "just right" :) –  Peter.O Jan 9 '11 at 16:56
    
Fred, you're right; I should have checked the documentation. Moreover gsub seems more appropriate than sub. I find it quite strange that no simple mulitline nonregex search/replace utility has turned up. There seems to be no easy way around escaping/quoting the search needle (in perl using quotemeta for example). –  loevborg Jan 9 '11 at 17:43
    
I've just become aware of grep -F --fixed-strings . That puts grep into a whole new category (for me) ... like read -r... literal interpretation.. If only sed had this, it would be all very simple.. (maybe awk has something like this(?) .... and btw. the easiest part of the whole thing is replacing the lines; a simple head (once the first line-number is identified) | cat replacement |tail the rest (the length of the removed piece is known)... So now, with grep -F -A{num} it may be stitched up soon :) –  Peter.O Jan 9 '11 at 18:26
add comment

UPDATE: loevborg's python script is certainly the simplest and best solution (there's no doubt about that) and I'm very happy with it, but I'd like to point out that the bash script I presented (at the end of the question) is nowhere near as complicated as it looks.. I trimmed out all the debugging dross which I used to test it.. and here it is again without the overburden (for anyone visiting this page).. It's basically a sed one-liner, with pre and post hex-conversions :

F=("$haystack"  "$needle"  "$replacement")
for f in "${F[@]}" ; do cat "$f" | hexdump -v -e '1/1 "%02x"' > "$f.hex" ; done
sed -i "s/$(cat "${F[1])}.hex")/$(cat "${F[2])}.hex")/p" "${F[0])}.hex"
cat "${F[0])}.hex" | xxd -r -p > "${F[0])}"
# delete the temp *.hex files.

Just to throw my hat into the ring, I've come up with a 'sed' solution which won't run into any problems with special regex characters, because it uses not even one! .. instead it works on Hexdumped versions of the files...

I think it is too "top heavy", but it works, and is apparently not restricted by any size limitations.. GNU sed has an unlimited pattern buffer size, and that is where the Hexdumped block of search-lines ends up.. So it's okay in that respect...

I am still looking for a diff solution, because it will be more flexible regarding white-space (and I would expect; faster)... but until then.. It's the famous Mr Sed. :)

This script is fully running as is, and is reasonably commented...
It looks bigger that it is; I has only 7 lines of essential code.
For a semi-realistic test, it downloads the book "Alice Through the Looking Glass" from Project Gutenberg (363.1 KB) ... and replaces the original Jabberwocky poem with a line-reversed version of itself.. (Interestingly, it's not much different reading it backwards :)

PS. I just realized that a weakness in this method is if your original uses \r\n (0xODOA) as it's newline, and your "text to match" is saved with \n (0x0A).. then this matching process is dead in the water... ('diff' doesn't have such a issues) ...


# In a text file, replace one block of lines with another block
#
# Keeping with the 'Jabberwocky' theme, 
#  and using 'sed' with 'hexdump', so 
#  there is no possible *special* char clash.
# 
# The current setup will replace only the first instance.
#   Using sed's 'g' command, it cah change all instances. 
#

  lookinglass="$HOME/Through the Looking-Glass by Lewis Carroll"
  jabberwocky="$lookinglass (jabberwocky)"
  ykcowrebbaj="$lookinglass (ykcowrebbaj)"

  ##### This section if FOR TEST PREPARATION ONLY
        fromURL="http://www.gutenberg.org/ebooks/12.txt.utf8"
        wget $fromURL -O "$lookinglass"
        if (($?==0))
        then  echo "Download OK"
        else  exit 1
        fi
        # Make a backup of the original (while testing)
        cp "$lookinglass" "$lookinglass(fromURL)"
        #
        # Extact the poem and write it to a file. (It runs from line 322-359)
        sed -n 322,359p "$lookinglass" > "$jabberwocky"
        cat "$jabberwocky"; read -p "This is the original.. (press Enter to continue)"
        #
        # Make a file containing a replacement block of lines
        tac "$jabberwocky" > "$ykcowrebbaj"
        cat "$ykcowrebbaj"; read -p "This is the REPLACEMENT.. (press Enter to continue)"
  ##### End TEST PREPARATION

# The main process
#
# Make 'hexdump' versions of the 3 files... source, expected, replacement 
  cat "$lookinglass" | hexdump -v -e '1/1 "%02x"' > "$lookinglass.xdig"
  cat "$jabberwocky" | hexdump -v -e '1/1 "%02x"' > "$jabberwocky.xdig"
  cat "$ykcowrebbaj" | hexdump -v -e '1/1 "%02x"' > "$ykcowrebbaj.xdig"
# Now use 'sed' in a safe (no special chrs) way.
# Note, all files are now each, a single line  ('\n' is now '0A')
  sed -i "s/$(cat "$jabberwocky.xdig")/$(cat "$ykcowrebbaj.xdig")/p" "$lookinglass.xdig"

  ##### This section if FOR CHECKING THE RESULTS ONLY
        # Check result 1
        read -p "About to test for the presence of  'jabberwocky.xdig'  within itself (Enter) "
        sed -n "/$(cat "$jabberwocky.xdig")/p"     "$jabberwocky.xdig"
        echo -e "\n\nA dump above this line, means: 'jabberwocky' is as expected\n" 
        # Check result 2
        read -p "About to test for the presence of  'ykcowrebbaj.xdig'  within itself (Enter) "
        sed -n "/$(cat "$ykcowrebbaj.xdig")/p"     "$ykcowrebbaj.xdig"
        echo -e "\n\nA dump above this line, means: 'ykcowrebbaj' is as expected\n" 
        # Check result 3
        read -p "About to test for the presence of  'lookinglass.xdig'  within itself (Enter) "
        sed -n "/$(cat "$ykcowrebbaj.xdig")/p"     "$lookinglass.xdig"
        echo -e "\n\nA dump above this line, means: 'lookinglass' is as expected\n" 
        # Check result 4
        read -p "About to test for the presence of  'lookinglass.xdig'  within itself (Enter) "
        sed -n "/$(cat "$jabberwocky.xdig")/p"     "$lookinglass.xdig"
        echo -e "\n\nNo dump above this line means: 'lookinglass' is as expected\n"
  ##### End of CHECKING THE RESULTS

# Now convert the hexdump to binary, and overwrite the original
  cat "$lookinglass.xdig" | xxd -r -p > "$lookinglass"
# Echo the "modified" poem to the screen
  sed -n 322,359p "$lookinglass"
  echo -e "\n\nYou are now looking at the REPLACEMENT text (dumped directly from the source 'book'"
share|improve this answer
    
This seems like an incredibly difficult solution. Much better to use regular expressions and quote the search string. –  loevborg Jan 9 '11 at 17:46
    
:) ... Yes, I agree, but I only been "bashing" away at this whole Linux Ubuntu thing for about 4 months, so when an idea like this grabs me, it sets a target, and that's great for getting some serious hand's-on bash/general experiance... but as I mention in the comments somewhere, there are only 7 lines of esential code; the rest was just "debugging" stuff and dowloadint a test file, etc.... (I've learnt a lot from it... –  Peter.O Jan 9 '11 at 20:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.