5

I have several files with text that needs replacing. The text starts and ends with the same pattern each time, but the content in between the patterns is variable. The patterns can appear in the middle of lines, and the content between them often spans multiple lines.

There will only be a single occurrence of the start and end pattern in each file.

I need a command line method to replace the text between the patterns, including the patterns themselves. Outputting to a new file, or editing in place is fine.

A command that operates on a single file will work, as I can loop through the files and apply the command myself. I attempted a sed solution but could only manage to replace entire lines.

An example of text would be:

Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: Security-Start Bs86gKI-734Lw#32_nP/5589Zfb8Wj-
sW93j9b Security-End, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

The start pattern is Security-Start and the end pattern is Security-End. I want to replace the patterns and everything in between with the word REDACTED.

I would like the output to be:

Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

Please note that the text between the two patterns may be so long that it spans several lines, it is fairly random in length. This is not clear in the example above

Any language which is available by default on an Ubuntu system will be fine. My first thoughts are 'sed' or 'awk', but whatever you're comfortable with will be fine.

8

It should work for you:

sed -e '/Security-Start/{ N; s/Security-Start.*Security-End/REDACTED/ }'
  • /Security-Start/ search for "Security-Start"
  • If you found it: "N;" means append the next line.
  • and do the replacements/Security-Start.*Security-End/REDACTED/ at the final result.

For more than of two line use this one:

sed -n '1h; 1!H; ${ g; s/Security-Start.*Security-End/REDACTED/p }'

Read here

  • That looks good, there is a possibility that the end pattern may be more than one line away from the start pattern though, which I think would break this. Not obvious by my example, I'll edit to make it more clear. – Arronical May 18 '17 at 11:42
  • @Arronical updated ;) – Ravexina May 18 '17 at 11:53
  • That's great, thanks for the link too, explains it all clearly. – Arronical May 18 '17 at 11:59
  • You're welcome, glad that it was helpful :-) – Ravexina May 18 '17 at 12:00
  • @Arronical To collect all lines a more straight forward approach is to use a loop: sed ':a;N;$!ba;s/start.*stop/FOO/' (use line breaks instead of ; for portability) – Philippos May 18 '17 at 12:07
8

If the files are not too large, then you could use perl in slurp mode:

$ perl -0777 -pe 's/Security-Start.*Security-End/REDACTED/s' file 
Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

The -0777 command line parameter effectively unsets the record separator so that the whole file is slurped. The s regex modifier causes perl to include newline characters in ., making the expression match across lines.


Alternatively, with a sed loop:

$ sed '/Security-Start/ {:a; $!N; s/Security-Start.*Security-End/REDACTED/; t; ba}' file
Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

With GNU sed, you can replace t; ba (branch out on successful replacement; (otherwise) branch to :a) by Ta (branch to :a on unsuccessful replacement).

  • I must get round to learning some perl. There are so many interesting things to learn though! – Arronical May 18 '17 at 12:06
  • I wish to give I could give you 200 upvotes – Pasupathi Dec 21 '18 at 22:44
4

A more manual approach would be to replace all newline character in the input file with NULLs, use a simple perl non-greedy regex to do the replacement and then put the newlines back:

$ tr '\n' '\0' < file | 
    perl -pe 's/Security-Start.*?Security-End/Security: REDACTED/g' |
        tr '\0' '\n'
Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...
  • +1 I didn't know about '\0' , now a simple sed will work too: tr '\n' '\0' | sed s/Security-Start.*-End?/REDACTED/ | tr '\0' '\n' – Ravexina May 18 '17 at 16:00
  • @Ravexina no, only if there is only one case of Security-End, otherwise, because sed doesn't support non-greedy matches, it will replace everything from the first Security-Start to the last Security-End. – terdon May 18 '17 at 16:06
  • another good point to know ;) – Ravexina May 18 '17 at 16:10
1

Here's how you could do it with awk:

awk -v RS='Security-Start.*Security-End' -v ORS= '1;NR==1{printf "REDACTED"}' file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.