7

I have several files with text that needs replacing. The text starts and ends with the same pattern each time, but the content in between the patterns is variable. The patterns can appear in the middle of lines, and the content between them often spans multiple lines.

There will only be a single occurrence of the start and end pattern in each file.

I need a command line method to replace the text between the patterns, including the patterns themselves. Outputting to a new file, or editing in place is fine.

A command that operates on a single file will work, as I can loop through the files and apply the command myself. I attempted a sed solution but could only manage to replace entire lines.

An example of text would be:

Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: Security-Start Bs86gKI-734Lw#32_nP/5589Zfb8Wj-
sW93j9b Security-End, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

The start pattern is Security-Start and the end pattern is Security-End. I want to replace the patterns and everything in between with the word REDACTED.

I would like the output to be:

Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

Please note that the text between the two patterns may be so long that it spans several lines, it is fairly random in length. This is not clear in the example above

Any language which is available by default on an Ubuntu system will be fine. My first thoughts are 'sed' or 'awk', but whatever you're comfortable with will be fine.

Improve this question
0

4 Answers 4

Reset to default
9

It should work for you:

sed -e '/Security-Start/{ N; s/Security-Start.*Security-End/REDACTED/ }'
  • /Security-Start/ search for "Security-Start"
  • If you found it: "N;" means append the next line.
  • and do the replacements/Security-Start.*Security-End/REDACTED/ at the final result.

For more than of two line use this one:

sed -n '1h; 1!H; ${ g; s/Security-Start.*Security-End/REDACTED/p }'

Read here

Improve this answer
6
  • That looks good, there is a possibility that the end pattern may be more than one line away from the start pattern though, which I think would break this. Not obvious by my example, I'll edit to make it more clear.
    – Arronical
    May 18, 2017 at 11:42
  • @Arronical updated ;)
    – Ravexina ♦
    May 18, 2017 at 11:53
  • That's great, thanks for the link too, explains it all clearly.
    – Arronical
    May 18, 2017 at 11:59
  • You're welcome, glad that it was helpful :-)
    – Ravexina ♦
    May 18, 2017 at 12:00
  • 1
    @Arronical To collect all lines a more straight forward approach is to use a loop: sed ':a;N;$!ba;s/start.*stop/FOO/' (use line breaks instead of ; for portability)
    – Philippos
    May 18, 2017 at 12:07
8

If the files are not too large, then you could use perl in slurp mode:

$ perl -0777 -pe 's/Security-Start.*Security-End/REDACTED/s' file 
Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

The -0777 command line parameter effectively unsets the record separator so that the whole file is slurped. The s regex modifier causes perl to include newline characters in ., making the expression match across lines.

Alternatively, with a sed loop:

$ sed '/Security-Start/ {:a; $!N; s/Security-Start.*Security-End/REDACTED/; t; ba}' file
Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...

With GNU sed, you can replace t; ba (branch out on successful replacement; (otherwise) branch to :a) by Ta (branch to :a on unsuccessful replacement).

Improve this answer
2
  • I must get round to learning some perl. There are so many interesting things to learn though!
    – Arronical
    May 18, 2017 at 12:06
  • I wish to give I could give you 200 upvotes
    – Pasupathi
    Dec 21, 2018 at 22:44
4

A more manual approach would be to replace all newline character in the input file with NULLs, use a simple perl non-greedy regex to do the replacement and then put the newlines back:

$ tr '\n' '\0' < file | 
    perl -pe 's/Security-Start.*?Security-End/Security: REDACTED/g' |
        tr '\0' '\n'
Cable Type ID:135, Installation ID:62, Alpha Conductor Origin:
Tolerance Report B74 - 3rd June 1996, Beta Conductor Origin: 
Tolerance Report B74 - 3rd June 1996, Phase Conductor Size: 
45mm, Security: Security: REDACTED, Location ID:889, Protective Earth Size:
67mm, Protective Earth Max Current (A): 4, Overload Time...
Improve this answer
3
  • +1 I didn't know about '\0' , now a simple sed will work too: tr '\n' '\0' | sed s/Security-Start.*-End?/REDACTED/ | tr '\0' '\n'
    – Ravexina ♦
    May 18, 2017 at 16:00
  • @Ravexina no, only if there is only one case of Security-End, otherwise, because sed doesn't support non-greedy matches, it will replace everything from the first Security-Start to the last Security-End.
    – terdon
    May 18, 2017 at 16:06
  • another good point to know ;)
    – Ravexina ♦
    May 18, 2017 at 16:10
2

Here's how you could do it with awk:

awk -v RS='Security-Start.*Security-End' -v ORS= '1;NR==1{printf "REDACTED"}' file
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .