0

I'd like to change all occurrences of \ntoken1\n and \ntoken2\n to token1 and token2 in a file).

I thought this would do it:

${
N
N
s/\ntoken1\n/token1/
s/\ntoken2\n/token2/
}

but it does not.

Thanks in advance for any clarification.

2
  • 2
    Is it a literal \n or an actual newline LF? ... I am now having a second thought about your requirement ... I wrote my answer with literal \n as I understood it ... If a real newline, then @BeastOfCaerbannog add an answer for that and you have both ;-) ... My bad if I understood wrong.
    – Raffa
    Feb 24 at 17:30
  • 6
    Please edit your question to include a minimal testable example of input and desired output Feb 24 at 18:41

4 Answers 4

8

Rather then treating the problem as a multi-line substitution, you might consider using the non-streaming ed editor instead, which provides a join command and supports relative addresses. Ex. given

$ cat file
abc
def
token1
ghi
jkl
token2
mno
pqr

then you can globally select token matches and then join each with one preceding and one following line:

$ printf '%s\n' 'g/token1\|token2/-1,+1j' ,p Q | ed -s file
abc
deftoken1ghi
jkltoken2mno
pqr

(if your tokens are really token1 and token2 you can simplify the regex to token[12]).

To edit in place, replace ,p Q (print then unconditionally quit) with wq (write and quit).


If you must use sed, then one approach would be to implement a loop:

$ sed -E -e :a -e 'N;s/\n(token1|token2)\n/\1/' -e '/\n.*\n/!ba' -e 'P;D'  file
abc
deftoken1ghi
jkltoken2mno
pqr

Here, /\n.*\n/!ba tests whether the pattern space contains two newlines (i.e. is three lines long) and if not, loops back and adds another line; otherwise P;D prints and removes a line before looping back. Together they maintain a 3-line sliding windows over which to apply the substitution.


Note: you haven't provided a test case, and in particular you haven't indicated the desired behavior in the case that `\ntoken1\n` and `\ntoken2\n` overlap ex. should
def
token1
ghi
token2

become

deftoken1ghitoken2

or

deftoken1ghi
token2
2
  • I am so bummed somebody beat me to ed man ed. I'm only 40, but even I have a lot of respect for ed and am glad somebody else thought of this too actually. Feb 25 at 9:07
  • 3
    @JasonNordwick thanks ;) I guess I should have noted that ed loads the whole file into memory, so may not be the best choice for processing large files (although the same is true of solutions using sed -z) Feb 25 at 13:01
4

To be able to use \n as newline character in a sed command you have to use the -z flag, which separates the lines by NUL characters instead of newlines (\n). So the command you can use is the following:

sed -z 's/\ntoken1\n/token1/g;s/\ntoken2\n/token2/g' /path/to/file

where /path/to/file is the path to the file you wish to modify.

The above command will not actually apply the changes to your file. To apply the changes you can either redirect the output to another file as:

sed -z 's/\ntoken1\n/token1/g;s/\ntoken2\n/token2/g' /path/to/file > /path/to/another_file

or, if you like to apply the changes to the same file, use the -i (in-place) flag as:

sed -z -i 's/\ntoken1\n/token1/g;s/\ntoken2\n/token2/g' /path/to/file

References:

4

A backslash is used in ERE's to escape (strip special meaning from) characters in order to match them literally ... However, in BRE's, it also adds a special meaning to some characters.

So, it has a special meaning in both and therefore has to be escaped itself in order to be matched as a literal \.

sed has a --debug option that you can use to see what is matched in your file:

$ cat file
\ntoken1\n
\ntoken2\n

... with your RegEx patterns \ntoken1\n and \ntoken1\n like so:

$ sed --debug -e 's/\ntoken1\n/token1/' -e 's/\ntoken2\n/token2/' file
SED PROGRAM:
  s/\ntoken1\n/token1/
  s/\ntoken2\n/token2/
INPUT:   'file' line 1
PATTERN: \\ntoken1\\n
COMMAND: s/\ntoken1\n/token1/
PATTERN: \\ntoken1\\n
COMMAND: s/\ntoken2\n/token2/
PATTERN: \\ntoken1\\n
END-OF-CYCLE:
\ntoken1\n
INPUT:   'file' line 2
PATTERN: \\ntoken2\\n
COMMAND: s/\ntoken1\n/token1/
PATTERN: \\ntoken2\\n
COMMAND: s/\ntoken2\n/token2/
PATTERN: \\ntoken2\\n
END-OF-CYCLE:
\ntoken2\n

... to see that no MATCHED REGEX is reported.

While if you escape it like so:

$ sed --debug -e 's/\\ntoken1\\n/token1/' -e 's/\\ntoken2\\n/token2/' file
SED PROGRAM:
  s/\\\\ntoken1\\\\n/token1/
  s/\\\\ntoken2\\\\n/token2/
INPUT:   'file' line 1
PATTERN: \\ntoken1\\n
COMMAND: s/\\\\ntoken1\\\\n/token1/
MATCHED REGEX REGISTERS
  regex[0] = 0-10 '\ntoken1\n'
PATTERN: token1
COMMAND: s/\\\\ntoken2\\\\n/token2/
PATTERN: token1
END-OF-CYCLE:
token1
INPUT:   'file' line 2
PATTERN: \\ntoken2\\n
COMMAND: s/\\\\ntoken1\\\\n/token1/
PATTERN: \\ntoken2\\n
COMMAND: s/\\\\ntoken2\\\\n/token2/
MATCHED REGEX REGISTERS
  regex[0] = 0-10 '\ntoken2\n'
PATTERN: token2
END-OF-CYCLE:
token2

... then, you get a MATCHED REGEX and your replacement works.

To find and replace all occurrences, you'll need to set the global flag like so:

sed -e 's/\\ntoken1\\n/token1/g' -e 's/\\ntoken2\\n/token2/g' file
0
1

Using any awk and only storing 1 line at a time in memory:

$ awk '
    /^(token1|token2)$/ { ors=ORS="" }
    { printf "%s%s", ors, $0; ors=ORS; ORS=RS }
    END { printf "%s", ors }
' file
abcdeftoken1ghi
jkltoken2mno
pqr
2
  • 1
    Okay ... So, basically, printf prints no line feeds \n by default but, you add it from RS before each following record (line) in { printf "%s%s", ors, $0; ors=ORS; ORS=RS } ... That is unless token1 or token2 are found alone between line start ^ and line end $ where you delay printing RS for an extra one record (two records in a row) in /^(token1|token2)$/ { ors=ORS="" } and at the END { printf "%s", ors }, you follow the last verdict on whether to print RS or not but after the record this time ... No doubt it'll works with actual LFs ... Am I right? :-)
    – Raffa
    Feb 27 at 8:49
  • 1
    @Raffa right, and yes it would.
    – Ed Morton
    Feb 27 at 11:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .