5

cat -e file.txt gives:

{"yellow":"mango"}^M$
^M$
{"yellow":"banana"}^M$
^M$
{"yellow":"blabla"}^M$
^M$

and I would like to just have:

{""yellow":"mango"}^M$
{"yellow":"banana"}^M$
{"yellow":"blabla"}^M$

in place for all files with txt extention in folder. So I tried:

find . -type f -name "*.txt" -print0 | xargs -0 sed -i "s/^M$^M$/^M$/g"

to no avail. Does anyone have a better idea?

head -n 3 file.txt | od -bc

yields:

0000000 173 042 171 145 154 154 157 167 042 072 042 155 141 156 147 157
          {   "   y   e   l   l   o   w   "   :   "   m   a   n   g   o
0000020 042 175 015 012 015 012 173 042 142 141 142 141 142 042 072 042
          "   }  \r  \n  \r  \n   {   "   b   a   b   a   b   "   :   "
0000040 155 141 156 147 157 042 175 015 012
          m   a   n   g   o   "   }  \r  \n
0000051

this:

awk 1 RS='\r\n' ORS= < file.txt

removes the new lines completely (so it's not good: I want to keep one of the successive two on each line, but it does something).

8
  • 1
    The $ from cat -e is an end-of-line indicator, and is not in the file. Inspect the file with head -n 3 fille.txt | od -bc and use xargs -0 sed -i "s/^M^M/^M/g.
    – waltinator
    Aug 24, 2020 at 20:19
  • added the result of inspect. strangely s/^M^M/^M/g doesn't help. Nor does sed -e 's/\\r\\n\\r\\n/\\r\\n/g' file.txt
    – user2413
    Aug 24, 2020 at 20:27
  • Does sed -e 's/\r\n\r\n/\r\n/g' file.txt work? Aug 24, 2020 at 20:30
  • @MaxSilvester: no
    – user2413
    Aug 24, 2020 at 20:33
  • 2
    When asking a question, state the goal, not the means you try to (maybe) achieve it : Do you really want to "replace to consecutive crlf with one" (and thus, if you ever have 4 consecutive crlf you end up with 2 consecutive crlf in the endfile, and thus have at least 1 empty line) ? or do you want to get rid of all empty lines instead? Aug 26, 2020 at 8:46

5 Answers 5

13

You can use sed -z 's/\r\n\r\n/\r\n/g'.

Normally sed only works on one line at a time. By using the -z option, sed will work on lines, which are seperated by 0 bytes, which normally don't exist in a text file, so the whole file will be treated as one line and newlines can be replaced.

(found on stackoverflow and added explanation)

4
  • You solution works. I had accidentally changed the file ending with one of my trials! But going back to the original file, it's this!
    – user2413
    Aug 24, 2020 at 20:50
  • 4
    Don't try this on a multi-GB file. Aug 25, 2020 at 7:00
  • 1
    Although this is for Ubuntu, warning to those that aren't using Ubuntu: This only works on GNU/sed, i.e. not Alpine, an initial ramdisk environment or similar.
    – ljrk
    Aug 25, 2020 at 21:36
  • @GuntramBlohmsupportsMonica sed -I on a large file is very slow no matter what operation you try to perform; but without -I, it should be reasonably fast.
    – jpaugh
    Aug 26, 2020 at 13:49
12

You can also delete lines that contain only the carriage return.

  • With GNU Sed:

    sed '/^\r$/d' file
    
  • For a minimal but POSIX compliant machine (here we need to generate the carriage return with Printf):

    sed "/^$(printf "\r")$/d" file
    

^ matches the start-of-line and the last $, the end-of-line (\n).

For example:

$ cat -e file
AB^M$
^M$
CB^M$
^M$
$ sed '/^\r$/d' file|cat -e
AB^M$
CB^M$
1
  • 1
    This is probably the better answer, depending on the actual requirements
    – JCRM
    Aug 26, 2020 at 12:43
2

If it's okay to remove all blank lines, you can do:

perl -wlne '/\S/ and print' old_file > new_file

And if you prefer to overwrite your file(s), you can use the -i (in-place) switch:

perl -wlni.bak -e '/\S/ and print' file1 file2 file3 ...

The above line will copy the original files as *.bak files. If you don't care about having backups, then you can just leave out the .bak part, like this:

perl -wlni -e '/\S/ and print' file1 file2 file3 ...

(You can even use wildcards, so instead of file1 file2 file3 ... you can write file* .)

The advantage of this approach is that it makes changes to your files all at once (instead of having to run it once for each file).

But remember: This will only keep lines that contain at least one non-whitespace character. So if a line consists only of five spaces, a tab, a carriage return, and a line-feed character, it won't be kept.

2

I think you could use awk's Record Separator and Output Record Separator to achieve the goal, which should be more efficient on very large files than sed -z ....

awk '
  BEGIN {
    RS = "\r\n\r\n"
    ORS = "\r\n"
  }

  {
    print $0
  }

' inputFile > outputFile
1
  • This is nice +1. It is also equivalent to awk 1 RS="\r\n\r\n" ORS="\r\n". Just bear in mind it adds a final blank line if the file does not terminate with a blank line and that a multi-character record separator is unespecified by POSIX. For example, GNU awk will give wrong results if given the --posix flag.
    – Quasímodo
    Aug 26, 2020 at 13:00
1

Using Raku (the language formerly known as Perl6)

~$ raku -ne '.put if /\S/ ;' test_blank.txt
{"yellow":"mango"}
{"yellow":"banana"}
{"yellow":"blabla"}

The example above only prints lines that contain non-whitespace characters (\S matches a single character that is not whitespace). A very readable version below:

~$ raku -ne '.put if .chars;' test_blank.txt
{"yellow":"mango"}
{"yellow":"banana"}
{"yellow":"blabla"}

HTH.

https://raku.org
https://rakudo.org/downloads

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .