5

cat -e file.txt gives:

{"yellow":"mango"}^M$
^M$
{"yellow":"banana"}^M$
^M$
{"yellow":"blabla"}^M$
^M$

and I would like to just have:

{""yellow":"mango"}^M$
{"yellow":"banana"}^M$
{"yellow":"blabla"}^M$

in place for all files with txt extention in folder. So I tried:

find . -type f -name "*.txt" -print0 | xargs -0 sed -i "s/^M$^M$/^M$/g"

to no avail. Does anyone have a better idea?

head -n 3 file.txt | od -bc

yields:

0000000 173 042 171 145 154 154 157 167 042 072 042 155 141 156 147 157
          {   "   y   e   l   l   o   w   "   :   "   m   a   n   g   o
0000020 042 175 015 012 015 012 173 042 142 141 142 141 142 042 072 042
          "   }  \r  \n  \r  \n   {   "   b   a   b   a   b   "   :   "
0000040 155 141 156 147 157 042 175 015 012
          m   a   n   g   o   "   }  \r  \n
0000051

this:

awk 1 RS='\r\n' ORS= < file.txt

removes the new lines completely (so it's not good: I want to keep one of the successive two on each line, but it does something).

8
  • 1
    The $ from cat -e is an end-of-line indicator, and is not in the file. Inspect the file with head -n 3 fille.txt | od -bc and use xargs -0 sed -i "s/^M^M/^M/g. – waltinator Aug 24 '20 at 20:19
  • added the result of inspect. strangely s/^M^M/^M/g doesn't help. Nor does sed -e 's/\\r\\n\\r\\n/\\r\\n/g' file.txt – user2413 Aug 24 '20 at 20:27
  • Does sed -e 's/\r\n\r\n/\r\n/g' file.txt work? – MaxSilvester Aug 24 '20 at 20:30
  • @MaxSilvester: no – user2413 Aug 24 '20 at 20:33
  • 2
    When asking a question, state the goal, not the means you try to (maybe) achieve it : Do you really want to "replace to consecutive crlf with one" (and thus, if you ever have 4 consecutive crlf you end up with 2 consecutive crlf in the endfile, and thus have at least 1 empty line) ? or do you want to get rid of all empty lines instead? – Olivier Dulac Aug 26 '20 at 8:46
13

You can use sed -z 's/\r\n\r\n/\r\n/g'.

Normally sed only works on one line at a time. By using the -z option, sed will work on lines, which are seperated by 0 bytes, which normally don't exist in a text file, so the whole file will be treated as one line and newlines can be replaced.

(found on stackoverflow and added explanation)

4
  • You solution works. I had accidentally changed the file ending with one of my trials! But going back to the original file, it's this! – user2413 Aug 24 '20 at 20:50
  • 4
    Don't try this on a multi-GB file. – Guntram Blohm Aug 25 '20 at 7:00
  • 1
    Although this is for Ubuntu, warning to those that aren't using Ubuntu: This only works on GNU/sed, i.e. not Alpine, an initial ramdisk environment or similar. – ljrk Aug 25 '20 at 21:36
  • @GuntramBlohmsupportsMonica sed -I on a large file is very slow no matter what operation you try to perform; but without -I, it should be reasonably fast. – jpaugh Aug 26 '20 at 13:49
12

You can also delete lines that contain only the carriage return.

  • With GNU Sed:

    sed '/^\r$/d' file
    
  • For a minimal but POSIX compliant machine (here we need to generate the carriage return with Printf):

    sed "/^$(printf "\r")$/d" file
    

^ matches the start-of-line and the last $, the end-of-line (\n).

For example:

$ cat -e file
AB^M$
^M$
CB^M$
^M$
$ sed '/^\r$/d' file|cat -e
AB^M$
CB^M$
1
  • 1
    This is probably the better answer, depending on the actual requirements – JCRM Aug 26 '20 at 12:43
2

If it's okay to remove all blank lines, you can do:

perl -wlne '/\S/ and print' old_file > new_file

And if you prefer to overwrite your file(s), you can use the -i (in-place) switch:

perl -wlni.bak -e '/\S/ and print' file1 file2 file3 ...

The above line will copy the original files as *.bak files. If you don't care about having backups, then you can just leave out the .bak part, like this:

perl -wlni -e '/\S/ and print' file1 file2 file3 ...

(You can even use wildcards, so instead of file1 file2 file3 ... you can write file* .)

The advantage of this approach is that it makes changes to your files all at once (instead of having to run it once for each file).

But remember: This will only keep lines that contain at least one non-whitespace character. So if a line consists only of five spaces, a tab, a carriage return, and a line-feed character, it won't be kept.

2

I think you could use awk's Record Separator and Output Record Separator to achieve the goal, which should be more efficient on very large files than sed -z ....

awk '
  BEGIN {
    RS = "\r\n\r\n"
    ORS = "\r\n"
  }

  {
    print $0
  }

' inputFile > outputFile
1
  • This is nice +1. It is also equivalent to awk 1 RS="\r\n\r\n" ORS="\r\n". Just bear in mind it adds a final blank line if the file does not terminate with a blank line and that a multi-character record separator is unespecified by POSIX. For example, GNU awk will give wrong results if given the --posix flag. – Quasímodo Aug 26 '20 at 13:00
1

Using Raku (the language formerly known as Perl6)

~$ raku -ne '.put if /\S/ ;' test_blank.txt
{"yellow":"mango"}
{"yellow":"banana"}
{"yellow":"blabla"}

The example above only prints lines that contain non-whitespace characters (\S matches a single character that is not whitespace). A very readable version below:

~$ raku -ne '.put if .chars;' test_blank.txt
{"yellow":"mango"}
{"yellow":"banana"}
{"yellow":"blabla"}

HTH.

https://raku.org
https://rakudo.org/downloads

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.