2

I have such a file with multiple blank lines in vscode

$ tail -n 20 draft3.py
            hi = len(a)  

        if lo < 0:
            raise ValueError('low must be non-negative')

        if lo == hi: 
            return None 

        mid = (lo + hi) // 2

        if x == a[mid]:
            return x
        if x > a[mid]:

            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:

            hi = mid
            return self.bi_search(a, x, lo, hi)

Tried multiple methods to remove the blank lines

`grep -v -e '^$' failed

$ tail -n 20 draft3.py | grep -v -e '^$'
            hi = len(a)  
        if lo < 0:
            raise ValueError('low must be non-negative')
        if lo == hi: 
            return None 
        mid = (lo + hi) // 2

        if x == a[mid]:
            return x
        if x > a[mid]:

            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:

            hi = mid
            return self.bi_search(a, x, lo, hi)

`grep -Ev "^$" failed

$ tail -n 20 draft3.py | grep -Ev "^$" 
            hi = len(a)  
        if lo < 0:
            raise ValueError('low must be non-negative')
        if lo == hi: 
            return None 
        mid = (lo + hi) // 2

        if x == a[mid]:
            return x
        if x > a[mid]:

            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:

            hi = mid
            return self.bi_search(a, x, lo, hi)

`sed '/^$/d' failed

$ tail -n 20 draft3.py | sed '/^$/d'
            hi = len(a)  
        if lo < 0:
            raise ValueError('low must be non-negative')
        if lo == hi: 
            return None 
        mid = (lo + hi) // 2

        if x == a[mid]:
            return x
        if x > a[mid]:

            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:

            hi = mid
            return self.bi_search(a, x, lo, hi)

What's the problem? How could remove the blank lines?

3

grep -v '^$' will remove empty lines. But what if we have spaces or tabs in some lines ? For example I added 3 spaces to parts of your text, and if we do cat -A we will see that it shows line terminator $, but it will be offset.

$
        mid = (lo + hi) // 2$
   $
        if x == a[mid]:$
            return x$
        if x > a[mid]:$

The second line there has 3 spaces, first one doesn't. So we also want to use [[:blank:]] character class to account for those as well:

$ grep -v '^[[:blank:]]*$' text.txt
           hi = len(a)  
        if lo < 0:
            raise ValueError('low must be non-negative')
        if lo == hi: 
            return None 
        mid = (lo + hi) // 2
        if x == a[mid]:
            return x
        if x > a[mid]:
            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:
            hi = mid
            return self.bi_search(a, x, lo, hi)

Now you should see that the line with 3 added spaces is gone. The * signifies zero or more repetitions of the characters, so the pattern ^[[:blank:]]*$ also implies ^$ when there are zero whitespace or tab characters on the line. So this pattern handles both truly empty and seemingly empty lines. It also applies exactly the same to grep or sed, because we're using basic regex expressions and [[:blank:]] is one of the POSIX character classes, so it is portable.


We could also do something like this in python but without regex patterns:

$ python3 -c 'import sys; print("\n".join([ l.rstrip() for l in sys.stdin if l.strip().split() ]))'  < text.txt
           hi = len(a)
        if lo < 0:
            raise ValueError('low must be non-negative')
        if lo == hi:
            return None
        mid = (lo + hi) // 2
        if x == a[mid]:
            return x
        if x > a[mid]:
            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:
            hi = mid
            return self.bi_search(a, x, lo, hi)

Why does this work ? Because .split() on a string will split at whitespaces to extract non-whitespace tokens. If a line contains only spaces, the resulting list from .split() will be empty.


As noted by ilkkachu in the comments, the issue can also occur if you use CRLF line endings ( used in DOS/Windows text files). It is easy to see if the file uses CRLF line endings via cat -A, they will be marked as ^M. For example,

$ printf 'hello\n\r\nWorld\n   \r\ntest\n\nnewtest\n' | cat -A
hello$
^M$
World$
   ^M$
test$
$
newtest$

One thing that could be done to account for carriage return is this:

$ printf 'hello\n\r\nWorld\n   \ntest\n\nnewtest\n' | sed '/^[[:blank:]]*\r*$/d'
hello
World
test
newtest

It may be simpler to first use a dos2unix utility designed specifically for converting DOS files to Unix files, and then use sed and grep. See ByteCommander's answer that shows example of how to do that.

  • 1
    "regular" whitespace like spaces could do that, or they might have CRLF line endings – ilkkachu Mar 23 at 11:53
  • @ilkkachu Good point, thank you. I've edited to account for CRLF as well – Sergiy Kolodyazhnyy Mar 23 at 19:48
  • 1
    oh, I just noticed you used [:blank:] and not [:space:]. The latter would match the CR too. – ilkkachu Mar 23 at 20:11
3

Presumably you want to remove not only empty lines, but also lines with only whitespace characters. For that, use:

sed '/^\s*$/d'   # or respectively
grep -v '^\s*$'

The sed expression deletes every line with any number (*) of whitespace characters (\s) in it. grep -v outputs any line which does not match the expression.

Example usage

$ sed '/^\s*$/d' <draft3.py 
            hi = len(a)  
        if lo < 0:
            raise ValueError('low must be non-negative')
        if lo == hi: 
            return None 
        mid = (lo + hi) // 2
        if x == a[mid]:
            return x
        if x > a[mid]:
            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:
            hi = mid
            return self.bi_search(a, x, lo, hi)
0

use

perl -p  -e 's/^[:blank:]*$//g' inputfile | sed '/^$/d' 

and you will have

tail -n 20 draft3.py
            hi = len(a)
        if lo < 0:
            raise ValueError('low must be non-negative')
        if lo == hi:
            return None
        mid = (lo + hi) // 2
        if x == a[mid]:
            return x
        if x > a[mid]:
            lo = mid + 1
            return self.bi_search(a, x, lo, hi)
        if x < a[mid]:
            hi = mid
            return self.bi_search(a, x, lo, hi)

Only with sed is

sed -r 's/^[[:space:]]*$//g' inputfile | sed '/^$/d'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.