3

I want to get delete all of the blank lines and lines with spaces (if any exist (ONLY from the bottom of the file)) and then to remove one more line (also ONLY from the bottom of the file).

I have this code:

while [[ "$last_line" =~ $ ]] || [[ "$last_line" =~ ^[[:space:]]+$ ]]
do
    sed -i -e '${/$/d}' "./file.txt"
done
    sed -i -e '${/$/d}' "./file.txt"

For some reason the loop doesn't stop and it deletes everything in the file. What is the matter?

  • 1
    if $last_line doesn't get updated, it will fall into the while forever. Also, the standard way to remove empty lines is grep -v '^\s*$' file, don't know what else you want to do here but it looks a bit too complicated to involve loops. – fedorqui Aug 5 '15 at 11:40
  • I'm not getting that $ before the curly braces. What is that for? – kos Aug 5 '15 at 11:42
  • I am a little confused as to what you want. You want to completely remove any line that has a space anywhere on it, and also remove the last line from the file? – Julian Stirling Aug 5 '15 at 12:02
  • 1
    Please add an example, the question is very hard to understand. – A.B. Aug 12 '15 at 13:59
4

Your script should work if fixed like so:

while
 last_line=$(tail -1 "./file.txt")
 [[ "$last_line" =~ ^$ ]] || [[ "$last_line" =~ ^[[:space:]]+$ ]]
do
 sed -i '$d' "./file.txt"
done

Your script had two main problems: (1) you never updated $last_line, so the loop's guard would always evaluate the same thing; (2) your [[ "$last_line" =~ $ ]] test matched any line, since any line has an end. (This is the reason why your script emptied your file completely.) You probably want to match against ^$ instead, which matches only empty lines. Additionally, I simplified the sed command to delete the last line in the loop's body (simply $d does the job).

However, this script is unnecessarily complicated. sed is there for just that kind of thing! This one-liner will do the same thing as the above script:

sed -i ':a;/^[ \n]*$/{$d;N;ba}' ./file.txt

Roughly,

  1. Match current line against ^[ \n]*$. (i.e, can only contain whitespaces and newlines)
  2. If it doesn't match, just print it. Read in next line and continue with step 1.
  3. If it does match,
    • If we are at the end of the file, delete it.
    • If we are not at the end of the file, append the next line to the current line, inserting a newline character between the two, and go back to step 1 with this new, longer line.

There are lots of awesome sed tutorials on the Internet. For example, I can recommend this one. Happy learning! :-)

Update: And of course, if you additionally want to remove the last (non-blank) line of the file after having truncated the trailing blank lines, you can just use another sed -i '$d' ./file.txt after either your script or the above one-liner. I intentionally did not want to include that in the sed one-liner since I thought that removing trailing blank lines is quite a reusable piece of code that may be interesting for other people; but removing the last non-blank line is really specific to your use case, and trivial anyway once you removed the trailing blank lines.

  • 1
    Didn't look at this before because I was a bit tired when I posted my answer, but the sed trick to recursively append the lines is very neat. Nice! – kos Aug 12 '15 at 16:59
4

This task is way more easily accomplished by processing the file's lines in the opposite order.

tac infile | awk 'flag {print} {if(NF) flag=1}' | tac | sponge infile

As pointed out in the comments by Malte Skoruppa and zwets, Ubuntu doesn't come with the moreutils package preinstalled, which contains sponge; an alternative solution is to use a command substitution inside a herestring to read the input file, so that, being the command substitution processed first, the file is safe to be truncated by the second tac command:

<<<"$(< infile)" tac | awk 'flag {print} {if(NF) flag=1}' | tac > infile
  • tac infile: ... does the opposite of cat infile (!): prints the file to stdout inverting the line's order;
  • awk [...]: processes the file;
  • tac: ... does the opposite of cat (!): prints the file to stdout inverting the line's order;
  • sponge infile: outputs to infile only when the left side of the pipe has terminated its execution, to avoid truncating infile before it's read by the first tac command;

awk command breakdown:

  • flag {print}: if flag is set, it will print the line; flag won't be set until a record whose NF value matches a number greater than 0 is processed, so until a record whose NF value matches a number greater than 0 is not found, the print command will be skipped;
  • {if(NF) flag=1}: if while flag is still unset a record whose NF value matches a number greater than 0 is processed, it will not be printed and flag will be set to 1, so the first record whose NF value matches a number greater than 0 won't be printed;

Test on a test file (mind that line 4 and line 7 contain 5 spaces, while line 5 and line 8 are empty):

user@debian ~ % cat infile                                           
line1
line2
line3


line6


user@debian ~ % tac infile | awk 'flag {print} {if(NF) flag=1}' | tac
line1
line2
line3


user@debian ~ % 

Line 7 and line 8 have been removed because they were both at the end of the file, containing only spaces (line 7) or containing nothing (line 8); line 6 was deleted because it was the first one, reading the file's lines in the opposite order, to have at least 1 field (hence not being empty or containing only spaces)

  • 1
    Oh, this is neat, +1 for using tac. :-) I intentionally did not remove the last non-blank line in my answer since I thought that removing the trailing blank lines is by far the more interesting part of the task and much more reusable (and removing one more line after having truncated the trailing blank lines is easy.) Minor quibble: sponge is not POSIX, nor even part of the GNU Coreutils that come preinstalled with Ubuntu. Having to install an extra package just to remove some blank lines may be undesirable for some people. – Malte Skoruppa Aug 12 '15 at 9:32
  • 1
    +1 for the cleverness of the double tac and another for pointing out sponge. I confess to doing the occasional diff -u a b | patch -p0 with fingers crossed. Just putting sponge in that pipeline resolves the race condition perfectly and elegantly. I agree with @MalteSkoruppa that the fact that sponge is not in coreutils is a hurdle, but then it made me apt-get install moreutils right away and admire some of the other little gems in that package. Their ingenuity is what makes working with Unix and GNU such a joy. – zwets Aug 12 '15 at 11:24
  • Add a line line 7 with a space and the awk script fails. OP said "delete all of the blank lines and lines with spaces" … " and then to remove one more line", but it's really hard to understand – A.B. Aug 12 '15 at 13:57
  • @A.B. If you look at the pattern OP used to match the lines to remove, he used $ (which is wrong since it matches every line, I think he meant to use ^$) and ^[[:space:]]+$, so he wants to delete empty lines and lines containing only spaces (^[[:space:]]+$), not every line containing at least one space (or it would have been [[:space:]]) – kos Aug 12 '15 at 14:26
  • Also he used [[:space:]], which I agree it's ambiguous since it matches any type of whitespace, so possibly another OP's error, since it doesn't match what he said he wants to do – kos Aug 12 '15 at 14:31
0

Edit

So originally, I've missed the point that OP wants to remove only last blank line, which my original solution does as well. Nonetheless, here's the version that removes only the last one, and if its blank.

awk -v numlines=$(wc -l file2|cut -f1 -d' ') 'NR < numlines; END {if (NF) print }' file2

What the code does is fairly straightforward - get number of lines, and print every line, until the last one. On the last one, we check if the line contains any fields; if there is any text, NF evaluates to an integer (true) thus printing last line, and if there's nothing or spaces only - NF evaluates to zero (false) and doesn't print anything else.

As for removing one more line, head -n -1 will do.

Bellow is the small demo. The trailing newline is designated with $ prompt is *$

*$ cat -A file2                                                                                                                                      
212$
1231$
$
324234$
213$
$
*$ awk -v numlines=$(wc -l file2|cut -f1 -d' ')  'NR < numlines ; END {if ( NF ) print }' file2 | head -n -1                                         
212
1231

324234

Original

awk solution.

awk 'NF' file1 > /tmp/tmpfile && cat /tmp/tmpfile > file1

Here we use Number of Fields variable as test for printing. For blank lines, number of fields is zero, hence it blank lines evaluating to false won't be printed. Now , unless your awk version supports in-line editing (which is gnu awk or gawk, I think), you have to redirect output to temporary output and back to the original file with cat, like I did here

Variation on the theme would be to use regex to test if lines contain some particular data, like digits or alphanumeric characters, e.g.

awk '$0~/[[:digit:]]||[[:alpha:]]/ ' file1 > /tmp/tmpfile && cat /tmp/tmpfile > file1

  • Hm I think you missed this two parts: "ONLY from the bottom of the file" and "and then to remove one more line (also ONLY from the bottom of the file)". I expressed my aversion towards cat file > file1 already so I won't bother you anymore with this one :) – kos Aug 12 '15 at 3:35
  • Downvote because this eliminates all empty lines and fails to delete the last non-empty line. – zwets Aug 12 '15 at 10:23
  • @zwets "all empty lines", that's true, but it deletes the latest empty line also … in my test. – A.B. Aug 12 '15 at 13:24
  • Edited my answer. Please review – Sergiy Kolodyazhnyy Aug 12 '15 at 14:40
  • Gotta go now, I'm gonna check it later along with A.B.'s and Matte Skoruppa's answer tough :) – kos Aug 12 '15 at 15:04
0

As I understand

  • Remove empty lines from the end of the file
  • Remove lines with spaces from the end of the file
  • And then remove one more line

Using awk and tac

tac foo | awk '! non_empty && ! /^$/ && ! /[ \t]/ {non_empty = 1} non_empty {skip++} skip > 1 {print}' | tac

More variations below the example …


Example

% cat -n foo                                                                
     1  line1
     2  line2
     3  line3
     4  
     5  
     6  line6
     7  line 7 
     8  

% tac foo | awk '! non_empty && ! /^$/ && ! /[ \t]/ {non_empty = 1} non_empty {skip++} skip > 1 {print}' | tac > bar

% cat -n bar
     1  line1
     2  line2
     3  line3
     4  
     5  

To remove only the blank lines at the end of the file

tac foo | awk '! non_empty && ! /^$  {non_empty = 1} non_empty {print}' | tac

To remove the blank lines at the end and one more line

tac foo | awk '! non_empty && ! /^$/ &&  {non_empty = 1} non_empty {skip++} skip > 1 {print}' | tac
  • Same as Serg, I think OP is asking something different; let's see if he provides some example, since everyone is understanding something different. – kos Aug 12 '15 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.