Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have an incoming stream at a serial port, with new lines appearing about once per second

wren@Raven:~$ cat /dev/ttyUSB0

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

I want to strip blank lines and timestamp the remainder.

sed will cull blank lines and add a timestamp, but I can't make the timestamp update, it just reports the time it was invoked:

wren@Raven:~$ cat /dev/ttyUSB0 | sed -e '/^$/d' -e "s/$/`date +\,%F\,%T`/"
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
^C

I've found ts, part of Moreutils, and can pipe into it to get an updating timestamp.

wren@Raven:~$ cat /dev/ttyUSB0 |  ts
May 14 09:49:26 A_Sensor1,B_22.00,C_50.00
May 14 09:49:26
May 14 09:49:27 A_Sensor1,B_22.00,C_50.00
^C

However, I can't properly combine ts with sed.

This, which looks like it should do what I want, produces no output at all

wren@Raven:~$ cat /dev/ttyUSB0 | sed -e '/^$/d' | ts
^C
wren@Raven:~$

However reversing the order of the pipes does produce an output, but of course doesn't strip lines which are no longer blank. Other substitutions work fine, so I know the pipe to sed is working.

wren@Raven:~$ cat /dev/ttyUSB0 |  ts | sed -e '/^$/d'
May 14 10:07:25 A_Sensor1,B_22.00,C_50.00
May 14 10:07:25
May 14 10:07:26 A_Sensor1,B_22.00,C_50.00
May 14 10:07:26
^C

So I'm a bit baffled. I can presumably make sed remove the unwanted lines, but timestamping them prior to removal must be the wrong approach.

I would appreciate an explanation and some help.

share|improve this question

2 Answers 2

up vote 7 down vote accepted

To answer the question directly, sed is buffering and that's the sole problem.
You can fix this by telling it not to buffer with its -u/--unbuffered flag:

sed -u '/^$/d' /dev/ttyUSB0 | ts

With a test harness (but you'll need to run it for proof):

$ (echo -e 'banana\n\n'; sleep 2; echo 'cheese') | sed -u '/^$/d' | ts
May 14 11:26:05 banana
May 14 11:26:07 cheese

You can run into similar predicaments with other stream editors. They seemingly all want to buffer a little bit. They all have workarounds though. Here are a bunch of commands I've tested:

... | mawk -W interactive '/./' | ts
... | gawk '/./ { print $0; fflush(); }' | ts
... | grep --line-buffered '.' | ts
... | perl -n -e 'print if /./' | ts

Another idea is to just let gawk handle it. It can filter for non-empty lines and do the date-printing for you (thanks to SO's Kieron):

awk '/./ { print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }' /dev/ttyUSB0

That flushes straight after lines come in. gawk is especially helpful here if you want to do other things... If you want to check that the fourth column of output (pre-ts) matches a regex, you can (eg $4~/\d{4}/). Awk (and its variants) are very flexible for stream processing.

Another test harness:

$ gawk '/./ { print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }' <(
      echo -e 'banana\n\n';
      sleep 2;
      echo 'cheese'
  )
2014-05-14 11:13:59 banana
2014-05-14 11:14:01 cheese
share|improve this answer
1  
+1 for sed -u. It is a block-buffering vs. line-buffering issue. –  J.F. Sebastian May 14 at 10:31
    
@Oli sed -u also works perfectly when piped into ts, so I'll read up about buffering. I'm no longer baffled, many thanks. –  perplexed May 14 at 10:38
    
awk is particularly well suited for things like this. awk code is generally a lot less dense and much more readable than sed and you can throw in as many print statements as you want to view partial results while debugging. You can put a whole awk program into a here document to avoid using a separate file and if you put quotes around the here document terminator string, then bash will ignore all the embedded tokens it would normally try to interpret. –  Joe May 15 at 2:40

bash can handle this in a while read loop

(echo -e 'banana\n\n'; sleep 2; echo 'cheese') | 
while IFS= read -r line; do 
    [[ $line ]] && echo "$(date "+%F %T") line"
done
2014-05-14 06:34:06 banana
2014-05-14 06:34:08 cheese

You can remove lines with only whitespace with a tricky parameter expansion: remove all leading whitespace and see if the line is empty:

shopt -s extglob

(echo -e '  banana\n\t\n'; sleep 2; echo 'cheese') |
while IFS= read -r line; do
    [[ "${line/#+([[:blank:]])/}" ]] && echo "$(date "+%F %T") $line"
done
share|improve this answer
    
I tried a variety of approaches like that, none of them worked. I can't get your code to work either. Using echo or cat to send /dev/ttyUSB0 to the while loop just results in a single line of output : 2014-05-14 12:23:32 line –  perplexed May 14 at 11:28
    
I'm sure there's a better way, but try tail -f /dev/ttyUSB0 instead of cat or echo. It will keep running. I didn't know how to test this on my system. –  Joe May 15 at 2:29
    
tail -f /dev/ttyUSB0 gives no output, with or without the while loop. tvm for your comments. –  perplexed May 15 at 12:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.