How can junk letters be removed from a document with sed?

Question

I have a list which is an output from a media program actually and i want to remove all the junk so i can use that list to my Zenity script but involves alot of sed commands in one string which is the way i prefer im open any suggestion that involves only sed.

I will explain exactly what i'm trying to succeed:

The actual list is List.txt: The * is just hex 1b

*line here is not needed*
*[1m0  *[22m : *[35mThis has SPACES inside, Must be DOTS!*[39m : *[34m123.123ABCD*[39m
*[1m1  *[22m : *[35mThis-DONT-have(SPACES)-inside*[39m : *[34m900.578CD*[39m
*[1m2  *[22m : *[35mThis SPACE!!! inside, # Must be DOTS!*[39m : *[34m9K99123AD*[39m
*[1m3  *[22m : *[35mSPACES *Must* be DOTS!*[39m : *[34mAB*[39m
*[1m4  *[22m : *[35mMust[be] [DOTS] !*[39m : *[34m12.AZZZ*[39m
*[1m5  *[22m : *[35mSPACES ~DOTS!*[39m : *[34m654KK45*[39m

This is my sed equation:

cat ~/List.txt | sed -e 1d -e 's/\x1b\[1m//g' -e 's/\x1b\[22m : \x1b\[35m/|---|/g' -e 's/\x1b\[39m : /|---| /g' -e 's/.|---|*//' -e 's/|---|*//' -e 's/\x1b*\[34m/(/g' -e 's/\x1b\[39m/)/g' -e 's/^/FALSE /' -e '0,/FALSE /s//TRUE /' > sedList.txt

This is the sedList.txt which has spaces between the words:

TRUE 0 This has SPACES inside, Must be DOTS! (123.123ABCD)
FALSE 1 This-DONT-have(SPACES)-inside (900.578CD)
FALSE 2 This SPACE!!! inside, # Must be DOTS! (9K99123AD)
FALSE 3 SPACES *Must* be DOTS! (AB)
FALSE 4 Must[be] [DOTS] ! (12.AZZZ)
FALSE 5 SPACES ~DOTS! (654KK45)

I want to replace those spaces with dots in the specific location (not everything) so the sedList.txt looks exacly like this:

TRUE 0 This.has.SPACES.inside,.Must.be.DOTS! (123.123ABCD)
FALSE 1 This-DONT-have(SPACES)-inside (900.578CD)
FALSE 2 This.SPACE!!!.inside,.#.Must.be.DOTS! (9K99123AD)
FALSE 3 SPACES.*Must*.be.DOTS! (AB)
FALSE 4 Must[be].[DOTS].! (12.AZZZ)
FALSE 5 SPACES.~DOTS! (654KK45)

So what is missing from my sed equation should i add something?

Answer 1

I guess you were almost there, I just replaced all spaces by dots first.

sed -e 1d \
        -e 's/\s/./g' \
        -e 's/x1b\[1m0/TRUE 0/g' \
        -e 's/x1b\[1m/FALSE /g' \
        -e 's/\.\./ /g' \
        -e 's/\!x1b\[39m/\!/g' \
        -e 's/x1b\[34m/\(/g' \
        -e 's/x1b\[39m/\)/g' \
        -e 's/x1b\[22m\.\:\.x1b\[35m//g' \
        -e 's/\.\./\./g' \
        -e 's/\.\:\./ /g' \
~/List.txt > SEDList.txt

Answer 2

Since you ask for sed:

sed -re 1d \
  -e 's/\x1b\[[0-9]+m//g' \
  -e 'y/ /./' \
  -e 's/^([0-9]+)\.*:\./FALSE \1 /'\
  -e 's/\.:\.(.*)/ (\1)/'\
  -e '1s/FALSE/TRUE/'

Answer 3

This one produces the desired output:

sed '1d;s/\x1b\[[0-9]*m//g;s/  *: /\n/g;y/ /./;s/^/FALSE /;2s/FALSE/TRUE/;s/$/)/;s/\n/ /;s// (/' yourfile

My hints for you:

• First of all: Don't cat and pipe to sed. Rather give the filename as argument to sed.
• Next, handle all Escape-sequences at once with a regular expression like \x1b\[[0-9]*m to simplify your script
• Avoid "magical" patterns like your |---|. With GNU sed you can use newline instead. This is unique and easy to handle.

Commented version:

sed '1d;                # drop the first line
     s/\x1b\[[0-9]*m//g;# remove all esc sequences
     s/  *: /\n/g;      # replace the colons with spaces by newlines
     y/ /./;            # replace the spaces with dots
     s/^/FALSE /;       # add the `FALSE` to the beginning
     2s/FALSE/TRUE/;    # except for the second line
     s// (/;            # first separator becomes space
     s/$/)/;            # second becomes space with (
     s/\n/ /;           # finally the trailing )' yourfile

Answer 4

While the following is not as fast as a single sed call, it adds flexibility.

sed \
    -e 1d  \
    -re 's/\x1b\[[0-9]+m//g' \
    -e 's/:/ /g' List.txt |
awk '{
    if ($1=="0")
        {printf("TRUE "$1" "); for (i=2; i<=(NF-1); i++) printf("%s%s",".",$i); printf(" ("$NF")\n")}
    else
        {printf("FALSE "$1" "); for (i=2; i<=(NF-1); i++) printf("%s%s",".",$i); printf(" ("$NF")\n")}
    }' |
sed 's/ \./ /g' > SEDList.txt

As a single line sutible for use in a script.

"$( sed -e 1d -re 's/\x1b\[[0-9]+m//g' -e 's/:/ /g' "$TheList" | awk '{ if ($1=="0") {printf("TRUE "$1" "); for (i=2; i<=(NF-1); i++) printf("%s%s",".",$i); printf(" ("$NF")\n")} else {printf("FALSE "$1" "); for (i=2; i<=(NF-1); i++) printf("%s%s",".",$i); printf(" ("$NF")\n")} }' | sed 's/ \./ /g' )"

Answer 5

I see this question as composed of 2 problems: (1) remove the ansi-seq and (2) adjust and clean the format. The first part obfuscates the second.

Here a give just a partial answer: I suggest the use of ansifilter for an initial cleaning of ansi-sequences:

ansifilter ~/List.txt | ...