Why do Brackets “][” need double escape “\\” to prevent interpretation by the shell?

Question

Im using 'bash' shell and executing below 'awk' command on a file with records of file separated with number of different chars like Brackets, Colons, Parentheses as below sample record

...(field#13[field#14:]]:filed#18[filed#19)[...

However when I'm escaping "][" with single escape '\', awk command failing and I must double escape '\\' Brackets only to get my expected result, how does it needs to use double escape Brackets (in 'csh' shell it's same)?

awk -F"[\\[\\]:)(]" '{print $18}' inFile
filed#18

also please note this, that I know for all I can escape them with double escape '\\' like below, I just want to know why for Brackets this is mandatory?

awk -F"[\\[\\]\\:\\)\\(]" '{print $18}' inFile
filed#18

even using signle escape gives warning (except for Brackets), but still command execeute and result coming, thanks

awk -F '[\\[\\]\:\)\(]' '{print $18}' inFile
awk: warning: escape sequence `\:' treated as plain `:'
awk: warning: escape sequence `\)' treated as plain `)'
awk: warning: escape sequence `\(' treated as plain `('
filed#18

Answer 1

There are multiple levels of quoting/escaping going on here. First, you have your FS regex (-F "[\\[\\]\:\)\(]") in double quotes. That's what's giving the warnings:

$ awk -F"[\\[\\]:)(]" '{print $2}' file 
awk: warning: escape sequence `\[' treated as plain `['
awk: warning: escape sequence `\]' treated as plain `]'
awk: fatal: :, [., or [=: /[[]:)(]/

While single quotes just work:

$ awk -F'[\\[\\]:)(]' '{print $2}' file 
field#13

That's because anything in double quotes is first expanded by the shell. So, the shell first expands \\[ to \[ and then passes it to awk. You can see this happening with set -x:

$ set -x
$ awk -F"[\\[\\]:)(]" '{print $2}' file 
+ awk '-F[\[\]:)(]' '{print $2}' file

As you can see above, the shell ate the first escape. So don't use " here at all.

The next issue is that awk itself interprets the escapes twice. Because the -F can accept special escapes like \t and \r etc, it will first try to read \[ as a single, escaped character. Since \[ is the same as [ (unlike, say \n which is not the same as n since \n is a newline), it gives you a warning message, explaining that it treated \[ as [.

So, you need the first escape to escape the \ itself, and the second escape to escape the [. In other words, in \\[, the 1st \ is escaping the second \ so that what awk finally receives is \[.

It might be easier to understand if you consider something like this:

$ echo -e 'a\tb'
a   b                  ## prints a tab character
$ echo -e 'a\\tb'
a\tb                   ## prints a literal \t
$ echo -e "a\\tb"      
a   b                  ## prints a tab because of the double quotes
$ echo -e "a\\\tb"     
a\tb                   ## 3ple escaping! Prints a literal `\t` again.

The last example above is the most telling. Since the string to be echoed is in double quotes, it is expanded by the shell first (that eats one \), then by echo -e (that eats another \) and is finally printed as a literal \t.