32

When I run these two commands, I get

$ type cd
cd is a shell builtin
$ type if
if is a shell keyword

It is clearly shown that cd is a shell builtin and if is a shell keyword. So what is the difference between shell builtin and keyword?

43
+200

There's a strong difference between a builtin and a keyword, in the way Bash parses your code. Before we talk about the difference, let's list all keywords and builtins:

Builtins:

$ compgen -b
.         :         [         alias     bg        bind      break     
builtin   caller    cd        command   compgen   complete  compopt   
continue  declare   dirs      disown    echo      enable    eval      
exec      exit      export    false     fc        fg        getopts   
hash      help      history   jobs      kill      let       local     
logout    mapfile   popd      printf    pushd     pwd       read      
readarray readonly  return    set       shift     shopt     source    
suspend   test      times     trap      true      type      typeset   
ulimit    umask     unalias   unset     wait                          

Keywords:

$ compgen -k
if        then      else      elif      fi        case      
esac      for       select    while     until     do        
done      in        function  time      {         }         
!         [[        ]]        coproc              

Notice that, for example [ is a builtin and that [[ is a keyword. I'll use these two to illustrate the difference below, since they are well-known operators: everybody knows them and uses them regularly (or should).

A keyword is scanned and understood by Bash very early in its parsing. This allows for example the following:

string_with_spaces='some spaces here'
if [[ -n $string_with_spaces ]]; then
    echo "The string is non-empty"
fi

This works fine, and Bash will happily output

The string is non-empty

Note that I didn't quote $string_with_spaces. Whereas the following:

string_with_spaces='some spaces here'
if [ -n $string_with_spaces ]; then
    echo "The string is non-empty"
fi

shows that Bash isn't happy:

bash: [: too many arguments

Why does it work with keywords and not with builtins? because when Bash parses the code, it sees [[ which is a keyword, and understands very early that it's special. So it will look for the closing ]] and will treat the inside in a special way. A builtin (or command) is treated as an actual command that is going to be called with arguments. In this last example, bash understands that it should run the command [ with arguments (shown one per line):

-n
some
spaces
here
]

since variable expansion, quote removal, pathname expansion and word splitting occurs. The command [ turns out to be built in the shell, so it executes it with these arguments, which results in an error, hence the complaint.

In practice, you see that this distinction allows for sophisticated behavior, that wouldn't be possible with builtins (or commands).

Still in practice, how can you possibly distinguish a builtin from a keyword? this is a fun experiment to perform:

$ a='['
$ $a -d . ]
$ echo $?
0

When Bash parses the line $a -d . ], it sees nothing special (i.e., no aliases, no redirections, no keywords), so it just performs variable expansion. After variable expansions, it sees:

[ -d . ]

so executes the command (builtin) [ with arguments -d, . and ], which, of course is true (this only tests whether . is a directory).

Now look:

$ a='[['
$ $a -d . ]]
bash: [[: command not found

Oh. That's because when Bash sees this line, it sees nothing special, and hence expands all variables, and eventually sees:

[[ -d . ]]

At this time, alias expansions and keyword scanning has long been performed and is not going to be performed anymore, so Bash tries to find the command called [[, doesn't find it, and complains.

Along the same lines:

$ '[' -d . ]
$ echo $?
0
$ '[[' -d . ]]
bash: [[: command not found

and

$ \[ -d . ]
$ echo $?
0
$ \[[ -d . ]]
bash: [[: command not found

Alias expansion is something rather special too. You've all done the following at least once:

$ alias ll='ls -l'
$ ll
.... <list of files in long format> ....
$ \ll
bash: ll: command not found
$ 'll'
bash: ll: command not found

The reasoning is the same: alias expansion occurs long before variable expansion and quote removal.


Keyword v.s. Alias

Now what do you think happens if we define an alias to be a keyword?

$ alias mytest='[['
$ mytest -d . ]]
$ echo $?
0

Oh, it works! so aliases can be used to alias keywords! nice to know.


Conclusion: builtins really behave like commands: they correspond to an action being executed with arguments that undergo direct variable expansion and word splitting and globbing. It's really just like having an external command somewhere in /bin or /usr/bin that is called with the arguments given after variable expansion, etc. Note that when I say it's really just like having an external command I only mean with respect to arguments, word splitting, globbing, variable expansion, etc. A builtin can modify the shell's internal state!

Keywords, on the other hand, are scanned and understood very early, and allow for sophisticated shell behavior: the shell will be able to forbid word splitting or pathname expansion, etc.

Now look at the list of builtins and keywords and try to figure out why some need to be keywords.


! is a keyword. It seems it would be possible to mimic its behavior with a function:

not() {
    if "$@"; then
        return false
    else
        return true
    fi
}

but this would forbid constructs like:

$ ! ! true
$ echo $?
0

or

$ ! { true; }
echo $?
1

Same for time: it's more powerful to have it a keyword so that it can time complex compound commands and pipelines with redirections:

$ time grep '^#' ~/.bashrc | { i=0; while read -r; do printf '%4d %s\n' "$((++i))" "$REPLY"; done; } > bashrc_numbered 2>/dev/null

If time where a mere command (even builtin), it would only see the arguments grep, ^# and /home/gniourf/.bashrc, time this, and then its output would go through the remaining parts of the pipeline. But with a keyword, Bash can handle everything! it can time the complete pipeline, including the redirections! If time were a mere command, we couldn't do:

$ time { printf 'hello '; echo world; }

Try it:

$ \time { printf 'hello '; echo world; }
bash: syntax error near unexpected token `}'

Try to fix (?) it:

$ \time { printf 'hello '; echo world;
time: cannot run {: No such file or directory

Hopeless.


Keyword vs alias?

$ alias mytime=time
$ alias myls=ls
$ mytime myls

What do you think happens?


Really, a builtin is like a command, except that it's built in the shell, whereas a keyword is something that allows for sophisticated behavior! we can say it's part of the shell's grammar.

  • 2
    Agreed with @JohnyTex, this is one of the most comprehensive and appropriate answer I've seen on the Stack sites. Thank you. One perhaps unrelated question: just for curiosity's sake I'm trying to find the documentation for the 'temporarily disable alias' functionality from preceding a command with =\' by using man, apropos, and help and I've not had any luck. Any idea where I would go about finding this info? Mostly so that in the future I can see what else is in there, because I think I'm missing a reference source. – nc. Feb 25 '16 at 8:21
  • @nc: you won't find it documented explicitly. The reason it works is explained in this answer. The closest you'll find is in the reference manual in the section Shell Operation. You'll see that alias expansion is done very early (something I tried to emphasize in this answer), in step 2. The quote removal, parameter expansion, globbing, etc. are performed later. So to disable an alias, you can use some kind of quoting to forbid the shell from understanding a token as an alias, e.g., \ll, "ll" or 'll'. – gniourf_gniourf Feb 25 '16 at 10:06
  • 1
    I actually did get the order thing, aliases and keywords getting fired happens first. Since we quote the [[ yielding \[[ it doesn't get parsed as an alias. Right so far? Where I got lost was not realizing the backslash was a quoting thing in Bash, and I tried to look it up under aliases and keywords and got completely lost... All good now. Under the Quoting section:> A non-quoted backslash () is the escape character. – nc. Feb 25 '16 at 13:49
  • 1
    So we can think of (2) in that Op list as Tokenization, and \[[ is a single token that of type LiteralQuoteToken, as opposed to [[ which will be an OpenTestKeywordToken, and that requires a closing ]] CloseTestKeywordToken for oroper compilation/correct syntax. Later, in (4) the LiteralQuoteToken will be evaluated to [[ as the name of the bulitin/command to execute, and in (6) Bash will barf because there's no [[ builtin or command. Nice. I may forget the precise details over time, but at this moment the way Bash runs stuff is much clearer to me; thank you. – nc. Feb 25 '16 at 14:07
8

man bash calls them SHELL BUILTIN COMMANDS. So, a "shell builtin" is just like a normal command, like grep, etc., but instead of being contained in a separate file, it's built into bash itself. This makes them perform more efficiently than external commands.

A keyword is also "hard-coded into Bash, but unlike a builtin, a keyword is not in itself a command, but a subunit of a command construct." I interpret this to mean that keywords have no function alone, but require commands to do anything. (From the link, other examples are for, while, do, and !, and there are more in my answer to your other question.)

  • 1
    Interesting fact: [[ is a shell keyword but [ is a shell builtin. I have no idea why. – Sparhawk Apr 10 '14 at 12:08
  • 1
    It's likely for historical reasons and POSIX standard to conform to old Bourne shell as closely as possible, since [ existed as separate command back in the day. [[ isn't specified by the standard, so the developers can choose to include that as keyword or as built-in. – Sergiy Kolodyazhnyy Jul 15 '18 at 22:44
1

The command-line manual that comes with Ubuntu doesn't give a definition of keywords, however the online manual (see sidenote) and POSIX Shell Command Language standard specifications, refer to these as "Reserved Words", and both provide lists of those. From the POSIX standard:

This recognition shall only occur when none of the characters is quoted and when the word is used as:

  • The first word of a command

  • The first word following one of the reserved words other than case, for, or in

  • The third word in a case command (only in is valid in this case)

  • The third word in a for command (only in and do are valid in this case)

The key here is that keywords/reserved words have special meaning because they facilitate the shell syntax, serve to signal certain blocks of code such as loops, compound commands, branching ( if/case ) statements, etc. They allow forming command statements, but by themselves - don't do anything, and in fact if you enter keywords such as for, until, case - the shell will expect a complete statement, otherwise - syntax error:

$ for
bash: syntax error near unexpected token `newline'
$  

On source code level, the reserved words for bash are defined in parese.y, while built-ins have whole directory dedicated to them.

Sidenote

The GNU index shows [ as reserved word,however it is in-fact built-in command. [[ by contrast is a reserved word.

See also: Differences between keyword, reserved word, and builtin?

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.