In Bash, every command in a pipline (|
) runs in a subshell. Changes to variables inside a subshell, including assignments made to array elements, do not propagate back up to the parent shell. In your test code, you have:
echo "dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935" | linesToArraySum
Shell functions do not generally run in subshells, but in this case you are running linesToArraySum
in a subshell because it appears in a pipline. In some other shells, like Ksh, the rightmost command in a pipeline is not run in a subshell, and your code will actually work in such a shell. But Bash runs even the last command being piped to in its own subshell.
Because linesToArraySum
runs in a subshell, the arraySum
array exists only in the subshell, is never created for the caller, and is newly recreated inside a new subshell each time the pipeline runs. Furthermore, even if the array already existed before the subshell was started, modifications made to it in the subshell would only modify the subshell's copy.
All you have to to do to fix this problem is to pass input to linesToArraySum
using a method that doesn't run the function in a subshell. One way to do that is to use a here string instead of a pipeline:
linesToArraySum <<<"dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935"
You can use that as a drop-in replacement for just that one line in the loop, though I suggest replacing your whole test loop with this or something like it:
for i in {0..9}; do linesToArraySum <<<"dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935"; done
(Of course, you may choose to write it over several lines.)
As Sergiy Kolodyazhnyy mentions, you are calling your function once per line, instead of passing it all the lines. The fixed code I showed above doesn't change that. Since you wrote the linesToArraySum
function to read multiple lines, you might want your test code to test that. But this is not why the values in arraySum
aren't being preserved. The first Bash script in Sergiy Kolodyazhnyy's answer avoids the problem by piping multiple lines of input at a time to the shell function, so that each modification of the array takes place in the same subshell. That's why it works. Furthermore:
- After the
generate_lines | sum_line_tokens
command finishes, subsequent commands will still not be able to read the sums from arraySum
, since the array is still created in a subshell which is destroyed at the end of the command.
- As long as you're using a pipeline, creating the
arraySum
array before calling the function in a pipline will not work to preserve the values, either. The subshell will receive a copy of arraySum
from the caller, so code that runs in the subshell will be able to access values that have been assigned to it, but when it writes to the array, that will only affect the subshell's copy of the array. And if you stop calling your function in a pipline, then you don't have to do anything else to make it work!
That second point bears explaining further, since it relates to a common point of confusion. In Bash, x=foo; IFS= read -r x <<<bar; echo "$x"
prints bar
, but x=foo; echo bar | IFS= read -r x; echo "$x"
prints foo
. Putting them in a function, declaring the variable with declare
or local
, and/or using an array does not change the fundamental principle that modifying a variable in a subshell does not modify it for the caller. For example, suppose you run this definition:
f() { local -ai a=(10 20 30); g() { IFS= read -r 'a[3]'; echo "${a[@]}"; }; echo 40 | g; echo "${a[@]}"; }
Then run f
. The output reveals that the array a
is modified in the pipeline where the function g
is called, but the modification does not persist after the echo 40 | g
command:
10 20 30 40
10 20 30
The reason the second Bash script in Sergiy Kolodyazhnyy's answer works is simply that it avoids using a pipeline, and thus avoids running its sum_line_tokens
function in a subshell. The way it does this is take input redirected from a file (< "$tempfile"
) rather than using a pipe:
generate_lines > "$tempfile"
sum_line_tokens "$1" < "$tempfile"
That script contains a comment explaining that sum_line_tokens
will run in a subshell if you use it in a pipeline, like generate_lines | sum_line_tokens
. That comment is actually the answer to your whole question. The other changes in that script--writing a main()
function, creating the array explicitly before calling the functions that use it, and using the local
builtin to do it--are completely irrelevant. (That script as a whole is still useful, though, both in that it shows one way to avoid using a pipeline and in that it shows a way to implement related behavior you asked about in comments.)
When you forgo placing a command in a pipline to prevent it from running in a subshell, which alternative you choose will depend on the circumstances. For text that appears in your script, use a here string (as shown above) or here document. For output from another command, writing to a temporary file and then reading from it--as in Sergiy Kolodyazhnyy's second Bash script--is often a reasonable choice. You could even create the temporary file as a named pipe with mkfifo
instead of a regular file, if you want it to have identical semantics and similar performance characteristics to a shell pipeline. But in most cases, I recommend using process substitution, which actually creates, uses, and destroys a named pipe for you, all behind the scenes:
sum_line_tokens "$1" < <(generate_lines)
To run that command, the shell:
- Creates a temporary named pipe.
- Runs
generate_lines
and redirects its output to the named pipe.
- Replaces
<(generate_lines)
with the name of the named pipe.
- Runs
sum_line_tokens "$1"
and redirects input from the named pipe to it (due to <
).
The command writing to the named pipe actually runs at the same time as the command reading from the named pipe. The order given above is for conceptual ease (I had to write them in some order). Note also that:
- The first
<
for input redirection and the second <
that's part of the process substitution syntax must be separated. This is to say that, where ...
is the command you want to take input from, write < <(...)
, not <<(...)
.
- Process substitution does use a subshell--but only for the substituted process. Thus the
generate_lines
command is being run in a subshell, but sum_line_tokens
is not. If you tried to modify the caller's variables in generate_lines
, those modifications would not persist afterwards. However, generate_lines
doesn't have to do that. Only sum_line_tokens
needs to modify variables that will be used afterwards, so it's sufficient that it not be run in a subshell.
- Process substitution--as well as here strings and
[[
--are not portable to all Bourne-style shells. (Here documents and test
/[
are portable.) But arrays are not portable either, so as long as you're using an array for this, you're already not writing a portable script--in the sense of being portable across different shells--so there's probably no reason for you to avoid using process substitution.
There are some other mistakes in your script. Since they are easy to make in any script--not just this one--and since I'm guessing you're writing this script for practice purposes, I'll list them here. However, as Sergiy Kolodyazhnyy says, you should consider using a tool like awk
for this. Many standard Unix utilities exist mainly for the purpose of processing text line-by-line, and awk
is one of them.
Processing text with a shell loop can sometimes be reasonable and is, on rare occasion, even the best choice. But for almost any task that can be done with a standard utility, it's better to do it that way than by writing a while
loop in your shell that uses the read
builtin. Shells are glue languages and if there is an external command that does the job, you should use it.
With that said, I recommend improving these other areas of the script if you choose to keep using it:
As Sergiy Kolodyazhnyy says, you cannot use return
to return arrays. In fact, you cannot even return a simple variable. You can only return an exit code, which should range from 0 to 255 and is not very versatile. The main purpose of passing an argument to the return
or exit
builtin is to indicate whether or not there was an error, or which of several possible errors occurred, or to return one of a small handful of possible pieces of information. (For example, the test
/[
builtin's return code indicates if the tested condition is true or false.) With the code you have, you should be seeing this error:
-bash: return: arraySum: numeric argument required
You should pass -r
when you use the read
builtin. Otherwise \
escapes are expanded. It is extremely rare that this is what you would want. So use read -r line
instead of read line
and use read -ra arrayLog
(or read -r -a arrayLog
, if you prefer that style) instead of read -a arrayLog
.
Even to read a line into a single variable, set IFS=
unless you have a specific reason you know you don't need to (or need not to). Instead of using while read line
, use while IFS= read -r line
. The reason is that read
strips IFS whitespace--anything in $IFS
--from the beginning and end of the line it reads. The exceptions are if you actually want that to happen and--for Bash--if you omit the variable name. In Bash, read -r
with no variable name is equivalent to IFS= read -r REPLY
.
Although not actually wrong, you don't have to use full parameter expansion syntax inside ((
))
to use the values of variables or array elements. Avoiding this makes such expressions far easier to read. Prefer $((arraySum[n] + arrayLog[n]))
over $((${arraySum[n]} + ${arrayLog[n]}))
.
With test
, [
, and [[
, the <
operator performs lexicographic string comparison and not numerical comparison. To check if $i
is less than 9
, you can use [[ $i -lt 9 ]]
. For example, with i=89
, [[ $i < 9 ]]
returns true! Similarly, you would use -gt
for numerical greater-than, -le
for numerical less-than-or-equal, and -ge
for numerical greater-than-or-equal. Or perhaps you meant to write (($i < 9))
, which would work, as would ((i < 9))
.
However, since in this case you just want to loop from 1
to 9
, it's much simpler, clearer, and easier to use a for
loop with brace expansion ({1..9}
) as shown near the beginning of this post.
Finally, I recommend taking advantage of the power of static code analysis by checking your shell scripts with ShellCheck. ShellCheck will catch most of the errors listed above. Many experienced shell scripters use it a lot, but it's very good for novices too, because it links to full explanations for each of its rules.
Occasionally ShellCheck will identify something as possibly wrong that is actually correct. For example, when I ran it on your script, it raised SC2086 for <<< $logLine
. Strictly speaking this is not necessary in the versions of Bash provided in currently supported Ubuntu systems, because text to the right of <<<
in a here string is not subject to pathname expansion or word splitting. However, earlier versions did not skip those expansions, plus it's a good idea to quote your variables anytime you don't have a specific reason not to. This is a common pattern: even with some of ShellCheck's warnings that you could safely ignore, you will write better code if you choose to heed them.