2

quite the bash beginner here. I am trying to log printed lines into an array via bash. I would like to perform mathematical operations (i.e. adding up elements at the same position) on certain elements in such an array, and ultimately return the array for further use outside of the function.

Here is what I have been fiddling with:

linesToArraySum() {
while read line
do
    logLine=$line # saves currently logged line in variable logLine
    IFS=';' read -a arrayLog <<< $logLine #redirect variable logLine as input for read command. read -a saves word of input string as array. InternalFieldSeparator set as ';' detects elements in input string which are separated by '; ' as words.
    for n in 1 3 5 7 9 11
    do
        arraySum[n]=$((${arraySum[n]} + ${arrayLog[n]})) # define element in arraySum at position n as sum of previous element and element in arrayLog at this position
        echo ${arraySum[n]}
    done
    return arraySum
done
}

As mentioned above, the logged lines are continuously printed via ttylog, but for troubleshooting, let's assume I would generate them with following script:

while [[ $i < 9 ]] 
do 
    i=$(($i + 1))
    echo "dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935" | linesToArraySum
done
commandDoSomethingWith_arraySum

My problem is that echo ${arraySum[n]} in function linesToArraySum() always returns the current value of ${array[n]} instead of the sum of values in the same column.

I would highly appreciate any hints towards my mistake(s). \o/

3
  • 1
    Have you considered using Perl or something a little bit fancier than bash?
    – user11553
    Sep 17, 2017 at 19:33
  • Bash is very limited when it comes to arithmetic. You'll be better of with other generic scripting languages like Python or Perl, or even specialised languages for mathematical tasks like Matlab/Octave. Sep 18, 2017 at 11:13
  • Reviewers: This question is on-topic. Sep 18, 2017 at 17:57

2 Answers 2

2

In Bash, every command in a pipline (|) runs in a subshell. Changes to variables inside a subshell, including assignments made to array elements, do not propagate back up to the parent shell. In your test code, you have:

echo "dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935" | linesToArraySum

Shell functions do not generally run in subshells, but in this case you are running linesToArraySum in a subshell because it appears in a pipline. In some other shells, like Ksh, the rightmost command in a pipeline is not run in a subshell, and your code will actually work in such a shell. But Bash runs even the last command being piped to in its own subshell.

Because linesToArraySum runs in a subshell, the arraySum array exists only in the subshell, is never created for the caller, and is newly recreated inside a new subshell each time the pipeline runs. Furthermore, even if the array already existed before the subshell was started, modifications made to it in the subshell would only modify the subshell's copy.

All you have to to do to fix this problem is to pass input to linesToArraySum using a method that doesn't run the function in a subshell. One way to do that is to use a here string instead of a pipeline:

linesToArraySum <<<"dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935"

You can use that as a drop-in replacement for just that one line in the loop, though I suggest replacing your whole test loop with this or something like it:

for i in {0..9}; do linesToArraySum <<<"dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935"; done

(Of course, you may choose to write it over several lines.)


As Sergiy Kolodyazhnyy mentions, you are calling your function once per line, instead of passing it all the lines. The fixed code I showed above doesn't change that. Since you wrote the linesToArraySum function to read multiple lines, you might want your test code to test that. But this is not why the values in arraySum aren't being preserved. The first Bash script in Sergiy Kolodyazhnyy's answer avoids the problem by piping multiple lines of input at a time to the shell function, so that each modification of the array takes place in the same subshell. That's why it works. Furthermore:

  • After the generate_lines | sum_line_tokens command finishes, subsequent commands will still not be able to read the sums from arraySum, since the array is still created in a subshell which is destroyed at the end of the command.
  • As long as you're using a pipeline, creating the arraySum array before calling the function in a pipline will not work to preserve the values, either. The subshell will receive a copy of arraySum from the caller, so code that runs in the subshell will be able to access values that have been assigned to it, but when it writes to the array, that will only affect the subshell's copy of the array. And if you stop calling your function in a pipline, then you don't have to do anything else to make it work!

That second point bears explaining further, since it relates to a common point of confusion. In Bash, x=foo; IFS= read -r x <<<bar; echo "$x" prints bar, but x=foo; echo bar | IFS= read -r x; echo "$x" prints foo. Putting them in a function, declaring the variable with declare or local, and/or using an array does not change the fundamental principle that modifying a variable in a subshell does not modify it for the caller. For example, suppose you run this definition:

 f() { local -ai a=(10 20 30); g() { IFS= read -r 'a[3]'; echo "${a[@]}"; }; echo 40 | g; echo "${a[@]}"; }

Then run f. The output reveals that the array a is modified in the pipeline where the function g is called, but the modification does not persist after the echo 40 | g command:

10 20 30 40
10 20 30

The reason the second Bash script in Sergiy Kolodyazhnyy's answer works is simply that it avoids using a pipeline, and thus avoids running its sum_line_tokens function in a subshell. The way it does this is take input redirected from a file (< "$tempfile") rather than using a pipe:

generate_lines > "$tempfile"
sum_line_tokens "$1" < "$tempfile"

That script contains a comment explaining that sum_line_tokens will run in a subshell if you use it in a pipeline, like generate_lines | sum_line_tokens. That comment is actually the answer to your whole question. The other changes in that script--writing a main() function, creating the array explicitly before calling the functions that use it, and using the local builtin to do it--are completely irrelevant. (That script as a whole is still useful, though, both in that it shows one way to avoid using a pipeline and in that it shows a way to implement related behavior you asked about in comments.)

When you forgo placing a command in a pipline to prevent it from running in a subshell, which alternative you choose will depend on the circumstances. For text that appears in your script, use a here string (as shown above) or here document. For output from another command, writing to a temporary file and then reading from it--as in Sergiy Kolodyazhnyy's second Bash script--is often a reasonable choice. You could even create the temporary file as a named pipe with mkfifo instead of a regular file, if you want it to have identical semantics and similar performance characteristics to a shell pipeline. But in most cases, I recommend using process substitution, which actually creates, uses, and destroys a named pipe for you, all behind the scenes:

sum_line_tokens "$1" < <(generate_lines)

To run that command, the shell:

  • Creates a temporary named pipe.
  • Runs generate_lines and redirects its output to the named pipe.
  • Replaces <(generate_lines) with the name of the named pipe.
  • Runs sum_line_tokens "$1" and redirects input from the named pipe to it (due to <).

The command writing to the named pipe actually runs at the same time as the command reading from the named pipe. The order given above is for conceptual ease (I had to write them in some order). Note also that:

  • The first < for input redirection and the second < that's part of the process substitution syntax must be separated. This is to say that, where ... is the command you want to take input from, write < <(...), not <<(...).
  • Process substitution does use a subshell--but only for the substituted process. Thus the generate_lines command is being run in a subshell, but sum_line_tokens is not. If you tried to modify the caller's variables in generate_lines, those modifications would not persist afterwards. However, generate_lines doesn't have to do that. Only sum_line_tokens needs to modify variables that will be used afterwards, so it's sufficient that it not be run in a subshell.
  • Process substitution--as well as here strings and [[--are not portable to all Bourne-style shells. (Here documents and test/[ are portable.) But arrays are not portable either, so as long as you're using an array for this, you're already not writing a portable script--in the sense of being portable across different shells--so there's probably no reason for you to avoid using process substitution.

There are some other mistakes in your script. Since they are easy to make in any script--not just this one--and since I'm guessing you're writing this script for practice purposes, I'll list them here. However, as Sergiy Kolodyazhnyy says, you should consider using a tool like awk for this. Many standard Unix utilities exist mainly for the purpose of processing text line-by-line, and awk is one of them.

Processing text with a shell loop can sometimes be reasonable and is, on rare occasion, even the best choice. But for almost any task that can be done with a standard utility, it's better to do it that way than by writing a while loop in your shell that uses the read builtin. Shells are glue languages and if there is an external command that does the job, you should use it.

With that said, I recommend improving these other areas of the script if you choose to keep using it:

  • As Sergiy Kolodyazhnyy says, you cannot use return to return arrays. In fact, you cannot even return a simple variable. You can only return an exit code, which should range from 0 to 255 and is not very versatile. The main purpose of passing an argument to the return or exit builtin is to indicate whether or not there was an error, or which of several possible errors occurred, or to return one of a small handful of possible pieces of information. (For example, the test/[ builtin's return code indicates if the tested condition is true or false.) With the code you have, you should be seeing this error:

    -bash: return: arraySum: numeric argument required
    
  • You should pass -r when you use the read builtin. Otherwise \ escapes are expanded. It is extremely rare that this is what you would want. So use read -r line instead of read line and use read -ra arrayLog (or read -r -a arrayLog, if you prefer that style) instead of read -a arrayLog.

  • Even to read a line into a single variable, set IFS= unless you have a specific reason you know you don't need to (or need not to). Instead of using while read line, use while IFS= read -r line. The reason is that read strips IFS whitespace--anything in $IFS--from the beginning and end of the line it reads. The exceptions are if you actually want that to happen and--for Bash--if you omit the variable name. In Bash, read -r with no variable name is equivalent to IFS= read -r REPLY.

  • Although not actually wrong, you don't have to use full parameter expansion syntax inside (( )) to use the values of variables or array elements. Avoiding this makes such expressions far easier to read. Prefer $((arraySum[n] + arrayLog[n])) over $((${arraySum[n]} + ${arrayLog[n]})).

  • With test, [, and [[, the < operator performs lexicographic string comparison and not numerical comparison. To check if $i is less than 9, you can use [[ $i -lt 9 ]]. For example, with i=89, [[ $i < 9 ]] returns true! Similarly, you would use -gt for numerical greater-than, -le for numerical less-than-or-equal, and -ge for numerical greater-than-or-equal. Or perhaps you meant to write (($i < 9)), which would work, as would ((i < 9)).

    However, since in this case you just want to loop from 1 to 9, it's much simpler, clearer, and easier to use a for loop with brace expansion ({1..9}) as shown near the beginning of this post.

Finally, I recommend taking advantage of the power of static code analysis by checking your shell scripts with ShellCheck. ShellCheck will catch most of the errors listed above. Many experienced shell scripters use it a lot, but it's very good for novices too, because it links to full explanations for each of its rules.

Occasionally ShellCheck will identify something as possibly wrong that is actually correct. For example, when I ran it on your script, it raised SC2086 for <<< $logLine. Strictly speaking this is not necessary in the versions of Bash provided in currently supported Ubuntu systems, because text to the right of <<< in a here string is not subject to pathname expansion or word splitting. However, earlier versions did not skip those expansions, plus it's a good idea to quote your variables anytime you don't have a specific reason not to. This is a common pattern: even with some of ShellCheck's warnings that you could safely ignore, you will write better code if you choose to heed them.

2

I took the liberty to edit your function a little bit and put all into a simple script. The core of the issue is that you had to echo after the while loop is done. Also, bash functions don't "return" arrays, you have to somehow echo them to stdout or use a main function and have a local array in main, which then can be accessible to child functions ( that's something I do quite often in my own scripts ).

Here's a test result. For 9 iterations with column 1 being always 110, we appropriately get 990.

$ ./generate_lines.sh                                                                                                       
990 1008 1035 1008 1017 1080

And here's the script:

#!/usr/bin/env bash

sum_line_tokens() {
while read line
do
    #echo "$line"
    logLine=$line # saves currently logged line in variable logLine
    # redirect variable logLine as input for read command. 
    # read -a saves word of input string as array. InternalFieldSeparator set as ';' 
    # detects elements in input string which are separated by '; ' as words.
    IFS=';' read -a arrayLog <<< $logLine     

    for n in 1 3 5 7 9 11
    do
        # define element in arraySum at position n as sum of previous element 
        # and element in arrayLog at this position
        arraySum[n]=$(( ${arraySum[n]} + ${arrayLog[n]} ))         
        #echo "${arraySum[n]}"
    done
done

# Functions in bash can only use return to indicate exit status
# This is more like int datatype for C or Java functions. If you want
# to return a string or array, you need to echo it to stdout
echo "${arraySum[@]}"
}

generate_lines(){
    while [[ $i < 9 ]] 
    do 
        i=$(($i + 1))
        echo "dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935"
    done
}

generate_lines |  sum_line_tokens

Simplifying the task with awk

While the script works, it's lengthy. We can shorten the solution with awk:

# again, same thing - the script now generates lines only, no summing. 
# We'll pipe it to awk
$ ./generate_lines.sh                                                                                                                                                                           
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935
dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935

$ ./generate_lines.sh  | awk -F ';' '{for(i=1;i<=11;i++) if(i%2 != 0) sum[i+1]+=$(i+1) }END{for(j in sum) printf "%d\t",sum[j];print ""}'                                                       
990 1008    1035    1008    1017    1080

Using main function and local array

I am a big advocate for using main functions in scripts because it keeps everything organized, and as an extra advantage you can declare a local variable which every function that you call from main will know, and will be able to overwrite.

Well, there's a bit of a problem in your case. We have two functions, one produces lines and the other - does something with those lines, and using pipes has a problem - whatever runs on the right side of the pipe runs in subshell and that means that all information from subshell is gone when subshell exits. See my older question for reference.

Therefore, we need some neutral ground - either we use temporary file or something known as "named pipe". In this example I simply used temp file. If the thing that you need to parse isn't too large, you can always store everything in a local variable and let two functions deal with that same variable - one write to variable, one - parse the variable. In case of long text that gets into thousands of lines it is better to use some temporary file.

So in this version of the script, I've covered several things, including main function and how main function gets command-line arguments, as well as what you requested in the comments. Basically, the script now gets 1 command line argument - which is number of lines you want, and give that to sum_line_tokens function. Without command-line arguments it sums all lines.

Test run:

$ ./generate_lines.sh 3                                                                                                                                                                         
330 336 345 336 339 360

$ ./generate_lines.sh 4                                                                                                                                                                         
440 448 460 448 452 480

And the script itself:

#!/usr/bin/env bash

sum_line_tokens() {
    # To perform counting for n number of lines, use a counter variable
    # In this case I am using argument passed from command-line
    linecount=0


    # IFS= and -r for better line reading to ensure that spaces won't mess you up
    while IFS='' read -r line
    do
        # Check if we have arg 1 to function and quit after n lines
        if [ -n $1  ] && [ $linecount -eq $1 ] 
        then
            break
        fi

        logLine=$line 
        IFS=';' read -a arrayLog <<< $logLine     


        for n in 1 3 5 7 9 11
        do
            arraySum[n]=$(( ${arraySum[n]} + ${arrayLog[n]} ))         
        done
        # increment line counter
        ((linecount++))
    done
}

generate_lines(){
    while [[ $i < 9 ]] 
    do 
        i=$(($i + 1))
        echo "dateTime;110;2930;112;2931;115;2932;112;2933;113;2934;120;2935"
    done
}

main(){
    # create local array. Any function called from main will know about it
    local -a arraySum

    # We can't just pipe lines to summing function. Whatever runs on the right-hand side
    # of the pipe runs in subshell, which means when that subshell exits, your variables are gone
    # See https://askubuntu.com/q/704154/295286
    tempfile=$(mktemp)
    generate_lines > "$tempfile"
    sum_line_tokens "$1" < "$tempfile"
    echo "${arraySum[@]}"
    rm "$tempfile"
}

# Call main function with the command-line arguments. This works sort of like int main(String[] args) in Java
main "$@"

Note about portability

Of course, because we're using whole lot of bash-specific things, if you run this on a system where bash is unavailable it won't work. Is this a good script ? Yes, it does the job. Is this portable script ? No. The awk solution above is probably more portable.

6
  • Thank you very much, @Sergiy Kolodyazhnyy, as far as I understood Your response, I would have to pipe the output string of function sum_line_tokens into another script, which - before performing another operation - would again convert it to an array. Would You be so kind to explain Your mentioned approach of instead having 'a local array in main, which then can be accessible to child functions'?
    – LUser
    Sep 17, 2017 at 19:57
  • @brunuser sure, I can add that as well. I'm working on adding awk version to the solution,too. Sep 17, 2017 at 20:17
  • if I may be free to ask another question: Do You know about an elegant way to perform the function sum_line_tokens for only n lines, independently of how many more lines (i.e. infinite in case of ttylog) are generated by function generate_lines?
    – LUser
    Sep 17, 2017 at 20:36
  • @brunuser done. I addressed that comment as well Sep 17, 2017 at 21:15
  • The middle section of this answer about pipes and subshells is the actual core issue, even for the OP's original script. Using an alternative to a pipe that doesn't run linesToArraySum/sum_line_tokens in a subshell is a complete fix, even without the other changes like pre-declaring arraySum and using a main function. The OP's original code will work if, instead of piping from echo, a here string--or anything that doesn't run the function in a subshell--is used. With that said, I do agree wholeheartedly with the advice to use awk for this! Sep 18, 2017 at 0:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .