2

I am running some query commands in remote host with xargs in parallel which is working efficiently and very well but I have some problem in finding in which host I am getting the query from.

Right now the part of the script which is doing it, looks exactly like this:

export commands="cmd1; cmd2; "
hosts=("host1" "host2" "host3" "host4")
MaxInstance=10
echo_var(){
   echo "Executing in $1:"; sleep 1; echo "Output of: $commands"; return 0;
}
export -f echo_var
printf "%s\n" ${hosts[@]} | xargs -n 2 -P $MaxInstance -I {} $(command -v bash) -c 'echo_var "$1"' _ {}

which should output like the following because of sleep 1:

Executing in host1:
Executing in host2:
Executing in host3:
Executing in host4:
Output of: cmd1; cmd2; 
Output of: cmd1; cmd2; 
Output of: cmd1; cmd2; 
Output of: cmd1; cmd2;

To achieve the output that may look similar to the one shown at the end, I would have to edit the existing functions to the following to get closer to what I wanted. But even here if I use printf "%s\n%s" "$name" "$output" , it breaks sometimes at the new line. For example:

echo_var(){
 name="Executing in $1:"; sleep 1; output="Output of: $commands";
 echo -e "$name: $output" ## will work
 # echo -e "$name\n$output" ## will not work or even
 # printf "%s\n%s" "$name" "$output" ## will not work
 return 0;
}

Without sacrificing the rapidness of parallel execution, I would like to have the output like the following. Is there any option in xargs to specify to show the output of the entire instance at a time?

Executing in host1:
Output of: cmd1; cmd2;
Executing in host2:
Output of: cmd1; cmd2;
Executing in host3:
Output of: cmd1; cmd2;
Executing in host4: 
Output of: cmd1; cmd2; 
5
  • why not use && instead of ;? Nov 24, 2019 at 15:03
  • @GeorgeUdosen Tried but gives the same output
    – Rakib Fiha
    Nov 24, 2019 at 15:08
  • Just now, I actually found a way around to solve this problem but I do not want to do it as I am lazy. Instead of printing the output directly, I should store all of them in variables and then output them all together side by side.
    – Rakib Fiha
    Nov 24, 2019 at 15:11
  • Even doing so I get new problems, which is outlined in the new edit @GeorgeUdosen
    – Rakib Fiha
    Nov 24, 2019 at 15:31
  • 2
    If you are running the same command on multiple hosts, checkout parallel --nonall.
    – Ole Tange
    Nov 25, 2019 at 6:49

1 Answer 1

5

Meet GNU Parallel (sudo apt install parallel):

GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially.
If you use xargs and tee today you will find GNU parallel very easy to use as GNU parallel is written to have the same options as xargs.

I can’t test it, but the command in your case should simply be:

parallel -n 2 -P $MaxInstance $(command -v bash) -c 'echo_var "$1"' _ {}

Note that the moreutils package provides a parallel command which is different from GNU parallel.

Example run

This example runs echo {} start; sleep 1; echo {} ready for the arguments 1, 2 and 3, first with xargs and then with parallel:

$ echo -e '1\n2\n3' | xargs -n1 -P3 -I{} bash -c 'echo {} start; sleep 1; echo {} ready'
2 start
1 start
3 start
2 ready
1 ready
3 ready
$ parallel 'echo {} start; sleep 1; echo {} ready' ::: 1 2 3
1 start
1 ready
2 start
2 ready
3 start
3 ready

Of course the echo | approach does work equally with parallel, I just like its argument list feature shown above – it does the exact same.

If you’re hesistant to use parallel because it’s not preinstalled, hard to install/update or anything like that I warmly recommend the program author’s blog article on Excuses for not installing GNU Parallel.

6
  • I have a parallel command from moreutils is it the same ? I didn't want to use parallel since xargs comes installed and is almost everywhere.
    – Rakib Fiha
    Nov 24, 2019 at 16:45
  • moreutils 's parallel and gnu parallel is not same: superuser.com/q/917577/978559
    – Rakib Fiha
    Nov 24, 2019 at 16:54
  • @RakibFiha No, not at all. If parallel is not an option you really should add that information to your question.
    – dessert
    Nov 24, 2019 at 16:57
  • 2
    @RakibFiha Does oletange.wordpress.com/2018/03/28/… solve your situation? (Especially parallel --embed)
    – Ole Tange
    Nov 25, 2019 at 6:41
  • 1
    @RakibFiha If you need moreutils installed as well the question you linked provides an answer to the name clash problem: On a system <17.04 you just force the installation of parallel and it takes care of the renaming for you.
    – dessert
    Nov 25, 2019 at 9:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .