10

Let's suppose I have five Bash (.sh) scripts in a folder (my_folder), and they are named as follows:

script_1.sh
script_2.sh
script_3.sh
script_4.sh
script_5.sh

How can I write a sixth Bash script or just a one liner that will start running all these scripts together?

I need the five scripts to start running all together at the same time and not one after the other.

12

GNU parallel is perfect for this sort of thing. Install with sudo apt install parallel and then:

parallel -j0 ::: my_folder/*sh

The -j controls how many processes t run in parallel, and -j0 means "all of them". Set it to another value (e.g. -j20) to run the scripts in batches instead.

  • 1
    note: there is also the parallel command from the moreutils package. The example usage: parallel -j 20 -- my_folder/*.sh (Unlike GNU parallel, -j0 would run the scripts one at a time). – jfs Jul 7 at 16:24
  • 1
    @jfs good point. But note that sudo apt install parallel will replace the file in /usr/bin/parallel with the one from GNU parallel. So as long as you install it, as my answer suggests, it should work as expected. – terdon Jul 7 at 16:37
  • Yes, I'd found out it the hard way when trying unsuccessfully parallel commands from the manual while writing my answer to suggest GNU parallel. – jfs Jul 7 at 16:42
9

To run all scripts at the same time (in parallel) use:

script_1.sh &
script_2.sh &
script_3.sh &
script_4.sh &
script_5.sh &

To run the one after the other (sequentially) use:

script_1.sh &&
script_2.sh &&
script_3.sh &&
script_4.sh &&
script_5.sh

Enhancement for comments

If you have 200 scripts you want to run at the same time (which might bog down the machine BTW) use this script:

#!/bin/bash
for Script in my_folder/*.sh ; do
    echo bash "$Script" &
done

Set the script attributes to executable with the command:

chmod a+x /path/to/script.sh

The first time you run the script it will only echo the names of the 200 scripts it will be executing. When you are happy the right names are being selected edit the script and change this line:

    echo bash "$Script" &

to:

    bash "$Script" &

There are three ways you can call a bash script from another as answered here:

  1. Make the other script executable, add the #!/bin/bash line at the top, and the path where the file is to the $PATH environment variable. Then you can call it as a normal command;

  2. Or call it with the source command (alias is .) like this: source /path/to/script;

  3. Or use the bash command to execute it: /bin/bash /path/to/script;

In OP's case one or more of the 200 scripts did not contain the shebang #!/bin/bash first line in the file. As such option 3. had to be used.


200 Scripts running at the same time

A comment has been raised about whether they are "running at the same time". On a typical 8 CPU system 25 scripts will be sharing one CPU at the same time but only one one script will execute at a time until it's time slice (measured in milliseconds) runs out. Then the next job will receive its fair share of milliseconds, then the next job, etc., etc.

In loose terms we can say 200 jobs are running "concurrently" but not "simultaneously" across 8 CPUs which equates to 25 jobs per CPU:

thread states.png

Above image and comments below from Linux kernel scheduler

time slice.png

  • thank you..the thing is that in my real case I have more than 200 scripts to run. so is there any way I can run all of them with a single command (i.e. without writing their file names)? – aaaaa Jul 6 at 17:18
  • @aaaaa I've changed the answer for 200 scripts. – WinEunuuchs2Unix Jul 6 at 17:32
  • thanks. it works.. the only thing I had to add to your last line was 'bash'. otherwise I get 'permission denied'. – aaaaa Jul 6 at 17:45
  • 1
    @aaaaa Do you actually prefer that all 200+ scripts to run at the same time, or would you rather have some smaller number (say, 10) run at a time, with the others queued to run when one or more of them finishes and the number of currently running scripts drops below that number? (If you're interested in such solutions, I suggest editing your question with information about how many scripts you have, how long they tend to take to run, what kind of work they do, and so forth.) – Eliah Kagan Jul 7 at 0:11
  • @aaaaa Thanks for the feedback. I've updated the answer to include bash prefix to calling the script. – WinEunuuchs2Unix Jul 7 at 15:33
3

There's a tool for this, read man run-parts.

For example, I do:

 run-parts ${visorhome}/pbackup.d/

in my Palm Pilot backup script. ${visorhome}/pbackup.d/:

01PopulateJpilot  02Extract_Pedometer  03URLs  04google  05Books  06Weight  07Sec  08Bkgm 50hardlinks
  • 1
    @eliah-kagan Running "more than 200 scripts" "at the same time" seems unwise. It might be a good way to stress test a system – waltinator Jul 6 at 23:59
  • Good point. I do think you may still want to mention that run-parts executes the scripts sequentially rather than all at once as the OP requested. You're right, though, that running 200 scripts at once is unlikely to perform the same as running 5--and even if running them all at once doesn't cause any problems, it may still be unnecessary. I've commented to suggest the OP clarify their preferences in this regard. – Eliah Kagan Jul 7 at 0:14
  • run-parts won't run the scripts (they have a dot in the name: my_folder/*.sh – jfs Jul 7 at 8:26
  • 1
    run-parts has very_ specific file name requirements for some strange reason. It won't launch file names with extensions, so I'm afraid this won't work here. – terdon Jul 7 at 12:12
1

To run all *.sh scripts in my_folder directory at the same time using xargs:

$ ls my_folder/*.sh | xargs -P0 -n1 bash

To limit the number of concurrently executed scripts to 10:

$ ls my_folder/*.sh | xargs -P10 -n1 bash
  • It's a bad practice to use the output of ls in a script. Using find is much safer. It's the very first item here: mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29 – Joe Jul 11 at 1:00
  • @Joe in general, it is true. I don't see issues in this particular case. – jfs Jul 11 at 15:22
  • That's because you already know about it. People coming here later might not. – Joe Jul 11 at 18:30
  • @Joe don't follow dogma blindly. Think for yourself than one option is better than another what are trade offs in your case. – jfs Jul 11 at 18:32
  • It's not a dogma. I'm alergic to those. But I've seen the stuff ls can emit. Parsing human readable output isn't reliable. There's no incentive for developers to keep – Joe Jul 11 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.