I would like to create a short but sweet script for wget to use a .list file. The catch is that I'd like to set directories that these files go into.


file: url.list
[group 1]

[group 2]

([group 1] and [group 2] are just here for readability, they are NOT in my real list file) (yeah I know 265 isn't real, that's why it's an example)

command (currently using which cannot parse folders) wget --continue --timestamping --content-disposition -i url.list

Of course, this currently downloads all 5 files to the same directory. my question is, is there a way to tell wget to use a different folder for group 1, and for group 2, in my case i'd like this to grab several tools that I use at work, i have a separate script in windows that creates a winpe usb key and injects all tools in these directories to the key.

So my ultimate question, can this be done super easy, or does it require me to use a full bash script to grab them and create the folders for them to go in and move them there? (using -o in wget screws with my timestamping, and timestamps are mission critical)

in theory when this finishes, i would like to have a fresh copy of (psuedo names):



Create a list that has the URL and target directory on the same line:

http://www.somehost.com/files/tool.7z tools/cool-tool/
http://www.someotherhost.com/files/icon36.png tools/cool-tool/
http://www.idunno.net/other-tool.tar.gz tools/special-tool/ tools/special-tool/
http://www.myownsite.com/tools/script-to-run-tool.cmd tools/special-tool/

Then, use a bash loop to read the file and feed it to wget:

while read url target; do 
  wget "$url" -P "$target";
done < url.list

You should be able to do what you want with the wget -P or --directory-prefix option

   -P prefix
       Set directory prefix to prefix.  The directory prefix is the
       directory where all other files and subdirectories will be saved
       to, i.e. the top of the retrieval tree.  The default is . (the
       current directory).

There are a number of other options for controlling how any source directory components are handled, which may also be useful depending whether you are downloading individual files or are using recursive wgets - see the Directory Options section of the wget manpages.


I came up with this:



while read -r line
    if [[ "$line" =~ (https?|ftp)\:\/\/ ]]
            wget "$line" -P "$currdir";
            if [ ! -d "$currdir" ]; then
                mkdir "$currdir"
done < "$filename"

for a text file of any name:


Dir 1
http://www.somehost.com/files/tool.7z tools/cool-tool/
https://www.someotherhost.com/files/icon36.png tools/cool-tool/
Dir 2
http://www.idunno.net/other-tool.tar.gz tools/special-tool/ tools/special-tool/
Dir 3
ftp://www.myownsite.com/tools/script-to-run-tool.cmd tools/special-tool/

This allows using any text file containing the directory name followed by multiple links. It creates the directories if they don't already exist. No checking is done so be sure to have directories followed by links as shown.

Just a little addition. This line will cover many situations:

wget "$line" -e robots=off -r -l 1 -nd -nc --wait 1 −−random−wait -R html,"index.*"  -P "$currdir"

It will ignore robots.txt (-e robots=off), recurse into a directory in case the link is to a directory (-r -l 1), will not download the directory hierarchy from the site--only the files (-nd), will not download already downloaded files (-nc), will randomly wait to download the next file to further help insure against rejection from the site (--wait 1 −−random−wait), and will not download html files or files of the form index.* (-R html,"index.*").

Also, to create a hierarchy of directories like Dir 1/subdir 1/sub-subdir 1:

mkdir -p "$currdir"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.