1

I would like to be able to write a bash script to automatically update my MediaWiki extensions and I think I will use this website https://extdist.wmflabs.org/dist/extensions/ to do so. But as you can see there each tarball file name has a random (or so it appears to me) combination of letters and numbers after the final dash (-). So what I would like is some way of downloading only files with this sort of name extname-REL1_25*.tar.gz where * is the place where the wild card in the name would come in and extname refers to the extension I want's name.

1
  • It is an impossibility if you need true wild cards - the server being reached has to serve the exact request and the spec for such communications isn't something the web server can process in most if not all cases.
    – Thomas Ward
    Jun 17, 2015 at 19:18

2 Answers 2

2

wget brace expansion

Use brace expansion with wget to download multiple files according to a pattern.

Example:

wget -c https://extdist.wmflabs.org/dist/extensions/GoogleDocTag-REL1_{22-45203b7,23-249c6f2,24-3186a71,25-10e101f}.tar.gz

The optional -c flag continues the download if it is paused or interrupted from the same point where it was stopped. The -c option is useful when you are downloading a large number of files and you want to download all of the files in a batch without interruptions.

wget download multiple files from a list

Example:

lynx -dump "https://extdist.wmflabs.org/dist/skins/" | awk '/http/{print $2}' | uniq >> list.txt  
wget -c -E -H -k -K -p -e robots=off -i ./list.txt    

wget recursive accept/reject options

Recursive Accept/Reject Options
    -A acclist --accept acclist
    -R rejlist --reject rejlist
        Specify comma-separated lists of file name suffixes or patterns to
        accept or reject. Note that if any of the wildcard characters, *,
        ?, [ or ], appear in an element of acclist or rejlist, it will be
        treated as a pattern, rather than a suffix.  In this case, you 
        have to enclose the pattern into quotes to prevent your shell 
        from expanding it, like in -A "*.mp3" or -A '*.mp3'.

Example:

lynx -dump "https://extdist.wmflabs.org/dist/skins/" | awk '/http/{print $2}' | uniq >> list.txt  
wget -c -A "Vector*.tar.gz" -E -H -k -K -p -e robots=off -i ./list.txt  

Both -A and -R options download the all the files and then the accept and reject options delete the downloaded files that don't match the pattern. This is horribly inefficient. A faster way to do it would be to do the pattern matching on the list.txt file to remove all the unwanted files from list.txt before downloading anything.

Suppose that you want to download all the files from https://extdist.wmflabs.org/dist/extensions/ that satisfy the following conditions:

  • version 1.25
  • only the selected MediaWiki extensions

Assuming that you have already made a list.txt file which contains all the links in https://extdist.wmflabs.org/dist/extensions/, you open list.txt in a text editor that supports pattern matching and filter the contents of list.txt to match lines having the string 1_25 (which refers to version 1.25) and then filter those results to match lines having the ~20 extensions (the number 20 is not important) that you want to download. Then write the results to a new file called new-list.txt (the file name new-list.txt is not important) and download the multiple files (~20 in number in this example) using new-list.txt after wget's -i option to read URLs from a local or external file.

A further improvement in this method would be to reduce the number of extensions to match by filtering the contents of list.txt to match extensions containing strings that match more than one extension that you are looking for. For example, if the string wiki matches multiple extensions that you are looking for, you can filter the contents of list.txt to match wiki and save a little bit of time. In https://extdist.wmflabs.org/dist/extensions/ there are five links that contain both 1_25 and wiki and you can find all five links with two pattern matches.

1
  • @Brace expansion. I don't want to download the tarballs for every MW version. I had the wildcard operator after the 1_25 (which refers to MW version 1.25, like 1_21 refers to 1.21 and so on), not after _ because the 25 refers to the MW version. I had the wildcard operator * just before the .tar.gz and not just after the underscore because I didn't want to manually look for what came after the 1_25 for the ~20 different extensions I wanted myself. In order to do it your way I'd have to go to the site and find the suffix everytime, which is exactly what I didn't want to do.
    – Josh Pinto
    Jun 17, 2015 at 23:53
0

No, it's impossible with wildcards. But in this situation wget will work with recursive download - https://unix.stackexchange.com/questions/25340/download-recursively-with-wget

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .