1

So I'm currently trying to set up a shell script utilizing wget to download a huge amount of files from a server. I'm using the -A 'pattern*.extension' option to download only files with a specific pattern and extension that I'm interested in. (Besides, there are loads of files that I don't want wasting space on my PC.) If I execute wget via command line, everything works just fine and it downloads exactly all the files matching the pattern. Now if I try to run the script with the exact same wget command as before, it suddenly says: "wget: No match". (I should mention that the shell script is being executed by a python script that is provided with a list of IDs to download files from specific subfolders.) But: If I print the exact wget line the shell script executes and copy&paste it into the command line, it works again. That just doesn't make sense to me.

Here is how my wget looks:

wget -r -c -nH -np -nd -e robots=off -P PATH -A 'pattern*.extension' -a logfile.log --progress=bar:force --no-check-certificate  https://.../ID/

And here is how my shell script in principle looks like:

#!/usr/bin/tcsh
set ID=$1  #just an ID for subfolders
set OPT2=$3  #additional options that can be passed to wget
set OPT="-r -c -nH -np -nd -e robots=off -P PATH/$ID -A 'pattern*.extension' -a $ID.log"
set OOP="--progress=bar:force --no-check-certificate" 
while ($1 != '')    
     echo "wget $OPT $OOP $OPT2 https://.../$ID/"
     wget $OPT $OOP $OPT2 https://.../$ID/
 shift
end 

The output then is as follows:

wget -r -c -nH -np -nd -e robots=off -P PATH/ID -A 'pattern*.extension'  -a ID.log --progress=bar:force --no-check-certificate https://.../ID/
wget: No match.

But now, if I copy that exact line that my script echoes, it works! Please, enlighten me. I was trying so hard to fix this, but, obviously, I just couldn't manage to do so.

Also: If I replace -A 'pattern*.extension' by -A.extenstion, it downloads all the files with that extension. For some reason it just doesn't work when using a pattern.

As I mentioned above, the shell script is called by a python script:

for ID in IDs:
    cmd = 'csh PATH/script.csh %s' % (ID)
    sub.call( cmd, shell=True )

Maybe that also helps and maybe I should also mention that I'm not really an advanced programmer.

Thanks in advance.

  • Why is this tagged bash? Looks like you're using (t)csh. – muru May 23 '18 at 11:46
  • As I said I'm not really experienced with this and I didn't know there was a difference between bash and tcsh. – F. Emmerich May 23 '18 at 11:59
  • Broadly speaking there are two main families of shells: those based on the Bourne shell (sh) and those based on the C shell (csh). Ubuntu has two sh-based shells (bash, dash) installed by default, but no csh-based one is installed by default. So I'm curious: why tcsh? – muru May 23 '18 at 12:01
  • So basically I'm working at a physics institute as a students assistant and I was provided with this script to download files from a server. The script was written years ago (so I can't tell you why tcsh) for that specific server, but recently something with the servers changed so that the script was not working. Thus, I needed to mess around with it to get it working again. In principle, I achieved what I was asked for, but I'm not happy that it's not possible to get files matching a certain pattern, because there are so huge amounts of scientific data on these servers that we don't need. – F. Emmerich May 23 '18 at 12:39
  • So you're not tied to tcsh and can use bash? – muru May 23 '18 at 12:40
1

I'm not familiar with tcsh, so there might be a good fix for this in tcsh.

That said, I am familiar with bash, and can see what might be causing this problem. Note the quoting here:

set OPT="-r -c -nH -np -nd -e robots=off -P PATH/$ID -A 'pattern*.extension' -a $ID.log"

There's an outer layer of quotes and additionally, an inner layer of quotes for pattern*.expansion. Presumably you're using the inner layer because that's how it would look like you if you actually typed the command in the shell.

The way this works, the shell performs various expansions on the command line, like variable expansion, wildcard expansion, etc. So in a command like:

wget $OPT ...

the shell will expand $OPT to its contents, perform field splitting, separating out the contents of $OPT to separate words, and wildcard expansion (or globbing), which is where the command fails:

> set foo="'*'"
> echo $foo
echo: No match.
> echo "$foo"
'*'

Note how the quotes around $foo prevented the error? But you cannot use quotes in your script, since quotes will prevent field splitting as well, and you are relying on field splitting so that the various options in $OPT (-r, -c, etc.) are passed as separate arguments to wget.

Here's a demo of the differences with and without quoting:

> printf "|%s|\n" "$OPT"
|-r -c -nH -np -nd -e robots=off -P PATH/foo -A 'pattern*.extension' -a foo.log|
> printf "|%s|\n" $OPT
printf: No match.
> printf "|%s|\n" -r -c -nH -np -nd -e robots=off -P PATH/$ID -A 'pattern*.extension' -a $ID.log
|-r|
|-c|
|-nH|
|-np|
|-nd|
|-e|
|robots=off|
|-P|
|PATH/foo|
|-A|
|pattern*.extension|
|-a|
|foo.log|

I imagine this might work if you used:

set OPT="-r -c -nH -np -nd -e robots=off -P PATH/$ID -A pattern*.extension -a $ID.log"

But that risks wildcard expansion if you don't use "$OPT", and the working directory has files matching pattern*.extension:

> touch pattern-abc.extension
> printf "|%s|\n" $OPT
printf: No match.
> set set OPT="-r -c -nH -np -nd -e robots=off -P PATH/$ID -A pattern*.extension -a $ID.log"
> printf "|%s|\n" $OPT
|-r|
|-c|
|-nH|
|-np|
|-nd|
|-e|
|robots=off|
|-P|
|PATH/foo|
|-A|
|pattern-abc.extension|      <---- tcsh expanded pattern*.extension
|-a|
|foo.log|

Again, I am not skilled enough in tcsh to suggest a good solution, but I do know a decent solution in bash:

Use arrays to build command lines.

In bash, your script would look like:

#!/bin/bash
ID=$1  #just an ID for subfolders
shift

OPT2=("$@")  #additional options that can be passed to wget
OPT=(-r -c -nH -np -nd -e robots=off -P PATH/"$ID" -A 'pattern*.extension' -a "$ID".log)
OOP=(--progress=bar:force --no-check-certificate)

echo wget "${OPT[@]}" "${OOP[@]}" "${OPT2[@]}" "https://.../$ID/"
wget "${OPT[@]}" "${OOP[@]}" "${OPT2[@]}" "https://.../$ID/"

I have made a few changes here. You have assigned $3 to OPT2, but then loop over all arguments (in effect, that's what while ($1 != '') ... shift will do), but without using those arguments in the loop, which doesn't make sense - the second argument to the script is effectively ignored, but the third argument is used as an option. Given the Python snippet, I'm going to assume only the first argument is the ID and the rest are options to wget.

Now, using an array like so:

OPT=(-r -c -nH -np -nd -e robots=off -P PATH/"$ID" -A 'pattern*.extension' -a "$ID".log)

Allows us to expand to each separate word in the array, while not risking field splitting or filename generation, using "${OPT[@]}" in bash:

$ printf "|%s|\n" "${OPT[@]}"
|-r|
|-c|
|-nH|
|-np|
|-nd|
|-e|
|robots=off|
|-P|
|PATH/foo|
|-A|
|pattern*.extension|
|-a|
|foo.log|

Every argument perfectly preserved.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.